PCI/CXL: Add CXL reset support for Type 2 devices
Subsystems (3)
v9 cover PCI/CXL: Add CXL reset support for Type 2 devices
Hi folks!
This series adds CXL Reset support for CXL Type 2 devices through the existing PCI reset_method ABI. The reset sequence follows the CXL 4.0 specification [1], including CXL.cache disable, optional cache writeback, CXL Reset initiation, ResetComplete polling, and ResetError reporting.
The userspace ABI is the existing PCI reset interface:
/sys/bus/pci/devices/.../reset_method
/sys/bus/pci/devices/.../resetUserspace can select "cxl_reset" in reset_method and then trigger reset through the existing reset attribute.
Following Dan's v6 feedback, this replaces the proposed memdev sysfs ABI with the existing PCI reset_method interface.
v9 changes from v8 [2]:
- Rebase the series onto v7.2-rc2 so it applies cleanly on a Linus rc tag.
- Include the cxl_bus HDM restore add-on as patch 11/11 so b4 orders it after the main CXL reset series.
- Address only patch application issues in v9 to gather feedback on the current direction. Other review comments, including Dave's code structure/readability feedback, are deferred to v10.
v8 changes from v7 [3]:
- Drop the PCI helper export patch.
- Drop the multi-function sibling coordination patch. cxl_reset is only exposed as a function-scoped reset method when the CXL reset scope is limited to the target function.
- Keep the reset-scope discovery needed to reject non-function-scoped CXL resets.
- Cache HDM location as BAR-relative metadata instead of keeping an enum-time MMIO mapping.
- Restore HDM through a temporary mapping based on the current BAR assignment after PCI config state is restored.
- Cache raw HDM decoder register state so uncommitted decoders can be restored as uncommitted, while committed decoders are recommitted through the normal HDM commit flow.
- Share HDM decoder decode and validation between normal CXL core enumeration and early PCI HDM caching.
- Put cxl_reset ahead of FLR in reset_method priority because FLR does not reset CXL.cache or CXL.mem protocol state.
Motivation:
-----------
- Type 2 devices need a CXL-specific reset mechanism beyond existing PCI reset methods.
- FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is the architectural reset mechanism for those protocols.
- The PCI reset_method ABI lets userspace select this narrower CXL reset before falling back to broader bus reset methods.
Change Description:
-------------------
Patch 1: cxl: Split decoder programming into a reusable helper
- Move shared decoder settings to include/cxl/cxl.h.
- Factor low-level HDM register programming into cxl_commit().
Patch 2: cxl: Cache decoder settings on PCI devices
- Cache CXL core HDM decoder settings in pci_dev->hdm.
- Refresh the cache as decoders are enumerated, committed, or reset.
Patch 3: cxl: Share HDM decoder decode logic
- Share HDM decoder decode and validation between normal CXL core enumeration and early PCI HDM cache setup.
Patch 4: cxl: Cache endpoint decoder settings during PCI enumeration
- Snapshot endpoint HDM state during PCI capability initialization.
- Cache the HDM register locator as BAR-relative metadata.
- Cache raw decoder register state in addition to committed decoder settings.
Patch 5: cxl: Add CXL Device Reset helper
- Add the internal DVSEC reset sequence.
- Disable CXL.cache, perform cache writeback where supported, initiate CXL Reset, and wait for completion.
Patch 6: cxl: Validate HDM ranges before CXL reset
- Collect enabled cached HDM ranges.
- Reject reset if affected ranges are busy.
- Invalidate CPU caches when runtime cache-invalidation support is available, otherwise continue after warning.
Patch 7: PCI/CXL: Discover the CXL reset scope
- Discover whether CXL Reset is function-scoped using the Non-CXL Function Map and CXL cache/mem capability bits.
Patch 8: cxl: Restore CXL HDM state after PCI reset
- Restore cached global and per-decoder HDM state after reset.
- Re-map HDM registers from the current BAR assignment during restore.
- Replay raw decoder state for uncommitted decoders and recommit decoders that were committed before reset.
Patch 9: PCI/CXL: Expose CXL Reset as a PCI reset method
- Add "cxl_reset" to the PCI reset-method table for Type 2 reset-capable CXL devices.
- Prioritize cxl_reset ahead of FLR.
Patch 10: Documentation/ABI: Document CXL Reset PCI reset method
- Document the new reset_method value and reset behavior.
Patch 11: PCI/CXL: Restore HDM state after CXL bus reset
- Restore cached CXL HDM decoder state after cxl_bus reset.
The CPU cache invalidation step depends on cpu_cache_invalidate_memregion() support for the affected address ranges. If no runtime provider is available, the kernel emits a warning and continues after the affected HDM ranges have been reserved.
Example:
echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/resetTesting:
- Ran 100 iterations of cxl_reset through the PCI reset sysfs ABI on a CXL Type 2 device. All iterations completed successfully and ResetComplete was observed.
- Exercised cxl_bus reset separately with the HDM restore add-on patch.
References:
[1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
[2] https://lore.kernel.org/linux-cxl/[email protected]/
[3] https://lore.kernel.org/linux-cxl/[email protected]/
Srirangan Madhavan (11):
cxl: Split decoder programming into a reusable helper
cxl: Cache decoder settings on PCI devices
cxl: Share HDM decoder decode logic
cxl: Cache endpoint decoder settings during PCI enumeration
cxl: Add CXL Device Reset helper
cxl: Validate HDM ranges before CXL reset PCI/CXL: Discover the CXL reset scope
cxl: Restore CXL HDM state after PCI reset PCI/CXL: Expose CXL Reset as a PCI reset method Documentation/ABI: Document CXL Reset PCI reset method PCI/CXL: Restore HDM state after CXL bus reset
Documentation/ABI/testing/sysfs-bus-pci | 14 + drivers/cxl/Kconfig | 4 + drivers/cxl/core/Makefile | 2 +- drivers/cxl/core/hdm.c | 257 ++--- drivers/cxl/core/region.c | 6 +- drivers/cxl/core/regs.c | 4 + drivers/cxl/core/reset.c | 1359 +++++++++++++++++++++++ drivers/cxl/cxl.h | 43 - drivers/pci/pci.c | 5 + drivers/pci/probe.c | 3 + include/cxl/cxl.h | 108 +- include/linux/pci.h | 8 +- include/uapi/linux/pci_regs.h | 14 + tools/testing/cxl/test/cxl.c | 10 +- 14 files changed, 1628 insertions(+), 209 deletions(-) create mode 100644 drivers/cxl/core/reset.c
base-commit: 8cdeaa50eae8dad34885515f62559ee83e7e8dda
Signature · 1 line
2.43.0
PATCH v9 1/11 cxl: Split decoder programming into a reusable helper
Move common HDM decoder settings to include/cxl/cxl.h and route the register programming sequence through cxl_commit(). This lets reset code restore cached HDM state without depending on private cxl_core types while keeping hdm.c in charge of the existing commit policy checks.
Build the low-level HDM helper under CONFIG_CXL_HDM so it is available even when cxl_core is modular.
--- drivers/cxl/Kconfig | 4 ++ drivers/cxl/core/Makefile | 1 + drivers/cxl/core/hdm.c | 122 ++++------------------------------- drivers/cxl/core/region.c | 6 +- drivers/cxl/core/reset.c | 118 +++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 43 ------------ include/cxl/cxl.h | 55 +++++++++++++++- tools/testing/cxl/test/cxl.c | 10 +-- 8 files changed, 197 insertions(+), 162 deletions(-) create mode 100644 drivers/cxl/core/reset.c
drivers/cxl/Kconfig +4 -0
@@ -6,6 +6,7 @@ menuconfig CXL_BUS select FW_UPLOAD select PCI_DOE select FIRMWARE_TABLE + select CXL_HDM select NUMA_KEEP_MEMINFO if NUMA_MEMBLKS select FWCTL if CXL_FEATURES help
@@ -243,4 +244,7 @@ config CXL_ATL depends on CXL_REGION depends on ACPI_PRMT && AMD_NB +config CXL_HDM + bool + endif
drivers/cxl/core/Makefile +1 -0
@@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_BUS) += cxl_core.o +obj-$(CONFIG_CXL_HDM) += reset.o obj-$(CONFIG_CXL_SUSPEND) += suspend.o ccflags-y += -I$(srctree)/drivers/cxl
drivers/cxl/core/hdm.c +12 -110
166 lines not on this page · raw message
drivers/cxl/core/region.c +3 -3
@@ -2963,9 +2963,9 @@ static int poison_by_decoder(struct device *dev, void *arg) cxlds = cxlmd->cxlds; mode = cxlds->part[cxled->part].mode; -if (cxled->skip) { - offset = cxled->dpa_res->start - cxled->skip; - length = cxled->skip; +if (cxled->cxld.skip) { + offset = cxled->dpa_res->start - cxled->cxld.skip; + length = cxled->cxld.skip; rc =cxl_mem_get_poison (cxlmd, offset, length,NULL );if (rc == -EFAULT && mode ==CXL_PARTMODE_RAM ) rc =0 ;
drivers/cxl/core/reset.c new file +118 -0
118 lines not on this page · raw message
drivers/cxl/cxl.h +0 -43
63 lines not on this page · raw message
include/cxl/cxl.h +54 -1
75 lines not on this page · raw message
tools/testing/cxl/test/cxl.c +5 -5
@@ -847,11 +847,11 @@ static int cxld_registry_restore(struct cxl_decoder *cxld,dbg_cxld (port,"restore" , &td->cxled.cxld);cxld_copy (cxld, &td->cxled.cxld); cxled->state = td->cxled.state; - cxled->skip = td->cxled.skip; + cxld->skip = td->cxled.cxld.skip;if (range_len (&td->dpa_range)) { rc =devm_cxl_dpa_reserve (cxled, td->dpa_range.start,range_len (&td->dpa_range), - td->cxled.skip); + td->cxled.cxld.skip);if (rc) {init_disabled_mock_decoder (cxld);return rc;
@@ -889,7 +889,7 @@ static void __cxld_registry_save(struct cxl_test_decoder *td,cxld_copy (&td->cxled.cxld, cxld); td->cxled.state = cxled->state; - td->cxled.skip = cxled->skip; + td->cxled.cxld.skip = cxled->cxld.skip;if (!(cxld->flags &CXL_DECODER_F_ENABLE )) { td->dpa_range.start =0 ;
@@ -977,7 +977,7 @@ static void mock_decoder_reset(struct cxl_decoder *cxld)to_cxl_endpoint_decoder (&cxld->dev); cxled->state =CXL_DECODER_STATE_MANUAL ; - cxled->skip =0 ; + cxled->cxld.skip =0 ; }if (decoder_reset_preserve_registry) dev_dbg(port->uport_dev, "decoder%d: skip registry update\n",
@@ -1028,7 +1028,7 @@ static void init_disabled_mock_decoder(struct cxl_decoder *cxld)to_cxl_endpoint_decoder (&cxld->dev); cxled->state =CXL_DECODER_STATE_MANUAL ; - cxled->skip =0 ; + cxled->cxld.skip =0 ; } }
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 01/11] cxl: Split decoder programming into a reusable helper
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Move common HDM decoder settings to
include/cxl/cxl.hand route the register programming sequence throughcxl_commit(). This lets reset code restore cached HDM state without depending on privatecxl_coretypes while keepinghdm.cin charge of the existing commit policy checks.Build the low-level HDM helper under
CONFIG_CXL_HDMso it is available even whencxl_coreis modular.Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH v9 01/11] cxl: Split decoder programming into a reusable helper
In reply to Srirangan MadhavanQuoted 20 lines · Srirangan Madhavan
Move common HDM decoder settings to
include/cxl/cxl.hand route the register programming sequence throughcxl_commit(). This lets reset code restore cached HDM state without depending on privatecxl_coretypes while keepinghdm.cin charge of the existing commit policy checks.Build the low-level HDM helper under
CONFIG_CXL_HDMso it is available even whencxl_coreis modular.Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/Kconfig | 4 ++ drivers/cxl/core/Makefile | 1 + drivers/cxl/core/hdm.c | 122 ++++------------------------------- drivers/cxl/core/region.c | 6 +- drivers/cxl/core/reset.c | 118 +++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 43 ------------ include/cxl/cxl.h | 55 +++++++++++++++- tools/testing/cxl/test/cxl.c | 10 +-- 8 files changed, 197 insertions(+), 162 deletions(-)create mode 100644
drivers/cxl/core/reset.c
<snip>
Quoted 12 lines
/*
* Track whether this decoder is free for userspace provisioning, reserved for
* region autodiscovery, whether it is started connecting (awaiting otherdrivers/cxl/cxl.h +0
-1@@ -322,7 +281,6 @@ enum cxl_decoder_state {* struct cxl_endpoint_decoder - Endpoint / SPA to DPA decoder * @cxld: base cxl_decoder_object * @dpa_res: actively claimed DPA span of this decoder -* @skip: offset into @dpa_res where @cxld.hpa_range maps * @state: autodiscovery state * @part: partition index this decoder maps * @pos: interleave position in @cxld.region
drivers/cxl/cxl.h enum cxl_decoder_state { · line 333
Quoted 4 lines
drivers/cxl/cxl.h +0
-1@@ -330,7 +288,6 @@ enum cxl_decoder_state {struct cxl_endpoint_decoder {struct cxl_decoder cxld;struct resource *dpa_res; -resource_size_t skip;
I understand this change moving skip to the cxl_decoder struct, but I do not undertand why dpa_res is not moved as well since skip refers to an offset inside dpa_res. Doesn't it?
If I am right this assumes only one decoder and mapping all the DPA. Is this the intention?
PATCH v9 2/11 cxl: Cache decoder settings on PCI devices
In reply to Srirangan MadhavanCache CXL core's HDM decoder settings in pci_dev->hdm as decoders are enumerated, committed, or reset. PCI reset paths can use this snapshot to restore HDM programming without walking CXL topology during reset recovery.
--- drivers/cxl/core/hdm.c | 81 +++++++++++++++++++++++++++++++++++++++++- include/cxl/cxl.h | 12 +++++++ include/linux/pci.h | 6 ++++ 3 files changed, 98 insertions(+), 1 deletion(-)
drivers/cxl/core/hdm.c +80 -1
@@ -84,6 +84,76 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) cxlhdm->iw_cap_mask |=BIT (16 ); } +static void clear_hdm_info (void *data) +{ +struct pci_dev *pdev = data; + +WRITE_ONCE (pdev->hdm,NULL ); +} + +static int devm_cxl_pci_setup_hdm_info (struct cxl_hdm *cxlhdm) +{ +struct cxl_port *port = cxlhdm->port; +struct cxl_hdm_info *info; +struct pci_dev *pdev; +struct device *uport; + +if (is_cxl_endpoint (port)) { +struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport_dev); + + uport = cxlmd->dev.parent; + }else { + uport = port->uport_dev; + } + +if (!dev_is_pci (uport)) +return 0 ; + + pdev =to_pci_dev (uport); + info =devm_kzalloc (&pdev->dev, +struct_size (info, settings, cxlhdm->decoder_count), +GFP_KERNEL ); +if (!info) +return -ENOMEM ; + + info->decoder_count = cxlhdm->decoder_count; +WRITE_ONCE (pdev->hdm, info); + +return devm_add_action_or_reset (&pdev->dev, clear_hdm_info, pdev); +} + +static void cxl_hdm_info_set_decoder (struct cxl_hdm *cxlhdm, +struct cxl_decoder *cxld) +{ +struct cxl_port *port = cxlhdm->port; +struct cxl_hdm_info *info; +struct pci_dev *pdev; +struct device *uport; + +if (is_cxl_endpoint (port)) { +struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport_dev); + + uport = cxlmd->dev.parent; + }else { + uport = port->uport_dev; + } + +if (!dev_is_pci (uport)) +return ; + + pdev =to_pci_dev (uport); + info =READ_ONCE (pdev->hdm); +if (!info || cxld->id >= info->decoder_count) +return ; + +if (cxld->flags &CXL_DECODER_F_ENABLE ) + info->settings[cxld->id] = cxld->settings; +else + info->settings[cxld->id] = (struct cxl_decoder_settings ) { + .id = cxld->id, + }; +} +static bool should_emulate_decoders (struct cxl_endpoint_dvsec_info *info) {struct cxl_hdm *cxlhdm;
@@ -747,6 +817,7 @@ static int cxl_decoder_commit(struct cxl_decoder *cxld)return rc; } port->commit_end++; +cxl_hdm_info_set_decoder (cxlhdm, cxld);return 0 ; }
@@ -819,6 +890,7 @@ static void cxl_decoder_reset(struct cxl_decoder *cxld)writel (0 , hdm +CXL_HDM_DECODER0_BASE_LOW_OFFSET (id)); cxld->flags &= ~CXL_DECODER_F_ENABLE ; +cxl_hdm_info_set_decoder (cxlhdm, cxld);/* Userspace is now responsible for reconfiguring this decoder */ if (is_endpoint_decoder (&cxld->dev)) {
@@ -989,6 +1061,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, lo =readl (hdm +CXL_HDM_DECODER0_TL_LOW (which)); hi =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (which)); target_list.value = (hi <<32 ) + lo; + cxld->targets = target_list.value;for (i =0 ; i < cxld->interleave_ways; i++) cxld->target_map[i] = target_list.target_id[i];
@@ -1062,11 +1135,16 @@ static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,struct cxl_port *port = cxlhdm->port;int i;u64 dpa_base =0 ; +int rc;cxl_settle_decoders (cxlhdm); + rc =devm_cxl_pci_setup_hdm_info (cxlhdm); +if (rc) +return rc; +for (i =0 ; i < cxlhdm->decoder_count; i++) { -int rc, target_count = cxlhdm->target_count; +int target_count = cxlhdm->target_count;struct cxl_decoder *cxld;if (is_cxl_endpoint (port)) {
@@ -1101,6 +1179,7 @@ static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,put_device (&cxld->dev);return rc; } +cxl_hdm_info_set_decoder (cxlhdm, cxld); rc =add_hdm_decoder (port, cxld);if (rc) { dev_warn(&port->dev,
include/cxl/cxl.h +12 -0
@@ -121,6 +121,18 @@ struct cxl_regs { ); }; +/** +* struct cxl_hdm_info - PCI device HDM decoder programming cache +* @decoder_count: number of decoder settings entries +* @regs: mapped CXL component registers for this HDM decoder block +* @settings: cached per-decoder programming state +*/ +struct cxl_hdm_info { +int decoder_count; +struct cxl_component_regs regs; + struct cxl_decoder_settings settings[] __counted_by(decoder_count); +}; +int cxl_commit (struct cxl_decoder_settings *settings,void __iomem *hdm);struct cxl_reg_map {
include/linux/pci.h +6 -0
@@ -335,6 +335,9 @@ struct pcie_link_state;struct pci_sriov ;struct pci_p2pdma ;struct rcec_ea ; +#ifdef CONFIG_CXL_HDM +struct cxl_hdm_info ; +#endif /* struct pci_dev - describes a PCI device *
@@ -562,6 +565,9 @@ struct pci_dev {#ifdef CONFIG_PCI_DOE struct xarray doe_mbs;/* Data Object Exchange mailboxes */ #endif +#ifdef CONFIG_CXL_HDM +struct cxl_hdm_info *hdm;/* CXL HDM decoder reset state */ +#endif #ifdef CONFIG_PCI_NPEM struct npem *npem;/* Native PCIe Enclosure Management */ #endif
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 02/11] cxl: Cache decoder settings on PCI devices
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
New issues:
- [High] Resource Leak / Devres Lifecycle Bug
Pre-existing issues:
- [Critical] Out-of-bounds Read / Information Leak
- [High] Endianness / Byte Order Logic Error --
--- Patch [2]: [PATCH v9 02/11] cxl: Cache decoder settings on PCI devices --- commit b30e35d89911fd38d088f7e8ed1637175c1c6ae9
Author: Srirangan Madhavan <[email protected]>
cxl: Cache decoder settings on PCI devices
This commit caches CXL core's HDM decoder settings in pci_dev->hdm as decoders are enumerated, committed, or reset. PCI reset paths can use this snapshot to restore HDM programming without walking CXL topology during reset recovery.
Quoted 4 lines
drivers/cxl/core/hdm.c +0
-0
[ ... ]
Quoted 6 lines
drivers/cxl/core/hdm.c +6
-0+static int devm_cxl_pci_setup_hdm_info (struct cxl_hdm *cxlhdm) +{ +struct cxl_port *port = cxlhdm->port; +struct cxl_hdm_info *info; +struct pci_dev *pdev; +struct device *uport;
[ ... ]
Quoted 5 lines
drivers/cxl/core/hdm.c +5
-0+if (!dev_is_pci (uport)) +return 0 ; + + pdev =to_pci_dev (uport); + info = devm_kzalloc(&pdev->dev,
drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 123
Quoted 10 lines
drivers/cxl/core/hdm.c +10
-0+struct_size (info, settings, cxlhdm->decoder_count), + GFP_KERNEL); +if (!info) +return -ENOMEM ; + + info->decoder_count = cxlhdm->decoder_count; +WRITE_ONCE (pdev->hdm, info); + +return devm_add_action_or_reset (&pdev->dev, clear_hdm_info, pdev); +}
[Severity: High]
Could this cause a memory leak if the CXL port driver is unbound and rebound by a user via sysfs?
Because pdev is the long-lived parent PCI device rather than the child cxl_port device being probed, these devres resources are tied to the parent's lifecycle. Repeated bind/unbind cycles on the port driver could cause the devres allocations and cleanup actions to accumulate infinitely on the parent device.
[ ... ]
drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1063
Quoted 4 lines
drivers/cxl/core/hdm.c +0
-0@@ -989,6 +1061,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, lo =readl (hdm +CXL_HDM_DECODER0_TL_LOW (which)); hi =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (which)); target_list.value = (hi <<32 ) + lo;
[Severity: High]
This is a pre-existing issue, but does this byte-by-byte access of a native 64-bit integer break on big-endian architectures?
Since hi and lo are correctly read as native 32-bit CPU-endian integers and combined into a native 64-bit integer, accessing individual bytes via target_list.target_id[i] on a big-endian system will place the most significant byte at index 0. This would reverse the target IDs and result in incorrect downstream port routing.
drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1066
Quoted 3 lines
drivers/cxl/core/hdm.c +1
-0+ cxld->targets = target_list.value;for (i =0 ; i < cxld->interleave_ways; i++) cxld->target_map[i] = target_list.target_id[i];
[Severity: Critical]
This isn't a bug introduced by this patch, but can this loop read past the end of the 8-byte union and leak kernel stack memory?
Looking at parse_hdm_decoder_caps(), cxld->interleave_ways can be set to 16. The target_list union is only 8 bytes long (containing an 8-byte u64 value and an 8-byte unsigned char target_id[8]). If a device reports a 16-way interleave, this loop will read uninitialized stack memory from indices 8 through 15, which could then be exposed to userspace via the sysfs target_list attribute.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v9 3/11 cxl: Share HDM decoder decode logic
In reply to Srirangan MadhavanMove HDM decoder register decoding into a helper shared by normal CXL core enumeration and early PCI HDM cache setup. This keeps validation of base, size, interleave, target type, and enable state in one place before adding another HDM parser.
--- drivers/cxl/core/hdm.c | 61 +++++++++++----------------------------- drivers/cxl/core/reset.c | 45 +++++++++++++++++++++++++++++ include/cxl/cxl.h | 3 ++ 3 files changed, 64 insertions(+), 45 deletions(-)
drivers/cxl/core/hdm.c +16 -45
@@ -952,6 +952,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,u32 remainder;int i, rc;u32 ctrl; +struct cxl_decoder_settings settings;union {u64 value;unsigned char target_id[8 ];
@@ -968,35 +969,28 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, lo =readl (hdm +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (which)); hi =readl (hdm +CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (which)); size = (hi <<32 ) + lo; - committed = !!(ctrl &CXL_HDM_DECODER0_CTRL_COMMITTED ); + lo =readl (hdm +CXL_HDM_DECODER0_TL_LOW (which)); + hi =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (which)); + target_list.value = (hi <<32 ) + lo; + rc =cxl_hdm_decode_decoder (&settings, which, ctrl, base, size, + target_list.value, &committed); +if (rc) { +dev_warn (&port->dev, +"decoder%d.%d: Invalid decoder configuration (ctrl: %#x): %d\n" , + port->id, cxld->id, ctrl, rc); +return rc; + } + cxld->commit = cxl_decoder_commit; cxld->reset = cxl_decoder_reset; - -if (!committed) - size =0 ; -if (base ==U64_MAX || size ==U64_MAX ) { -dev_warn (&port->dev,"decoder%d.%d: Invalid resource range\n" , - port->id, cxld->id); -return -ENXIO ; - } + cxld->settings = settings; + size =range_len (&cxld->hpa_range);if (info) cxled =to_cxl_endpoint_decoder (&cxld->dev); - cxld->hpa_range = (struct range ) { - .start = base, - .end = base + size -1 , - };/* decoders are enabled if committed */ if (committed) { - cxld->flags |=CXL_DECODER_F_ENABLE ; -if (ctrl &CXL_HDM_DECODER0_CTRL_LOCK ) - cxld->flags |=CXL_DECODER_F_LOCK ; -if (FIELD_GET (CXL_HDM_DECODER0_CTRL_HOSTONLY , ctrl)) - cxld->target_type =CXL_DECODER_HOSTONLYMEM ; -else - cxld->target_type =CXL_DECODER_DEVMEM ; -guard (rwsem_write)(&cxl_rwsem.region);if (cxld->id !=cxl_num_decoders_committed (port)) { dev_warn(&port->dev,
@@ -1036,32 +1030,11 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (which)); } } - rc =eiw_to_ways (FIELD_GET (CXL_HDM_DECODER0_CTRL_IW_MASK , ctrl), - &cxld->interleave_ways); -if (rc) { -dev_warn (&port->dev, -"decoder%d.%d: Invalid interleave ways (ctrl: %#x)\n" , - port->id, cxld->id, ctrl); -return rc; - } - rc =eig_to_granularity (FIELD_GET (CXL_HDM_DECODER0_CTRL_IG_MASK , ctrl), - &cxld->interleave_granularity); -if (rc) { -dev_warn (&port->dev, -"decoder%d.%d: Invalid interleave granularity (ctrl: %#x)\n" , - port->id, cxld->id, ctrl); -return rc; - } -dev_dbg (&port->dev,"decoder%d.%d: range: %#llx-%#llx iw: %d ig: %d\n" , port->id, cxld->id, cxld->hpa_range.start, cxld->hpa_range.end, cxld->interleave_ways, cxld->interleave_granularity);if (!cxled) { - lo =readl (hdm +CXL_HDM_DECODER0_TL_LOW (which)); - hi =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (which)); - target_list.value = (hi <<32 ) + lo; - cxld->targets = target_list.value;for (i =0 ; i < cxld->interleave_ways; i++) cxld->target_map[i] = target_list.target_id[i];
@@ -1078,9 +1051,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, port->id, cxld->id, size, cxld->interleave_ways);return -ENXIO ; } - lo =readl (hdm +CXL_HDM_DECODER0_SKIP_LOW (which)); - hi =readl (hdm +CXL_HDM_DECODER0_SKIP_HIGH (which)); - skip = (hi <<32 ) + lo; + skip = cxld->skip; rc =devm_cxl_dpa_reserve (cxled, *dpa_base + skip, dpa_size, skip);if (rc) { dev_err(&port->dev,
drivers/cxl/core/reset.c +45 -0
48 lines not on this page · raw message
include/cxl/cxl.h +3 -0
@@ -133,6 +133,9 @@ struct cxl_hdm_info { struct cxl_decoder_settings settings[] __counted_by(decoder_count); }; +int cxl_hdm_decode_decoder (struct cxl_decoder_settings *settings,int id, +u32 ctrl,u64 base,u64 size,u64 target_or_skip, +bool *committed);int cxl_commit (struct cxl_decoder_settings *settings,void __iomem *hdm);struct cxl_reg_map {
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 03/11] cxl: Share HDM decoder decode logic
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] Out-of-bounds stack read leaking kernel memory to userspace via decoder target maps when processing decoders with more than 8 interleave ways.
- [High] Endianness bug when extracting 8-bit Target IDs via a 64-bit union, leading to broken routing on big-endian architectures. --
--- Patch [3]: [PATCH v9 03/11] cxl: Share HDM decoder decode logic --- commit 0c55ebf0416f52a7748df44eabd5450eb7aba244
Author: Srirangan Madhavan <[email protected]>
cxl: Share HDM decoder decode logic
This commit moves HDM decoder register decoding into a shared helper function. It consolidates the validation of base, size, interleave, target type, and enable state to simplify early PCI HDM cache setup.
Quoted 13 lines
drivers/cxl/core/hdm.c +1
-0@@ -952,6 +952,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,u32 remainder;int i, rc;u32 ctrl; +struct cxl_decoder_settings settings;union {u64 value;unsigned char target_id[8 ];}
target_list;
[ ... ]
Quoted 1 line
drivers/cxl/core/hdm.c +0
-0@@ -1036,32 +1030,11 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
[ ... ]
drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1039
Quoted 7 lines
drivers/cxl/core/hdm.c +0
-4if (!cxled) { - lo =readl (hdm +CXL_HDM_DECODER0_TL_LOW (which)); - hi =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (which)); - target_list.value = (hi <<32 ) + lo; - cxld->targets = target_list.value;for (i =0 ; i < cxld->interleave_ways; i++) cxld->target_map[i] = target_list.target_id[i];
[Severity: High]
This isn't a bug introduced by this patch, but does this loop cause an out-of-bounds stack read if cxld->interleave_ways is greater than 8?
The target_list union is only 8 bytes wide, but CXL decoders can have up to 16 interleave ways. This would result in reading uninitialized stack memory into the target_map array.
[Severity: High]
This is a pre-existing issue, but can using a union to map a 64-bit integer to a byte array lead to endianness issues here?
On big-endian CPUs, target_list.value would store the bytes in reverse order compared to little-endian CPUs, causing the loop to map memory to the wrong routing targets.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v9 4/11 cxl: Cache endpoint decoder settings during PCI enumeration
In reply to Srirangan MadhavanPopulate pci_dev->hdm from PCI capability initialization for CXL.mem functions. If Memory Space Enable is clear, temporarily set it while reading HDM MMIO and restore the original PCI_COMMAND value before returning. This gives driver-free reset paths an early HDM snapshot.
CXL core later reuses and refreshes the same cache. Move the register helpers into the built-in CONFIG_CXL_HDM set so the early cache path is available without cxl_core.
--- drivers/cxl/core/Makefile | 3 +- drivers/cxl/core/hdm.c | 59 ++++---- drivers/cxl/core/regs.c | 4 + drivers/cxl/core/reset.c | 288 ++++++++++++++++++++++++++++++++++++++ drivers/pci/probe.c | 3 + include/cxl/cxl.h | 31 +++- 6 files changed, 349 insertions(+), 39 deletions(-)
drivers/cxl/core/Makefile +1 -2
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_BUS) += cxl_core.o -obj-$(CONFIG_CXL_HDM) += reset.o +obj-$(CONFIG_CXL_HDM) += regs.o reset.o obj-$(CONFIG_CXL_SUSPEND) += suspend.o ccflags-y += -I$(srctree)/drivers/cxl
@@ -8,7 +8,6 @@ CFLAGS_trace.o = -DTRACE_INCLUDE_PATH=. -I$(src) cxl_core-y := port.o cxl_core-y += pmem.o -cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.o cxl_core-y += pci.o
drivers/cxl/core/hdm.c +25 -34
96 lines not on this page · raw message
drivers/cxl/core/regs.c +4 -0
@@ -199,6 +199,7 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,return ret_val; } +EXPORT_SYMBOL_NS_GPL (devm_cxl_iomap_block,"CXL" ); int cxl_map_component_regs(const struct cxl_register_map *map, struct cxl_component_regs *regs,
@@ -517,6 +518,7 @@ u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb)return offset; } +EXPORT_SYMBOL_NS_GPL (cxl_rcrb_to_aer,"CXL" );static resource_size_t cxl_rcrb_to_linkcap (struct device *dev,struct cxl_dport *dport) {
@@ -633,6 +635,7 @@ resource_size_t __rcrb_to_component(struct device *dev, struct cxl_rcrb_info *rireturn component_reg_phys; } +EXPORT_SYMBOL_NS_GPL (__rcrb_to_component,"CXL" );resource_size_t cxl_rcd_component_reg_phys (struct device *dev,struct cxl_dport *dport)
@@ -641,3 +644,4 @@ resource_size_t cxl_rcd_component_reg_phys(struct device *dev,return CXL_RESOURCE_NONE ;return __rcrb_to_component (dev, &dport->rcrb,CXL_RCRB_UPSTREAM ); } +EXPORT_SYMBOL_NS_GPL (cxl_rcd_component_reg_phys,"CXL" );
drivers/cxl/core/reset.c +288 -0
300 lines not on this page · raw message
drivers/pci/probe.c +3 -0
@@ -24,6 +24,7 @@#include <linux/pm_runtime.h> #include <linux/bitfield.h> #include <trace/events/pci.h> +#include <cxl/cxl.h> #include "pci.h" static struct resource busn_resource = {
@@ -2484,6 +2485,7 @@ static void pci_release_dev(struct device *dev)struct pci_dev *pci_dev; pci_dev =to_pci_dev (dev); +pci_cxl_hdm_release (pci_dev);pci_release_capabilities (pci_dev);pci_release_of_node (pci_dev);pcibios_release_device (pci_dev);
@@ -2674,6 +2676,7 @@ static void pci_init_capabilities(struct pci_dev *dev)pci_rebar_init (dev);/* Resizable BAR */ pci_dev3_init (dev);/* Device 3 capabilities */ pci_ide_init (dev);/* Link Integrity and Data Encryption */ +pci_cxl_hdm_init (dev);/* CXL HDM Decoder Capability */ pcie_report_downtraining (dev);pci_init_reset_methods (dev);
include/cxl/cxl.h +28 -3
@@ -26,6 +26,7 @@ enum cxl_devtype { };struct cxl_region ; +struct pci_dev ;enum cxl_decoder_type {CXL_DECODER_DEVMEM =2 ,
@@ -121,22 +122,46 @@ struct cxl_regs { ); }; +#define CXL_HDM_DECODER_MAX_COUNT 32 + +struct cxl_hdm_decoder_state ; +/** * struct cxl_hdm_info - PCI device HDM decoder programming cache * @decoder_count: number of decoder settings entries -* @regs: mapped CXL component registers for this HDM decoder block +* @hdm_bar: BAR containing the HDM decoder registers +* @hdm_offset: HDM decoder register offset relative to @hdm_bar +* @hdm_size: HDM decoder register resource size +* @global_ctrl: cached HDM decoder global control register +* @decoder_state: cached raw per-decoder register state * @settings: cached per-decoder programming state */ struct cxl_hdm_info {int decoder_count; -struct cxl_component_regs regs; - struct cxl_decoder_settings settings[] __counted_by(decoder_count); +int hdm_bar; +resource_size_t hdm_offset; +resource_size_t hdm_size; +u32 global_ctrl; +struct cxl_hdm_decoder_state *decoder_state; +struct cxl_decoder_settings settings[CXL_HDM_DECODER_MAX_COUNT ]; };int cxl_hdm_decode_decoder (struct cxl_decoder_settings *settings,int id,u32 ctrl,u64 base,u64 size,u64 target_or_skip,bool *committed);int cxl_commit (struct cxl_decoder_settings *settings,void __iomem *hdm); +#ifdef CONFIG_CXL_HDM +void pci_cxl_hdm_init (struct pci_dev *pdev); +void pci_cxl_hdm_release (struct pci_dev *pdev); +#else +static inline void pci_cxl_hdm_init (struct pci_dev *pdev) +{ +} + +static inline void pci_cxl_hdm_release (struct pci_dev *pdev) +{ +} +#endif struct cxl_reg_map {bool valid;
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 04/11] cxl: Cache endpoint decoder settings during PCI enumeration
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Enabling PCI_COMMAND_MEMORY and mapping unassigned BARs during early PCI enumeration causes severe…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Enabling PCI_COMMAND_MEMORY and mapping unassigned BARs during early PCI enumeration causes severe resource conflicts and system instability.
- [Medium] Mixing goto-based error handling with scope-based cleanup helpers (scoped_guard()), leading to a logical fall-through bug. --
--- Patch [4]: [PATCH v9 04/11] cxl: Cache endpoint decoder settings during PCI enumeration --- commit 64ea438bc1dd54d340a46889f9f340251bdf644f
Author: Srirangan Madhavan <[email protected]>
cxl: Cache endpoint decoder settings during PCI enumeration
This patch populates pci_dev->hdm from PCI capability initialization for CXL.mem functions. It temporarily sets memory space enable to read HDM MMIO and restores the original configuration value.
Quoted 3 lines
drivers/cxl/core/reset.c +0
-0
[ ... ]
Quoted 2 lines
drivers/cxl/core/reset.c +2
-0+static int __pci_cxl_hdm_init (struct pci_dev *pdev) +{
[ ... ]
drivers/cxl/core/reset.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 339
Quoted 6 lines
drivers/cxl/core/reset.c +6
-0+if (!(command &PCI_COMMAND_MEMORY )) + restore_command = true; + +if (restore_command) { + rc =pci_write_config_word (pdev,PCI_COMMAND , + command |PCI_COMMAND_MEMORY );
[Severity: High]
Is it safe to forcefully enable memory decoding before the PCI core has verified or assigned safe base address registers?
Since this function is called during early enumeration in pci_init_capabilities():
drivers/pci/probe.c:pci_init_capabilities() {
...
pci_ide_init(dev);
pci_cxl_hdm_init(dev);
...
}Could uninitialized or overlapping physical addresses cause the endpoint to decode transactions intended for system memory, leading to resource conflicts or machine check exceptions?
Quoted 3 lines
drivers/cxl/core/reset.c +3
-0+if (rc) +return pcibios_err_to_errno (rc); + }
[ ... ]
drivers/cxl/core/reset.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 414
Quoted 3 lines
drivers/cxl/core/reset.c +3
-0+ scoped_guard(rwsem_write, &cxl_rwsem.dpa) { +if (pdev->hdm) + goto out_unmap;
[Severity: Medium]
Could this goto inside a scoped_guard() block lead to logical fall-through bugs?
The cleanup subsystem guidelines advise against mixing goto and scope-based cleanups in the same function. If pdev->hdm is already set, execution jumps to out_unmap and then falls through to out_restore_command.
Could this cause the configuration write to be re-executed redundantly during the error handling fall-through?
Quoted 4 lines
drivers/cxl/core/reset.c +4
-0+ pdev->hdm = info; + } +iounmap (hdm); +return 0 ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Alison Schofield
maintainerRe: [PATCH v9 04/11] cxl: Cache endpoint decoder settings during PCI enumeration
In reply to Srirangan MadhavanQuoted 24 lines · Srirangan Madhavan
Populate
pci_dev->hdmfrom PCI capability initialization for CXL.mem functions. If Memory Space Enable is clear, temporarily set it while reading HDM MMIO and restore the originalPCI_COMMANDvalue before returning. This gives driver-free reset paths an early HDM snapshot.CXL core later reuses and refreshes the same cache. Move the register helpers into the built-in
CONFIG_CXL_HDMset so the early cache path is available withoutcxl_core.Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/core/Makefile | 3 +- drivers/cxl/core/hdm.c | 59 ++++---- drivers/cxl/core/regs.c | 4 + drivers/cxl/core/reset.c | 288 ++++++++++++++++++++++++++++++++++++++ drivers/pci/probe.c | 3 + include/cxl/cxl.h | 31 +++- 6 files changed, 349 insertions(+), 39 deletions(-)drivers/cxl/core/Makefile +0
-0
drivers/cxl/core/Makefile line 4
Quoted 5 lines
drivers/cxl/core/Makefile +1
-1@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_BUS) += cxl_core.o -obj-$(CONFIG_CXL_HDM) += reset.o +obj-$(CONFIG_CXL_HDM) += regs.o reset.o obj-$(CONFIG_CXL_SUSPEND) += suspend.o
Hi Srirangan,
This breaks cxl-test module builds with an 'exported twice' error, 14 symbols are both built-in and in cxl_core.ko.
All from drivers/cxl/core/regs.c
With that fixed up, this set passes the cxl unit test suite :)
tools/testing/cxl/Kbuild +0 -1
@@ -55,7 +55,6 @@ obj-m += cxl_core.o cxl_core-y := $(CXL_CORE_SRC)/port.o cxl_core-y += $(CXL_CORE_SRC)/pmem.o -cxl_core-y += $(CXL_CORE_SRC)/regs.o cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o
-- Alison
Dan Williams (nvidia)
maintainerRe: [PATCH v9 04/11] cxl: Cache endpoint decoder settings during PCI enumeration
In reply to Srirangan MadhavanQuoted 10 lines · Srirangan Madhavan
Populate
pci_dev->hdmfrom PCI capability initialization for CXL.mem functions. If Memory Space Enable is clear, temporarily set it while reading HDM MMIO and restore the originalPCI_COMMANDvalue before returning. This gives driver-free reset paths an early HDM snapshot.CXL core later reuses and refreshes the same cache. Move the register helpers into the built-in
CONFIG_CXL_HDMset so the early cache path is available withoutcxl_core.Signed-off-by Srirangan Madhavan <[email protected]>
[..]
drivers/cxl/core/reset.c
Quoted 4 lines
drivers/cxl/core/reset.c +0
-0
To Dave's point, and now the fact that Manish needs this information too for VFIO, it probably best to call this file drivers/cxl/core/resource.c. The scheme being PCI Core caches PCI MMIO resource in the pdev, and CXL Core caches CXL HPA resources in the pdev.
drivers/cxl/core/reset.c line 2
Quoted 2 lines
drivers/cxl/core/reset.c +0
-0@@ -2,9 +2,16 @@/* Copyright(c) 2026 NVIDIA Corporation. All rights reserved. */
Should be "Copyright (c) 2026 NVIDIA Corporation & Affiliates"
...no requirement for All rights reserved.
Quoted 163 lines
drivers/cxl/core/reset.c +151
-0#include <linux/delay.h> #include <linux/bug.h> +#include <linux/bitfield.h> #include <linux/errno.h> #include <linux/export.h> +#include <linux/io.h> +#include <linux/ioport.h> #include <linux/kernel.h> +#include <linux/pci.h> +#include <linux/slab.h> + +#include <cxlpci.h> #include "cxl.h" #include "core.h" @@ -161,3 +168,284 @@ int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, &settings->interleave_granularity); }EXPORT_SYMBOL_FOR_MODULES (cxl_hdm_decode_decoder,"cxl_core" ); + +struct cxl_hdm_decoder_state { +u32 ctrl; +u32 base_low; +u32 base_high; +u32 size_low; +u32 size_high; +u32 target_low; +u32 target_high; +}; + +void pci_cxl_hdm_release (struct pci_dev *pdev) +{ +struct cxl_hdm_info *info; + + scoped_guard(rwsem_write, &cxl_rwsem.dpa) { + info = pdev->hdm; + pdev->hdm =NULL ; + } +if (!info) +return ; + +kfree (info->decoder_state); +kfree (info); +} + +static int cxl_pci_hdm_find_bar (struct pci_dev *pdev,resource_size_t hdm_start, +resource_size_t hdm_size,int *bar, +resource_size_t *offset) +{ +resource_size_t hdm_end = hdm_start + hdm_size -1 ; + +for (int i =0 ; i <PCI_STD_NUM_BARS ; i++) { +struct resource *res = &pdev->resource[i]; + +if (!pci_resource_len (pdev, i)) +continue ; +if (resource_type (res) !=IORESOURCE_MEM ) +continue ; +if (hdm_start < res->start || hdm_end > res->end) +continue ; + + *bar = i; + *offset = hdm_start - res->start; +return 0 ; + } + +return -ENODEV ; +} + +static void __iomem *cxl_pci_hdm_map (struct pci_dev *pdev, +struct cxl_register_map *map, +struct cxl_hdm_info *info) +{ +struct cxl_reg_map *hdm_map = &map->component_map.hdm_decoder; +resource_size_t hdm_start; +void __iomem *hdm; +int rc; + + hdm_start = map->resource + hdm_map->offset; + info->hdm_size = hdm_map->size; + + rc =cxl_pci_hdm_find_bar (pdev, hdm_start, info->hdm_size, + &info->hdm_bar, &info->hdm_offset); +if (rc) +return ERR_PTR (rc); + + hdm =ioremap (hdm_start, info->hdm_size); +if (!hdm) { +pci_err (pdev,"failed to map CXL HDM decoder registers\n" ); +return ERR_PTR (-ENOMEM ); + } + +return hdm; +} + +static void cxl_pci_hdm_read_decoder_state (struct cxl_hdm_decoder_state *state, +void __iomem *hdm,int id) +{ + state->ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); + state->base_low =readl (hdm +CXL_HDM_DECODER0_BASE_LOW_OFFSET (id)); + state->base_high =readl (hdm +CXL_HDM_DECODER0_BASE_HIGH_OFFSET (id)); + state->size_low =readl (hdm +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (id)); + state->size_high =readl (hdm +CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (id)); + state->target_low =readl (hdm +CXL_HDM_DECODER0_TL_LOW (id)); + state->target_high =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (id)); +} + +static int cxl_pci_hdm_read_decoder (struct pci_dev *pdev, +struct cxl_hdm_decoder_state *state, +struct cxl_decoder_settings *settings, +void __iomem *hdm,int id) +{ +u64 target_or_skip, base, size; +bool committed; +int rc; + +cxl_pci_hdm_read_decoder_state (state, hdm, id); + + base = ((u64 )state->base_high <<32 ) | state->base_low; + size = ((u64 )state->size_high <<32 ) | state->size_low; + target_or_skip = ((u64 )state->target_high <<32 ) | state->target_low; + + rc =cxl_hdm_decode_decoder (settings, id, state->ctrl, base, size, + target_or_skip, &committed); +if (rc) { +pci_err (pdev,"CXL HDM decoder %d has invalid configuration: %d\n" , + id, rc); +return rc; + } +if (!committed) +return 0 ; + +return 0 ; +} + +static int cxl_pci_hdm_capable (struct pci_dev *pdev) +{ +u16 cap; +int dvsec; +int rc; + + dvsec =pci_find_dvsec_capability (pdev,PCI_VENDOR_ID_CXL , +PCI_DVSEC_CXL_DEVICE ); +if (!dvsec) +return -ENOTTY ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CAP , &cap); +if (rc) +return pcibios_err_to_errno (rc); + +if (!(cap &PCI_DVSEC_CXL_MEM_CAPABLE )) +return -ENOTTY ; + +return 0 ; +} + +static int __pci_cxl_hdm_init (struct pci_dev *pdev) +{ +struct cxl_decoder_settings *settings; +struct cxl_register_map map = {0 }; +struct cxl_hdm_info *info; +void __iomem *hdm =NULL ; +bool restore_command = false;
drivers/cxl/core/reset.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 324
Quoted 10 lines
drivers/cxl/core/reset.c +10
-0+bool allocated_info = false; +int decoder_count; +u16 command; +int rc; + + scoped_guard(rwsem_read, &cxl_rwsem.dpa) { + info = pdev->hdm; +if (info) +return 0 ; + }
Is this function going to be reused for refreshing the state of the decoder cache? Does not look like it in this set, so it should be impossible for pdev->hdm to already be enabled, right?
Quoted 82 lines
drivers/cxl/core/reset.c +82
-0+ + rc =cxl_pci_hdm_capable (pdev); +if (rc) +return rc; + + rc =pci_read_config_word (pdev,PCI_COMMAND , &command); +if (rc) +return pcibios_err_to_errno (rc); + +if (!(command &PCI_COMMAND_MEMORY )) + restore_command = true; + +if (restore_command) { + rc =pci_write_config_word (pdev,PCI_COMMAND , + command |PCI_COMMAND_MEMORY ); +if (rc) +return pcibios_err_to_errno (rc); + } + +if (!info) { + info =kzalloc_obj (*info,GFP_KERNEL ); +if (!info) + goto err_nomem; + allocated_info = true; + } + + rc =cxl_find_regblock (pdev,CXL_REGLOC_RBI_COMPONENT , &map); +if (rc) + goto out_restore_command; + + rc =cxl_setup_regs (&map); +if (rc) + goto out_restore_command; + +if (!map.component_map.hdm_decoder.valid) { + rc = -ENODEV ; + goto out_restore_command; + } + + hdm =cxl_pci_hdm_map (pdev, &map, info); +if (IS_ERR (hdm)) { + rc =PTR_ERR (hdm); + hdm =NULL ; + goto out_restore_command; + } + + decoder_count =cxl_hdm_decoder_count (readl (hdm + +CXL_HDM_DECODER_CAP_OFFSET )); +if (decoder_count <0 ) { + rc = decoder_count; + goto out_unmap; + } + +if (decoder_count >CXL_HDM_DECODER_MAX_COUNT ) { + rc = -ENXIO ; + goto out_unmap; + } + +if (info->decoder_count && info->decoder_count != decoder_count) { + rc = -ENXIO ; + goto out_unmap; + } + + info->decoder_count = decoder_count; + info->global_ctrl =readl (hdm +CXL_HDM_DECODER_CTRL_OFFSET ); + info->decoder_state =kcalloc (decoder_count, +sizeof (*info->decoder_state), +GFP_KERNEL ); +if (!info->decoder_state) { + rc = -ENOMEM ; + goto out_unmap; + } + + settings = info->settings; +for (int i =0 ; i < info->decoder_count; i++) { + rc =cxl_pci_hdm_read_decoder (pdev, &info->decoder_state[i], + &settings[i], hdm, i); +if (rc) + goto out_unmap; + } + +if (restore_command) {
drivers/cxl/core/reset.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 416
Quoted 10 lines
drivers/cxl/core/reset.c +10
-0+ rc =pci_write_config_word (pdev,PCI_COMMAND , command); +if (rc) + goto out_restore_failed; + } + + scoped_guard(rwsem_write, &cxl_rwsem.dpa) { +if (pdev->hdm) + goto out_unmap; + pdev->hdm = info; + }
If the plan is to later use this routine to refresh the cached information then how does it know that pdev->hdm is more fresh then the recent @info snapshot? Otherwise this looks out of place because there is no driver to race in this early context.
Quoted 56 lines
drivers/cxl/core/reset.c +35
-0+iounmap (hdm); +return 0 ; + +out_restore_failed: + rc =pcibios_err_to_errno (rc); + goto out_unmap; +err_nomem: + rc = -ENOMEM ; + goto out_restore_command; +out_unmap: +if (hdm) +iounmap (hdm); +out_restore_command: +if (allocated_info) { +kfree (info->decoder_state); +kfree (info); + } +if (restore_command) { +int rc2; + + rc2 =pci_write_config_word (pdev,PCI_COMMAND , command); +if (rc2 && !rc) + rc =pcibios_err_to_errno (rc2); + } +return rc; +} + +void pci_cxl_hdm_init (struct pci_dev *pdev) +{ +int rc; + + rc =__pci_cxl_hdm_init (pdev); +if (rc && rc != -ENOTTY && rc != -ENODEV ) +pci_dbg (pdev,"CXL HDM cache init failed: %d\n" , rc); +}drivers/pci/probe.c +2
-0@@ -24,6 +24,7 @@#include <linux/pm_runtime.h> #include <linux/bitfield.h> #include <trace/events/pci.h> +#include <cxl/cxl.h> #include "pci.h" static struct resource busn_resource = {@@ -2484,6 +2485,7 @@ static void pci_release_dev(struct device *dev)struct pci_dev *pci_dev; pci_dev =to_pci_dev (dev); +pci_cxl_hdm_release (pci_dev);pci_release_capabilities (pci_dev);pci_release_of_node (pci_dev);pcibios_release_device (pci_dev);
drivers/pci/probe.c static void pci_init_capabilities(struct pci_dev *dev) · line 2679
Quoted 4 lines
drivers/pci/probe.c +1
-0@@ -2674,6 +2676,7 @@ static void pci_init_capabilities(struct pci_dev *dev)pci_rebar_init (dev);/* Resizable BAR */ pci_dev3_init (dev);/* Device 3 capabilities */ pci_ide_init (dev);/* Link Integrity and Data Encryption */ +pci_cxl_hdm_init (dev);/* CXL HDM Decoder Capability */
I think Richard has a point that this is too early to be using device MMIO resources. I took a look and the proper place appears to be pci_bus_add_device(). Note how some of the quirks invoked by:
pci_fixup_device(pci_fixup_final, dev);
...do their own command register fixups and ioremap().
PATCH v9 5/11 cxl: Add CXL Device Reset helper
In reply to Srirangan MadhavanAdd an internal CXL Device Reset helper for Type 2 functions that advertise CXL Reset in the CXL Device DVSEC. The helper disables CXL.cache, performs cache writeback when supported, initiates reset with Memory Clear disabled, waits for completion, and re-enables CXL.cache on exit.
Leave the helper unregistered until range validation and reset-scope validation are in place.
--- drivers/cxl/core/reset.c | 224 ++++++++++++++++++++++++++++++++++ include/cxl/cxl.h | 7 ++ include/uapi/linux/pci_regs.h | 13 ++ 3 files changed, 244 insertions(+)
drivers/cxl/core/reset.c +224 -0
@@ -7,6 +7,8 @@#include <linux/export.h> #include <linux/io.h> #include <linux/ioport.h> +#include <linux/iommu.h> +#include <linux/jiffies.h> #include <linux/kernel.h> #include <linux/pci.h> #include <linux/slab.h>
@@ -449,3 +451,225 @@ void pci_cxl_hdm_init(struct pci_dev *pdev)if (rc && rc != -ENOTTY && rc != -ENODEV )pci_dbg (pdev,"CXL HDM cache init failed: %d\n" , rc); } + +/* +* CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings. +* Sec 9.7.3 leaves config-space access behavior undefined for 100 ms after +* initiating CXL Reset, then limits software to CXL Status2 access until +* reset completion, timeout, or error. +*/ +#define CXL_RESET_RRS_WAIT_MS 100 +#define CXL_RESET_STATUS_POLL_MS 20 +static const u32 cxl_reset_timeout_ms[] = { +10 ,100 ,1000 ,10000 ,100000 , +}; + +#define CXL_CACHE_WBI_TIMEOUT_US 100000 +#define CXL_CACHE_WBI_POLL_US 100 + +static int cxl_reset_dvsec (struct pci_dev *pdev) +{ +int dvsec, rc; +u16 cap; + + dvsec =pci_find_dvsec_capability (pdev,PCI_VENDOR_ID_CXL , +PCI_DVSEC_CXL_DEVICE ); +if (!dvsec) +return -ENOTTY ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CAP , &cap); +if (rc) +return pcibios_err_to_errno (rc); + +if ((cap & (PCI_DVSEC_CXL_CACHE_CAPABLE | +PCI_DVSEC_CXL_MEM_CAPABLE )) != + (PCI_DVSEC_CXL_CACHE_CAPABLE |PCI_DVSEC_CXL_MEM_CAPABLE )) +return -ENOTTY ; + +if (!(cap &PCI_DVSEC_CXL_RST_CAPABLE )) +return -ENOTTY ; + +return dvsec; +} + +static int cxl_reset_update_ctrl2 (struct pci_dev *pdev,int dvsec,u16 set, +u16 clear) +{ +u16 cmd =PCI_DVSEC_CXL_INIT_CACHE_WBI |PCI_DVSEC_CXL_INIT_CXL_RST ; +u16 ctrl2; +int rc; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL2 , &ctrl2); +if (rc) +return pcibios_err_to_errno (rc); + + ctrl2 &= ~cmd; + ctrl2 |= set; + ctrl2 &= ~clear; + + rc =pci_write_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL2 , ctrl2); +if (rc) +return pcibios_err_to_errno (rc); + +return 0 ; +} + +static int cxl_reset_enable_cache (struct pci_dev *pdev,int dvsec) +{ +return cxl_reset_update_ctrl2 (pdev, dvsec,0 , +PCI_DVSEC_CXL_DISABLE_CACHING ); +} + +static int cxl_reset_disable_cache (struct pci_dev *pdev,int dvsec,u16 cap) +{ +int remaining_us =CXL_CACHE_WBI_TIMEOUT_US ; +u16 status2; +int rc, rc2; + + rc =cxl_reset_update_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_DISABLE_CACHING ,0 ); +if (rc) +return rc; + +if (!(cap &PCI_DVSEC_CXL_CACHE_WBI_CAPABLE )) +return 0 ; + + rc =cxl_reset_update_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_INIT_CACHE_WBI ,0 ); +if (rc) + goto err_enable_cache; + +do { +usleep_range (CXL_CACHE_WBI_POLL_US ,CXL_CACHE_WBI_POLL_US +1 ); + remaining_us -=CXL_CACHE_WBI_POLL_US ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_STATUS2 , + &status2); +if (rc) { + rc =pcibios_err_to_errno (rc); + goto err_enable_cache; + } + }while (!(status2 &PCI_DVSEC_CXL_CACHE_INV ) && remaining_us >0 ); + +if (!(status2 &PCI_DVSEC_CXL_CACHE_INV )) { + rc = -ETIMEDOUT ; + goto err_enable_cache; + } + +return 0 ; + +err_enable_cache: +/* +* DISABLE_CACHING can be rolled back here. INIT_CACHE_WBI is +* self-clearing on completion, so leave any in-flight writeback alone. +*/ + rc2 =cxl_reset_enable_cache (pdev, dvsec); +if (rc2) +pci_warn (pdev,"failed to re-enable CXL caching: %d\n" , rc2); +return rc; +} + +static int cxl_reset_wait_done (struct pci_dev *pdev,int dvsec,u16 cap) +{ +unsigned long deadline; +u32 timeout_ms; +u16 status2; +int idx, rc; + + idx =FIELD_GET (PCI_DVSEC_CXL_RST_TIMEOUT , cap); +if (idx >=ARRAY_SIZE (cxl_reset_timeout_ms)) { +int last =ARRAY_SIZE (cxl_reset_timeout_ms) -1 ; + +pci_warn (pdev, +"unknown CXL reset timeout encoding %d; using %u ms\n" , + idx, cxl_reset_timeout_ms[last]); + idx = last; + } + + timeout_ms =max_t (u32, cxl_reset_timeout_ms[idx], +CXL_RESET_RRS_WAIT_MS ); + deadline = jiffies +msecs_to_jiffies (timeout_ms); +msleep (CXL_RESET_RRS_WAIT_MS ); + +do { + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_STATUS2 , + &status2); +if (rc || status2 ==U16_MAX ) + goto not_ready; + +if (status2 &PCI_DVSEC_CXL_RST_ERR ) +return -EIO ; + +if (status2 &PCI_DVSEC_CXL_RST_DONE ) +return 0 ; + +not_ready: +if (time_after_eq (jiffies, deadline)) +return -ETIMEDOUT ; + +msleep (CXL_RESET_STATUS_POLL_MS ); + }while (true); +} + +static int cxl_reset_execute (struct pci_dev *pdev,int dvsec) +{ +bool cache_disabled = false; +u16 cap; +int rc; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CAP , &cap); +if (rc) +return pcibios_err_to_errno (rc); + +if (!pci_wait_for_pending_transaction (pdev)) +pci_err (pdev,"timed out waiting for pending transactions\n" ); + + rc =pci_dev_reset_iommu_prepare (pdev); +if (rc) { +pci_err (pdev,"failed to stop IOMMU for CXL reset: %d\n" , rc); +return rc; + } + + rc =cxl_reset_disable_cache (pdev, dvsec, cap); +if (rc) + goto out; + cache_disabled = true; + + rc =cxl_reset_update_ctrl2 (pdev, dvsec,PCI_DVSEC_CXL_INIT_CXL_RST , +PCI_DVSEC_CXL_RST_MEM_CLR_EN ); +if (rc) + goto out; + + rc =cxl_reset_wait_done (pdev, dvsec, cap); +if (rc) + goto out; + +out: +if (cache_disabled) { +int rc2; + + rc2 =cxl_reset_enable_cache (pdev, dvsec); +if (rc2 && rc) +pci_warn (pdev,"failed to re-enable CXL caching: %d\n" , + rc2); +else if (rc2) + rc = rc2; + } + +pci_dev_reset_iommu_done (pdev); +return rc; +} + +int cxl_reset_function (struct pci_dev *pdev,bool probe) +{ +int dvsec; + + dvsec =cxl_reset_dvsec (pdev); +if (dvsec <0 ) +return dvsec; + +if (probe) +return 0 ; + +return cxl_reset_execute (pdev, dvsec); +}
include/cxl/cxl.h +7 -0
@@ -9,6 +9,7 @@#include <linux/node.h> #include <linux/ioport.h> #include <linux/range.h> +#include <linux/errno.h> #include <cxl/mailbox.h> /**
@@ -153,6 +154,7 @@ int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm);#ifdef CONFIG_CXL_HDM void pci_cxl_hdm_init (struct pci_dev *pdev);void pci_cxl_hdm_release (struct pci_dev *pdev); +int cxl_reset_function (struct pci_dev *pdev,bool probe);#else static inline void pci_cxl_hdm_init (struct pci_dev *pdev) {
@@ -161,6 +163,11 @@ static inline void pci_cxl_hdm_init(struct pci_dev *pdev)static inline void pci_cxl_hdm_release (struct pci_dev *pdev) { } + +static inline int cxl_reset_function (struct pci_dev *pdev,bool probe) +{ +return -ENOTTY ; +}#endif struct cxl_reg_map {
include/uapi/linux/pci_regs.h +13 -0
@@ -1352,8 +1352,21 @@#define PCI_DVSEC_CXL_CACHE_CAPABLE _BITUL(0)#define PCI_DVSEC_CXL_MEM_CAPABLE _BITUL(2)#define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4) +#define PCI_DVSEC_CXL_CACHE_WBI_CAPABLE _BITUL(6) +#define PCI_DVSEC_CXL_RST_CAPABLE _BITUL(7) +#define PCI_DVSEC_CXL_RST_TIMEOUT __GENMASK(10, 8) +#define PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE _BITUL(11)#define PCI_DVSEC_CXL_CTRL 0xC#define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2) +#define PCI_DVSEC_CXL_CTRL2 0x10 +#define PCI_DVSEC_CXL_DISABLE_CACHING _BITUL(0) +#define PCI_DVSEC_CXL_INIT_CACHE_WBI _BITUL(1) +#define PCI_DVSEC_CXL_INIT_CXL_RST _BITUL(2) +#define PCI_DVSEC_CXL_RST_MEM_CLR_EN _BITUL(3) +#define PCI_DVSEC_CXL_STATUS2 0x12 +#define PCI_DVSEC_CXL_CACHE_INV _BITUL(0) +#define PCI_DVSEC_CXL_RST_DONE _BITUL(1) +#define PCI_DVSEC_CXL_RST_ERR _BITUL(2)#define PCI_DVSEC_CXL_RANGE_SIZE_HIGH (i) (0x18 + (i * 0x10))#define PCI_DVSEC_CXL_RANGE_SIZE_LOW (i) (0x1C + (i * 0x10))#define PCI_DVSEC_CXL_MEM_INFO_VALID _BITUL(0)
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 05/11] cxl: Add CXL Device Reset helper
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Incorrect ordering of CXL cache writeback (WBI), pending transaction wait, and IOMMU reset…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Incorrect ordering of CXL cache writeback (WBI), pending transaction wait, and IOMMU reset preparation.
- [Medium] Premature success of WBI wait loop on device disconnect due to unhandled U16_MAX status.
- [Medium] Potential false timeout in the reset wait loop due to a missing final hardware status check. --
--- Patch [5]: [PATCH v9 05/11] cxl: Add CXL Device Reset helper --- commit c832f1294f231fc19fa036ee29639b20e723c1a1
Author: Srirangan Madhavan <[email protected]>
cxl: Add CXL Device Reset helper
This commit adds an internal CXL Device Reset helper for Type 2 functions that advertise CXL Reset in the CXL Device DVSEC. The helper handles cache disabling, writeback, reset initiation, and re-enabling caching.
Quoted 4 lines
drivers/cxl/core/reset.c +0
-0
[ ... ]
Quoted 20 lines
drivers/cxl/core/reset.c +20
-0+static int cxl_reset_disable_cache (struct pci_dev *pdev,int dvsec,u16 cap) +{ +int remaining_us =CXL_CACHE_WBI_TIMEOUT_US ; +u16 status2; +int rc, rc2; + + rc =cxl_reset_update_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_DISABLE_CACHING ,0 ); +if (rc) +return rc; + +if (!(cap &PCI_DVSEC_CXL_CACHE_WBI_CAPABLE )) +return 0 ; + + rc =cxl_reset_update_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_INIT_CACHE_WBI ,0 ); +if (rc) + goto err_enable_cache; + + do {
drivers/cxl/core/reset.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 552
Quoted 10 lines
drivers/cxl/core/reset.c +10
-0+usleep_range (CXL_CACHE_WBI_POLL_US ,CXL_CACHE_WBI_POLL_US +1 ); + remaining_us -=CXL_CACHE_WBI_POLL_US ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_STATUS2 , + &status2); +if (rc) { + rc =pcibios_err_to_errno (rc); + goto err_enable_cache; + } + }while (!(status2 &PCI_DVSEC_CXL_CACHE_INV ) && remaining_us >0 );
[Severity: Medium]
If the device is disconnected or offline, pci_read_config_word() will return success (0) and set status2 to U16_MAX (0xFFFF). Would this cause the loop condition !(status2 & PCI_DVSEC_CXL_CACHE_INV) to evaluate to false and bypass the wait entirely, falsely reporting success?
Quoted 7 lines
drivers/cxl/core/reset.c +7
-0+ +if (!(status2 &PCI_DVSEC_CXL_CACHE_INV )) { + rc = -ETIMEDOUT ; + goto err_enable_cache; + } + +return 0 ;
[ ... ]
Quoted 2 lines
drivers/cxl/core/reset.c +2
-0+static int cxl_reset_wait_done (struct pci_dev *pdev,int dvsec,u16 cap) +{
[ ... ]
Quoted 5 lines
drivers/cxl/core/reset.c +5
-0+ do { + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2, + &status2); + if (rc || status2 == U16_MAX) + goto not_ready;
drivers/cxl/core/reset.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 608
Quoted 10 lines
drivers/cxl/core/reset.c +10
-0+ +if (status2 &PCI_DVSEC_CXL_RST_ERR ) +return -EIO ; + +if (status2 &PCI_DVSEC_CXL_RST_DONE ) +return 0 ; + +not_ready: +if (time_after_eq (jiffies, deadline)) +return -ETIMEDOUT ;
[Severity: Medium]
Does this timeout check need a final hardware status read?
If a thread preemption, interrupt, or scheduling delay occurs after the config space read but before the timeout check, the hardware condition may be met concurrently while the jiffies have advanced past the deadline. This could potentially lead to a false -ETIMEDOUT error despite the device being ready.
Quoted 15 lines
drivers/cxl/core/reset.c +15
-0+ +msleep (CXL_RESET_STATUS_POLL_MS ); + }while (true); +} + +static int cxl_reset_execute (struct pci_dev *pdev,int dvsec) +{ +bool cache_disabled = false; +u16 cap; +int rc; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CAP , &cap); +if (rc) +return pcibios_err_to_errno (rc); +
drivers/cxl/core/reset.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 633
Quoted 10 lines
drivers/cxl/core/reset.c +10
-0+if (!pci_wait_for_pending_transaction (pdev)) +pci_err (pdev,"timed out waiting for pending transactions\n" ); + + rc =pci_dev_reset_iommu_prepare (pdev); +if (rc) { +pci_err (pdev,"failed to stop IOMMU for CXL reset: %d\n" , rc); +return rc; + } + + rc =cxl_reset_disable_cache (pdev, dvsec, cap);
[Severity: High]
Is it safe to wait for pending transactions and prepare the IOMMU for reset before initiating the CXL cache Write-Back Invalidate (WBI)?
Initiating WBI in cxl_reset_disable_cache() forces the device to flush its dirty cache lines via new DMA transactions. If the IOMMU has already been prepared for reset (which may disable ATS, PRI, and block translations), could these WBI transactions fault and lead to silent data loss or IOMMU errors?
Additionally, creating new pending transactions immediately before asserting the CXL Reset might violate the PCIe requirement to drain all transactions prior to reset. Should cxl_reset_disable_cache() happen before the IOMMU preparation and pending transaction wait?
Quoted 6 lines
drivers/cxl/core/reset.c +6
-0+if (rc) + goto out; + cache_disabled = true; + + rc =cxl_reset_update_ctrl2 (pdev, dvsec,PCI_DVSEC_CXL_INIT_CXL_RST , +PCI_DVSEC_CXL_RST_MEM_CLR_EN );
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
PATCH v9 6/11 cxl: Validate HDM ranges before CXL reset
In reply to Srirangan MadhavanBefore reset, collect enabled cached HDM decoder ranges and reserve them with request_mem_region(). This rejects reset while affected CXL memory is busy and keeps the validation stable through reset.
If CPU cache invalidation support is available, invalidate the affected ranges before reset. If the runtime backend is unavailable, continue after the range reservation succeeds.
--- drivers/cxl/core/reset.c | 240 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 239 insertions(+), 1 deletion(-)
drivers/cxl/core/reset.c +239 -1
@@ -10,6 +10,8 @@#include <linux/iommu.h> #include <linux/jiffies.h> #include <linux/kernel.h> +#include <linux/list.h> +#include <linux/memregion.h> #include <linux/pci.h> #include <linux/slab.h>
@@ -467,6 +469,231 @@ static const u32 cxl_reset_timeout_ms[] = {#define CXL_CACHE_WBI_TIMEOUT_US 100000#define CXL_CACHE_WBI_POLL_US 100 +struct cxl_hdm_range { +struct list_head list; +struct pci_dev *pdev; +struct range hpa_range; +struct resource *res; +}; + +struct cxl_hdm_range_context { +struct list_head ranges; +}; + +static void cxl_hdm_range_context_init (struct cxl_hdm_range_context *ctx) +{ +INIT_LIST_HEAD (&ctx->ranges); +} + +static void cxl_hdm_range_context_destroy (struct cxl_hdm_range_context *ctx) +{ +struct cxl_hdm_range *range, *next; + +list_for_each_entry_safe (range, next, &ctx->ranges, list) { +list_del (&range->list); +if (range->res) +release_mem_region (range->hpa_range.start, +resource_size (range->res)); +kfree (range); + } +} + +static int cxl_hdm_range_add (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev,const struct range *hpa_range) +{ +struct cxl_hdm_range *range; + +if (hpa_range->end < hpa_range->start) +return -EINVAL ; + +list_for_each_entry (range, &ctx->ranges, list) +if (range->hpa_range.start == hpa_range->start && + range->hpa_range.end == hpa_range->end) +return 0 ; + + range =kzalloc_obj (*range); +if (!range) +return -ENOMEM ; + + range->pdev = pdev; + range->hpa_range = *hpa_range; +list_add_tail (&range->list, &ctx->ranges); + +return 0 ; +} + +static int cxl_hdm_ranges_collect (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev) +{ +struct cxl_hdm_info *info =READ_ONCE (pdev->hdm); +int rc; + +if (!info) { +pci_err (pdev,"CXL HDM decoder state unavailable\n" ); +return -ENXIO ; + } + +for (int i =0 ; i < info->decoder_count; i++) { +struct cxl_decoder_settings *settings = &info->settings[i]; + +if (!(settings->flags &CXL_DECODER_F_ENABLE )) +continue ; + +if (settings->flags &CXL_DECODER_F_NORMALIZED_ADDRESSING ) { +pci_err (pdev, +"CXL reset does not support normalized address decoders\n" ); +return -EOPNOTSUPP ; + } + + rc =cxl_hdm_range_add (ctx, pdev, &settings->hpa_range); +if (rc) +return rc; + } + +return 0 ; +} + +static int cxl_hdm_range_len (struct pci_dev *pdev, +const struct range *hpa_range,u64 *len) +{ +if (sizeof (resource_size_t) <sizeof (hpa_range->start) && + (hpa_range->start > (resource_size_t )~0ULL || + hpa_range->end > (resource_size_t )~0ULL )) { +pci_err (pdev, +"CXL reset range [%#llx-%#llx] exceeds resource address size\n" , + hpa_range->start, hpa_range->end); +return -EOVERFLOW ; + } + +if (hpa_range->end < hpa_range->start) +return -EINVAL ; + +if (!hpa_range->start && hpa_range->end ==U64_MAX ) { +pci_err (pdev, +"CXL reset range [%#llx-%#llx] exceeds resource size\n" , + hpa_range->start, hpa_range->end); +return -EOVERFLOW ; + } + + *len =range_len (hpa_range); +if (sizeof (resource_size_t) <sizeof (*len) && + *len > (resource_size_t )~0ULL ) { +pci_err (pdev, +"CXL reset range [%#llx-%#llx] exceeds resource size\n" , + hpa_range->start, hpa_range->end); +return -EOVERFLOW ; + } + +if (sizeof (size_t ) <sizeof (*len) && *len >SIZE_MAX ) { +pci_err (pdev, +"CXL reset range [%#llx-%#llx] exceeds cache flush size\n" , + hpa_range->start, hpa_range->end); +return -EOVERFLOW ; + } + +return 0 ; +} + +static int cxl_hdm_range_request (struct cxl_hdm_range *range) +{ +struct pci_dev *pdev = range->pdev; +const struct range *hpa_range = &range->hpa_range; +u64 len; +int rc; + + rc =cxl_hdm_range_len (pdev, hpa_range, &len); +if (rc) +return rc; + + range->res =request_mem_region (hpa_range->start, len,"cxl_reset" ); +if (!range->res) { +pci_err (pdev, +"cannot reset while CXL memory range is busy [%#llx-%#llx]\n" , + hpa_range->start, hpa_range->end); +return -EBUSY ; + } + +return 0 ; +} + +static int cxl_hdm_ranges_request (struct cxl_hdm_range_context *ctx) +{ +struct cxl_hdm_range *range; +int rc; + +lockdep_assert_held_write (&cxl_rwsem.region); + +list_for_each_entry (range, &ctx->ranges, list) { + rc =cxl_hdm_range_request (range); +if (rc) +return rc; + } + +return 0 ; +} + +static int cxl_hdm_range_flush_cache (struct cxl_hdm_range *range) +{ +struct pci_dev *pdev = range->pdev; +const struct range *hpa_range = &range->hpa_range; +u64 len; +int rc; + + rc =cxl_hdm_range_len (pdev, hpa_range, &len); +if (rc) +return rc; + + rc =cpu_cache_invalidate_memregion (hpa_range->start, len); +if (rc) +pci_err (pdev, +"failed to invalidate CPU cache [%#llx-%#llx]: %d\n" , + hpa_range->start, hpa_range->end, rc); + +return rc; +} + +static int cxl_hdm_ranges_flush_cpu_caches (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev) +{ +struct cxl_hdm_range *range; +int rc; + +if (list_empty (&ctx->ranges)) +return 0 ; + +if (!cpu_cache_has_invalidate_memregion ()) { +pci_warn (pdev, +"CPU cache synchronization unavailable; continuing without cache invalidation\n" ); +return 0 ; + } + +list_for_each_entry (range, &ctx->ranges, list) { + rc =cxl_hdm_range_flush_cache (range); +if (rc) +return rc; + } + +return 0 ; +} + +static int cxl_hdm_ranges_prepare (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev) +{ +int rc; + +lockdep_assert_held_write (&cxl_rwsem.region); + + rc =cxl_hdm_ranges_collect (ctx, pdev); +if (rc) +return rc; + + rc =cxl_hdm_ranges_request (ctx); +if (rc) +return rc; + +return cxl_hdm_ranges_flush_cpu_caches (ctx, pdev); +} +static int cxl_reset_dvsec (struct pci_dev *pdev) {int dvsec, rc;
@@ -662,7 +889,9 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)int cxl_reset_function (struct pci_dev *pdev,bool probe) { +struct cxl_hdm_range_context range_ctx;int dvsec; +int rc; dvsec =cxl_reset_dvsec (pdev);if (dvsec <0 )
@@ -671,5 +900,14 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)if (probe)return 0 ; -return cxl_reset_execute (pdev, dvsec); +cxl_hdm_range_context_init (&range_ctx); + + scoped_guard(rwsem_write, &cxl_rwsem.region) { + rc =cxl_hdm_ranges_prepare (&range_ctx, pdev); +if (!rc) + rc =cxl_reset_execute (pdev, dvsec); + } + +cxl_hdm_range_context_destroy (&range_ctx); +return rc; }
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 06/11] cxl: Validate HDM ranges before CXL reset
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] cxl_hdm_ranges_collect() reads HDM decoder settings without holding the required cxl_rwsem.dpa…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] cxl_hdm_ranges_collect() reads HDM decoder settings without holding the required cxl_rwsem.dpa lock, leading to data races with concurrent modifications.
- [High] cxl_hdm_range_context_destroy() releases memory regions outside the cxl_rwsem.region lock, causing spurious -EBUSY errors for concurrent resets.
- [Medium] Functions contain both goto-based error handling and scope-based cleanup (__free() / scoped_guard()), violating subsystem guidelines. --
--- Patch [6]: [PATCH v9 06/11] cxl: Validate HDM ranges before CXL reset --- commit c90294fef588f0141e340a7198edc3285f1a401d
Author: Srirangan Madhavan <[email protected]>
cxl: Validate HDM ranges before CXL reset
Before reset, collect enabled cached HDM decoder ranges and reserve them with request_mem_region(). This rejects reset while affected CXL memory is busy and keeps the validation stable through reset.
[Severity: Medium]
Is it expected that cxl_restore_hdm() and __pci_cxl_hdm_init() mix goto-based error handling with scope-based cleanup (__free() and scoped_guard())?
The cleanup subsystem guidelines strictly mandate converting all resources to scope-based cleanup or none of them to avoid LIFO ordering mistakes. Mixing these paradigms increases the risk of double-free or resource leak bugs due to confusing ownership semantics.
Quoted 4 lines
drivers/cxl/core/reset.c +0
-0
[ ... ]
Quoted 1 line
drivers/cxl/core/reset.c +0
-0@@ -467,6 +469,231 @@ static const u32 cxl_reset_timeout_ms[] = {
[ ... ]
drivers/cxl/core/reset.c static const u32 cxl_reset_timeout_ms[] = { · line 537
Quoted 13 lines
drivers/cxl/core/reset.c +13
-0+static int cxl_hdm_ranges_collect (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev) +{ +struct cxl_hdm_info *info =READ_ONCE (pdev->hdm); +int rc; + +if (!info) { +pci_err (pdev,"CXL HDM decoder state unavailable\n" ); +return -ENXIO ; + } + +for (int i =0 ; i < info->decoder_count; i++) { +struct cxl_decoder_settings *settings = &info->settings[i];
[Severity: High]
Is it safe to read info->settings[i] here without holding the cxl_rwsem.dpa lock?
If a concurrent userspace PCI reset via sysfs triggers cxl_reset_function() and calls cxl_hdm_ranges_collect(), this code iterates over info->settings holding only cxl_rwsem.region.
Concurrently, an asynchronous device probe calling cxl_hdm_info_set_decoder() could modify info->settings holding only cxl_rwsem.dpa for write.
Could this lead to a torn read of settings->hpa_range, which is then passed to request_mem_region(), potentially causing memory corruption or a kernel panic by requesting a garbage memory region?
Quoted 3 lines
drivers/cxl/core/reset.c +3
-0+ +if (!(settings->flags &CXL_DECODER_F_ENABLE )) +continue ;
[ ... ]
Quoted 1 line
drivers/cxl/core/reset.c +0
-0@@ -662,7 +889,9 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)
[ ... ]
Quoted 12 lines
int
cxl_reset_function(struct pci_dev *pdev, bool probe){drivers/cxl/core/reset.c +2
-0+struct cxl_hdm_range_context range_ctx;int dvsec; +int rc; dvsec =cxl_reset_dvsec (pdev);if (dvsec <0 )@@ -671,5 +900,14 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)if (probe)return 0 ;
drivers/cxl/core/reset.c int cxl_reset_function(struct pci_dev *pdev, bool probe) · line 911
Quoted 10 lines
drivers/cxl/core/reset.c +9
-1-return cxl_reset_execute (pdev, dvsec); +cxl_hdm_range_context_init (&range_ctx); + + scoped_guard(rwsem_write, &cxl_rwsem.region) { + rc =cxl_hdm_ranges_prepare (&range_ctx, pdev); +if (!rc) + rc =cxl_reset_execute (pdev, dvsec); + } + +cxl_hdm_range_context_destroy (&range_ctx);
[Severity: High]
Could this cause spurious -EBUSY errors for concurrent resets?
When two endpoints of an interleaved CXL region are reset concurrently, they both attempt to lock the same hpa_range.
If thread A acquires cxl_rwsem.region, calls request_mem_region(), and finishes the reset block, it then releases cxl_rwsem.region before calling cxl_hdm_range_context_destroy() to release the memory region.
If thread B immediately acquires cxl_rwsem.region and calls request_mem_region() on the same hpa_range, it seems thread B would fail with -EBUSY because thread A still holds the global iomem_resource for this range.
Would it be safer to place the context destruction inside the lock scope to ensure proper serialization?
Quoted 2 lines
drivers/cxl/core/reset.c +1
-0+return rc; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
PATCH v9 7/11 PCI/CXL: Discover the CXL reset scope
In reply to Srirangan MadhavanAdd reset context support to validate that CXL Reset is function-scoped before advertising it as a PCI reset method. Use the Non-CXL Function Map, ARI/devfn rules, and CXL.cache/mem capability bits to reject reset when another same-scope function would also be affected.
If the Function Map cannot be read, warn and conservatively treat all candidate same-scope functions as CXL-capable for scope validation.
--- drivers/cxl/core/reset.c | 152 +++++++++++++++++++++++++++++++++- include/uapi/linux/pci_regs.h | 1 + 2 files changed, 151 insertions(+), 2 deletions(-)
drivers/cxl/core/reset.c +150 -2
@@ -1,5 +1,6 @@// SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2026 NVIDIA Corporation. All rights reserved. */ +#include <linux/bitmap.h> #include <linux/delay.h> #include <linux/bug.h> #include <linux/bitfield.h>
@@ -469,6 +470,21 @@ static const u32 cxl_reset_timeout_ms[] = {#define CXL_CACHE_WBI_TIMEOUT_US 100000#define CXL_CACHE_WBI_POLL_US 100 +/* CXL r4.0 sec 8.1.4 defines 256 bits of Non-CXL Function Map. */ +#define CXL_RESET_MAX_FUNCTIONS 256 +#define CXL_RESET_FUNCTION_MAP_REGS (CXL_RESET_MAX_FUNCTIONS / 32) + +struct cxl_reset_context { +struct pci_dev *target; +}; + +struct cxl_reset_walk_context { +struct cxl_reset_context *ctx; + DECLARE_BITMAP(non_cxl_func_map, CXL_RESET_MAX_FUNCTIONS); +bool ari; +int rc; +}; +struct cxl_hdm_range {struct list_head list;struct pci_dev *pdev;
@@ -480,6 +496,137 @@ struct cxl_hdm_range_context {struct list_head ranges; }; +static void cxl_reset_context_init (struct cxl_reset_context *ctx, +struct pci_dev *pdev) +{ + *ctx = (struct cxl_reset_context ) { + .target = pdev, + }; +} + +static void cxl_reset_read_non_cxl_func_map (struct pci_dev *pdev, +unsigned long *map) +{ +u32 words[CXL_RESET_FUNCTION_MAP_REGS ]; +int dvsec, reg; + +bitmap_zero (map,CXL_RESET_MAX_FUNCTIONS ); + + dvsec =pci_find_dvsec_capability (pdev,PCI_VENDOR_ID_CXL , +PCI_DVSEC_CXL_FUNCTION_MAP ); +if (!dvsec) +return ; + +for (reg =0 ; reg <CXL_RESET_FUNCTION_MAP_REGS ; reg++) { +int offset = dvsec +PCI_DVSEC_CXL_FUNCTION_MAP_REG + + reg *sizeof (u32); +int rc; + + rc =pci_read_config_dword (pdev, offset, &words[reg]); +if (rc) { +pci_warn (pdev, +"failed to read Non-CXL Function Map; treating same-scope functions as CXL\n" ); +bitmap_zero (map,CXL_RESET_MAX_FUNCTIONS ); +return ; + } + } + +bitmap_from_arr32 (map, words,CXL_RESET_MAX_FUNCTIONS ); +} + +static int cxl_reset_func_map_bit (struct pci_dev *sibling,bool ari) +{ +if (ari) +return sibling->devfn; + +/* +* Without ARI, the Function Map is organized as 32 device slots per +* conventional 3-bit function number. +*/ +return PCI_FUNC (sibling->devfn) *32 +PCI_SLOT (sibling->devfn); +} + +static int cxl_reset_read_cxl_cap (struct pci_dev *pdev,u16 *cap) +{ +int dvsec, rc; + + dvsec =pci_find_dvsec_capability (pdev,PCI_VENDOR_ID_CXL , +PCI_DVSEC_CXL_DEVICE ); +if (!dvsec) +return -ENODEV ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CAP , cap); +if (rc) { + rc =pcibios_err_to_errno (rc); +pci_warn (pdev,"failed to read CXL capability: %d\n" , rc); +return rc; + } + +return 0 ; +} + +static int cxl_reset_has_cache_or_mem (struct pci_dev *pdev) +{ +u16 cap; +int rc; + + rc =cxl_reset_read_cxl_cap (pdev, &cap); +if (rc == -ENODEV ) +return 0 ; +if (rc) +return rc; + +return !!(cap & (PCI_DVSEC_CXL_CACHE_CAPABLE | +PCI_DVSEC_CXL_MEM_CAPABLE )); +} + +static int cxl_reset_validate_function_scope (struct pci_dev *sibling, +void *data) +{ +struct cxl_reset_walk_context *wctx = data; +struct cxl_reset_context *ctx = wctx->ctx; +struct pci_dev *pdev = ctx->target; +int fn, rc; + +if (sibling == pdev) +return 0 ; + +if (sibling->bus != pdev->bus) +return 0 ; + +if (!wctx->ari &&PCI_SLOT (sibling->devfn) !=PCI_SLOT (pdev->devfn)) +return 0 ; + + fn =cxl_reset_func_map_bit (sibling, wctx->ari); +if (test_bit (fn, wctx->non_cxl_func_map)) +return 0 ; + + rc =cxl_reset_has_cache_or_mem (sibling); +if (rc <0 ) { + wctx->rc = rc; +return rc; + } +if (!rc) +return 0 ; + + wctx->rc = -ENOTTY ; +return wctx->rc; +} + +static int cxl_reset_validate_function_scoped (struct cxl_reset_context *ctx) +{ +struct pci_dev *pdev = ctx->target; +struct cxl_reset_walk_context wctx = { + .ctx = ctx, + .ari =pci_ari_enabled (pdev->bus), + }; + +cxl_reset_read_non_cxl_func_map (pdev, wctx.non_cxl_func_map); +pci_walk_bus (pdev->bus, cxl_reset_validate_function_scope, &wctx); + +return wctx.rc; +} +static void cxl_hdm_range_context_init (struct cxl_hdm_range_context *ctx) {INIT_LIST_HEAD (&ctx->ranges);
@@ -890,6 +1037,7 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)int cxl_reset_function (struct pci_dev *pdev,bool probe) {struct cxl_hdm_range_context range_ctx; +struct cxl_reset_context ctx;int dvsec;int rc;
@@ -897,8 +1045,9 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)if (dvsec <0 )return dvsec; +cxl_reset_context_init (&ctx, pdev);if (probe) -return 0 ; +return cxl_reset_validate_function_scoped (&ctx);cxl_hdm_range_context_init (&range_ctx);
@@ -907,7 +1056,6 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)if (!rc) rc =cxl_reset_execute (pdev, dvsec); } -cxl_hdm_range_context_destroy (&range_ctx);return rc; }
include/uapi/linux/pci_regs.h +1 -0
@@ -1381,6 +1381,7 @@/* CXL r4.0, 8.1.4: Non-CXL Function Map DVSEC */ #define PCI_DVSEC_CXL_FUNCTION_MAP 2 +#define PCI_DVSEC_CXL_FUNCTION_MAP_REG 0x0C/* CXL r4.0, 8.1.5: Extensions DVSEC for Ports */ #define PCI_DVSEC_CXL_PORT 3
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 07/11] PCI/CXL: Discover the CXL reset scope
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] CXL Reset scope validation is conditionally bypassed during actual reset execution, creating a…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] CXL Reset scope validation is conditionally bypassed during actual reset execution, creating a TOCTOU vulnerability that fails to protect siblings. --
--- Patch [7]: [PATCH v9 07/11] PCI/CXL: Discover the CXL reset scope --- commit 363800e2ebd7dad78179683be8d0a027b5bd8a7e
Author: Srirangan Madhavan <[email protected]>
PCI/CXL: Discover the CXL reset scope
This commit adds reset context support to validate that a CXL Reset is function-scoped before advertising it as a PCI reset method. It uses the Non-CXL Function Map, ARI/devfn rules, and CXL capability bits to reject the reset if another same-scope function would be affected.
Quoted 4 lines
drivers/cxl/core/reset.c +0
-0
[ ... ]
Quoted 9 lines
drivers/cxl/core/reset.c +1
-0@@ -890,6 +1037,7 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)int cxl_reset_function (struct pci_dev *pdev,bool probe) {struct cxl_hdm_range_context range_ctx; +struct cxl_reset_context ctx;int dvsec;int rc;
drivers/cxl/core/reset.c int cxl_reset_function(struct pci_dev *pdev, bool probe) · line 1050
Quoted 7 lines
drivers/cxl/core/reset.c +2
-1@@ -897,8 +1045,9 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)if (dvsec <0 )return dvsec; +cxl_reset_context_init (&ctx, pdev);if (probe) -return 0 ; +return cxl_reset_validate_function_scoped (&ctx);
[Severity: High]
Is it safe to conditionally bypass scope validation when probe is false?
During the early probe phase, when this is called from pci_init_capabilities() for Function 0, higher-numbered sibling functions might not have been enumerated and added to the bus list yet. As a result, cxl_reset_validate_function_scoped() could find no siblings and incorrectly report the reset as safe.
Later, when a user or driver actually triggers the reset and this function is called with probe=0, it skips validation and executes the reset directly. Could this inadvertently reset and destroy the state of active sibling functions that were enumerated after the initial probe?
Quoted 2 lines
drivers/cxl/core/reset.c +0
-0cxl_hdm_range_context_init (&range_ctx);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Dan Williams (nvidia)
maintainerRe: [PATCH v9 07/11] PCI/CXL: Discover the CXL reset scope
In reply to Srirangan MadhavanQuoted 7 lines · Srirangan Madhavan
Add reset context support to validate that CXL Reset is function-scoped before advertising it as a PCI reset method. Use the Non-CXL Function Map, ARI/devfn rules, and CXL.cache/mem capability bits to reject reset when another same-scope function would also be affected.
If the Function Map cannot be read, warn and conservatively treat all candidate same-scope functions as CXL-capable for scope validation.
I was hoping that all the sibling logic could be dropped after the realization that function scoped reset *must* not affect siblings.
The small corner case of multi-function device, but only function0 is impacted by CXL Reset can be an add-on patch if someone really needs it, otherwise just do a simple:
if (dev->multifunction)
...check for now.
It would also be good to fix the bug in cxl_reset_bus_function() with the same check. That handler currently allows reset for multifunction devices. Give Alex credit with a Reported-by.
PATCH v9 8/11 cxl: Restore CXL HDM state after PCI reset
In reply to Srirangan MadhavanAfter CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, and then run the normal PCI restore callbacks.
Keep the target IOMMU reset block active until HDM restore completes so Bus Master Enable cannot reopen DMA before decoder state is valid.
--- drivers/cxl/core/hdm.c | 4 + drivers/cxl/core/reset.c | 321 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 311 insertions(+), 14 deletions(-)
drivers/cxl/core/hdm.c +4 -0
@@ -137,6 +137,10 @@ static void cxl_hdm_info_set_decoder(struct cxl_hdm *cxlhdm,if (!info || cxld->id >= info->decoder_count)return ; +if (cxlhdm->regs.hdm_decoder) + info->global_ctrl =readl (cxlhdm->regs.hdm_decoder + +CXL_HDM_DECODER_CTRL_OFFSET ); +if (cxld->flags &CXL_DECODER_F_ENABLE ) info->settings[cxld->id] = cxld->settings;else
drivers/cxl/core/reset.c +307 -14
408 lines not on this page · raw message
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 08/11] cxl: Restore CXL HDM state after PCI reset
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Hardware PCI BARs are not restored before attempting HDM MMIO access post-reset, causing the…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Hardware PCI BARs are not restored before attempting HDM MMIO access post-reset, causing the restore to fail silently or trigger bus errors.
- [Medium] Mixing goto-based error handling with scope-based cleanups (__free() and scoped_guard()) in the same function. --
--- Patch [8]: [PATCH v9 08/11] cxl: Restore CXL HDM state after PCI reset --- commit a1cce28cbb5bb68a25127de7ce247a2950193943
Author: Srirangan Madhavan <[email protected]>
cxl: Restore CXL HDM state after PCI reset
After CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, and then run the normal PCI restore callbacks.
Quoted 4 lines
drivers/cxl/core/reset.c +0
-0@@ -455,6 +531,164 @@ void pci_cxl_hdm_init(struct pci_dev *pdev)
[ ... ]
Quoted 4 lines
drivers/cxl/core/reset.c +4
-0+static int cxl_restore_hdm (struct pci_dev *pdev) +{ + struct cxl_hdm_decoder_state *decoder_state __free(kfree) = NULL; + struct cxl_decoder_settings *settings __free(kfree) = NULL;
[ ... ]
drivers/cxl/core/reset.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 663
Quoted 5 lines
drivers/cxl/core/reset.c +5
-0+ hdm =cxl_pci_hdm_ioremap_current (pdev, hdm_bar, hdm_offset, hdm_size); +if (IS_ERR (hdm)) { + first_rc =PTR_ERR (hdm); + goto out_restore_command; + }
[Severity: Medium]
Is it safe to mix goto-based error handling with scope-based cleanups like __free() in this function?
The kernel cleanup guidelines suggest converting all resources that need a goto cleanup to scope-based cleanup, or converting none of them, because mixing them creates confusing ownership semantics.
drivers/cxl/core/reset.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 670
Quoted 7 lines
drivers/cxl/core/reset.c +7
-0+ +/* +* Restore global HDM control before per-decoder commit. PCI config +* memory decoding is enabled for MMIO access, but IOMMU reset blocks +* remain active until HDM restore completes. +*/ +writel (global_ctrl, hdm +CXL_HDM_DECODER_CTRL_OFFSET );
[Severity: High]
Will this MMIO access fail or trigger bus errors?
The CXL reset clears the device's PCI configuration space, which sets the hardware BARs to 0. Although cxl_hdm_enable_mem() sets the PCI_COMMAND_MEMORY bit earlier in this function, it doesn't appear to write the physical address back into the hardware BARs before we attempt to access the MMIO region here.
[ ... ]
Quoted 2 lines
int
cxl_reset_function(struct pci_dev *pdev, bool probe){
[ ... ]
drivers/cxl/core/reset.c int cxl_reset_function(struct pci_dev *pdev, bool probe) · line 1342
Quoted 12 lines
drivers/cxl/core/reset.c +9
-0cxl_hdm_range_context_init (&range_ctx); +if (!cxl_reset_scope_hdm_available (&ctx)) { + rc = -ENOTTY ; + goto out; + } + + rc =cxl_pci_target_reset_prepare (&ctx); +if (rc) + goto out; + scoped_guard(rwsem_write, &cxl_rwsem.region) {
[Severity: Medium]
Could we also avoid mixing the legacy goto error paths with the scoped_guard() block in this function?
Quoted 13 lines
drivers/cxl/core/reset.c +6
-1- rc =cxl_hdm_ranges_prepare (&range_ctx, pdev); + rc =cxl_hdm_ranges_prepare (&range_ctx, &ctx);if (!rc) rc =cxl_reset_execute (pdev, dvsec); +if (!rc) + rc =cxl_restore_hdm_decoders (&ctx); } + +cxl_pci_target_reset_done (&ctx); +out:cxl_hdm_range_context_destroy (&range_ctx);return rc; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
PATCH v9 9/11 PCI/CXL: Expose CXL Reset as a PCI reset method
In reply to Srirangan MadhavanAdd the CXL Reset helper to the PCI reset-method table so userspace can select it through the existing reset_method ABI.
Advertise the method for Type 2 CXL devices that report CXL Reset support in the CXL Device DVSEC and pass the function-scope check. Reset execution requires cached HDM decoder state for the target so affected ranges can be validated and HDM programming can be restored. If that state is unavailable at reset time, return -ENOTTY so PCI can try the next reset method.
--- drivers/pci/pci.c | 2 ++ include/linux/pci.h | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
drivers/pci/pci.c +2 -0
@@ -34,6 +34,7 @@#include <linux/aer.h> #include <linux/bitfield.h> #include <linux/suspend.h> +#include <cxl/cxl.h> #include "pci.h" DEFINE_MUTEX (pci_slot_mutex);
@@ -5092,6 +5093,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { }, { pci_dev_specific_reset, .name = "device_specific" }, { pci_dev_acpi_reset, .name = "acpi" }, + { cxl_reset_function, .name = "cxl_reset" }, { pcie_reset_flr, .name = "flr" }, { pci_af_flr, .name = "af_flr" }, { pci_pm_reset, .name = "pm" },
include/linux/pci.h +1 -1
@@ -52,7 +52,7 @@ PCI_STATUS_PARITY)/* Number of reset methods used in pci_reset_fn_methods array in pci.c */ -#define PCI_NUM_RESET_METHODS 8 +#define PCI_NUM_RESET_METHODS 9#define PCI_RESET_PROBE true#define PCI_RESET_DO_RESET false
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 09/11] PCI/CXL: Expose CXL Reset as a PCI reset method
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The CXL reset method skips function-scoped validation during execution, relying entirely on a…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The CXL reset method skips function-scoped validation during execution, relying entirely on a flawed probe-time check that occurs before sibling functions are added to the PCI bus. --
--- Patch [9]: [PATCH v9 09/11] PCI/CXL: Expose CXL Reset as a PCI reset method --- commit f2ff500e92a2699aaf86703b4ef5bcf21ad27f7b
Author: Srirangan Madhavan <[email protected]>
PCI/CXL: Expose CXL Reset as a PCI reset method
Add the CXL Reset helper to the PCI reset-method table so userspace can select it through the existing reset_method ABI. Advertise the method for Type 2 CXL devices that report CXL Reset support in the CXL Device DVSEC.
Quoted 4 lines
drivers/pci/pci.c +0
-0
[ ... ]
drivers/pci/pci.c const struct pci_reset_fn_method pci_reset_fn_methods[] = { · line 5096
Quoted 5 lines
drivers/pci/pci.c +1
-0@@ -5092,6 +5093,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { }, { pci_dev_specific_reset, .name = "device_specific" }, { pci_dev_acpi_reset, .name = "acpi" }, + { cxl_reset_function, .name = "cxl_reset" },
[Severity: High]
Does adding this reset method here introduce a regression where multi-function CXL devices are exposed to state corruption when a reset is triggered?
During early system boot, PCI enumeration calls pci_init_reset_methods() which evaluates this method with probe = true. The method then calls cxl_reset_validate_function_scoped() to walk the bus looking for siblings:
drivers/cxl/core/reset.c:cxl_reset_validate_function_scoped() {
...
pci_walk_bus(pdev->bus, cxl_reset_validate_function_scope, &wctx);
...
}Because function 0 has not yet been added to the bus list at this early stage and sibling functions have not been scanned, the bus walk is empty. The reset method is then incorrectly recorded as valid for function 0.
When the reset is actually executed later via VFIO or sysfs, cxl_reset_function() is called with probe = false. This skips the validation:
drivers/cxl/core/reset.c:cxl_reset_function() {
...
if (probe)
return cxl_reset_validate_function_scoped(&ctx);
...
}This means the bus is not re-evaluated when the siblings are actually present. Will this proceed to reset the entire device without isolating sibling functions, resulting in corrupted or lost state for those siblings?
Quoted 1 line
drivers/pci/pci.c +0
-0{ pcie_reset_flr, .name = "flr" },
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
PATCH v9 10/11 Documentation/ABI: Document CXL Reset PCI reset method
In reply to Srirangan MadhavanDocument the "cxl_reset" PCI reset_method value for Type 2 CXL devices. CXL Reset is device scoped, requires affected memory to be idle, invalidates CPU caches, restores cached HDM decoder state, and does not request Memory Clear.
--- Documentation/ABI/testing/sysfs-bus-pci | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +14 -0
@@ -153,6 +153,20 @@ Description: "default" enables all supported reset methods in the default ordering. + If present, "cxl_reset" selects CXL Reset for CXL Type 2 + devices that advertise CXL Reset support and whose reset scope is + limited to the selected PCI function. The method is not listed + when CXL Reset would affect another CXL.cache or CXL.mem function. + + Before issuing CXL Reset, the kernel quiesces the PCI function, + rejects the operation if cached HDM state is unavailable or + affected CXL memory is busy, invalidates CPU caches for enabled + HDM ranges, disables CXL.cache, and initiates cache write-back + where supported. After reset, the kernel restores PCI config + state needed to access HDM MMIO, restores cached HDM decoder + state, and completes PCI reset recovery. "cxl_reset" does not + request CXL Reset Memory Clear. + What: /sys/bus/pci/devices/.../reset Date: July 2009 Contact: Michael S. Tsirkin <[email protected]>
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 10/11] Documentation/ABI: Document CXL Reset PCI reset method
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines
Document the "
cxl_reset" PCIreset_methodvalue for Type 2 CXL devices. CXL Reset is device scoped, requires affected memory to be idle, invalidates CPU caches, restores cached HDM decoder state, and does not request Memory Clear.Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Re: [PATCH v9 10/11] Documentation/ABI: Document CXL Reset PCI reset method
In reply to Srirangan MadhavanQuoted 15 lines · Srirangan Madhavan
Document the "
cxl_reset" PCIreset_methodvalue for Type 2 CXL devices. CXL Reset is device scoped, requires affected memory to be idle, invalidates CPU caches, restores cached HDM decoder state, and does not request Memory Clear.Signed-off-by Srirangan Madhavan <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 14 ++++++++++++++ 1 file changed, 14 insertions(+)Documentation/ABI/testing/sysfs-bus-pci +0
-0
Documentation/ABI/testing/sysfs-bus-pci Description: · line 159
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-pci +4
-0@@ -153,6 +153,20 @@ Description: "default" enables all supported reset methods in the default ordering. + If present, "cxl_reset" selects CXL Reset for CXL Type 2 + devices that advertise CXL Reset support and whose reset scope is + limited to the selected PCI function. The method is not listed + when CXL Reset would affect another CXL.cache or CXL.mem function.
Not sure about CXL.cache, but as I replied to the cover letter, this last sentence is at least confusing. Only PF0 can enable/initialize CXL.mem so if there exists a memdev, any function using CXL will be affected ... but the reset can not happen because the range is in use by the PF0 initialization. If the reset can happen implies no PF is using it.
Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci +10
-0+ + Before issuing CXL Reset, the kernel quiesces the PCI function, + rejects the operation if cached HDM state is unavailable or + affected CXL memory is busy, invalidates CPU caches for enabled + HDM ranges, disables CXL.cache, and initiates cache write-back + where supported. After reset, the kernel restores PCI config + state needed to access HDM MMIO, restores cached HDM decoder + state, and completes PCI reset recovery. "cxl_reset" does not + request CXL Reset Memory Clear. + What: /sys/bus/pci/devices/.../reset Date: July 2009 Contact: Michael S. Tsirkin <[email protected]>
PATCH v9 11/11 PCI/CXL: Restore HDM state after CXL bus reset
In reply to Srirangan MadhavanCXL bus reset can clear HDM global control and decoder programming. Restore cached HDM state after a successful cxl_bus reset while the IOMMU reset block remains active.
This intentionally restores only the HDM component decoder state cached by the CXL reset support. It does not attempt to restore CXL Device DVSEC range/control registers.
This was exercised with cxl_bus reset on a CXL Type 2 device. Testing confirmed that CXL Device DVSEC range/control state is a separate restore class from HDM component decoder state.
This depends on the CXL reset series because it reuses the HDM state cache and restore helper introduced there.
--- drivers/cxl/core/reset.c | 5 +++++ drivers/pci/pci.c | 3 +++ include/cxl/cxl.h | 6 ++++++ 3 files changed, 14 insertions(+)
drivers/cxl/core/reset.c +5 -0
@@ -894,6 +894,11 @@ static int cxl_restore_hdm_decoders(struct cxl_reset_context *ctx)return cxl_restore_hdm (ctx->target); } +int cxl_restore_hdm_after_pci_reset (struct pci_dev *pdev) +{ +return cxl_restore_hdm (pdev); +} +static void cxl_hdm_range_context_init (struct cxl_hdm_range_context *ctx) {INIT_LIST_HEAD (&ctx->ranges);
drivers/pci/pci.c +3 -0
@@ -5004,6 +5004,9 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)pci_write_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , reg); +if (!rc) + rc =cxl_restore_hdm_after_pci_reset (dev); +pci_dev_reset_iommu_done (dev);return rc; }
include/cxl/cxl.h +6 -0
@@ -154,6 +154,7 @@ int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm);#ifdef CONFIG_CXL_HDM void pci_cxl_hdm_init (struct pci_dev *pdev);void pci_cxl_hdm_release (struct pci_dev *pdev); +int cxl_restore_hdm_after_pci_reset (struct pci_dev *pdev);int cxl_reset_function (struct pci_dev *pdev,bool probe);#else static inline void pci_cxl_hdm_init (struct pci_dev *pdev)
@@ -164,6 +165,11 @@ static inline void pci_cxl_hdm_release(struct pci_dev *pdev) { } +static inline int cxl_restore_hdm_after_pci_reset (struct pci_dev *pdev) +{ +return 0 ; +} +static inline int cxl_reset_function (struct pci_dev *pdev,bool probe) {return -ENOTTY ;
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v9 11/11] PCI/CXL: Restore HDM state after CXL bus reset
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] HDM decoder restoration is performed before the PCI device's BARs are restored, causing MMIO…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] HDM decoder restoration is performed before the PCI device's BARs are restored, causing MMIO writes to be dropped by the hardware or triggering Unsupported Requests. --
--- Patch [11]: [PATCH v9 11/11] PCI/CXL: Restore HDM state after CXL bus reset --- commit 4e02f0f18a7abbebc3cc9b8384269906531ad726
Author: Srirangan Madhavan <[email protected]>
PCI/CXL: Restore HDM state after CXL bus reset
This commit restores cached HDM state after a successful CXL bus reset while the IOMMU reset block remains active. It introduces a helper to restore the state and calls it directly from the PCI reset path.
drivers/pci/pci.c static int cxl_reset_bus_function(struct pci_dev *dev, bool probe) · line 5009
Quoted 11 lines
drivers/pci/pci.c +3
-0@@ -5004,6 +5004,9 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)pci_write_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , reg); +if (!rc) + rc =cxl_restore_hdm_after_pci_reset (dev); +
[Severity: High]
Is it safe to restore the HDM decoders here before the PCI state (including BARs) has been restored?
When pci_reset_bus_function() is called earlier in this function, the hardware reset clears the device's BARs to 0. The caller of the reset function typically restores the saved PCI state after this returns.
However, cxl_restore_hdm_after_pci_reset() calls cxl_restore_hdm(), which enables memory decoding and writes to the cached BAR address:
drivers/cxl/core/reset.c:cxl_restore_hdm() {
...
rc = cxl_hdm_enable_mem(pdev, &command, &restore_command);
...
writel(global_ctrl, hdm + CXL_HDM_DECODER_CTRL_OFFSET);
...
}Because the hardware BARs are still 0 at this point, will enabling PCI_COMMAND_MEMORY cause the device to incorrectly claim physical address 0?
Furthermore, will the MMIO writes to the cached BAR address result in PCIe Unsupported Requests (Master Aborts) since the hardware isn't actually decoding the cached address yet?
Quoted 3 lines
drivers/pci/pci.c +0
-0pci_dev_reset_iommu_done (dev);return rc; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Re: [PATCH v9 11/11] PCI/CXL: Restore HDM state after CXL bus reset
In reply to Srirangan MadhavanQuoted 8 lines · Srirangan Madhavan
CXL bus reset can clear HDM global control and decoder programming. Restore cached HDM state after a successful
cxl_busreset while the IOMMU reset block remains active.This intentionally restores only the HDM component decoder state cached by the CXL reset support. It does not attempt to restore CXL Device DVSEC range/control registers.
Quoted 7 lines
This was exercised with
cxl_busreset on a CXL Type 2 device. Testing confirmed that CXL Device DVSEC range/control state is a separate restore class from HDM component decoder state.This depends on the CXL reset series because it reuses the HDM state cache and restore helper introduced there.
Quoted 64 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/core/reset.c | 5 +++++ drivers/pci/pci.c | 3 +++ include/cxl/cxl.h | 6 ++++++ 3 files changed, 14 insertions(+)drivers/cxl/core/reset.c +5
-0@@ -894,6 +894,11 @@ static int cxl_restore_hdm_decoders(struct cxl_reset_context *ctx)return cxl_restore_hdm (ctx->target); } +int cxl_restore_hdm_after_pci_reset (struct pci_dev *pdev) +{ +return cxl_restore_hdm (pdev); +} +static void cxl_hdm_range_context_init (struct cxl_hdm_range_context *ctx) {INIT_LIST_HEAD (&ctx->ranges);drivers/pci/pci.c +3
-0@@ -5004,6 +5004,9 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)pci_write_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , reg); +if (!rc) + rc =cxl_restore_hdm_after_pci_reset (dev); +pci_dev_reset_iommu_done (dev);return rc; }include/cxl/cxl.h +6
-0@@ -154,6 +154,7 @@ int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm);#ifdef CONFIG_CXL_HDM void pci_cxl_hdm_init (struct pci_dev *pdev);void pci_cxl_hdm_release (struct pci_dev *pdev); +int cxl_restore_hdm_after_pci_reset (struct pci_dev *pdev);int cxl_reset_function (struct pci_dev *pdev,bool probe);#else static inline void pci_cxl_hdm_init (struct pci_dev *pdev)@@ -164,6 +165,11 @@ static inline void pci_cxl_hdm_release(struct pci_dev *pdev) { } +static inline int cxl_restore_hdm_after_pci_reset (struct pci_dev *pdev) +{ +return 0 ; +} +static inline int cxl_reset_function (struct pci_dev *pdev,bool probe) {return -ENOTTY ;Signature · 1 line
2.43.0
Quoted 7 lines · Srirangan Madhavan
Hi folks!
This series adds CXL Reset support for CXL Type 2 devices through the existing PCI
reset_methodABI. The reset sequence follows the CXL 4.0 specification [1], including CXL.cache disable, optional cache writeback, CXL Reset initiation, ResetComplete polling, and ResetError reporting.
Hi Srirangan,
I'm working on understanding all the implications in this patchset, and I have to say my mind is on adding multipf support to Type2 drivers, with a RFC coming soon to the list.
Some comments below.
Thanks.
Quoted 4 lines
The userspace ABI is the existing PCI reset interface:
/sys/bus/pci/devices/.../reset_method /sys/bus/pci/devices/.../reset
Not sure if this will be available in any device PFs, but I think it should only be available for PF0 as the specs define this function as the one having the CXL DVSEC. In any case, the reset will only be possible if PF0 is not attached to the Type2 driver and therefore the memdev is not there. With the multipf support that will also be the case as any none PF0 device will rely on PF0 CXL driver initialization and if PF0 is detached from the driver, so will it be for non PF0 devices.
Note this is not the same than the "Non-CXL Function Map DVSEC" you are checking later on.
<snip>
Quoted 5 lines
The CPU cache invalidation step depends on
cpu_cache_invalidate_memregion()support for the affected address ranges. If no runtime provider is available, the kernel emits a warning and continues after the affected HDM ranges have been reserved.
Because no memdev should be there this step should not be necessary. Maybe as a sanity check.