Skip to contentSkip to conversation

vfio/pci: Add CXL Type-2 device passthrough support

PATCH Superseded [email protected] 4 participants 54 messages Mar 11 – Mar 23

Subsystems (7)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover vfio/pci: Add CXL Type-2 device passthrough support

From: Manish Honap <[email protected]>

This series adds support for passthrough of CXL Type-2 devices to virtual machines through VFIO; The goal is to expose CXL functionality through the generic vfio-pci core, without any need for a variant driver.

Current design is based on CXL core APIs provided by Alejandro's CXL type-2 device support patch series which is currently in upstream review. (see drivers/net/ethernet/sfc/efx_cxl.c) [1].

This patchset should be applied on the cxl next branch using the base specified at the end of this cover letter + Alejandro's v23 mentioned in [1].

This patch series introduces CONFIG_VFIO_CXL_CORE, a new optional module source compiled into vfio-pci-core, that hooks into the vfio-pci open/close and reset paths to provide:

* Automatic CXL Type-2 detection at device open time via the CXL Device

    DVSEC capability (Vendor ID 0x1E98, ID 0x0000) and HDM Decoder
    Capability block.

* Kernel-owned HDM decoder management. The VMM never programs HDM decoders directly; instead it reads and writes an emulated shadow copy of the HDM register block through a dedicated COMP_REGS VFIO region. All bit-field rules (reserved bits, read-only bits, the COMMIT/COMMITTED latch) are enforced by the kernel.

* A DPA VFIO region backed by the kernel-assigned Host Physical Address (HPA). The VMM maps this region with mmap(); PTEs are inserted lazily on first fault. During FLR/reset all PTEs are invalidated atomically under memory_lock and re-inserted after the reset path re-enables the decoder.

* CXL DVSEC configuration-space emulation. Writes to the CXL Control, Status, Control2, Status2, Lock, and Range Base registers in the device's PCI extended configuration space are intercepted and replayed through a per-device shadow (vconfig), enforcing CXL 3.1 register semantics including the RWL/RW1CS/RWO access types and the CONFIG_LOCK one-shot latch.

* A new VFIO_DEVICE_INFO_CAP_CXL capability (id=6) returned in the VFIO_DEVICE_GET_INFO capability chain, carrying all the information a VMM (e.g. QEMU) needs: HDM decoder count, BAR index and offset of the component registers, total DPA size, and indices of the two new VFIO regions.

* Two new VFIO region subtypes under the PCI_VENDOR_ID_CXL vendor
namespace: VFIO_REGION_SUBTYPE_CXL (DPA memory) and VFIO_REGION_SUBTYPE_CXL_COMP_REGS (emulated HDM registers).

* A module parameter (disable_cxl=1) and a per-device flag (vdev->disable_cxl) so that the feature can be suppressed for individual devices or globally without recompiling.

* Comprehensive selftests in tools/testing/selftests/vfio/ covering device detection, capability parsing, region enumeration, HDM register emulation, DPA mmap with page-fault insertion, FLR invalidation, and DVSEC register emulation.

This new design is moved away from variant driver approach and all the CXL functionality is now made part of vfio-pci driver.

The reasons for this change are:

* Generic CXL Type-2 support features (DVSEC, HDM, regions, reset) are common to all CXL adapters and don't belong in variant drivers. When something is vendor-specific (e.g. live migration, proprietary features), a variant is appropriate; generic CXL behavior should not require a vendor-specific driver. Generic CXL support belongs in the core, not behind a variant.

* With this new approach, the user always binds to vfio-pci. No need to choose or document a CXL-specific or vendor-specific driver for standard CXL Type-2 passthrough.

* For any CXL Type-2 device, enlightened vfio-pci works with any device that presents CXL Device DVSEC and the expected component layout.

* CXL detection, state, register emulation, region creation, and reset live in a CXL-aware layer invoked from the core (optionally built via CONFIG_VFIO_CXL_CORE). The core stays a single entry point; CXL is an optional extension, not a separate driver stack.

* Pushing CXL into the pci-core avoids per-device CXL detection and feature toggling inside vendor-specific drivers.

Series structure
================

* Patches 1-5 extend the CXL subsystem to export the interfaces and defines that vfio-pci-core needs.

* Patches 6-8 lay the vfio-pci-core plumbing.

* Patches 9-12 implement the core device lifecycle and DPA region.

* Patches 13-15 implement configuration-space and register emulation.

* Patches 16-18 wire everything together.

* Patches 19-20 add documentation and testing.

Limitations and future work
===========================

* This series does not yet support switched topologies with more than one caching agent; that is planned for a future series.

* RAS / ECC / CCA / Reset Support
This design will integrate RAS and ECC handling in generic vfio-pci by leveraging CXL core and RAS capabilities in next patch updates.

* cxl_reset support [2]
Integrate changes from Srirangan to have VFIO-CXL reset support.

Dependencies
============

[1] Type2 device basic support https://lore.kernel.org/linux-cxl/[email protected]/
[2] CXL Reset support for Type 2 devices https://lore.kernel.org/linux-cxl/[email protected]/

Cc Alex Williamson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Alejandro Lucero <[email protected]>

--

Manish Honap (20):
cxl: Introduce cxl_get_hdm_reg_info()
cxl: Expose cxl subsystem specific functions for vfio
cxl: Move CXL spec defines to public header
cxl: Media ready check refactoring
cxl: Expose BAR index and offset from register map vfio/cxl: Add UAPI for CXL Type-2 device passthrough vfio/pci: Add CXL state to vfio_pci_core_device vfio/pci: Add vfio-cxl Kconfig and build infrastructure vfio/cxl: Implement CXL device detection and HDM register probing vfio/cxl: CXL region management
vfio/cxl: Expose DPA memory region to userspace with fault+zap mmap vfio/pci: Export config access helpers
vfio/cxl: Introduce HDM decoder register emulation framework vfio/cxl: Check media readiness and create CXL memdev vfio/cxl: Introduce CXL DVSEC configuration space emulation vfio/pci: Expose CXL device and region info via VFIO ioctl vfio/cxl: Provide opt-out for CXL feature
docs: vfio-pci: Document CXL Type-2 device passthrough selftests/vfio: Add CXL Type-2 passthrough tests selftests/vfio: Fix VLA initialisation in vfio_pci_irq_set()

 Documentation/driver-api/index.rst            |   1 +
 Documentation/driver-api/vfio-pci-cxl.rst     | 216 +++++
 drivers/cxl/core/pci.c                        |  80 +-
 drivers/cxl/core/regs.c                       |  29 +
 drivers/cxl/cxl.h                             |  34 -
 drivers/vfio/pci/Kconfig                      |   2 +
 drivers/vfio/pci/Makefile                     |   1 +
 drivers/vfio/pci/cxl/Kconfig                  |   7 +
 drivers/vfio/pci/cxl/vfio_cxl_config.c        | 304 +++++++
 drivers/vfio/pci/cxl/vfio_cxl_core.c          | 713 +++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_emu.c           | 414 +++++++++
 drivers/vfio/pci/cxl/vfio_cxl_priv.h          | 123 +++
 drivers/vfio/pci/vfio_pci.c                   |  32 +
 drivers/vfio/pci/vfio_pci_config.c            |  58 +-
 drivers/vfio/pci/vfio_pci_core.c              |  31 +
 drivers/vfio/pci/vfio_pci_priv.h              |  72 ++
 drivers/vfio/pci/vfio_pci_rdwr.c              |   8 +
 include/cxl/cxl.h                             |  52 ++
 include/linux/vfio_pci_core.h                 |  10 +
 include/uapi/linux/vfio.h                     |  52 ++
 tools/testing/selftests/vfio/Makefile         |   1 +
 .../selftests/vfio/lib/vfio_pci_device.c      |   4 +-
 .../selftests/vfio/vfio_cxl_type2_test.c      | 816 ++++++++++++++++++
 23 files changed, 3013 insertions(+), 47 deletions(-)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst
 create mode 100644 drivers/vfio/pci/cxl/Kconfig
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_config.c
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_core.c
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_emu.c
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_priv.h
 create mode 100644 tools/testing/selftests/vfio/vfio_cxl_type2_test.c

base-commit: 3f7938b1aec7f06d5b23adca83e4542fcf027001 --
2.25.1

#1 lore raw

PATCH 1/20 cxl: Introduce cxl_get_hdm_reg_info()

From: Manish Honap <[email protected]>

CXL core has the information of what CXL register groups a device has. When initializing the device, the CXL core probes the register groups and saves the information. The probing sequence is quite complicated.

vfio-cxl requires the HDM register information to emulate the HDM decoder registers.

Introduce cxl_get_hdm_reg_info() for vfio-cxl to leverage the HDM register information in the CXL core. Thus, it doesn't need to implement its own probing sequence.

---
 drivers/cxl/core/pci.c | 45 ++++++++++++++++++++++++++++++++++++++++++
 include/cxl/cxl.h      |  3 +++
 2 files changed, 48 insertions(+)
drivers/cxl/core/pci.c +45 -0
index ba2d393c540a..52ed0b4f5e78 100644
@@ -449,6 +449,51 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
 }
 EXPORT_SYMBOL_NS_GPL(cxl_hdm_decode_init, "CXL");
 
+/**
+ * cxl_get_hdm_reg_info - Get HDM decoder register block location and count
+ * @cxlds: CXL device state (must have component regs enumerated)
+ * @count: Output: number of HDM decoders (from DVSEC cap). Only set when
+ *         the device has a valid HDM decoder capability.
+ * @offset: Output: byte offset of the HDM decoder register block within the
+ *          component register BAR. Only set when valid.
+ * @size: Output: size in bytes of the HDM decoder register block. Only set
+ *        when valid.
+ *
+ * Reads the CXL component register map and DVSEC capability to return the
+ * Host Managed Device Memory (HDM) decoder register block offset and size,
+ * and the number of HDM decoders. This function requires cxlds->cxl_dvsec
+ * to be non-zero.
+ *
+ * Return: 0 on success. A negative errno is returned when config read
+ * failure or when the decoder registers are not valid.
+ */
+int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
+			 resource_size_t *offset, resource_size_t *size)
+{
+	struct pci_dev *pdev = to_pci_dev(cxlds->dev);
+	struct cxl_component_reg_map *map =
+		&cxlds->reg_map.component_map;
+	int d = cxlds->cxl_dvsec;
+	u16 cap;
+	int rc;
+
+	/* HDM decoder registers not implemented */
+	if (!map->hdm_decoder.valid || !d)
+		return -ENODEV;
+
+	rc = pci_read_config_word(pdev,
+				  d + PCI_DVSEC_CXL_CAP, &cap);
+	if (rc)
+		return rc;
+
+	*count = FIELD_GET(PCI_DVSEC_CXL_HDM_COUNT, cap);
+	*offset = map->hdm_decoder.offset;
+	*size = map->hdm_decoder.size;
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_hdm_reg_info, "CXL");
+
 #define CXL_DOE_TABLE_ACCESS_REQ_CODE		0x000000ff
 #define   CXL_DOE_TABLE_ACCESS_REQ_CODE_READ	0
 #define CXL_DOE_TABLE_ACCESS_TABLE_TYPE		0x0000ff00
include/cxl/cxl.h +3 -0
index 50acbd13bcf8..8456177b523e 100644
@@ -284,4 +284,7 @@ int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
 struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
 				     struct cxl_endpoint_decoder **cxled,
 				     int ways);
+
+int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
+			 resource_size_t *offset, resource_size_t *size);
 #endif /* __CXL_CXL_H__ */
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH 01/20] cxl: Introduce cxl_get_hdm_reg_info()

Quoted 9 lines · On Thu, 12 Mar 2026 02:04:21 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

CXL core has the information of what CXL register groups a device has. When initializing the device, the CXL core probes the register groups and saves the information. The probing sequence is quite complicated.

vfio-cxl requires the HDM register information to emulate the HDM decoder registers.

Quoted 7 lines

Introduce cxl_get_hdm_reg_info() for vfio-cxl to leverage the HDM register information in the CXL core. Thus, it doesn't need to implement its own probing sequence.

Hi Manish, Zhi,

Taking a first pass look at this so likely comments will be trivial whilst I get my head around it.

Jonathan

Quoted 16 lines
---
 drivers/cxl/core/pci.c | 45 ++++++++++++++++++++++++++++++++++++++++++
 include/cxl/cxl.h      |  3 +++
 2 files changed, 48 insertions(+)
drivers/cxl/core/pci.c +3 -0
index ba2d393c540a..52ed0b4f5e78 100644
@@ -449,6 +449,51 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
 }
 EXPORT_SYMBOL_NS_GPL(cxl_hdm_decode_init, "CXL");
 
+/**
+ * cxl_get_hdm_reg_info - Get HDM decoder register block location and count
+ * @cxlds: CXL device state (must have component regs enumerated)

drivers/cxl/core/pci.c int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, · line 464

Quoted 10 lines
drivers/cxl/core/pci.c +10 -0
+ * @count: Output: number of HDM decoders (from DVSEC cap). Only set when
+ *         the device has a valid HDM decoder capability.
+ * @offset: Output: byte offset of the HDM decoder register block within the
+ *          component register BAR. Only set when valid.
+ * @size: Output: size in bytes of the HDM decoder register block. Only set
+ *        when valid.
+ *
+ * Reads the CXL component register map and DVSEC capability to return the
+ * Host Managed Device Memory (HDM) decoder register block offset and size,
+ * and the number of HDM decoders.

No it doesn't, see below.

Quoted 7 lines

This function requires cxlds->cxl_dvsec
+ * to be non-zero.
+ *
+ * Return: 0 on success. A negative errno is returned when config read
+ * failure or when the decoder registers are not valid.
+ */
+int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,

If we are going to use a fixed size for count (which is fine) maybe pick a smaller one. It's never bigger than a u8. Actually never bigger than 2... See below.

drivers/cxl/core/pci.c int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, · line 475

Quoted 5 lines
drivers/cxl/core/pci.c +5 -0
+			 resource_size_t *offset, resource_size_t *size)
+{
+	struct pci_dev *pdev = to_pci_dev(cxlds->dev);
+	struct cxl_component_reg_map *map =
+		&cxlds->reg_map.component_map;

Trivial: Fits on one < 80 char line.

drivers/cxl/core/pci.c int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, · line 485

Quoted 10 lines
drivers/cxl/core/pci.c +10 -0
+	int d = cxlds->cxl_dvsec;
+	u16 cap;
+	int rc;
+
+	/* HDM decoder registers not implemented */
+	if (!map->hdm_decoder.valid || !d)
+		return -ENODEV;
+
+	rc = pci_read_config_word(pdev,
+				  d + PCI_DVSEC_CXL_CAP, &cap);

Single line.

Why do you want this? You are using HDM decoders, but checking the number of ranges. Historical artifact before the HDM Decoder stuff was added to the spec. That could really do with a rename!

In theory the spec recommends keeping these HDM ranges aligned with the first 2 HDM decoders, but it doesn't require it and functionally that doesn't matter so I'd not bother.

Quoted 26 lines
drivers/cxl/core/pci.c +11 -0
+	if (rc)
+		return rc;
+
+	*count = FIELD_GET(PCI_DVSEC_CXL_HDM_COUNT, cap);
+	*offset = map->hdm_decoder.offset;
+	*size = map->hdm_decoder.size;
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_hdm_reg_info, "CXL");
+
 #define CXL_DOE_TABLE_ACCESS_REQ_CODE		0x000000ff
 #define   CXL_DOE_TABLE_ACCESS_REQ_CODE_READ	0
 #define CXL_DOE_TABLE_ACCESS_TABLE_TYPE		0x0000ff00
include/cxl/cxl.h +3 -0
index 50acbd13bcf8..8456177b523e 100644
@@ -284,4 +284,7 @@ int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
 struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
 				     struct cxl_endpoint_decoder **cxled,
 				     int ways);
+
+int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
+			 resource_size_t *offset, resource_size_t *size);
 #endif /* __CXL_CXL_H__ */
maintainer
#3 lore raw

Re: [PATCH 01/20] cxl: Introduce cxl_get_hdm_reg_info()

In reply to [email protected]
Quoted 7 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

CXL core has the information of what CXL register groups a device has. When initializing the device, the CXL core probes the register groups and saves the information. The probing sequence is quite complicated.

vfio-cxl requires the HDM register information to emulate the HDM decoder

/register/register block/

Otherwise it implies a single register.

Quoted 4 lines

registers.

Introduce cxl_get_hdm_reg_info() for vfio-cxl to leverage the HDM register information in the CXL core. Thus, it doesn't need to implement

same here. "register block"

Quoted 15 lines

its own probing sequence.

---
 drivers/cxl/core/pci.c | 45 ++++++++++++++++++++++++++++++++++++++++++
 include/cxl/cxl.h      |  3 +++
 2 files changed, 48 insertions(+)
drivers/cxl/core/pci.c +0 -0
index ba2d393c540a..52ed0b4f5e78 100644

drivers/cxl/core/pci.c int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, · line 453

Quoted 5 lines
drivers/cxl/core/pci.c +2 -0
@@ -449,6 +449,51 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
 }
 EXPORT_SYMBOL_NS_GPL(cxl_hdm_decode_init, "CXL");
 
+/**
+ * cxl_get_hdm_reg_info - Get HDM decoder register block location and count

cxl_get_hdm_info() to be consistent with CXL core naming convention.

Quoted 7 lines
drivers/cxl/core/pci.c +7 -0
+ * @cxlds: CXL device state (must have component regs enumerated)
+ * @count: Output: number of HDM decoders (from DVSEC cap). Only set when
+ *         the device has a valid HDM decoder capability.
+ * @offset: Output: byte offset of the HDM decoder register block within the
+ *          component register BAR. Only set when valid.
+ * @size: Output: size in bytes of the HDM decoder register block. Only set
+ *        when valid.

drivers/cxl/core/pci.c int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, · line 470

Quoted 10 lines
drivers/cxl/core/pci.c +10 -0
+ *
+ * Reads the CXL component register map and DVSEC capability to return the
+ * Host Managed Device Memory (HDM) decoder register block offset and size,
+ * and the number of HDM decoders. This function requires cxlds->cxl_dvsec
+ * to be non-zero.
+ *
+ * Return: 0 on success. A negative errno is returned when config read
+ * failure or when the decoder registers are not valid.
+ */
+int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,

u8 for count probably sufficient as Jonathan mentioned. Decoder cap is only 4 bits for count.

drivers/cxl/core/pci.c int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, · line 476

Quoted 6 lines
drivers/cxl/core/pci.c +6 -0
+			 resource_size_t *offset, resource_size_t *size)
+{
+	struct pci_dev *pdev = to_pci_dev(cxlds->dev);
+	struct cxl_component_reg_map *map =
+		&cxlds->reg_map.component_map;
+	int d = cxlds->cxl_dvsec;

Name it dvsec for easier correlation when reading the code.

drivers/cxl/core/pci.c int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, · line 489

Quoted 13 lines
drivers/cxl/core/pci.c +13 -0
+	u16 cap;
+	int rc;
+
+	/* HDM decoder registers not implemented */
+	if (!map->hdm_decoder.valid || !d)
+		return -ENODEV;
+
+	rc = pci_read_config_word(pdev,
+				  d + PCI_DVSEC_CXL_CAP, &cap);
+	if (rc)
+		return rc;
+
+	*count = FIELD_GET(PCI_DVSEC_CXL_HDM_COUNT, cap);

This is not the correct place to find number of decoders. You should be looking at CXL HDM Decoder Capability Register (CXL r4.0 8.2.4.20.1) for the 'Decoder Count' field.

DJ

Quoted 22 lines
drivers/cxl/core/pci.c +7 -0
+	*offset = map->hdm_decoder.offset;
+	*size = map->hdm_decoder.size;
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_hdm_reg_info, "CXL");
+
 #define CXL_DOE_TABLE_ACCESS_REQ_CODE		0x000000ff
 #define   CXL_DOE_TABLE_ACCESS_REQ_CODE_READ	0
 #define CXL_DOE_TABLE_ACCESS_TABLE_TYPE		0x0000ff00
include/cxl/cxl.h +3 -0
index 50acbd13bcf8..8456177b523e 100644
@@ -284,4 +284,7 @@ int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
 struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
 				     struct cxl_endpoint_decoder **cxled,
 				     int ways);
+
+int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
+			 resource_size_t *offset, resource_size_t *size);
 #endif /* __CXL_CXL_H__ */
#4 lore raw

PATCH 2/20 cxl: Expose cxl subsystem specific functions for vfio

In reply to [email protected]

From: Manish Honap <[email protected]>

Below functions from CXL subsystem will be required in vfio-cxl for supporting the type-2 device passthrough:
cxl_find_regblock - To find component registers cxl_probe_component_regs - Probe HDM/RAS capabilities

Make these functions available via declaring them from include header instead of subsystem-specific header.

---
 drivers/cxl/cxl.h | 4 ----
 include/cxl/cxl.h | 7 +++++++
 2 files changed, 7 insertions(+), 4 deletions(-)
drivers/cxl/cxl.h +0 -4
index 2b1f7d687a0e..10ddab3949ee 100644
@@ -198,8 +198,6 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
 #define   CXLDEV_MBOX_BG_CMD_COMMAND_VENDOR_MASK GENMASK_ULL(63, 48)
 #define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20
 
-void cxl_probe_component_regs(struct device *dev, void __iomem *base,
-			      struct cxl_component_reg_map *map);
 void cxl_probe_device_regs(struct device *dev, void __iomem *base,
 			   struct cxl_device_reg_map *map);
 int cxl_map_device_regs(const struct cxl_register_map *map,
@@ -211,8 +209,6 @@ enum cxl_regloc_type;
 int cxl_count_regblock(struct pci_dev *pdev, enum cxl_regloc_type type);
 int cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_type type,
 			       struct cxl_register_map *map, unsigned int index);
-int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
-		      struct cxl_register_map *map);
 int cxl_setup_regs(struct cxl_register_map *map);
 struct cxl_dport;
 int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev, struct cxl_dport *dport);
include/cxl/cxl.h +7 -0
index 8456177b523e..610711e861d4 100644
@@ -287,4 +287,11 @@ struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
 
 int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
 			 resource_size_t *offset, resource_size_t *size);
+struct pci_dev;
+enum cxl_regloc_type;
+int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
+		      struct cxl_register_map *map);
+void cxl_probe_component_regs(struct device *dev, void __iomem *base,
+			      struct cxl_component_reg_map *map);
+
 #endif /* __CXL_CXL_H__ */
Signature · 1 line
2.25.1
maintainer
#5 lore raw

Re: [PATCH 02/20] cxl: Expose cxl subsystem specific functions for vfio

Quoted 43 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Below functions from CXL subsystem will be required in vfio-cxl for supporting the type-2 device passthrough:
cxl_find_regblock - To find component registers cxl_probe_component_regs - Probe HDM/RAS capabilities

Make these functions available via declaring them from include header instead of subsystem-specific header.

---
 drivers/cxl/cxl.h | 4 ----
 include/cxl/cxl.h | 7 +++++++
 2 files changed, 7 insertions(+), 4 deletions(-)
drivers/cxl/cxl.h +0 -4
index 2b1f7d687a0e..10ddab3949ee 100644
@@ -198,8 +198,6 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
 #define   CXLDEV_MBOX_BG_CMD_COMMAND_VENDOR_MASK GENMASK_ULL(63, 48)
 #define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20
 
-void cxl_probe_component_regs(struct device *dev, void __iomem *base,
-			      struct cxl_component_reg_map *map);
 void cxl_probe_device_regs(struct device *dev, void __iomem *base,
 			   struct cxl_device_reg_map *map);
 int cxl_map_device_regs(const struct cxl_register_map *map,
@@ -211,8 +209,6 @@ enum cxl_regloc_type;
 int cxl_count_regblock(struct pci_dev *pdev, enum cxl_regloc_type type);
 int cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_type type,
 			       struct cxl_register_map *map, unsigned int index);
-int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
-		      struct cxl_register_map *map);
 int cxl_setup_regs(struct cxl_register_map *map);
 struct cxl_dport;
 int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev, struct cxl_dport *dport);
include/cxl/cxl.h +0 -0
index 8456177b523e..610711e861d4 100644

include/cxl/cxl.h struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd, · line 295

Quoted 9 lines
include/cxl/cxl.h +6 -0
@@ -287,4 +287,11 @@ struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
 
 int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
 			 resource_size_t *offset, resource_size_t *size);
+struct pci_dev;
+enum cxl_regloc_type;
+int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
+		      struct cxl_register_map *map);
+void cxl_probe_component_regs(struct device *dev, void __iomem *base,
+			      struct cxl_component_reg_map *map);

I do wonder if this needs an ifdef of CONFIG_CXL_BUS given cxl_core can be made as a module or disabled.

Quoted 2 lines
include/cxl/cxl.h +1 -0
+
 #endif /* __CXL_CXL_H__ */
#6 lore raw

RE: [PATCH 02/20] cxl: Expose cxl subsystem specific functions for vfio

Quoted 71 lines

-----Original Message-----
From: Dave Jiang <[email protected]>
Sent: 12 March 2026 22:19
To: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]
Cc: Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 02/20] cxl: Expose cxl subsystem specific functions for vfio

External email: Use caution opening links or attachments

drivers/cxl/cxl.h static inline int ways_to_eiw(unsigned int ways, u8

Quoted 21 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Below functions from CXL subsystem will be required in vfio-cxl for supporting the type-2 device passthrough:
cxl_find_regblock - To find component registers cxl_probe_component_regs - Probe HDM/RAS capabilities

Make these functions available via declaring them from include header instead of subsystem-specific header.

---
 drivers/cxl/cxl.h | 4 ----
 include/cxl/cxl.h | 7 +++++++
 2 files changed, 7 insertions(+), 4 deletions(-)
drivers/cxl/cxl.h index renamed from drivers/cxl/cxl.h +0 -0

2b1f7d687a0e..10ddab3949ee 100644

drivers/cxl/cxl.h +0 -0
@@ -198,8 +198,6 @@ static inline int ways_to_eiw(unsigned int ways, u8

*eiw)

Quoted 11 lines
drivers/cxl/cxl.h +0 -2
 #define   CXLDEV_MBOX_BG_CMD_COMMAND_VENDOR_MASK GENMASK_ULL(63, 48)
 #define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20
 
-void cxl_probe_component_regs(struct device *dev, void __iomem *base,
-                           struct cxl_component_reg_map *map);
 void cxl_probe_device_regs(struct device *dev, void __iomem *base,
                         struct cxl_device_reg_map *map);  int

cxl_map_device_regs(const struct cxl_register_map *map, @@ -211,8 +209,6 @@ enum cxl_regloc_type; int cxl_count_regblock(struct pci_dev *pdev, enum cxl_regloc_type type); int cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_type

type,

Quoted 2 lines

struct cxl_register_map *map, unsigned
int index); -int cxl_find_regblock(struct pci_dev *pdev, enum

cxl_regloc_type type,

Quoted 10 lines

- struct cxl_register_map *map);
int cxl_setup_regs(struct cxl_register_map *map); struct cxl_dport;
int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev, struct cxl_dport *dport); diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index 8456177b523e..610711e861d4 100644

include/cxl/cxl.h +0 -0
@@ -287,4 +287,11 @@ struct cxl_region *cxl_create_region(struct
 cxl_root_decoder *cxlrd,
 
Quoted 13 lines

include/cxl/cxl.h struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd, · line 295

Quoted 9 lines
include/cxl/cxl.h +0 -0
 int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
                       resource_size_t *offset, resource_size_t

*size);

include/cxl/cxl.h +6 -0
+struct pci_dev;
+enum cxl_regloc_type;
+int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
+                   struct cxl_register_map *map); void
+cxl_probe_component_regs(struct device *dev, void __iomem *base,
+                           struct cxl_component_reg_map *map);

I do wonder if this needs an ifdef of CONFIG_CXL_BUS given cxl_core can be made as a module or disabled.

Okay, I will update this section for #ifdef protection.

Quoted 3 lines
Quoted 3 lines

+
#endif /* __CXL_CXL_H__ */

#7 lore raw

PATCH 3/20 cxl: Move CXL spec defines to public header

In reply to [email protected]

From: Manish Honap <[email protected]>

HDM decoder capability structure and component reg block size needs to be used by VFIO subsystem. Move the macros from private CXL header to public one.

---
 drivers/cxl/cxl.h | 30 ------------------------------
 include/cxl/cxl.h | 30 ++++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 30 deletions(-)
drivers/cxl/cxl.h +0 -30
index 10ddab3949ee..7146059e0dae 100644
@@ -24,9 +24,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
  * (port-driver, region-driver, nvdimm object-drivers... etc).
  */
 
-/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */
-#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K
-
 /* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/
 #define CXL_CM_OFFSET 0x1000
 #define CXL_CM_CAP_HDR_OFFSET 0x0
@@ -39,33 +36,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
 #define   CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24)
 #define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)
 
-/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
-#define CXL_HDM_DECODER_CAP_OFFSET 0x0
-#define   CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8)
-#define   CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9)
-#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11)
-#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12)
-#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
-#define   CXL_HDM_DECODER_ENABLE BIT(1)
-#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
-#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
-#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
-#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
-#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
-#define   CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER0_CTRL_LOCK BIT(8)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT BIT(9)
-#define   CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11)
-#define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
-#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
-#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
-#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
-#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
-
 /* HDM decoder control register constants CXL 3.0 8.2.5.19.7 */
 #define CXL_DECODER_MIN_GRANULARITY 256
 #define CXL_DECODER_MAX_ENCODED_IG 6
include/cxl/cxl.h +30 -0
index 610711e861d4..27c006fa53c3 100644
@@ -75,6 +75,36 @@ struct cxl_regs {
 #define   CXL_CM_CAP_CAP_ID_HDM 0x5
 #define   CXL_CM_CAP_CAP_HDM_VERSION 1
 
+/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */
+#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K
+
+/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
+#define CXL_HDM_DECODER_CAP_OFFSET 0x0
+#define   CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
+#define   CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4)
+#define   CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8)
+#define   CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9)
+#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11)
+#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12)
+#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
+#define   CXL_HDM_DECODER_ENABLE BIT(1)
+#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
+#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
+#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
+#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
+#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
+#define   CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0)
+#define   CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4)
+#define   CXL_HDM_DECODER0_CTRL_LOCK BIT(8)
+#define   CXL_HDM_DECODER0_CTRL_COMMIT BIT(9)
+#define   CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10)
+#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11)
+#define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
+#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
+#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
+#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
+#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
+
 struct cxl_reg_map {
 	bool valid;
 	int id;
Signature · 1 line
2.25.1
#8 lore raw

Re: [PATCH 03/20] cxl: Move CXL spec defines to public header

Quoted 7 lines · On Thu, 12 Mar 2026 02:04:23 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

HDM decoder capability structure and component reg block size needs to be used by VFIO subsystem. Move the macros from private CXL header to public one.

Not really related to this patch, but...

Maybe this is the time to think about doing a PCI like uapi/linux/cxl_regs.h header?

So far we have these duplicated in at least a couple of code bases and that is less than ideal!

Move seems fine to me otherwise.

Quoted 94 lines
---
 drivers/cxl/cxl.h | 30 ------------------------------
 include/cxl/cxl.h | 30 ++++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 30 deletions(-)
drivers/cxl/cxl.h +0 -30
index 10ddab3949ee..7146059e0dae 100644
@@ -24,9 +24,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
  * (port-driver, region-driver, nvdimm object-drivers... etc).
  */
 
-/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */
-#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K
-
 /* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/
 #define CXL_CM_OFFSET 0x1000
 #define CXL_CM_CAP_HDR_OFFSET 0x0
@@ -39,33 +36,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
 #define   CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24)
 #define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)
 
-/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
-#define CXL_HDM_DECODER_CAP_OFFSET 0x0
-#define   CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8)
-#define   CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9)
-#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11)
-#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12)
-#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
-#define   CXL_HDM_DECODER_ENABLE BIT(1)
-#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
-#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
-#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
-#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
-#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
-#define   CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER0_CTRL_LOCK BIT(8)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT BIT(9)
-#define   CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11)
-#define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
-#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
-#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
-#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
-#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
-
 /* HDM decoder control register constants CXL 3.0 8.2.5.19.7 */
 #define CXL_DECODER_MIN_GRANULARITY 256
 #define CXL_DECODER_MAX_ENCODED_IG 6
include/cxl/cxl.h +30 -0
index 610711e861d4..27c006fa53c3 100644
@@ -75,6 +75,36 @@ struct cxl_regs {
 #define   CXL_CM_CAP_CAP_ID_HDM 0x5
 #define   CXL_CM_CAP_CAP_HDM_VERSION 1
 
+/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */
+#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K
+
+/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
+#define CXL_HDM_DECODER_CAP_OFFSET 0x0
+#define   CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
+#define   CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4)
+#define   CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8)
+#define   CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9)
+#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11)
+#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12)
+#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
+#define   CXL_HDM_DECODER_ENABLE BIT(1)
+#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
+#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
+#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
+#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
+#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
+#define   CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0)
+#define   CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4)
+#define   CXL_HDM_DECODER0_CTRL_LOCK BIT(8)
+#define   CXL_HDM_DECODER0_CTRL_COMMIT BIT(9)
+#define   CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10)
+#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11)
+#define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
+#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
+#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
+#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
+#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
+
 struct cxl_reg_map {
 	bool valid;
 	int id;
maintainer
#9 lore raw

Re: [PATCH 03/20] cxl: Move CXL spec defines to public header

Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Thu, 12 Mar 2026 02:04:23 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

HDM decoder capability structure and component reg block size needs to be used by VFIO subsystem. Move the macros from private CXL header to public one.

Quoted 6 lines
Quoted 1 line

Not really related to this patch, but...

Maybe this is the time to think about doing a PCI like uapi/linux/cxl_regs.h header?

Sounds reasonable. Maybe makes sense to be in uapi/cxl/? There's already a features.h header in there.

Quoted 102 lines

So far we have these duplicated in at least a couple of code bases and that is less than ideal!

Move seems fine to me otherwise.

Quoted 94 lines
---
 drivers/cxl/cxl.h | 30 ------------------------------
 include/cxl/cxl.h | 30 ++++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 30 deletions(-)
drivers/cxl/cxl.h +0 -30
index 10ddab3949ee..7146059e0dae 100644
@@ -24,9 +24,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
  * (port-driver, region-driver, nvdimm object-drivers... etc).
  */
 
-/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */
-#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K
-
 /* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/
 #define CXL_CM_OFFSET 0x1000
 #define CXL_CM_CAP_HDR_OFFSET 0x0
@@ -39,33 +36,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
 #define   CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24)
 #define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)
 
-/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
-#define CXL_HDM_DECODER_CAP_OFFSET 0x0
-#define   CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8)
-#define   CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9)
-#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11)
-#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12)
-#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
-#define   CXL_HDM_DECODER_ENABLE BIT(1)
-#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
-#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
-#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
-#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
-#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
-#define   CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER0_CTRL_LOCK BIT(8)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT BIT(9)
-#define   CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11)
-#define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
-#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
-#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
-#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
-#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
-
 /* HDM decoder control register constants CXL 3.0 8.2.5.19.7 */
 #define CXL_DECODER_MIN_GRANULARITY 256
 #define CXL_DECODER_MAX_ENCODED_IG 6
include/cxl/cxl.h +30 -0
index 610711e861d4..27c006fa53c3 100644
@@ -75,6 +75,36 @@ struct cxl_regs {
 #define   CXL_CM_CAP_CAP_ID_HDM 0x5
 #define   CXL_CM_CAP_CAP_HDM_VERSION 1
 
+/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */
+#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K
+
+/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
+#define CXL_HDM_DECODER_CAP_OFFSET 0x0
+#define   CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
+#define   CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4)
+#define   CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8)
+#define   CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9)
+#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11)
+#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12)
+#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
+#define   CXL_HDM_DECODER_ENABLE BIT(1)
+#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
+#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
+#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
+#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
+#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
+#define   CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0)
+#define   CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4)
+#define   CXL_HDM_DECODER0_CTRL_LOCK BIT(8)
+#define   CXL_HDM_DECODER0_CTRL_COMMIT BIT(9)
+#define   CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10)
+#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11)
+#define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
+#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
+#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
+#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
+#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
+
 struct cxl_reg_map {
 	bool valid;
 	int id;
#10 lore raw

Re: [PATCH 03/20] cxl: Move CXL spec defines to public header

Quoted 10 lines · On Fri, 13 Mar 2026 09:56:10 -0700 Dave Jiang <[email protected]> wrote:
Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Thu, 12 Mar 2026 02:04:23 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

HDM decoder capability structure and component reg block size needs to be used by VFIO subsystem. Move the macros from private CXL header to public one.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Not really related to this patch, but...

Maybe this is the time to think about doing a PCI like uapi/linux/cxl_regs.h header?

Sounds reasonable. Maybe makes sense to be in uapi/cxl/? There's already a features.h header in there.

Sure. I'd forgotten about that directory.
Jonathan

#11 lore raw

RE: [PATCH 03/20] cxl: Move CXL spec defines to public header

Quoted 39 lines

-----Original Message-----
From: Jonathan Cameron <[email protected]>
Sent: 18 March 2026 20:27
To: Dave Jiang <[email protected]>
Cc: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH 03/20] cxl: Move CXL spec defines to public header

External email: Use caution opening links or attachments

Quoted 14 lines · On Fri, 13 Mar 2026 09:56:10 -0700 Dave Jiang <[email protected]> wrote:
Quoted 13 lines · Jonathan Cameron
Quoted 7 lines · On Thu, 12 Mar 2026 02:04:23 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

HDM decoder capability structure and component reg block size needs to be used by VFIO subsystem. Move the macros from private CXL header to public one.

Not really related to this patch, but...

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Maybe this is the time to think about doing a PCI like uapi/linux/cxl_regs.h header?

Sounds reasonable. Maybe makes sense to be in uapi/cxl/? There's already

a features.h header in there.

Sure. I'd forgotten about that directory.
Jonathan

Okay, I have moved the CXL r4.0 spec defined register bits and flags in include/uapi/cxl/cxl_regs.h but for the bits/flags which are required for emulation handling, I have kept in vfio_cxl_priv.h.

#12 lore raw

PATCH 4/20 cxl: Media ready check refactoring

In reply to [email protected]

From: Manish Honap <[email protected]>

Before accessing CXL device memory after reset/power-on, the driver must ensure media is ready. Not every CXL device implements the CXL Memory Device register group (many Type-2 devices do not). cxl_await_media_ready() reads cxlds->regs.memdev; calling it on a Type-2 without that block can result in kernel panic.

This commit refactors the HDM range based check in a new function which can be safely used for type-2 and type-3 devices. cxl_await_media_ready still uses the same format of checking the HDM range and memory device register status.

---
 drivers/cxl/core/pci.c | 35 ++++++++++++++++++++++++++++++-----
 include/cxl/cxl.h      |  1 +
 2 files changed, 31 insertions(+), 5 deletions(-)
drivers/cxl/core/pci.c +30 -5
index 52ed0b4f5e78..2b7e4d73a6dd 100644
@@ -142,16 +142,24 @@ static int cxl_dvsec_mem_range_active(struct cxl_dev_state *cxlds, int id)
 	return 0;
 }
 
-/*
- * Wait up to @media_ready_timeout for the device to report memory
- * active.
+/**
+ * cxl_await_range_active - Wait for all HDM DVSEC memory ranges to be active
+ * @cxlds: CXL device state (DVSEC and HDM count must be valid)
+ *
+ * For each HDM decoder range reported in the CXL DVSEC capability, waits for
+ * the range to report MEM INFO VALID (up to 1s per range), then MEM ACTIVE
+ * (up to media_ready_timeout seconds per range, default 60s). Used by
+ * cxl_await_media_ready() and by callers that only need range readiness
+ * without checking the memory device status register.
+ *
+ * Return: 0 if all ranges become valid and active, -ETIMEDOUT if a timeout
+ * occurs, or a negative errno from config read on failure.
  */
-int cxl_await_media_ready(struct cxl_dev_state *cxlds)
+int cxl_await_range_active(struct cxl_dev_state *cxlds)
 {
 	struct pci_dev *pdev = to_pci_dev(cxlds->dev);
 	int d = cxlds->cxl_dvsec;
 	int rc, i, hdm_count;
-	u64 md_status;
 	u16 cap;
 
 	rc = pci_read_config_word(pdev,
@@ -172,6 +180,23 @@ int cxl_await_media_ready(struct cxl_dev_state *cxlds)
 			return rc;
 	}
 
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_await_range_active, "CXL");
+
+/*
+ * Wait up to @media_ready_timeout for the device to report memory
+ * active.
+ */
+int cxl_await_media_ready(struct cxl_dev_state *cxlds)
+{
+	u64 md_status;
+	int rc;
+
+	rc = cxl_await_range_active(cxlds);
+	if (rc)
+		return rc;
+
 	md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET);
 	if (!CXLMDEV_READY(md_status))
 		return -EIO;
include/cxl/cxl.h +1 -0
index 27c006fa53c3..684603799fb1 100644
@@ -323,5 +323,6 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
 		      struct cxl_register_map *map);
 void cxl_probe_component_regs(struct device *dev, void __iomem *base,
 			      struct cxl_component_reg_map *map);
+int cxl_await_range_active(struct cxl_dev_state *cxlds);
 
 #endif /* __CXL_CXL_H__ */
Signature · 1 line
2.25.1
maintainer Reviewed-by on v1 4/20
#13 lore raw

Re: [PATCH 04/20] cxl: Media ready check refactoring

Quoted 7 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Before accessing CXL device memory after reset/power-on, the driver must ensure media is ready. Not every CXL device implements the CXL Memory Device register group (many Type-2 devices do not). cxl_await_media_ready() reads cxlds->regs.memdev; calling it on a Type-2 without that block can result in kernel panic.

please consider:
Access the memory device registers on a Type-2 device may result in kernel panic.

Quoted 9 lines

This commit refactors the HDM range based check in a new function which can be safely used for type-2 and type-3 devices. cxl_await_media_ready still uses the same format of checking the HDM range and memory device register status.

Quoted 74 lines
---
 drivers/cxl/core/pci.c | 35 ++++++++++++++++++++++++++++++-----
 include/cxl/cxl.h      |  1 +
 2 files changed, 31 insertions(+), 5 deletions(-)
drivers/cxl/core/pci.c +30 -5
index 52ed0b4f5e78..2b7e4d73a6dd 100644
@@ -142,16 +142,24 @@ static int cxl_dvsec_mem_range_active(struct cxl_dev_state *cxlds, int id)
 	return 0;
 }
 
-/*
- * Wait up to @media_ready_timeout for the device to report memory
- * active.
+/**
+ * cxl_await_range_active - Wait for all HDM DVSEC memory ranges to be active
+ * @cxlds: CXL device state (DVSEC and HDM count must be valid)
+ *
+ * For each HDM decoder range reported in the CXL DVSEC capability, waits for
+ * the range to report MEM INFO VALID (up to 1s per range), then MEM ACTIVE
+ * (up to media_ready_timeout seconds per range, default 60s). Used by
+ * cxl_await_media_ready() and by callers that only need range readiness
+ * without checking the memory device status register.
+ *
+ * Return: 0 if all ranges become valid and active, -ETIMEDOUT if a timeout
+ * occurs, or a negative errno from config read on failure.
  */
-int cxl_await_media_ready(struct cxl_dev_state *cxlds)
+int cxl_await_range_active(struct cxl_dev_state *cxlds)
 {
 	struct pci_dev *pdev = to_pci_dev(cxlds->dev);
 	int d = cxlds->cxl_dvsec;
 	int rc, i, hdm_count;
-	u64 md_status;
 	u16 cap;
 
 	rc = pci_read_config_word(pdev,
@@ -172,6 +180,23 @@ int cxl_await_media_ready(struct cxl_dev_state *cxlds)
 			return rc;
 	}
 
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_await_range_active, "CXL");
+
+/*
+ * Wait up to @media_ready_timeout for the device to report memory
+ * active.
+ */
+int cxl_await_media_ready(struct cxl_dev_state *cxlds)
+{
+	u64 md_status;
+	int rc;
+
+	rc = cxl_await_range_active(cxlds);
+	if (rc)
+		return rc;
+
 	md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET);
 	if (!CXLMDEV_READY(md_status))
 		return -EIO;
include/cxl/cxl.h +1 -0
index 27c006fa53c3..684603799fb1 100644
@@ -323,5 +323,6 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
 		      struct cxl_register_map *map);
 void cxl_probe_component_regs(struct device *dev, void __iomem *base,
 			      struct cxl_component_reg_map *map);
+int cxl_await_range_active(struct cxl_dev_state *cxlds);
 
 #endif /* __CXL_CXL_H__ */
#14 lore raw

RE: [PATCH 04/20] cxl: Media ready check refactoring

Quoted 31 lines

-----Original Message-----
From: Dave Jiang <[email protected]>
Sent: 13 March 2026 01:59
To: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]
Cc: Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/20] cxl: Media ready check refactoring

External email: Use caution opening links or attachments

Quoted 7 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Before accessing CXL device memory after reset/power-on, the driver must ensure media is ready. Not every CXL device implements the CXL Memory Device register group (many Type-2 devices do not). cxl_await_media_ready() reads cxlds->regs.memdev; calling it on a Type-2 without that block can result in kernel panic.

Quoted 3 lines

please consider:
Access the memory device registers on a Type-2 device may result in kernel panic.

Okay, I will update this wording accordingly.

Quoted 95 lines
Quoted 10 lines

This commit refactors the HDM range based check in a new function which can be safely used for type-2 and type-3 devices. cxl_await_media_ready still uses the same format of checking the HDM range and memory device register status.

drivers/cxl/core/pci.c static int cxl_dvsec_mem_range_active(struct

Quoted 10 lines
---
 drivers/cxl/core/pci.c | 35 ++++++++++++++++++++++++++++++-----
 include/cxl/cxl.h      |  1 +
 2 files changed, 31 insertions(+), 5 deletions(-)
drivers/cxl/core/pci.c index renamed from drivers/cxl/core/pci.c +0 -0

52ed0b4f5e78..2b7e4d73a6dd 100644

drivers/cxl/core/pci.c +0 -0
@@ -142,16 +142,24 @@ static int cxl_dvsec_mem_range_active(struct

cxl_dev_state *cxlds, int id)

drivers/cxl/core/pci.c int cxl_await_media_ready(struct cxl_dev_state

Quoted 36 lines
drivers/cxl/core/pci.c +19 -5
      return 0;
 }
 
-/*
- * Wait up to @media_ready_timeout for the device to report memory
- * active.
+/**
+ * cxl_await_range_active - Wait for all HDM DVSEC memory ranges to
+be active
+ * @cxlds: CXL device state (DVSEC and HDM count must be valid)
+ *
+ * For each HDM decoder range reported in the CXL DVSEC capability,
+waits for
+ * the range to report MEM INFO VALID (up to 1s per range), then MEM
+ACTIVE
+ * (up to media_ready_timeout seconds per range, default 60s). Used
+by
+ * cxl_await_media_ready() and by callers that only need range
+readiness
+ * without checking the memory device status register.
+ *
+ * Return: 0 if all ranges become valid and active, -ETIMEDOUT if a
+timeout
+ * occurs, or a negative errno from config read on failure.
  */
-int cxl_await_media_ready(struct cxl_dev_state *cxlds)
+int cxl_await_range_active(struct cxl_dev_state *cxlds)
 {
      struct pci_dev *pdev = to_pci_dev(cxlds->dev);
      int d = cxlds->cxl_dvsec;
      int rc, i, hdm_count;
-     u64 md_status;
      u16 cap;
 
      rc = pci_read_config_word(pdev,
@@ -172,6 +180,23 @@ int cxl_await_media_ready(struct cxl_dev_state

*cxlds)

include/cxl/cxl.h int cxl_find_regblock(struct pci_dev *pdev, enum

Quoted 27 lines
drivers/cxl/core/pci.c +16 -0
                      return rc;
      }
 
+     return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_await_range_active, "CXL");
+
+/*
+ * Wait up to @media_ready_timeout for the device to report memory
+ * active.
+ */
+int cxl_await_media_ready(struct cxl_dev_state *cxlds) {
+     u64 md_status;
+     int rc;
+
+     rc = cxl_await_range_active(cxlds);
+     if (rc)
+             return rc;
+
      md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET);
      if (!CXLMDEV_READY(md_status))
              return -EIO;
include/cxl/cxl.h index renamed from include/cxl/cxl.h +0 -0

27c006fa53c3..684603799fb1 100644

include/cxl/cxl.h +0 -0
@@ -323,5 +323,6 @@ int cxl_find_regblock(struct pci_dev *pdev, enum

cxl_regloc_type type,

Quoted 6 lines
include/cxl/cxl.h +1 -0
                    struct cxl_register_map *map);  void
 cxl_probe_component_regs(struct device *dev, void __iomem *base,
                            struct cxl_component_reg_map *map);
+int cxl_await_range_active(struct cxl_dev_state *cxlds);
 
 #endif /* __CXL_CXL_H__ */
#15 lore raw

PATCH 5/20 cxl: Expose BAR index and offset from register map

In reply to [email protected]

From: Manish Honap <[email protected]>

The Register Locator DVSEC (CXL 2.0 8.1.9) describes register blocks by BAR index (BIR) and offset within the BAR. CXL core currently only stores the resolved HPA (resource + offset) in struct cxl_register_map, so callers that need to use pci_iomap() or report the BAR to userspace must reverse-engineer the BAR from the HPA.

Add bar_index and bar_offset to struct cxl_register_map and fill them in cxl_decode_regblock() when the regblock is BAR-backed (BIR 0-5). Add cxl_regblock_get_bar_info() so callers (e.g. vfio-cxl) can get BAR index and offset directly and use pci_iomap() instead of ioremap(HPA).

---
 drivers/cxl/core/regs.c | 29 +++++++++++++++++++++++++++++
 include/cxl/cxl.h       | 11 +++++++++++
 2 files changed, 40 insertions(+)
drivers/cxl/core/regs.c +29 -0
index 20c2d9fbcfe7..720eb6eb5a45 100644
@@ -287,9 +287,37 @@ static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi,
 	map->reg_type = reg_type;
 	map->resource = pci_resource_start(pdev, bar) + offset;
 	map->max_size = pci_resource_len(pdev, bar) - offset;
+	map->bar_index = (bar >= 0 && bar < PCI_STD_NUM_BARS) ? (u8)bar : 0xFF;
+	map->bar_offset = offset;
 	return true;
 }
 
+/**
+ * cxl_regblock_get_bar_info() - Get BAR index and offset for a BAR-backed regblock
+ * @map: Register map from cxl_find_regblock() or cxl_find_regblock_instance()
+ * @bar_index: Output BAR index (0-5). Optional, may be NULL.
+ * @bar_offset: Output offset within the BAR. Optional, may be NULL.
+ *
+ * When the register block was found via the Register Locator DVSEC and
+ * lives in a PCI BAR (BIR 0-5), this returns the BAR index and the offset
+ * within that BAR. Callers can use pci_iomap(pdev, bar_index, size) and
+ * base + bar_offset instead of ioremap(map->resource).
+ *
+ * Return: 0 if the regblock is BAR-backed (bar_index <= 5), -EINVAL otherwise.
+ */
+int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8 *bar_index,
+			      resource_size_t *bar_offset)
+{
+	if (!map || map->bar_index > PCI_STD_NUM_BARS - 1)
+		return -EINVAL;
+	if (bar_index)
+		*bar_index = map->bar_index;
+	if (bar_offset)
+		*bar_offset = map->bar_offset;
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_regblock_get_bar_info, "CXL");
+
 /*
  * __cxl_find_regblock_instance() - Locate a register block or count instances by type / index
  * Use CXL_INSTANCES_COUNT for @index if counting instances.
@@ -308,6 +336,7 @@ static int __cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_ty
 
 	*map = (struct cxl_register_map) {
 		.host = &pdev->dev,
+		.bar_index = 0xFF,
 		.resource = CXL_RESOURCE_NONE,
 	};
 
include/cxl/cxl.h +11 -0
index 684603799fb1..08e327a929ba 100644
@@ -134,9 +134,16 @@ struct cxl_pmu_reg_map {
  * @resource: physical resource base of the register block
  * @max_size: maximum mapping size to perform register search
  * @reg_type: see enum cxl_regloc_type
+ * @bar_index: PCI BAR index (0-5) when regblock is BAR-backed; 0xFF otherwise
+ * @bar_offset: offset within the BAR; only valid when bar_index <= 5
  * @component_map: cxl_reg_map for component registers
  * @device_map: cxl_reg_maps for device registers
  * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units
+ *
+ * When the register block is described by the Register Locator DVSEC with
+ * a BAR Indicator (BIR 0-5), bar_index and bar_offset are set so callers can
+ * use pci_iomap(pdev, bar_index, size) and base + bar_offset instead of
+ * ioremap(resource).
  */
 struct cxl_register_map {
 	struct device *host;
@@ -144,6 +151,8 @@ struct cxl_register_map {
 	resource_size_t resource;
 	resource_size_t max_size;
 	u8 reg_type;
+	u8 bar_index;
+	resource_size_t bar_offset;
 	union {
 		struct cxl_component_reg_map component_map;
 		struct cxl_device_reg_map device_map;
@@ -319,6 +328,8 @@ int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
 			 resource_size_t *offset, resource_size_t *size);
 struct pci_dev;
 enum cxl_regloc_type;
+int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8 *bar_index,
+			      resource_size_t *bar_offset);
 int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
 		      struct cxl_register_map *map);
 void cxl_probe_component_regs(struct device *dev, void __iomem *base,
Signature · 1 line
2.25.1
maintainer
#16 lore raw

Re: [PATCH 05/20] cxl: Expose BAR index and offset from register map

Quoted 3 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

The Register Locator DVSEC (CXL 2.0 8.1.9) describes register blocks by

CXL r4.0

Let's keep it to the latest spec version.

Quoted 21 lines

BAR index (BIR) and offset within the BAR. CXL core currently only stores the resolved HPA (resource + offset) in struct cxl_register_map, so callers that need to use pci_iomap() or report the BAR to userspace must reverse-engineer the BAR from the HPA.

Add bar_index and bar_offset to struct cxl_register_map and fill them in cxl_decode_regblock() when the regblock is BAR-backed (BIR 0-5). Add cxl_regblock_get_bar_info() so callers (e.g. vfio-cxl) can get BAR index and offset directly and use pci_iomap() instead of ioremap(HPA).

---
 drivers/cxl/core/regs.c | 29 +++++++++++++++++++++++++++++
 include/cxl/cxl.h       | 11 +++++++++++
 2 files changed, 40 insertions(+)
drivers/cxl/core/regs.c +0 -0
index 20c2d9fbcfe7..720eb6eb5a45 100644

drivers/cxl/core/regs.c static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi, · line 290

Quoted 4 lines
drivers/cxl/core/regs.c +1 -0
@@ -287,9 +287,37 @@ static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi,
 	map->reg_type = reg_type;
 	map->resource = pci_resource_start(pdev, bar) + offset;
 	map->max_size = pci_resource_len(pdev, bar) - offset;
+	map->bar_index = (bar >= 0 && bar < PCI_STD_NUM_BARS) ? (u8)bar : 0xFF;

map->bar_index = bar should be fine. Otherwise the pci_resource_start() and pci_resource_len() would also be invalid.

Quoted 11 lines
drivers/cxl/core/regs.c +8 -0
+	map->bar_offset = offset;
 	return true;
 }
 
+/**
+ * cxl_regblock_get_bar_info() - Get BAR index and offset for a BAR-backed regblock
+ * @map: Register map from cxl_find_regblock() or cxl_find_regblock_instance()
+ * @bar_index: Output BAR index (0-5). Optional, may be NULL.
+ * @bar_offset: Output offset within the BAR. Optional, may be NULL.
+ *
+ * When the register block was found via the Register Locator DVSEC and

drivers/cxl/core/regs.c static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi, · line 311

Quoted 10 lines
drivers/cxl/core/regs.c +10 -0
+ * lives in a PCI BAR (BIR 0-5), this returns the BAR index and the offset
+ * within that BAR. Callers can use pci_iomap(pdev, bar_index, size) and
+ * base + bar_offset instead of ioremap(map->resource).
+ *
+ * Return: 0 if the regblock is BAR-backed (bar_index <= 5), -EINVAL otherwise.
+ */
+int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8 *bar_index,
+			      resource_size_t *bar_offset)
+{
+	if (!map || map->bar_index > PCI_STD_NUM_BARS - 1)

map->bar_index == 0xff? Otherwise it's probably a hardware issue right?

DJ

Quoted 59 lines
drivers/cxl/core/regs.c +10 -0
+		return -EINVAL;
+	if (bar_index)
+		*bar_index = map->bar_index;
+	if (bar_offset)
+		*bar_offset = map->bar_offset;
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_regblock_get_bar_info, "CXL");
+
 /*
  * __cxl_find_regblock_instance() - Locate a register block or count instances by type / index
  * Use CXL_INSTANCES_COUNT for @index if counting instances.
@@ -308,6 +336,7 @@ static int __cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_ty
 
 	*map = (struct cxl_register_map) {
 		.host = &pdev->dev,
+		.bar_index = 0xFF,
 		.resource = CXL_RESOURCE_NONE,
 	};
 
include/cxl/cxl.h +11 -0
index 684603799fb1..08e327a929ba 100644
@@ -134,9 +134,16 @@ struct cxl_pmu_reg_map {
  * @resource: physical resource base of the register block
  * @max_size: maximum mapping size to perform register search
  * @reg_type: see enum cxl_regloc_type
+ * @bar_index: PCI BAR index (0-5) when regblock is BAR-backed; 0xFF otherwise
+ * @bar_offset: offset within the BAR; only valid when bar_index <= 5
  * @component_map: cxl_reg_map for component registers
  * @device_map: cxl_reg_maps for device registers
  * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units
+ *
+ * When the register block is described by the Register Locator DVSEC with
+ * a BAR Indicator (BIR 0-5), bar_index and bar_offset are set so callers can
+ * use pci_iomap(pdev, bar_index, size) and base + bar_offset instead of
+ * ioremap(resource).
  */
 struct cxl_register_map {
 	struct device *host;
@@ -144,6 +151,8 @@ struct cxl_register_map {
 	resource_size_t resource;
 	resource_size_t max_size;
 	u8 reg_type;
+	u8 bar_index;
+	resource_size_t bar_offset;
 	union {
 		struct cxl_component_reg_map component_map;
 		struct cxl_device_reg_map device_map;
@@ -319,6 +328,8 @@ int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
 			 resource_size_t *offset, resource_size_t *size);
 struct pci_dev;
 enum cxl_regloc_type;
+int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8 *bar_index,
+			      resource_size_t *bar_offset);
 int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
 		      struct cxl_register_map *map);
 void cxl_probe_component_regs(struct device *dev, void __iomem *base,
#17 lore raw

RE: [PATCH 05/20] cxl: Expose BAR index and offset from register map

Quoted 24 lines

-----Original Message-----
From: Dave Jiang <[email protected]>
Sent: 13 March 2026 02:29
To: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]
Cc: Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 05/20] cxl: Expose BAR index and offset from register map

External email: Use caution opening links or attachments

On 3/11/26 1:34 PM, [email protected] wrote:

Quoted 8 lines
Quoted 4 lines

From: Manish Honap <[email protected]>

The Register Locator DVSEC (CXL 2.0 8.1.9) describes register blocks by

CXL r4.0

Let's keep it to the latest spec version.

Okay, I will update the spec version references to r4.0

Quoted 23 lines

drivers/cxl/core/regs.c static bool cxl_decode_regblock(struct pci_dev

Quoted 22 lines

BAR index (BIR) and offset within the BAR. CXL core currently only stores the resolved HPA (resource + offset) in struct cxl_register_map, so callers that need to use pci_iomap() or report the BAR to userspace must reverse-engineer the BAR from the HPA.

Add bar_index and bar_offset to struct cxl_register_map and fill them in cxl_decode_regblock() when the regblock is BAR-backed (BIR 0-5). Add cxl_regblock_get_bar_info() so callers (e.g. vfio-cxl) can get BAR index and offset directly and use pci_iomap() instead of ioremap(HPA).

---
 drivers/cxl/core/regs.c | 29 +++++++++++++++++++++++++++++
 include/cxl/cxl.h       | 11 +++++++++++
 2 files changed, 40 insertions(+)
drivers/cxl/core/regs.c index renamed from drivers/cxl/core/regs.c +0 -0

20c2d9fbcfe7..720eb6eb5a45 100644

drivers/cxl/core/regs.c +0 -0
@@ -287,9 +287,37 @@ static bool cxl_decode_regblock(struct pci_dev

*pdev, u32 reg_lo, u32 reg_hi,

Quoted 8 lines

drivers/cxl/core/regs.c static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi,

Quoted 5 lines
drivers/cxl/core/regs.c +2 -0
      map->reg_type = reg_type;
      map->resource = pci_resource_start(pdev, bar) + offset;
      map->max_size = pci_resource_len(pdev, bar) - offset;
+     map->bar_index = (bar >= 0 && bar < PCI_STD_NUM_BARS) ? (u8)bar
+ : 0xFF;

map->bar_index = bar should be fine. Otherwise the pci_resource_start() and pci_resource_len() would also be invalid.

Okay, I will update this.

Quoted 24 lines
Quoted 13 lines
drivers/cxl/core/regs.c +8 -0
+     map->bar_offset = offset;
      return true;
 }
 
+/**
+ * cxl_regblock_get_bar_info() - Get BAR index and offset for a
+BAR-backed regblock
+ * @map: Register map from cxl_find_regblock() or
+cxl_find_regblock_instance()
+ * @bar_index: Output BAR index (0-5). Optional, may be NULL.
+ * @bar_offset: Output offset within the BAR. Optional, may be NULL.

drivers/cxl/core/regs.c static bool cxl_decode_regblock(struct pci_dev · line 312

Quoted 10 lines
drivers/cxl/core/regs.c +10 -0
+ *
+ * When the register block was found via the Register Locator DVSEC
+and
+ * lives in a PCI BAR (BIR 0-5), this returns the BAR index and the
+offset
+ * within that BAR. Callers can use pci_iomap(pdev, bar_index, size)
+and
+ * base + bar_offset instead of ioremap(map->resource).
+ *
+ * Return: 0 if the regblock is BAR-backed (bar_index <= 5), -EINVAL

otherwise.

Quoted 8 lines

drivers/cxl/core/regs.c static bool cxl_decode_regblock(struct pci_dev · line 314

Quoted 2 lines
drivers/cxl/core/regs.c +2 -0
+ */
+int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8

*bar_index,

drivers/cxl/core/regs.c static bool cxl_decode_regblock(struct pci_dev · line 316

Quoted 2 lines
drivers/cxl/core/regs.c +2 -0
+                           resource_size_t *bar_offset) {
+     if (!map || map->bar_index > PCI_STD_NUM_BARS - 1)

map->bar_index == 0xff? Otherwise it's probably a hardware issue right?

Okay, agreed.

Quoted 67 lines

DJ

drivers/cxl/core/regs.c static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi,

Quoted 11 lines
drivers/cxl/core/regs.c +9 -0
+             return -EINVAL;
+     if (bar_index)
+             *bar_index = map->bar_index;
+     if (bar_offset)
+             *bar_offset = map->bar_offset;
+     return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_regblock_get_bar_info, "CXL");
+
 /*
  * __cxl_find_regblock_instance() - Locate a register block or count

instances by type / index

Quoted 44 lines

* Use CXL_INSTANCES_COUNT for @index if counting instances.

drivers/cxl/core/regs.c +0 -0
@@ -308,6 +336,7 @@ static int __cxl_find_regblock_instance(struct

pci_dev *pdev, enum cxl_regloc_ty

      *map = (struct cxl_register_map) {
              .host = &pdev->dev,
drivers/cxl/core/regs.c +1 -0
+             .bar_index = 0xFF,
              .resource = CXL_RESOURCE_NONE,
      };
include/cxl/cxl.h index renamed from include/cxl/cxl.h +0 -0

684603799fb1..08e327a929ba 100644

include/cxl/cxl.h +13 -0
@@ -134,9 +134,16 @@ struct cxl_pmu_reg_map {
  * @resource: physical resource base of the register block
  * @max_size: maximum mapping size to perform register search
  * @reg_type: see enum cxl_regloc_type
+ * @bar_index: PCI BAR index (0-5) when regblock is BAR-backed; 0xFF
+ otherwise
+ * @bar_offset: offset within the BAR; only valid when bar_index <= 5
  * @component_map: cxl_reg_map for component registers
  * @device_map: cxl_reg_maps for device registers
  * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units
+ *
+ * When the register block is described by the Register Locator DVSEC
+ with
+ * a BAR Indicator (BIR 0-5), bar_index and bar_offset are set so
+ callers can
+ * use pci_iomap(pdev, bar_index, size) and base + bar_offset instead
+ of
+ * ioremap(resource).
  */
 struct cxl_register_map {
      struct device *host;
@@ -144,6 +151,8 @@ struct cxl_register_map {
      resource_size_t resource;
      resource_size_t max_size;
      u8 reg_type;
+     u8 bar_index;
+     resource_size_t bar_offset;
      union {
              struct cxl_component_reg_map component_map;
              struct cxl_device_reg_map device_map; @@ -319,6 +328,8
Quoted 4 lines

@@ int cxl_get_hdm_reg_info(struct cxl_dev_state *cxlds, u32 *count,
resource_size_t *offset, resource_size_t
*size); struct pci_dev; enum cxl_regloc_type;
+int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8

*bar_index,

Quoted 4 lines

+ resource_size_t *bar_offset);
int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
struct cxl_register_map *map); void
cxl_probe_component_regs(struct device *dev, void __iomem *base,

#18 lore raw

PATCH 6/20 vfio/cxl: Add UAPI for CXL Type-2 device passthrough

In reply to [email protected]

From: Manish Honap <[email protected]>

CXL capabilities include:
- hdm_count: Number of HDM decoders available
- capacity: Total device memory (DPA)
- flags: COMMITTED, PRECOMMITTED

This UAPI enables VMMs like QEMU to passthrough CXL Type-2 devices (GPUs, accelerators) with coherent memory to VMs.

Also added user-kernel API definitions for CXL Type-2 device passthrough. Document how VFIO_DEVICE_FLAGS_CXL relates to VFIO_DEVICE_FLAGS_PCI and VFIO_DEVICE_FLAGS_CAPS, and add field and flag descriptions for the CXL capability.

---
 include/uapi/linux/vfio.h | 52 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
include/uapi/linux/vfio.h +52 -0
index ac2329f24141..7ec0f96cc2d9 100644
@@ -215,6 +215,13 @@ struct vfio_device_info {
 #define VFIO_DEVICE_FLAGS_FSL_MC (1 << 6)	/* vfio-fsl-mc device */
 #define VFIO_DEVICE_FLAGS_CAPS	(1 << 7)	/* Info supports caps */
 #define VFIO_DEVICE_FLAGS_CDX	(1 << 8)	/* vfio-cdx device */
+/*
+ * CXL Type-2 device (memory coherent; e.g. GPU, accelerator). When set,
+ * VFIO_DEVICE_FLAGS_PCI is also set (same device is a PCI device). The
+ * capability chain (VFIO_DEVICE_FLAGS_CAPS) contains VFIO_DEVICE_INFO_CAP_CXL
+ * describing HDM decoders, DPA size, and CXL-specific options.
+ */
+#define VFIO_DEVICE_FLAGS_CXL   (1 << 9)        /* Device supports CXL */
 	__u32	num_regions;	/* Max region index + 1 */
 	__u32	num_irqs;	/* Max IRQ index + 1 */
 	__u32   cap_offset;	/* Offset within info struct of first cap */
@@ -257,6 +264,39 @@ struct vfio_device_info_cap_pci_atomic_comp {
 	__u32 reserved;
 };
 
+/*
+ * VFIO_DEVICE_INFO_CAP_CXL - CXL Type-2 device capability
+ *
+ * Present in the device info capability chain when VFIO_DEVICE_FLAGS_CXL
+ * is set. Describes Host Managed Device Memory (HDM) layout and CXL
+ * memory options so that userspace (e.g. QEMU) can expose the CXL region
+ * and component registers correctly to the guest.
+ */
+#define VFIO_DEVICE_INFO_CAP_CXL		6
+struct vfio_device_info_cap_cxl {
+	struct vfio_info_cap_header header;
+	__u8  hdm_count; /* Number of HDM decoders */
+	__u8  hdm_regs_bar_index; /* PCI BAR containing HDM registers */
+	__u16 pad;
+	__u32 flags;
+/* Decoder was committed by host firmware/BIOS */
+#define VFIO_CXL_CAP_COMMITTED		(1 << 0)
+/*
+ * Memory was pre-committed (firmware-programmed); VMM need not allocate
+ * from CXL pool
+ */
+#define VFIO_CXL_CAP_PRECOMMITTED	(1 << 1)
+	__u64 hdm_regs_size; /* Size in bytes of HDM register block */
+	__u64 hdm_regs_offset; /* Byte offset within the BAR to the HDM decoder block */
+	__u64 dpa_size; /* Device Physical Address (DPA) size in bytes */
+	/*
+	 * Region indices for the two CXL VFIO device regions.
+	 * Avoids forcing userspace to scan all regions by type/subtype.
+	 */
+	__u32  dpa_region_index;       /* VFIO_REGION_SUBTYPE_CXL */
+	__u32  comp_regs_region_index; /* VFIO_REGION_SUBTYPE_CXL_COMP_REGS */
+};
+
 /**
  * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
  *				       struct vfio_region_info)
@@ -370,6 +410,18 @@ struct vfio_region_info_cap_type {
  */
 #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD	(1)
 
+/* 1e98 vendor PCI sub-types (CXL Consortium) */
+/*
+ * CXL memory region. Use with region type
+ * (PCI_VENDOR_ID_CXL | VFIO_REGION_TYPE_PCI_VENDOR_TYPE).
+ * DPA memory region (fault+zap mmap)
+ */
+#define VFIO_REGION_SUBTYPE_CXL                 (1)
+/*
+ * HDM decoder register emulation region (read/write only, no mmap).
+ */
+#define VFIO_REGION_SUBTYPE_CXL_COMP_REGS       (2)
+
 /* sub-types for VFIO_REGION_TYPE_GFX */
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
 
Signature · 1 line
2.25.1
maintainer
#19 lore raw

Re: [PATCH 06/20] vfio/cxl: Add UAPI for CXL Type-2 device passthrough

Quoted 48 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

CXL capabilities include:
- hdm_count: Number of HDM decoders available
- capacity: Total device memory (DPA)
- flags: COMMITTED, PRECOMMITTED

This UAPI enables VMMs like QEMU to passthrough CXL Type-2 devices (GPUs, accelerators) with coherent memory to VMs.

Also added user-kernel API definitions for CXL Type-2 device passthrough. Document how VFIO_DEVICE_FLAGS_CXL relates to VFIO_DEVICE_FLAGS_PCI and VFIO_DEVICE_FLAGS_CAPS, and add field and flag descriptions for the CXL capability.

---
 include/uapi/linux/vfio.h | 52 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
include/uapi/linux/vfio.h +13 -0
index ac2329f24141..7ec0f96cc2d9 100644
@@ -215,6 +215,13 @@ struct vfio_device_info {
 #define VFIO_DEVICE_FLAGS_FSL_MC (1 << 6)	/* vfio-fsl-mc device */
 #define VFIO_DEVICE_FLAGS_CAPS	(1 << 7)	/* Info supports caps */
 #define VFIO_DEVICE_FLAGS_CDX	(1 << 8)	/* vfio-cdx device */
+/*
+ * CXL Type-2 device (memory coherent; e.g. GPU, accelerator). When set,
+ * VFIO_DEVICE_FLAGS_PCI is also set (same device is a PCI device). The
+ * capability chain (VFIO_DEVICE_FLAGS_CAPS) contains VFIO_DEVICE_INFO_CAP_CXL
+ * describing HDM decoders, DPA size, and CXL-specific options.
+ */
+#define VFIO_DEVICE_FLAGS_CXL   (1 << 9)        /* Device supports CXL */
 	__u32	num_regions;	/* Max region index + 1 */
 	__u32	num_irqs;	/* Max IRQ index + 1 */
 	__u32   cap_offset;	/* Offset within info struct of first cap */
@@ -257,6 +264,39 @@ struct vfio_device_info_cap_pci_atomic_comp {
 	__u32 reserved;
 };
 
+/*
+ * VFIO_DEVICE_INFO_CAP_CXL - CXL Type-2 device capability
+ *
+ * Present in the device info capability chain when VFIO_DEVICE_FLAGS_CXL
+ * is set. Describes Host Managed Device Memory (HDM) layout and CXL
+ * memory options so that userspace (e.g. QEMU) can expose the CXL region

include/uapi/linux/vfio.h struct vfio_device_info_cap_pci_atomic_comp { · line 282

Quoted 10 lines
include/uapi/linux/vfio.h +10 -0
+ * and component registers correctly to the guest.
+ */
+#define VFIO_DEVICE_INFO_CAP_CXL		6
+struct vfio_device_info_cap_cxl {
+	struct vfio_info_cap_header header;
+	__u8  hdm_count; /* Number of HDM decoders */
+	__u8  hdm_regs_bar_index; /* PCI BAR containing HDM registers */
+	__u16 pad;
+	__u32 flags;
+/* Decoder was committed by host firmware/BIOS */

I'm confused by COMMITTED vs PRECOMMITTED. Should it just say "Decoder is committed" here? Otherwise what is the difference? Also can you explain a little the usage for COMMITTED vs PRECOMMITTED in the commit log please? i.e why does VFIO CXL needs to know a decoder is pre-committed?

DJ

Quoted 39 lines
include/uapi/linux/vfio.h +29 -0
+#define VFIO_CXL_CAP_COMMITTED		(1 << 0)
+/*
+ * Memory was pre-committed (firmware-programmed); VMM need not allocate
+ * from CXL pool
+ */
+#define VFIO_CXL_CAP_PRECOMMITTED	(1 << 1)
+	__u64 hdm_regs_size; /* Size in bytes of HDM register block */
+	__u64 hdm_regs_offset; /* Byte offset within the BAR to the HDM decoder block */
+	__u64 dpa_size; /* Device Physical Address (DPA) size in bytes */
+	/*
+	 * Region indices for the two CXL VFIO device regions.
+	 * Avoids forcing userspace to scan all regions by type/subtype.
+	 */
+	__u32  dpa_region_index;       /* VFIO_REGION_SUBTYPE_CXL */
+	__u32  comp_regs_region_index; /* VFIO_REGION_SUBTYPE_CXL_COMP_REGS */
+};
+
 /**
  * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
  *				       struct vfio_region_info)
@@ -370,6 +410,18 @@ struct vfio_region_info_cap_type {
  */
 #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD	(1)
 
+/* 1e98 vendor PCI sub-types (CXL Consortium) */
+/*
+ * CXL memory region. Use with region type
+ * (PCI_VENDOR_ID_CXL | VFIO_REGION_TYPE_PCI_VENDOR_TYPE).
+ * DPA memory region (fault+zap mmap)
+ */
+#define VFIO_REGION_SUBTYPE_CXL                 (1)
+/*
+ * HDM decoder register emulation region (read/write only, no mmap).
+ */
+#define VFIO_REGION_SUBTYPE_CXL_COMP_REGS       (2)
+
 /* sub-types for VFIO_REGION_TYPE_GFX */
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
 
#20 lore raw

PATCH 7/20 vfio/pci: Add CXL state to vfio_pci_core_device

In reply to [email protected]

From: Manish Honap <[email protected]>

Add CXL-specific state to vfio_pci_core_device structure to support CXL Type-2 device passthrough.

The new vfio_pci_cxl_state structure embeds CXL core objects:
- struct cxl_dev_state: CXL device state (from CXL core)
- struct cxl_memdev: CXL memory device
- struct cxl_region: CXL region object
- Root and endpoint decoders

Key design point: The CXL state pointer is NULL for non-CXL devices, allowing vfio-pci-core to handle both CXL and standard PCI devices with minimal overhead.

This will follow the approach where vfio-pci-core itself gains CXL awareness, rather than requiring a separate variant driver.

---
 drivers/vfio/pci/cxl/vfio_cxl_priv.h | 29 ++++++++++++++++++++++++++++
 include/linux/vfio_pci_core.h        |  3 +++
 2 files changed, 32 insertions(+)
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_priv.h
drivers/vfio/pci/cxl/vfio_cxl_priv.h new file +29 -0
new file mode 100644
index 000000000000..818a83a3809d
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Common infrastructure for CXL Type-2 device variant drivers
+ *
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#ifndef __LINUX_VFIO_CXL_PRIV_H
+#define __LINUX_VFIO_CXL_PRIV_H
+
+#include <cxl/cxl.h>
+#include <linux/types.h>
+
+/* CXL device state embedded in vfio_pci_core_device */
+struct vfio_pci_cxl_state {
+	struct cxl_dev_state         cxlds;
+	struct cxl_memdev           *cxlmd;
+	struct cxl_root_decoder     *cxlrd;
+	struct cxl_endpoint_decoder *cxled;
+	resource_size_t              hdm_reg_offset;
+	size_t                       hdm_reg_size;
+	resource_size_t              comp_reg_offset;
+	size_t                       comp_reg_size;
+	u32                          hdm_count;
+	u16                          dvsec;
+	u8                           comp_reg_bar;
+};
+
+#endif /* __LINUX_VFIO_CXL_PRIV_H */
include/linux/vfio_pci_core.h +3 -0
index 1ac86896875c..cd8ed98a82a3 100644
@@ -30,6 +30,8 @@ struct vfio_pci_region;
 struct p2pdma_provider;
 struct dma_buf_phys_vec;
 struct dma_buf_attachment;
+struct vfio_pci_cxl_state;
+
 
 struct vfio_pci_eventfd {
 	struct eventfd_ctx	*ctx;
@@ -138,6 +140,7 @@ struct vfio_pci_core_device {
 	struct mutex		ioeventfds_lock;
 	struct list_head	ioeventfds_list;
 	struct vfio_pci_vf_token	*vf_token;
+	struct vfio_pci_cxl_state *cxl;
 	struct list_head		sriov_pfs_item;
 	struct vfio_pci_core_device	*sriov_pf_core_dev;
 	struct notifier_block	nb;
Signature · 1 line
2.25.1
#21 lore raw

PATCH 8/20 vfio/pci: Add vfio-cxl Kconfig and build infrastructure

In reply to [email protected]

From: Manish Honap <[email protected]>

Introduce the Kconfig option CONFIG_VFIO_CXL_CORE and the necessary build rules to compile CXL Type-2 passthrough support into the vfio-pci-core module. The new option depends on VFIO_PCI_CORE, CXL_BUS and CXL_MEM.

Wire up the detection and cleanup entry-point stubs in vfio_pci_core_register_device() and vfio_pci_core_unregister_device() so that subsequent patches can fill in the CXL-specific logic without touching the vfio-pci-core flow again.

The vfio_cxl_core.c file added here is an empty skeleton; the actual CXL detection and initialisation code is introduced in the following patch to keep this build-system patch reviewable on its own.

---
 drivers/vfio/pci/Kconfig             |  2 ++
 drivers/vfio/pci/Makefile            |  1 +
 drivers/vfio/pci/cxl/Kconfig         |  7 ++++++
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 35 ++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_core.c     |  4 ++++
 drivers/vfio/pci/vfio_pci_priv.h     | 14 +++++++++++
 6 files changed, 63 insertions(+)
 create mode 100644 drivers/vfio/pci/cxl/Kconfig
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_core.c
drivers/vfio/pci/Kconfig +2 -0
index 1e82b44bda1a..b981a7c164ca 100644
@@ -68,6 +68,8 @@ source "drivers/vfio/pci/virtio/Kconfig"
 
 source "drivers/vfio/pci/nvgrace-gpu/Kconfig"
 
+source "drivers/vfio/pci/cxl/Kconfig"
+
 source "drivers/vfio/pci/qat/Kconfig"
 
 source "drivers/vfio/pci/xe/Kconfig"
drivers/vfio/pci/Makefile +1 -0
index e0a0757dd1d2..ecb0eacbc089 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
+vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
drivers/vfio/pci/cxl/Kconfig new file +7 -0
new file mode 100644
index 000000000000..41d60dc0de2d
@@ -0,0 +1,7 @@
+config VFIO_CXL_CORE
+	bool "VFIO CXL core"
+	depends on VFIO_PCI_CORE && CXL_BUS && CXL_MEM
+	help
+	  Core library for VFIO CXL Type-2 device support (enlightened path).
+	  When enabled, vfio-pci-core can detect and manage CXL Type-2 devices
+	  without a separate variant driver.
drivers/vfio/pci/cxl/vfio_cxl_core.c new file +35 -0
new file mode 100644
index 000000000000..7698d94e16be
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * VFIO CXL Core - Common infrastructure for CXL Type-2 device variant drivers
+ *
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ *
+ * This module provides common functionality for VFIO variant drivers that
+ * support CXL Type-2 devices (cache-coherent accelerators with attached memory).
+ */
+
+#include <linux/vfio_pci_core.h>
+#include <linux/pci.h>
+#include <cxl/cxl.h>
+#include <cxl/pci.h>
+
+#include "../vfio_pci_priv.h"
+#include "vfio_cxl_priv.h"
+
+MODULE_IMPORT_NS("CXL");
+
+/**
+ * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2 device
+ * @vdev: VFIO PCI device
+ *
+ * Called from vfio_pci_core_register_device(). Detects CXL DVSEC capability
+ * and initializes CXL features. On failure vdev->cxl remains NULL and the
+ * device operates as a standard PCI device.
+ */
+void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
+{
+}
+
+void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
+{
+}
drivers/vfio/pci/vfio_pci_core.c +4 -0
index 3a11e6f450f7..b7364178e23d 100644
@@ -2181,6 +2181,8 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)
 	if (ret)
 		goto out_vf;
 
+	vfio_pci_cxl_detect_and_init(vdev);
+
 	vfio_pci_probe_power_state(vdev);
 
 	/*
@@ -2224,6 +2226,8 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev)
 	vfio_pci_vf_uninit(vdev);
 	vfio_pci_vga_uninit(vdev);
 
+	vfio_pci_cxl_cleanup(vdev);
+
 	if (!disable_idle_d3)
 		pm_runtime_get_noresume(&vdev->pdev->dev);
 
drivers/vfio/pci/vfio_pci_priv.h +14 -0
index 27ac280f00b9..d7df5538dcde 100644
@@ -133,4 +133,18 @@ static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev,
 }
 #endif
 
+#if IS_ENABLED(CONFIG_VFIO_CXL_CORE)
+
+void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev);
+void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev);
+
+#else
+
+static inline void
+vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) { }
+static inline void
+vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) { }
+
+#endif /* CONFIG_VFIO_CXL_CORE */
+
 #endif
Signature · 1 line
2.25.1
#22 lore raw

Re: [PATCH 08/20] vfio/pci: Add vfio-cxl Kconfig and build infrastructure

Quoted 12 lines · On Thu, 12 Mar 2026 02:04:28 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

Introduce the Kconfig option CONFIG_VFIO_CXL_CORE and the necessary build rules to compile CXL Type-2 passthrough support into the vfio-pci-core module. The new option depends on VFIO_PCI_CORE, CXL_BUS and CXL_MEM.

Wire up the detection and cleanup entry-point stubs in vfio_pci_core_register_device() and vfio_pci_core_unregister_device() so that subsequent patches can fill in the CXL-specific logic without touching the vfio-pci-core flow again.

Quoted 5 lines

The vfio_cxl_core.c file added here is an empty skeleton; the actual CXL detection and initialisation code is introduced in the following patch to keep this build-system patch reviewable on its own.

Hi Manish,
A few trivial things inline.

Quoted 6 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c new file +0 -0
new file mode 100644
index 000000000000..7698d94e16be

drivers/vfio/pci/cxl/vfio_cxl_core.c line 8

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c new file +8 -0
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * VFIO CXL Core - Common infrastructure for CXL Type-2 device variant drivers
+ *
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ *
+ * This module provides common functionality for VFIO variant drivers that
+ * support CXL Type-2 devices (cache-coherent accelerators with attached memory).

As I mentioned for docs, that definition needs some finessing as also CXL Type3 devices, though intention is not the ones compliant with the class code as those can be nicely paravirtualized.

There is a whole class of CXL.mem only devices with various forms of accelerator that never need CXL.cache and so aren't Type 2.

E.g. Compressed memory type 3 devices are known to be in the wild.

drivers/vfio/pci/cxl/vfio_cxl_core.c line 19

Quoted 11 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c new file +11 -0
+ */
+
+#include <linux/vfio_pci_core.h>
+#include <linux/pci.h>
+#include <cxl/cxl.h>
+#include <cxl/pci.h>
+
+#include "../vfio_pci_priv.h"
+#include "vfio_cxl_priv.h"
+
+MODULE_IMPORT_NS("CXL");

Most often I've seen this added at the end of file next other MODULE_X calls. Whilse we don't have any of those here, still feels like a sensible place to put it.

Quoted 16 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c new file +16 -0
+
+/**
+ * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2 device
+ * @vdev: VFIO PCI device
+ *
+ * Called from vfio_pci_core_register_device(). Detects CXL DVSEC capability
+ * and initializes CXL features. On failure vdev->cxl remains NULL and the
+ * device operates as a standard PCI device.
+ */
+void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
+{
+}
+
+void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
+{
+}
#23 lore raw

RE: [PATCH 08/20] vfio/pci: Add vfio-cxl Kconfig and build infrastructure

Quoted 69 lines

-----Original Message-----
From: Jonathan Cameron <[email protected]>
Sent: 13 March 2026 17:57
To: Manish Honap <[email protected]>
Cc: Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH 08/20] vfio/pci: Add vfio-cxl Kconfig and build infrastructure

External email: Use caution opening links or attachments

Quoted 12 lines · On Thu, 12 Mar 2026 02:04:28 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

Introduce the Kconfig option CONFIG_VFIO_CXL_CORE and the necessary build rules to compile CXL Type-2 passthrough support into the vfio-pci-core module. The new option depends on VFIO_PCI_CORE, CXL_BUS and CXL_MEM.

Wire up the detection and cleanup entry-point stubs in vfio_pci_core_register_device() and vfio_pci_core_unregister_device() so that subsequent patches can fill in the CXL-specific logic without touching the vfio-pci-core flow again.

Quoted 5 lines

The vfio_cxl_core.c file added here is an empty skeleton; the actual CXL detection and initialisation code is introduced in the following patch to keep this build-system patch reviewable on its own.

Hi Manish,
A few trivial things inline.

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c new file +1 -0
new file mode 100644
index 000000000000..7698d94e16be
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0-only

drivers/vfio/pci/cxl/vfio_cxl_core.c line 11

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c new file +10 -0
+/*
+ * VFIO CXL Core - Common infrastructure for CXL Type-2 device
+variant drivers
+ *
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights
+reserved
+ *
+ * This module provides common functionality for VFIO variant drivers
+that
+ * support CXL Type-2 devices (cache-coherent accelerators with

attached memory).
As I mentioned for docs, that definition needs some finessing as also CXL Type3 devices, though intention is not the ones compliant with the class code as those can be nicely paravirtualized.

Quoted 4 lines

There is a whole class of CXL.mem only devices with various forms of accelerator that never need CXL.cache and so aren't Type 2.

E.g. Compressed memory type 3 devices are known to be in the wild.

Okay, I have updated this wording to specify the CXL capable device expectation.

Quoted 18 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 19

Quoted 12 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c new file +11 -0
+ */
+
+#include <linux/vfio_pci_core.h>
+#include <linux/pci.h>
+#include <cxl/cxl.h>
+#include <cxl/pci.h>
+
+#include "../vfio_pci_priv.h"
+#include "vfio_cxl_priv.h"
+
+MODULE_IMPORT_NS("CXL");

Most often I've seen this added at the end of file next other MODULE_X calls.
Whilse we don't have any of those here, still feels like a sensible place to put it.

Moved this to EOF.

Quoted 17 lines
Quoted 17 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c new file +16 -0
+
+/**
+ * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2
+device
+ * @vdev: VFIO PCI device
+ *
+ * Called from vfio_pci_core_register_device(). Detects CXL DVSEC
+capability
+ * and initializes CXL features. On failure vdev->cxl remains NULL
+and the
+ * device operates as a standard PCI device.
+ */
+void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
+{ }
+
+void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) { }
#24 lore raw

PATCH 9/20 vfio/cxl: Implement CXL device detection and HDM register probing

In reply to [email protected]

From: Manish Honap <[email protected]>

Implement the core CXL Type-2 device detection and component register probing logic in vfio_pci_cxl_detect_and_init().

Three private helpers are introduced:

vfio_cxl_create_device_state() allocates the per-device vfio_pci_cxl_state structure using devm_cxl_dev_state_create() so that lifetime is tied to the PCI device binding.

vfio_cxl_find_bar() locates the PCI BAR that contains a given HPA range, returning the BAR index and offset within it.

vfio_cxl_setup_regs() uses the CXL core helpers cxl_find_regblock() and cxl_probe_component_regs() to enumerate the HDM decoder register block, then records its BAR index, offset and size in the CXL state.

vfio_pci_cxl_detect_and_init() orchestrates detection:
1. Check for CXL DVSEC via pcie_is_cxl() + pci_find_dvsec_capability().
2. Allocate CXL device state.
3. Temporarily call pci_enable_device_mem() for ioremap, then disable.
4. Probe component registers to find the HDM decoder block.

On any failure vdev->cxl is devm_kfree'd so that device falls back to plain PCI mode transparently.

---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 151 +++++++++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |   8 ++
 2 files changed, 159 insertions(+)
drivers/vfio/pci/cxl/vfio_cxl_core.c +151 -0
index 7698d94e16be..2da6da1c0605 100644
@@ -18,6 +18,114 @@
 
 MODULE_IMPORT_NS("CXL");
 
+static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
+					u16 dvsec)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	struct device *dev = &pdev->dev;
+	struct vfio_pci_cxl_state *cxl;
+	bool cxl_mem_capable, is_cxl_type3;
+	u16 cap_word;
+
+	/*
+	 * The devm allocation for the CXL state remains for the entire time
+	 * the PCI device is bound to vfio-pci. From successful CXL init
+	 * in probe until the device is released on unbind.
+	 * No extra explicit free is needed; devm handles it when
+	 * pdev->dev is released.
+	 */
+	vdev->cxl = devm_cxl_dev_state_create(dev,
+					      CXL_DEVTYPE_DEVMEM,
+					      pdev->dev.id, dvsec,
+					      struct vfio_pci_cxl_state,
+					      cxlds, false);
+	if (!vdev->cxl)
+		return -ENOMEM;
+
+	cxl = vdev->cxl;
+	cxl->dvsec = dvsec;
+
+	pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
+			     &cap_word);
+
+	cxl_mem_capable = !!(cap_word & CXL_DVSEC_MEM_CAPABLE);
+	is_cxl_type3 = ((pdev->class >> 8) == PCI_CLASS_MEMORY_CXL);
+
+	/*
+	 * Type 2 = CXL memory capable but NOT Type 3 (e.g. accelerator/GPU)
+	 * Unsupported for non cxl type-2 class of devices.
+	 */
+	if (!(cxl_mem_capable && !is_cxl_type3)) {
+		devm_kfree(&pdev->dev, vdev->cxl);
+		vdev->cxl = NULL;
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	struct cxl_register_map *map = &cxl->cxlds.reg_map;
+	resource_size_t offset, bar_offset, size;
+	struct pci_dev *pdev = vdev->pdev;
+	void __iomem *base;
+	u32 count;
+	int ret;
+	u8 bar;
+
+	if (WARN_ON_ONCE(!pci_is_enabled(pdev)))
+		return -EINVAL;
+
+	/* Find component register block via Register Locator DVSEC */
+	ret = cxl_find_regblock(pdev, CXL_REGLOC_RBI_COMPONENT, map);
+	if (ret)
+		return ret;
+
+	/* Temporarily map the register block */
+	base = ioremap(map->resource, map->max_size);
+	if (!base)
+		return -ENOMEM;
+
+	/* Probe component register capabilities */
+	cxl_probe_component_regs(&pdev->dev, base, &map->component_map);
+
+	/* Unmap immediately */
+	iounmap(base);
+
+	/* Check if HDM decoder was found */
+	if (!map->component_map.hdm_decoder.valid)
+		return -ENODEV;
+
+	pci_dbg(pdev,
+		"vfio_cxl: HDM decoder at offset=0x%lx, size=0x%lx\n",
+		map->component_map.hdm_decoder.offset,
+		map->component_map.hdm_decoder.size);
+
+	/* Get HDM register info */
+	ret = cxl_get_hdm_reg_info(&cxl->cxlds, &count, &offset, &size);
+	if (ret)
+		return ret;
+
+	if (!count || !size)
+		return -ENODEV;
+
+	cxl->hdm_count = count;
+	cxl->hdm_reg_offset = offset;
+	cxl->hdm_reg_size = size;
+
+	ret = cxl_regblock_get_bar_info(map, &bar, &bar_offset);
+	if (ret)
+		return ret;
+
+	cxl->comp_reg_bar = bar;
+	cxl->comp_reg_offset = bar_offset;
+	cxl->comp_reg_size = CXL_COMPONENT_REG_BLOCK_SIZE;
+
+	return 0;
+}
+
 /**
  * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2 device
  * @vdev: VFIO PCI device
@@ -28,8 +136,51 @@ MODULE_IMPORT_NS("CXL");
  */
 void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 {
+	struct pci_dev *pdev = vdev->pdev;
+	struct vfio_pci_cxl_state *cxl;
+	u16 dvsec;
+	int ret;
+
+	if (!pcie_is_cxl(pdev))
+		return;
+
+	dvsec = pci_find_dvsec_capability(pdev,
+					  PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return;
+
+	ret = vfio_cxl_create_device_state(vdev, dvsec);
+	if (ret)
+		return;
+
+	cxl = vdev->cxl;
+
+	/*
+	 * Required for ioremap of the component register block and
+	 * calls to cxl_probe_component_regs().
+	 */
+	ret = pci_enable_device_mem(pdev);
+	if (ret)
+		goto failed;
+
+	ret = vfio_cxl_setup_regs(vdev);
+	if (ret) {
+		pci_disable_device(pdev);
+		goto failed;
+	}
+
+	pci_disable_device(pdev);
+
+	return;
+
+failed:
+	devm_kfree(&pdev->dev, vdev->cxl);
+	vdev->cxl = NULL;
 }
 
 void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
 {
+	if (!vdev->cxl)
+		return;
 }
drivers/vfio/pci/cxl/vfio_cxl_priv.h +8 -0
index 818a83a3809d..57fed39a80da 100644
@@ -26,4 +26,12 @@ struct vfio_pci_cxl_state {
 	u8                           comp_reg_bar;
 };
 
+/*
+ * CXL DVSEC for CXL Devices - register offsets within the DVSEC
+ * (CXL 2.0+ 8.1.3).
+ * Offsets are relative to the DVSEC capability base (cxl->dvsec).
+ */
+#define CXL_DVSEC_CAPABILITY_OFFSET 0xa
+#define CXL_DVSEC_MEM_CAPABLE	    BIT(2)
+
 #endif /* __LINUX_VFIO_CXL_PRIV_H */
Signature · 1 line
2.25.1
maintainer
#25 lore raw

Re: [PATCH 09/20] vfio/cxl: Implement CXL device detection and HDM register probing

Quoted 63 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Implement the core CXL Type-2 device detection and component register probing logic in vfio_pci_cxl_detect_and_init().

Three private helpers are introduced:

vfio_cxl_create_device_state() allocates the per-device vfio_pci_cxl_state structure using devm_cxl_dev_state_create() so that lifetime is tied to the PCI device binding.

vfio_cxl_find_bar() locates the PCI BAR that contains a given HPA range, returning the BAR index and offset within it.

vfio_cxl_setup_regs() uses the CXL core helpers cxl_find_regblock() and cxl_probe_component_regs() to enumerate the HDM decoder register block, then records its BAR index, offset and size in the CXL state.

vfio_pci_cxl_detect_and_init() orchestrates detection:
1. Check for CXL DVSEC via pcie_is_cxl() + pci_find_dvsec_capability().
2. Allocate CXL device state.
3. Temporarily call pci_enable_device_mem() for ioremap, then disable.
4. Probe component registers to find the HDM decoder block.

On any failure vdev->cxl is devm_kfree'd so that device falls back to plain PCI mode transparently.

---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 151 +++++++++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |   8 ++
 2 files changed, 159 insertions(+)
drivers/vfio/pci/cxl/vfio_cxl_core.c +22 -0
index 7698d94e16be..2da6da1c0605 100644
@@ -18,6 +18,114 @@
 
 MODULE_IMPORT_NS("CXL");
 
+static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
+					u16 dvsec)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	struct device *dev = &pdev->dev;
+	struct vfio_pci_cxl_state *cxl;
+	bool cxl_mem_capable, is_cxl_type3;
+	u16 cap_word;
+
+	/*
+	 * The devm allocation for the CXL state remains for the entire time
+	 * the PCI device is bound to vfio-pci. From successful CXL init
+	 * in probe until the device is released on unbind.
+	 * No extra explicit free is needed; devm handles it when
+	 * pdev->dev is released.
+	 */
+	vdev->cxl = devm_cxl_dev_state_create(dev,
+					      CXL_DEVTYPE_DEVMEM,
+					      pdev->dev.id, dvsec,
+					      struct vfio_pci_cxl_state,
+					      cxlds, false);
+	if (!vdev->cxl)

drivers/vfio/pci/cxl/vfio_cxl_core.c line 52

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+		return -ENOMEM;
+
+	cxl = vdev->cxl;
+	cxl->dvsec = dvsec;
+
+	pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
+			     &cap_word);
+
+	cxl_mem_capable = !!(cap_word & CXL_DVSEC_MEM_CAPABLE);
+	is_cxl_type3 = ((pdev->class >> 8) == PCI_CLASS_MEMORY_CXL);

Both of these can use FIELD_GET().

Quoted 25 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +25 -0
+
+	/*
+	 * Type 2 = CXL memory capable but NOT Type 3 (e.g. accelerator/GPU)
+	 * Unsupported for non cxl type-2 class of devices.
+	 */
+	if (!(cxl_mem_capable && !is_cxl_type3)) {
+		devm_kfree(&pdev->dev, vdev->cxl);
+		vdev->cxl = NULL;
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	struct cxl_register_map *map = &cxl->cxlds.reg_map;
+	resource_size_t offset, bar_offset, size;
+	struct pci_dev *pdev = vdev->pdev;
+	void __iomem *base;
+	u32 count;
+	int ret;
+	u8 bar;
+

drivers/vfio/pci/cxl/vfio_cxl_core.c line 87

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+	if (WARN_ON_ONCE(!pci_is_enabled(pdev)))
+		return -EINVAL;
+
+	/* Find component register block via Register Locator DVSEC */
+	ret = cxl_find_regblock(pdev, CXL_REGLOC_RBI_COMPONENT, map);
+	if (ret)
+		return ret;
+
+	/* Temporarily map the register block */
+	base = ioremap(map->resource, map->max_size);

Request the mem region before mapping it?

DJ

Quoted 113 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +84 -0
+	if (!base)
+		return -ENOMEM;
+
+	/* Probe component register capabilities */
+	cxl_probe_component_regs(&pdev->dev, base, &map->component_map);
+
+	/* Unmap immediately */
+	iounmap(base);
+
+	/* Check if HDM decoder was found */
+	if (!map->component_map.hdm_decoder.valid)
+		return -ENODEV;
+
+	pci_dbg(pdev,
+		"vfio_cxl: HDM decoder at offset=0x%lx, size=0x%lx\n",
+		map->component_map.hdm_decoder.offset,
+		map->component_map.hdm_decoder.size);
+
+	/* Get HDM register info */
+	ret = cxl_get_hdm_reg_info(&cxl->cxlds, &count, &offset, &size);
+	if (ret)
+		return ret;
+
+	if (!count || !size)
+		return -ENODEV;
+
+	cxl->hdm_count = count;
+	cxl->hdm_reg_offset = offset;
+	cxl->hdm_reg_size = size;
+
+	ret = cxl_regblock_get_bar_info(map, &bar, &bar_offset);
+	if (ret)
+		return ret;
+
+	cxl->comp_reg_bar = bar;
+	cxl->comp_reg_offset = bar_offset;
+	cxl->comp_reg_size = CXL_COMPONENT_REG_BLOCK_SIZE;
+
+	return 0;
+}
+
 /**
  * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2 device
  * @vdev: VFIO PCI device
@@ -28,8 +136,51 @@ MODULE_IMPORT_NS("CXL");
  */
 void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 {
+	struct pci_dev *pdev = vdev->pdev;
+	struct vfio_pci_cxl_state *cxl;
+	u16 dvsec;
+	int ret;
+
+	if (!pcie_is_cxl(pdev))
+		return;
+
+	dvsec = pci_find_dvsec_capability(pdev,
+					  PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return;
+
+	ret = vfio_cxl_create_device_state(vdev, dvsec);
+	if (ret)
+		return;
+
+	cxl = vdev->cxl;
+
+	/*
+	 * Required for ioremap of the component register block and
+	 * calls to cxl_probe_component_regs().
+	 */
+	ret = pci_enable_device_mem(pdev);
+	if (ret)
+		goto failed;
+
+	ret = vfio_cxl_setup_regs(vdev);
+	if (ret) {
+		pci_disable_device(pdev);
+		goto failed;
+	}
+
+	pci_disable_device(pdev);
+
+	return;
+
+failed:
+	devm_kfree(&pdev->dev, vdev->cxl);
+	vdev->cxl = NULL;
 }
 
 void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
 {
+	if (!vdev->cxl)
+		return;
 }
drivers/vfio/pci/cxl/vfio_cxl_priv.h +8 -0
index 818a83a3809d..57fed39a80da 100644
@@ -26,4 +26,12 @@ struct vfio_pci_cxl_state {
 	u8                           comp_reg_bar;
 };
 
+/*
+ * CXL DVSEC for CXL Devices - register offsets within the DVSEC
+ * (CXL 2.0+ 8.1.3).
+ * Offsets are relative to the DVSEC capability base (cxl->dvsec).
+ */
+#define CXL_DVSEC_CAPABILITY_OFFSET 0xa
+#define CXL_DVSEC_MEM_CAPABLE	    BIT(2)
+
 #endif /* __LINUX_VFIO_CXL_PRIV_H */
#26 lore raw

Re: [PATCH 09/20] vfio/cxl: Implement CXL device detection and HDM register probing

Quoted 25 lines · On Thu, 12 Mar 2026 15:31:03 -0700 Dave Jiang <[email protected]> wrote:
Quoted 24 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Implement the core CXL Type-2 device detection and component register probing logic in vfio_pci_cxl_detect_and_init().

Three private helpers are introduced:

vfio_cxl_create_device_state() allocates the per-device vfio_pci_cxl_state structure using devm_cxl_dev_state_create() so that lifetime is tied to the PCI device binding.

vfio_cxl_find_bar() locates the PCI BAR that contains a given HPA range, returning the BAR index and offset within it.

vfio_cxl_setup_regs() uses the CXL core helpers cxl_find_regblock() and cxl_probe_component_regs() to enumerate the HDM decoder register block, then records its BAR index, offset and size in the CXL state.

vfio_pci_cxl_detect_and_init() orchestrates detection:
1. Check for CXL DVSEC via pcie_is_cxl() + pci_find_dvsec_capability().
2. Allocate CXL device state.
3. Temporarily call pci_enable_device_mem() for ioremap, then disable.
4. Probe component registers to find the HDM decoder block.

Quoted 4 lines
Quoted 4 lines

On any failure vdev->cxl is devm_kfree'd so that device falls back to plain PCI mode transparently.

Given Dave didn't crop anything I'll just reply on top and avoid duplication.

Mostly lifetime handling comments. I get nervous when devm occurs in the middle of non devm code. It needs to be done very carefully. I don't think you have a bug here, but I'm not keen on the resulting difference in order of setup and tear down. I'd like cleanup to tidy it up even though it would be safe to do later.

Quoted 22 lines
Quoted 22 lines
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 151 +++++++++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |   8 ++
 2 files changed, 159 insertions(+)
drivers/vfio/pci/cxl/vfio_cxl_core.c +9 -0
index 7698d94e16be..2da6da1c0605 100644
@@ -18,6 +18,114 @@
 
 MODULE_IMPORT_NS("CXL");
 
+static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
+					u16 dvsec)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	struct device *dev = &pdev->dev;
+	struct vfio_pci_cxl_state *cxl;
+	bool cxl_mem_capable, is_cxl_type3;
+	u16 cap_word;
+
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 37

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +8 -0
+	/*
+	 * The devm allocation for the CXL state remains for the entire time
+	 * the PCI device is bound to vfio-pci. From successful CXL init
+	 * in probe until the device is released on unbind.
+	 * No extra explicit free is needed; devm handles it when
+	 * pdev->dev is released.
+	 */
+	vdev->cxl = devm_cxl_dev_state_create(dev,

Rather than assigning this in here, I'd use a local variable for the return of this, operate on that and return it from the function. That both creates a clean separation and possibly make error handling simpler later...

Also similar to some of the feedback Alejandro had on his type 2 series, be very careful with mixing devm calls and non devm, generally that's a path to hard to read and reason about code. I know you've stated this is fine because it's tied to the PCI device lifetime and you are probably right on that, but I'm not keen.

This might be a case where manual unwinding of devres is needed (maybe using devres groups if we end up with a bunch of stuff
to undo).

Quoted 9 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 46

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +9 -0
+					      CXL_DEVTYPE_DEVMEM,
+					      pdev->dev.id, dvsec,
+					      struct vfio_pci_cxl_state,
+					      cxlds, false);
+	if (!vdev->cxl)
+		return -ENOMEM;
+
+	cxl = vdev->cxl;
+	cxl->dvsec = dvsec;

That's a bit odd given we pass dvsec into devm_cxl_dev_state_create() why doesn't it assign it?

Quoted 9 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 52

Quoted 6 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +6 -0
+
+	pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
+			     &cap_word);
+
+	cxl_mem_capable = !!(cap_word & CXL_DVSEC_MEM_CAPABLE);
+	is_cxl_type3 = ((pdev->class >> 8) == PCI_CLASS_MEMORY_CXL);  

Both of these can use FIELD_GET().

Quoted 4 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 56

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +4 -0
+
+	/*
+	 * Type 2 = CXL memory capable but NOT Type 3 (e.g. accelerator/GPU)
+	 * Unsupported for non cxl type-2 class of devices.

As in other places, type 3 doesn't mean that. What you mean is class code compliant type 3.

Quoted 3 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 59

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +3 -0
+	 */
+	if (!(cxl_mem_capable && !is_cxl_type3)) {
+		devm_kfree(&pdev->dev, vdev->cxl);

As below. That needs a name that makes it clear it is the right devm call.

Quoted 88 lines
Quoted 18 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +18 -0
+		vdev->cxl = NULL;
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	struct cxl_register_map *map = &cxl->cxlds.reg_map;
+	resource_size_t offset, bar_offset, size;
+	struct pci_dev *pdev = vdev->pdev;
+	void __iomem *base;
+	u32 count;
+	int ret;
+	u8 bar;
+

drivers/vfio/pci/cxl/vfio_cxl_core.c line 87

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+	if (WARN_ON_ONCE(!pci_is_enabled(pdev)))
+		return -EINVAL;
+
+	/* Find component register block via Register Locator DVSEC */
+	ret = cxl_find_regblock(pdev, CXL_REGLOC_RBI_COMPONENT, map);
+	if (ret)
+		return ret;
+
+	/* Temporarily map the register block */
+	base = ioremap(map->resource, map->max_size);  

Request the mem region before mapping it?

DJ

Quoted 55 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +48 -0
+	if (!base)
+		return -ENOMEM;
+
+	/* Probe component register capabilities */
+	cxl_probe_component_regs(&pdev->dev, base, &map->component_map);
+
+	/* Unmap immediately */
+	iounmap(base);
+
+	/* Check if HDM decoder was found */
+	if (!map->component_map.hdm_decoder.valid)
+		return -ENODEV;
+
+	pci_dbg(pdev,
+		"vfio_cxl: HDM decoder at offset=0x%lx, size=0x%lx\n",
+		map->component_map.hdm_decoder.offset,
+		map->component_map.hdm_decoder.size);
+
+	/* Get HDM register info */
+	ret = cxl_get_hdm_reg_info(&cxl->cxlds, &count, &offset, &size);
+	if (ret)
+		return ret;
+
+	if (!count || !size)
+		return -ENODEV;
+
+	cxl->hdm_count = count;
+	cxl->hdm_reg_offset = offset;
+	cxl->hdm_reg_size = size;
+
+	ret = cxl_regblock_get_bar_info(map, &bar, &bar_offset);
+	if (ret)
+		return ret;
+
+	cxl->comp_reg_bar = bar;
+	cxl->comp_reg_offset = bar_offset;
+	cxl->comp_reg_size = CXL_COMPONENT_REG_BLOCK_SIZE;
+
+	return 0;
+}
+
 /**
  * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2 device
  * @vdev: VFIO PCI device
@@ -28,8 +136,51 @@ MODULE_IMPORT_NS("CXL");
  */
 void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 {
+	struct pci_dev *pdev = vdev->pdev;
+	struct vfio_pci_cxl_state *cxl;
+	u16 dvsec;
+	int ret;
+
+	if (!pcie_is_cxl(pdev))
+		return;
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c MODULE_IMPORT_NS("CXL"); · line 153

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +8 -0
+
+	dvsec = pci_find_dvsec_capability(pdev,
+					  PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return;
+
+	ret = vfio_cxl_create_device_state(vdev, dvsec);

Suggestion above would lead to.

	cxl = vfio_cxl_create_device_state(vdev, dvsec);
	if (IS_ERR(cxl))
		return PTR_ERR(cxl); //assuming failing at this point is fatal.

then only set vdev->cxl once you are sure this function succeeded. Thus removing the need to set it to NULL on failure. You need to pass it into a few more calls though.

Quoted 13 lines
Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +13 -0
+	if (ret)
+		return;
+
+	cxl = vdev->cxl;
+
+	/*
+	 * Required for ioremap of the component register block and
+	 * calls to cxl_probe_component_regs().
+	 */
+	ret = pci_enable_device_mem(pdev);
+	if (ret)
+		goto failed;
+
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c MODULE_IMPORT_NS("CXL"); · line 174

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +8 -0
+	ret = vfio_cxl_setup_regs(vdev);
+	if (ret) {
+		pci_disable_device(pdev);
+		goto failed;
+	}
+
+	pci_disable_device(pdev);
+

AS above, I think this would be cleaner if only here do we have

	vdev->cxl = cxl;
Quoted 4 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c MODULE_IMPORT_NS("CXL"); · line 178

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +4 -0
+	return;
+
+failed:
+	devm_kfree(&pdev->dev, vdev->cxl);

If you get here you found a CXL device but couldn't handle it. Is it useful
to continue? I'd suggest probably not. If returning an error then devm
cleanup happens. Also, we should have a wrapper around devm_kfree to make it clear that it is valid to use it for undoing the
devm_cxl_dev_state_create() (I haven't even checked it is valid)

Quoted 2 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c MODULE_IMPORT_NS("CXL"); · line 180

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -0
+	vdev->cxl = NULL;
 }

if /* __LINUX_VFIO_CXL_PRIV_H */

Quoted 2 lines
#27 lore raw

RE: [PATCH 09/20] vfio/cxl: Implement CXL device detection and HDM register probing

Quoted 57 lines

-----Original Message-----
From: Jonathan Cameron <[email protected]>
Sent: 13 March 2026 18:14
To: Dave Jiang <[email protected]>
Cc: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH 09/20] vfio/cxl: Implement CXL device detection and HDM register probing

External email: Use caution opening links or attachments

Quoted 15 lines · On Thu, 12 Mar 2026 15:31:03 -0700 Dave Jiang <[email protected]> wrote:
Quoted 14 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Implement the core CXL Type-2 device detection and component register probing logic in vfio_pci_cxl_detect_and_init().

Three private helpers are introduced:

vfio_cxl_create_device_state() allocates the per-device vfio_pci_cxl_state structure using devm_cxl_dev_state_create() so that lifetime is tied to the PCI device binding.

vfio_cxl_find_bar() locates the PCI BAR that contains a given HPA range, returning the BAR index and offset within it.

Quoted 6 lines
Quoted 6 lines

vfio_cxl_setup_regs() uses the CXL core helpers cxl_find_regblock() and cxl_probe_component_regs() to enumerate the HDM decoder register block, then records its BAR index, offset and size in the CXL state.

vfio_pci_cxl_detect_and_init() orchestrates detection:
1. Check for CXL DVSEC via pcie_is_cxl() +

pci_find_dvsec_capability().

Quoted 2 lines
Quoted 2 lines

2. Allocate CXL device state.
3. Temporarily call pci_enable_device_mem() for ioremap, then

disable.

Quoted 6 lines
Quoted 6 lines

4. Probe component registers to find the HDM decoder block.

On any failure vdev->cxl is devm_kfree'd so that device falls back to plain PCI mode transparently.

Quoted 9 lines

Given Dave didn't crop anything I'll just reply on top and avoid duplication.

Mostly lifetime handling comments. I get nervous when devm occurs in the middle of non devm code. It needs to be done very carefully. I don't think you have a bug here, but I'm not keen on the resulting difference in order of setup and tear down. I'd like cleanup to tidy it up even though it would be safe to do later.

I have addressed these comments by creating a local variable before assigning cxl to vdev->cxl.

Quoted 61 lines
Quoted 3 lines
Quoted 3 lines
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 151

+++++++++++++++++++++++++++

Quoted 10 lines
Quoted 10 lines
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |   8 ++
 2 files changed, 159 insertions(+)
drivers/vfio/pci/cxl/vfio_cxl_core.c +0 -0
index 7698d94e16be..2da6da1c0605 100644
@@ -18,6 +18,114 @@
 
Quoted 3 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -0
 MODULE_IMPORT_NS("CXL");
 
+static int vfio_cxl_create_device_state(struct vfio_pci_core_device

*vdev,

Quoted 9 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +9 -0
+                                   u16 dvsec) {
+   struct pci_dev *pdev = vdev->pdev;
+   struct device *dev = &pdev->dev;
+   struct vfio_pci_cxl_state *cxl;
+   bool cxl_mem_capable, is_cxl_type3;
+   u16 cap_word;
+
+   /*
+    * The devm allocation for the CXL state remains for the entire

time

Quoted 6 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 37

Quoted 6 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +6 -0
+    * the PCI device is bound to vfio-pci. From successful CXL init
+    * in probe until the device is released on unbind.
+    * No extra explicit free is needed; devm handles it when
+    * pdev->dev is released.
+    */
+   vdev->cxl = devm_cxl_dev_state_create(dev,

Rather than assigning this in here, I'd use a local variable for the return of this, operate on that and return it from the function. That both creates a clean separation and possibly make error handling simpler later...

Also similar to some of the feedback Alejandro had on his type 2 series, be very careful with mixing devm calls and non devm, generally that's a path to hard to read and reason about code.
I know you've stated this is fine because it's tied to the PCI device lifetime and you are probably right on that, but I'm not keen.

This might be a case where manual unwinding of devres is needed (maybe using devres groups if we end up with a bunch of stuff to undo).

Quoted 9 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 46

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +9 -0
+                                         CXL_DEVTYPE_DEVMEM,
+                                         pdev->dev.id, dvsec,
+                                         struct vfio_pci_cxl_state,
+                                         cxlds, false);
+   if (!vdev->cxl)
+           return -ENOMEM;
+
+   cxl = vdev->cxl;
+   cxl->dvsec = dvsec;
Quoted 2 lines

That's a bit odd given we pass dvsec into devm_cxl_dev_state_create() why doesn't it assign it?

Yes, dvsec was redundant in cxl->dvsec and cxl->cxlds.cxl_dvsec Removed it from cxl.

Quoted 9 lines
Quoted 9 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 52

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +6 -0
+
+   pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
+                        &cap_word);
+
+   cxl_mem_capable = !!(cap_word & CXL_DVSEC_MEM_CAPABLE);
+   is_cxl_type3 = ((pdev->class >> 8) == PCI_CLASS_MEMORY_CXL);

Both of these can use FIELD_GET().

I have used FIELD_GET for MEM_CAPABLE flag. For pdev->class, I see there are other occurrences (drivers/pci/pcie/aer_cxl_rch.c) which use same check for detecting memory class. So, I have kept the is_cxl_type3 check as it is.

Quoted 9 lines
Quoted 4 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +3 -0
+
+   /*
+    * Type 2 = CXL memory capable but NOT Type 3 (e.g.

accelerator/GPU)

Quoted 1 line

drivers/vfio/pci/cxl/vfio_cxl_core.c line 56

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -0
+    * Unsupported for non cxl type-2 class of devices.

As in other places, type 3 doesn't mean that. What you mean is class code compliant type 3.

Updated the comments and checks.

Quoted 36 lines
Quoted 4 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c line 59

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +3 -0
+    */
+   if (!(cxl_mem_capable && !is_cxl_type3)) {
+           devm_kfree(&pdev->dev, vdev->cxl);

As below. That needs a name that makes it clear it is the right devm call.

Quoted 29 lines
Quoted 17 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +17 -0
+           vdev->cxl = NULL;
+           return -ENODEV;
+   }
+
+   return 0;
+}
+
+static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) {
+   struct vfio_pci_cxl_state *cxl = vdev->cxl;
+   struct cxl_register_map *map = &cxl->cxlds.reg_map;
+   resource_size_t offset, bar_offset, size;
+   struct pci_dev *pdev = vdev->pdev;
+   void __iomem *base;
+   u32 count;
+   int ret;
+   u8 bar;
+

drivers/vfio/pci/cxl/vfio_cxl_core.c line 87

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+   if (WARN_ON_ONCE(!pci_is_enabled(pdev)))
+           return -EINVAL;
+
+   /* Find component register block via Register Locator DVSEC */
+   ret = cxl_find_regblock(pdev, CXL_REGLOC_RBI_COMPONENT, map);
+   if (ret)
+           return ret;
+
+   /* Temporarily map the register block */
+   base = ioremap(map->resource, map->max_size);

Request the mem region before mapping it?

Yes, added the call to request region first before mapping.

Quoted 68 lines
Quoted 38 lines

DJ

Quoted 35 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +35 -0
+   if (!base)
+           return -ENOMEM;
+
+   /* Probe component register capabilities */
+   cxl_probe_component_regs(&pdev->dev, base, &map->component_map);
+
+   /* Unmap immediately */
+   iounmap(base);
+
+   /* Check if HDM decoder was found */
+   if (!map->component_map.hdm_decoder.valid)
+           return -ENODEV;
+
+   pci_dbg(pdev,
+           "vfio_cxl: HDM decoder at offset=0x%lx, size=0x%lx\n",
+           map->component_map.hdm_decoder.offset,
+           map->component_map.hdm_decoder.size);
+
+   /* Get HDM register info */
+   ret = cxl_get_hdm_reg_info(&cxl->cxlds, &count, &offset, &size);
+   if (ret)
+           return ret;
+
+   if (!count || !size)
+           return -ENODEV;
+
+   cxl->hdm_count = count;
+   cxl->hdm_reg_offset = offset;
+   cxl->hdm_reg_size = size;
+
+   ret = cxl_regblock_get_bar_info(map, &bar, &bar_offset);
+   if (ret)
+           return ret;
+
+   cxl->comp_reg_bar = bar;
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +6 -0
+   cxl->comp_reg_offset = bar_offset;
+   cxl->comp_reg_size = CXL_COMPONENT_REG_BLOCK_SIZE;
+
+   return 0;
+}
+
 /**
  * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2

device

Quoted 12 lines
Quoted 12 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +7 -0
  * @vdev: VFIO PCI device
@@ -28,8 +136,51 @@ MODULE_IMPORT_NS("CXL");
  */
 void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device
 *vdev)  {
+   struct pci_dev *pdev = vdev->pdev;
+   struct vfio_pci_cxl_state *cxl;
+   u16 dvsec;
+   int ret;
+
+   if (!pcie_is_cxl(pdev))
+           return;
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c MODULE_IMPORT_NS("CXL"); · line 153

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +8 -0
+
+   dvsec = pci_find_dvsec_capability(pdev,
+                                     PCI_VENDOR_ID_CXL,
+                                     PCI_DVSEC_CXL_DEVICE);
+   if (!dvsec)
+           return;
+
+   ret = vfio_cxl_create_device_state(vdev, dvsec);

Suggestion above would lead to.

Quoted 8 lines
        cxl = vfio_cxl_create_device_state(vdev, dvsec);
        if (IS_ERR(cxl))
                return PTR_ERR(cxl); //assuming failing at this point is

fatal.

then only set vdev->cxl once you are sure this function succeeded. Thus removing the need to set it to NULL on failure. You need to pass it into a few more calls though.

Yes, I have added a local variable for this purpose.

Quoted 27 lines
Quoted 15 lines
Quoted 15 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +13 -0
+   if (ret)
+           return;
+
+   cxl = vdev->cxl;
+
+   /*
+    * Required for ioremap of the component register block and
+    * calls to cxl_probe_component_regs().
+    */
+   ret = pci_enable_device_mem(pdev);
+   if (ret)
+           goto failed;
+
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c MODULE_IMPORT_NS("CXL"); · line 174

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +8 -0
+   ret = vfio_cxl_setup_regs(vdev);
+   if (ret) {
+           pci_disable_device(pdev);
+           goto failed;
+   }
+
+   pci_disable_device(pdev);
+

AS above, I think this would be cleaner if only here do we have

        vdev->cxl = cxl;
Quoted 12 lines
Quoted 4 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c MODULE_IMPORT_NS("CXL"); · line 178

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +4 -0
+   return;
+
+failed:
+   devm_kfree(&pdev->dev, vdev->cxl);

If you get here you found a CXL device but couldn't handle it. Is it useful
to continue? I'd suggest probably not. If returning an error then devm
cleanup happens. Also, we should have a wrapper around devm_kfree to make it clear that it is valid to use it for undoing the
devm_cxl_dev_state_create() (I haven't even checked it is valid)

Added a wrapper around devm_kfree.

Quoted 5 lines
Quoted 2 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c MODULE_IMPORT_NS("CXL"); · line 180

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -0
+   vdev->cxl = NULL;
 }

if /* __LINUX_VFIO_CXL_PRIV_H */

Quoted 2 lines
#28 lore raw

PATCH 10/20 vfio/cxl: CXL region management

In reply to [email protected]

From: Manish Honap <[email protected]>

Add CXL region management for future guest access.

Region Management makes use of APIs provided by CXL_CORE as below:

CREATE_REGION flow:
1. Validate request (size, decoder availability)
2. Allocate HPA via cxl_get_hpa_freespace()
3. Allocate DPA via cxl_request_dpa()
4. Create region via cxl_create_region() - commits HDM decoder!
5. Get HPA range via cxl_get_region_range()

DESTROY_REGION flow:
1. Detach decoder via cxl_decoder_detach()
2. Free DPA via cxl_dpa_free()
3. Release root decoder via cxl_put_root_decoder()

---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 118 ++++++++++++++++++++++++++-
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |   5 ++
 drivers/vfio/pci/vfio_pci_priv.h     |   8 ++
 3 files changed, 130 insertions(+), 1 deletion(-)
drivers/vfio/pci/cxl/vfio_cxl_core.c +117 -1
index 2da6da1c0605..9c71f592e74e 100644
@@ -126,6 +126,112 @@ static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)
 	return 0;
 }
 
+int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev, resource_size_t size)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	resource_size_t max_size;
+	int ret;
+
+	if (cxl->precommitted)
+		return 0;
+
+	cxl->cxlrd = cxl_get_hpa_freespace(cxl->cxlmd, 1,
+					   CXL_DECODER_F_RAM |
+					   CXL_DECODER_F_TYPE2,
+					   &max_size);
+	if (IS_ERR(cxl->cxlrd))
+		return PTR_ERR(cxl->cxlrd);
+
+	/* Insufficient HPA space */
+	if (max_size < size) {
+		cxl_put_root_decoder(cxl->cxlrd);
+		cxl->cxlrd = NULL;
+		return -ENOSPC;
+	}
+
+	cxl->cxled = cxl_request_dpa(cxl->cxlmd, CXL_PARTMODE_RAM, size);
+	if (IS_ERR(cxl->cxled)) {
+		ret = PTR_ERR(cxl->cxled);
+		goto err_free_hpa;
+	}
+
+	cxl->region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1);
+	if (IS_ERR(cxl->region)) {
+		ret = PTR_ERR(cxl->region);
+		goto err_free_dpa;
+	}
+
+	return 0;
+
+err_free_dpa:
+	cxl_dpa_free(cxl->cxled);
+err_free_hpa:
+	if (cxl->cxlrd)
+		cxl_put_root_decoder(cxl->cxlrd);
+
+	return ret;
+}
+
+void vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	if (!cxl->region)
+		return;
+
+	cxl_unregister_region(cxl->region);
+	cxl->region = NULL;
+
+	if (cxl->precommitted)
+		return;
+
+	cxl_dpa_free(cxl->cxled);
+	cxl_put_root_decoder(cxl->cxlrd);
+}
+
+static int vfio_cxl_create_region_helper(struct vfio_pci_core_device *vdev,
+					 resource_size_t capacity)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	int ret;
+
+	if (cxl->precommitted) {
+		cxl->cxled = cxl_get_committed_decoder(cxl->cxlmd,
+						       &cxl->region);
+		if (IS_ERR(cxl->cxled))
+			return PTR_ERR(cxl->cxled);
+	} else {
+		ret = vfio_cxl_create_cxl_region(vdev, capacity);
+		if (ret)
+			return ret;
+	}
+
+	if (cxl->region) {
+		struct range range;
+
+		ret = cxl_get_region_range(cxl->region, &range);
+		if (ret)
+			goto failed;
+
+		cxl->region_hpa = range.start;
+		cxl->region_size = range_len(&range);
+
+		pci_dbg(pdev, "Precommitted decoder: HPA 0x%llx size %lu MB\n",
+			cxl->region_hpa, cxl->region_size >> 20);
+	} else {
+		pci_err(pdev, "Failed to create CXL region\n");
+		ret = -ENODEV;
+		goto failed;
+	}
+
+	return 0;
+
+failed:
+	vfio_cxl_destroy_cxl_region(vdev);
+	return ret;
+}
+
 /**
  * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2 device
  * @vdev: VFIO PCI device
@@ -172,6 +278,12 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 
 	pci_disable_device(pdev);
 
+	ret = vfio_cxl_create_region_helper(vdev, SZ_256M);
+	if (ret)
+		goto failed;
+
+	cxl->precommitted = true;
+
 	return;
 
 failed:
@@ -181,6 +293,10 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 
 void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
 {
-	if (!vdev->cxl)
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	if (!cxl || !cxl->region)
 		return;
+
+	vfio_cxl_destroy_cxl_region(vdev);
 }
drivers/vfio/pci/cxl/vfio_cxl_priv.h +5 -0
index 57fed39a80da..985680842a13 100644
@@ -17,6 +17,10 @@ struct vfio_pci_cxl_state {
 	struct cxl_memdev           *cxlmd;
 	struct cxl_root_decoder     *cxlrd;
 	struct cxl_endpoint_decoder *cxled;
+	struct cxl_region	    *region;
+	resource_size_t		     region_hpa;
+	size_t			     region_size;
+	void __iomem		    *region_vaddr;
 	resource_size_t              hdm_reg_offset;
 	size_t                       hdm_reg_size;
 	resource_size_t              comp_reg_offset;
@@ -24,6 +28,7 @@ struct vfio_pci_cxl_state {
 	u32                          hdm_count;
 	u16                          dvsec;
 	u8                           comp_reg_bar;
+	bool                         precommitted;
 };
 
 /*
drivers/vfio/pci/vfio_pci_priv.h +8 -0
index d7df5538dcde..818d99f098bf 100644
@@ -137,6 +137,9 @@ static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev,
 
 void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev);
 void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev);
+int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev,
+			       resource_size_t size);
+void vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev);
 
 #else
 
@@ -144,6 +147,11 @@ static inline void
 vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) { }
 static inline void
 vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) { }
+static inline int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev,
+					     resource_size_t size)
+{ return 0; }
+static inline void
+vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev) { }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
 
Signature · 1 line
2.25.1
maintainer
#29 lore raw

Re: [PATCH 10/20] vfio/cxl: CXL region management

Quoted 36 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Add CXL region management for future guest access.

Region Management makes use of APIs provided by CXL_CORE as below:

CREATE_REGION flow:
1. Validate request (size, decoder availability)
2. Allocate HPA via cxl_get_hpa_freespace()
3. Allocate DPA via cxl_request_dpa()
4. Create region via cxl_create_region() - commits HDM decoder!
5. Get HPA range via cxl_get_region_range()

DESTROY_REGION flow:
1. Detach decoder via cxl_decoder_detach()
2. Free DPA via cxl_dpa_free()
3. Release root decoder via cxl_put_root_decoder()

---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 118 ++++++++++++++++++++++++++-
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |   5 ++
 drivers/vfio/pci/vfio_pci_priv.h     |   8 ++
 3 files changed, 130 insertions(+), 1 deletion(-)
drivers/vfio/pci/cxl/vfio_cxl_core.c +3 -0
index 2da6da1c0605..9c71f592e74e 100644
@@ -126,6 +126,112 @@ static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)
 	return 0;
 }
 
+int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev, resource_size_t size)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 141

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+	resource_size_t max_size;
+	int ret;
+
+	if (cxl->precommitted)
+		return 0;
+
+	cxl->cxlrd = cxl_get_hpa_freespace(cxl->cxlmd, 1,
+					   CXL_DECODER_F_RAM |
+					   CXL_DECODER_F_TYPE2,
+					   &max_size);

Not sure what VFIO subsystem's policy is on scoped base resource cleanup, but a __free() here can get you out of managing put() of the root decoder.

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 152

Quoted 11 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +11 -0
+	if (IS_ERR(cxl->cxlrd))
+		return PTR_ERR(cxl->cxlrd);
+
+	/* Insufficient HPA space */
+	if (max_size < size) {
+		cxl_put_root_decoder(cxl->cxlrd);
+		cxl->cxlrd = NULL;
+		return -ENOSPC;
+	}
+
+	cxl->cxled = cxl_request_dpa(cxl->cxlmd, CXL_PARTMODE_RAM, size);

Same comment here about __free().

Quoted 48 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +48 -0
+	if (IS_ERR(cxl->cxled)) {
+		ret = PTR_ERR(cxl->cxled);
+		goto err_free_hpa;
+	}
+
+	cxl->region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1);
+	if (IS_ERR(cxl->region)) {
+		ret = PTR_ERR(cxl->region);
+		goto err_free_dpa;
+	}
+
+	return 0;
+
+err_free_dpa:
+	cxl_dpa_free(cxl->cxled);
+err_free_hpa:
+	if (cxl->cxlrd)
+		cxl_put_root_decoder(cxl->cxlrd);
+
+	return ret;
+}
+
+void vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	if (!cxl->region)
+		return;
+
+	cxl_unregister_region(cxl->region);
+	cxl->region = NULL;
+
+	if (cxl->precommitted)
+		return;
+
+	cxl_dpa_free(cxl->cxled);
+	cxl_put_root_decoder(cxl->cxlrd);
+}
+
+static int vfio_cxl_create_region_helper(struct vfio_pci_core_device *vdev,
+					 resource_size_t capacity)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	int ret;
+
+	if (cxl->precommitted) {
+		cxl->cxled = cxl_get_committed_decoder(cxl->cxlmd,

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 210

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+						       &cxl->region);
+		if (IS_ERR(cxl->cxled))
+			return PTR_ERR(cxl->cxled);
+	} else {
+		ret = vfio_cxl_create_cxl_region(vdev, capacity);
+		if (ret)
+			return ret;
+	}
+
+	if (cxl->region) {

Maybe if you do 'if (!cxl->region)' first and just exit, then you don't need to indent the normal code path.

Quoted 28 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +24 -0
+		struct range range;
+
+		ret = cxl_get_region_range(cxl->region, &range);
+		if (ret)
+			goto failed;
+
+		cxl->region_hpa = range.start;
+		cxl->region_size = range_len(&range);
+
+		pci_dbg(pdev, "Precommitted decoder: HPA 0x%llx size %lu MB\n",
+			cxl->region_hpa, cxl->region_size >> 20);
+	} else {
+		pci_err(pdev, "Failed to create CXL region\n");
+		ret = -ENODEV;
+		goto failed;
+	}
+
+	return 0;
+
+failed:
+	vfio_cxl_destroy_cxl_region(vdev);
+	return ret;
+}
+
 /**
  * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2 device
  * @vdev: VFIO PCI device

drivers/vfio/pci/cxl/vfio_cxl_core.c void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) · line 281

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -0
@@ -172,6 +278,12 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 
 	pci_disable_device(pdev);
 
+	ret = vfio_cxl_create_region_helper(vdev, SZ_256M);

Maybe a comment on why this size?

DJ

Quoted 69 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -1
+	if (ret)
+		goto failed;
+
+	cxl->precommitted = true;
+
 	return;
 
 failed:
@@ -181,6 +293,10 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 
 void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
 {
-	if (!vdev->cxl)
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	if (!cxl || !cxl->region)
 		return;
+
+	vfio_cxl_destroy_cxl_region(vdev);
 }
drivers/vfio/pci/cxl/vfio_cxl_priv.h +5 -0
index 57fed39a80da..985680842a13 100644
@@ -17,6 +17,10 @@ struct vfio_pci_cxl_state {
 	struct cxl_memdev           *cxlmd;
 	struct cxl_root_decoder     *cxlrd;
 	struct cxl_endpoint_decoder *cxled;
+	struct cxl_region	    *region;
+	resource_size_t		     region_hpa;
+	size_t			     region_size;
+	void __iomem		    *region_vaddr;
 	resource_size_t              hdm_reg_offset;
 	size_t                       hdm_reg_size;
 	resource_size_t              comp_reg_offset;
@@ -24,6 +28,7 @@ struct vfio_pci_cxl_state {
 	u32                          hdm_count;
 	u16                          dvsec;
 	u8                           comp_reg_bar;
+	bool                         precommitted;
 };
 
 /*
drivers/vfio/pci/vfio_pci_priv.h +8 -0
index d7df5538dcde..818d99f098bf 100644
@@ -137,6 +137,9 @@ static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev,
 
 void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev);
 void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev);
+int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev,
+			       resource_size_t size);
+void vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev);
 
 #else
 
@@ -144,6 +147,11 @@ static inline void
 vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) { }
 static inline void
 vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) { }
+static inline int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev,
+					     resource_size_t size)
+{ return 0; }
+static inline void
+vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev) { }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
 
#30 lore raw

Re: [PATCH 10/20] vfio/cxl: CXL region management

Quoted 14 lines · On Thu, 12 Mar 2026 15:55:32 -0700 Dave Jiang <[email protected]> wrote:
Quoted 13 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Add CXL region management for future guest access.

Region Management makes use of APIs provided by CXL_CORE as below:

CREATE_REGION flow:
1. Validate request (size, decoder availability)
2. Allocate HPA via cxl_get_hpa_freespace()
3. Allocate DPA via cxl_request_dpa()
4. Create region via cxl_create_region() - commits HDM decoder!
5. Get HPA range via cxl_get_region_range()

Quoted 6 lines
Quoted 6 lines

DESTROY_REGION flow:
1. Detach decoder via cxl_decoder_detach()
2. Free DPA via cxl_dpa_free()
3. Release root decoder via cxl_put_root_decoder()

A few additional comments from me.

Quoted 30 lines
Quoted 17 lines
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 118 ++++++++++++++++++++++++++-
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |   5 ++
 drivers/vfio/pci/vfio_pci_priv.h     |   8 ++
 3 files changed, 130 insertions(+), 1 deletion(-)
drivers/vfio/pci/cxl/vfio_cxl_core.c +3 -0
index 2da6da1c0605..9c71f592e74e 100644
@@ -126,6 +126,112 @@ static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)
 	return 0;
 }
 
+int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev, resource_size_t size)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 141

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+	resource_size_t max_size;
+	int ret;
+
+	if (cxl->precommitted)
+		return 0;
+
+	cxl->cxlrd = cxl_get_hpa_freespace(cxl->cxlmd, 1,
+					   CXL_DECODER_F_RAM |
+					   CXL_DECODER_F_TYPE2,
+					   &max_size);  

Not sure what VFIO subsystem's policy is on scoped base resource cleanup, but a __free() here can get you out of managing put() of the root decoder.

Quoted 7 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 148

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +7 -0
+	if (IS_ERR(cxl->cxlrd))
+		return PTR_ERR(cxl->cxlrd);
+
+	/* Insufficient HPA space */
+	if (max_size < size) {
+		cxl_put_root_decoder(cxl->cxlrd);
+		cxl->cxlrd = NULL;

Similar to other cases, I'd keep assigning stuff in cxl to the point where there are no more error paths. Use local variables until then. (that would fit with using __free() as well which I'd also favor if
accepted in VFIO).

Quoted 7 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 152

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +4 -0
+		return -ENOSPC;
+	}
+
+	cxl->cxled = cxl_request_dpa(cxl->cxlmd, CXL_PARTMODE_RAM, size);  

Same comment here about __free().

Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 160

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +8 -0
+	if (IS_ERR(cxl->cxled)) {
+		ret = PTR_ERR(cxl->cxled);
+		goto err_free_hpa;
+	}
+
+	cxl->region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1);
+	if (IS_ERR(cxl->region)) {
+		ret = PTR_ERR(cxl->region);

You carefully NULL this in vfio_cxl_destroy_region, but if you fail here you end up with it containing an ERR_PTR(). I'd avoid that by using a local variable and only assigning cxl->region after this suceeds.

Quoted 66 lines
Quoted 40 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +40 -0
+		goto err_free_dpa;
+	}
+
+	return 0;
+
+err_free_dpa:
+	cxl_dpa_free(cxl->cxled);
+err_free_hpa:
+	if (cxl->cxlrd)
+		cxl_put_root_decoder(cxl->cxlrd);
+
+	return ret;
+}
+
+void vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	if (!cxl->region)
+		return;
+
+	cxl_unregister_region(cxl->region);
+	cxl->region = NULL;
+
+	if (cxl->precommitted)
+		return;
+
+	cxl_dpa_free(cxl->cxled);
+	cxl_put_root_decoder(cxl->cxlrd);
+}
+
+static int vfio_cxl_create_region_helper(struct vfio_pci_core_device *vdev,
+					 resource_size_t capacity)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	int ret;
+
+	if (cxl->precommitted) {
+		cxl->cxled = cxl_get_committed_decoder(cxl->cxlmd,

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 210

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+						       &cxl->region);
+		if (IS_ERR(cxl->cxled))
+			return PTR_ERR(cxl->cxled);
+	} else {
+		ret = vfio_cxl_create_cxl_region(vdev, capacity);
+		if (ret)
+			return ret;
+	}
+
+	if (cxl->region) {  

Maybe if you do 'if (!cxl->region)' first and just exit, then you don't need to indent the normal code path.

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +13 -0
+		struct range range;
+
+		ret = cxl_get_region_range(cxl->region, &range);
+		if (ret)
+			goto failed;
+
+		cxl->region_hpa = range.start;
+		cxl->region_size = range_len(&range);
+
+		pci_dbg(pdev, "Precommitted decoder: HPA 0x%llx size %lu MB\n",
+			cxl->region_hpa, cxl->region_size >> 20);
+	} else {
+		pci_err(pdev, "Failed to create CXL region\n");
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 231

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +8 -0
+		ret = -ENODEV;
+		goto failed;
+	}
+
+	return 0;
+
+failed:
+	vfio_cxl_destroy_cxl_region(vdev);

Little bit of refactoring and this could be replaced with __free() magic.

Quoted 8 lines
Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +3 -0
+	return ret;
+}
+
 /**
  * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2 device
  * @vdev: VFIO PCI device
@@ -172,6 +278,12 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 
Quoted 5 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) · line 281

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -0
 	pci_disable_device(pdev);
 
+	ret = vfio_cxl_create_region_helper(vdev, SZ_256M);  

Maybe a comment on why this size?

:) I wondered that as well. I'm guessing your bios isn't always providing the decoder and this lets you test.

Quoted 10 lines

DJ

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +5 -0
+	if (ret)
+		goto failed;
+
+	cxl->precommitted = true;
+
 	return;
 
Quoted 7 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) · line 296

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -1
 failed:
@@ -181,6 +293,10 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 
 void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
 {
-	if (!vdev->cxl)
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;

Do that in the earlier patch to reduce churn a tiny bit.

Quoted 6 lines
Quoted 6 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +4 -0
+
+	if (!cxl || !cxl->region)
 		return;
+
+	vfio_cxl_destroy_cxl_region(vdev);
 }
#31 lore raw

RE: [PATCH 10/20] vfio/cxl: CXL region management

Quoted 77 lines

-----Original Message-----
From: Jonathan Cameron <[email protected]>
Sent: 13 March 2026 18:23
To: Dave Jiang <[email protected]>
Cc: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH 10/20] vfio/cxl: CXL region management

External email: Use caution opening links or attachments

Quoted 13 lines · On Thu, 12 Mar 2026 15:55:32 -0700 Dave Jiang <[email protected]> wrote:
Quoted 12 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Add CXL region management for future guest access.

Region Management makes use of APIs provided by CXL_CORE as below:

CREATE_REGION flow:
1. Validate request (size, decoder availability) 2. Allocate HPA via cxl_get_hpa_freespace() 3. Allocate DPA via cxl_request_dpa() 4. Create region via cxl_create_region() - commits HDM decoder!
5. Get HPA range via cxl_get_region_range()

Quoted 5 lines
Quoted 5 lines

DESTROY_REGION flow:
1. Detach decoder via cxl_decoder_detach() 2. Free DPA via cxl_dpa_free() 3. Release root decoder via cxl_put_root_decoder()

A few additional comments from me.

Quoted 2 lines
Quoted 2 lines
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 118

++++++++++++++++++++++++++-

Quoted 10 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct

Quoted 10 lines
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |   5 ++
 drivers/vfio/pci/vfio_pci_priv.h     |   8 ++
 3 files changed, 130 insertions(+), 1 deletion(-)
drivers/vfio/pci/cxl/vfio_cxl_core.c +0 -0
index 2da6da1c0605..9c71f592e74e 100644
@@ -126,6 +126,112 @@ static int vfio_cxl_setup_regs(struct

vfio_pci_core_device *vdev)

Quoted 18 lines
Quoted 6 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +3 -0
    return 0;
 }
 
+int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev,
+resource_size_t size) {
+   struct vfio_pci_cxl_state *cxl = vdev->cxl;

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 141

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+   resource_size_t max_size;
+   int ret;
+
+   if (cxl->precommitted)
+           return 0;
+
+   cxl->cxlrd = cxl_get_hpa_freespace(cxl->cxlmd, 1,
+                                      CXL_DECODER_F_RAM |
+                                      CXL_DECODER_F_TYPE2,
+                                      &max_size);

Not sure what VFIO subsystem's policy is on scoped base resource

Quoted 2 lines

cleanup, but a __free() here can get you out of managing put() of the root decoder.

After having a word with Alex for this, I have added __free macros.

Quoted 8 lines
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 148

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +7 -0
+   if (IS_ERR(cxl->cxlrd))
+           return PTR_ERR(cxl->cxlrd);
+
+   /* Insufficient HPA space */
+   if (max_size < size) {
+           cxl_put_root_decoder(cxl->cxlrd);
+           cxl->cxlrd = NULL;
Quoted 5 lines

Similar to other cases, I'd keep assigning stuff in cxl to the point where there are no more error paths. Use local variables until then. (that would fit with using __free() as well which I'd also favor if accepted in VFIO).

Yes, addressed.

Quoted 17 lines
Quoted 6 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +5 -0
+           return -ENOSPC;
+   }
+
+   cxl->cxled = cxl_request_dpa(cxl->cxlmd, CXL_PARTMODE_RAM,
+ size);
Quoted 10 lines

Same comment here about __free().

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +8 -0
+   if (IS_ERR(cxl->cxled)) {
+           ret = PTR_ERR(cxl->cxled);
+           goto err_free_hpa;
+   }
+
+   cxl->region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1);
+   if (IS_ERR(cxl->region)) {
+           ret = PTR_ERR(cxl->region);
Quoted 3 lines

You carefully NULL this in vfio_cxl_destroy_region, but if you fail here you end up with it containing an ERR_PTR(). I'd avoid that by using a local variable and only assigning cxl->region after this suceeds.

Agreed.

Quoted 52 lines
Quoted 24 lines
Quoted 24 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +23 -0
+           goto err_free_dpa;
+   }
+
+   return 0;
+
+err_free_dpa:
+   cxl_dpa_free(cxl->cxled);
+err_free_hpa:
+   if (cxl->cxlrd)
+           cxl_put_root_decoder(cxl->cxlrd);
+
+   return ret;
+}
+
+void vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev)
+{
+   struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+   if (!cxl->region)
+           return;
+
+   cxl_unregister_region(cxl->region);
+   cxl->region = NULL;
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +8 -0
+
+   if (cxl->precommitted)
+           return;
+
+   cxl_dpa_free(cxl->cxled);
+   cxl_put_root_decoder(cxl->cxlrd); }
+
+static int vfio_cxl_create_region_helper(struct vfio_pci_core_device

*vdev,

Quoted 11 lines
Quoted 11 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +11 -0
+                                    resource_size_t capacity) {
+   struct vfio_pci_cxl_state *cxl = vdev->cxl;
+   struct pci_dev *pdev = vdev->pdev;
+   int ret;
+
+   if (cxl->precommitted) {
+           cxl->cxled = cxl_get_committed_decoder(cxl->cxlmd,
+                                                  &cxl->region);
+           if (IS_ERR(cxl->cxled))
+                   return PTR_ERR(cxl->cxled);
+   } else {
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 210

Quoted 6 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +6 -0
+           ret = vfio_cxl_create_cxl_region(vdev, capacity);
+           if (ret)
+                   return ret;
+   }
+
+   if (cxl->region) {

Maybe if you do 'if (!cxl->region)' first and just exit, then you don't

Quoted 1 line

need to indent the normal code path.

Okay, I will change this.

Quoted 36 lines
Quoted 11 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)

Quoted 11 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+           struct range range;
+
+           ret = cxl_get_region_range(cxl->region, &range);
+           if (ret)
+                   goto failed;
+
+           cxl->region_hpa = range.start;
+           cxl->region_size = range_len(&range);
+
+           pci_dbg(pdev, "Precommitted decoder: HPA 0x%llx size %lu

MB\n",

Quoted 11 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev) · line 231

Quoted 11 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +11 -0
+                   cxl->region_hpa, cxl->region_size >> 20);
+   } else {
+           pci_err(pdev, "Failed to create CXL region\n");
+           ret = -ENODEV;
+           goto failed;
+   }
+
+   return 0;
+
+failed:
+   vfio_cxl_destroy_cxl_region(vdev);

Little bit of refactoring and this could be replaced with __free() magic.

Quoted 5 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +3 -0
+   return ret;
+}
+
 /**
  * vfio_pci_cxl_detect_and_init - Detect and initialize CXL Type-2

device

Quoted 4 lines
Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +0 -0
  * @vdev: VFIO PCI device
@@ -172,6 +278,12 @@ void vfio_pci_cxl_detect_and_init(struct
 vfio_pci_core_device *vdev)
 
Quoted 7 lines
Quoted 5 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) · line 281

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -0
    pci_disable_device(pdev);
 
+   ret = vfio_cxl_create_region_helper(vdev, SZ_256M);

Maybe a comment on why this size?

:) I wondered that as well. I'm guessing your bios isn't always providing the decoder and this lets you test.

This was added for handling the patch organization so that the patches are cut at a short reviewable boundary. It seems this patch is not cut correctly.

I have removed this temporary size and refactored the patches to incorporate firmware committed decoder size here.

Quoted 16 lines
Quoted 16 lines

DJ

Quoted 11 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +5 -0
+   if (ret)
+           goto failed;
+
+   cxl->precommitted = true;
+
    return;
 

failed:

drivers/vfio/pci/cxl/vfio_cxl_core.c +0 -0
@@ -181,6 +293,10 @@ void vfio_pci_cxl_detect_and_init(struct
 vfio_pci_core_device *vdev)
 
Quoted 5 lines
Quoted 3 lines

drivers/vfio/pci/cxl/vfio_cxl_core.c void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) · line 296

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -1
 void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)  {
-   if (!vdev->cxl)
+   struct vfio_pci_cxl_state *cxl = vdev->cxl;

Do that in the earlier patch to reduce churn a tiny bit.

Okay, agreed.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +4 -0
+
+   if (!cxl || !cxl->region)
            return;
+
+   vfio_cxl_destroy_cxl_region(vdev);
 }
#32 lore raw

PATCH 11/20 vfio/cxl: Expose DPA memory region to userspace with fault+zap mmap

In reply to [email protected]

From: Manish Honap <[email protected]>

To directly access the device memory, a CXL region is required. For the userspace (e.g. QEMU) to access the CXL region, the region is required to be exposed via VFIO interfaces.

Introduce a new VFIO device region and region ops to expose the created CXL region. Introduce a new sub region type for userspace to identify a CXL region.

CXL region lifecycle:
- The CXL memory region is registered with VFIO layer during vfio_pci_open_device
- mmap() establishes the VMA with vm_ops but inserts no PTEs
- Each guest page fault calls vfio_cxl_region_page_fault() which inserts a single PFN under the memory_lock read side
- On device reset, vfio_cxl_zap_region_locked() sets region_active=false and calls unmap_mapping_range() to invalidate all DPA PTEs atomically while holding memory_lock for writing
- Faults racing with reset see region_active==false and return VM_FAULT_SIGBUS
- vfio_cxl_reactivate_region() restores region_active after successful hardware reset

Also integrate the zap/reactivate calls into vfio_pci_ioctl_reset() so that FLR correctly invalidates DPA mappings and restores them on success.

---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 222 +++++++++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |   2 +
 drivers/vfio/pci/vfio_pci.c          |   9 ++
 drivers/vfio/pci/vfio_pci_core.c     |  11 ++
 drivers/vfio/pci/vfio_pci_priv.h     |  13 ++
 5 files changed, 257 insertions(+)
drivers/vfio/pci/cxl/vfio_cxl_core.c +222 -0
index 9c71f592e74e..03846bd11c8a 100644
@@ -44,6 +44,7 @@ static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
 
 	cxl = vdev->cxl;
 	cxl->dvsec = dvsec;
+	cxl->dpa_region_idx = -1;
 
 	pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
 			     &cap_word);
@@ -300,3 +301,224 @@ void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
 
 	vfio_cxl_destroy_cxl_region(vdev);
 }
+
+/*
+ * Fault handler for the DPA region VMA.  Called under mm->mmap_lock read
+ * side by the fault path.  We take memory_lock read side here to exclude
+ * the write-side held by vfio_cxl_zap_region_locked() during reset.
+ */
+static vm_fault_t vfio_cxl_region_page_fault(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct vfio_pci_core_device *vdev = vma->vm_private_data;
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	unsigned long pfn;
+
+	guard(rwsem_read)(&vdev->memory_lock);
+
+	if (!READ_ONCE(cxl->region_active))
+		return VM_FAULT_SIGBUS;
+
+	pfn = PHYS_PFN(cxl->region_hpa) +
+		((vmf->address - vma->vm_start) >> PAGE_SHIFT);
+
+	/*
+	 * Scrub the page via the kernel ioremap_cache mapping before inserting
+	 * the user PFN. Prevent the stale device data from leaking across VFIO
+	 * device open/close boundaries.
+	 */
+	memset_io((u8 __iomem *)cxl->region_vaddr +
+		  ((pfn - PHYS_PFN(cxl->region_hpa)) << PAGE_SHIFT),
+		  0, PAGE_SIZE);
+
+	return vmf_insert_pfn(vma, vmf->address, pfn);
+}
+
+static const struct vm_operations_struct vfio_cxl_region_vm_ops = {
+	.fault = vfio_cxl_region_page_fault,
+};
+
+static int vfio_cxl_region_mmap(struct vfio_pci_core_device *vdev,
+				struct vfio_pci_region *region,
+				struct vm_area_struct *vma)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	unsigned long req_len;
+
+	if (!(region->flags & VFIO_REGION_INFO_FLAG_MMAP))
+		return -EINVAL;
+
+	if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len))
+		return -EOVERFLOW;
+
+	if (req_len > cxl->region_size)
+		return -EINVAL;
+
+	/*
+	 * Do not insert PTEs here (no remap_pfn_range).  PTEs are inserted
+	 * lazily on first fault via vfio_cxl_region_page_fault().  This
+	 * allows vfio_cxl_zap_region_locked() to safely invalidate them
+	 * during device reset without any userspace cooperation.
+	 * Leave vm_page_prot at its default.
+	 */
+
+	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
+	vma->vm_private_data = vdev;
+	vma->vm_ops = &vfio_cxl_region_vm_ops;
+
+	return 0;
+}
+
+/*
+ * vfio_cxl_zap_region_locked - Invalidate all DPA region PTEs.
+ *
+ * Must be called with vdev->memory_lock held for writing.  Sets
+ * region_active=false before zapping so any fault racing with zap sees
+ * the inactive state and returns VM_FAULT_SIGBUS rather than inserting
+ * a stale PFN.
+ */
+void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_device *core_vdev = &vdev->vdev;
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (!cxl || cxl->dpa_region_idx < 0)
+		return;
+
+	WRITE_ONCE(cxl->region_active, false);
+	unmap_mapping_range(core_vdev->inode->i_mapping,
+			    VFIO_PCI_INDEX_TO_OFFSET(VFIO_PCI_NUM_REGIONS +
+						     cxl->dpa_region_idx),
+			    cxl->region_size, true);
+}
+
+/*
+ * vfio_cxl_reactivate_region - Re-enable DPA region after successful reset.
+ *
+ * Must be called with vdev->memory_lock held for writing.  Re-reads the
+ * HDM decoder state from hardware (FLR cleared it) and sets region_active
+ * so that subsequent faults can re-insert PFNs without a new mmap.
+ */
+void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (!cxl)
+		return;
+}
+
+static ssize_t vfio_cxl_region_rw(struct vfio_pci_core_device *core_dev,
+				  char __user *buf, size_t count, loff_t *ppos,
+				  bool iswrite)
+{
+	unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) - VFIO_PCI_NUM_REGIONS;
+	struct vfio_pci_cxl_state *cxl = core_dev->region[i].data;
+	loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+
+	guard(rwsem_read)(&core_dev->memory_lock);
+
+	if (!READ_ONCE(cxl->region_active))
+		return -EIO;
+
+	if (!count)
+		return 0;
+
+	return vfio_pci_core_do_io_rw(core_dev, false,
+				      cxl->region_vaddr,
+				      (char __user *)buf, pos, count,
+				      0, 0, iswrite, VFIO_PCI_IO_WIDTH_8);
+}
+
+static void vfio_cxl_region_release(struct vfio_pci_core_device *vdev,
+				    struct vfio_pci_region *region)
+{
+	struct vfio_pci_cxl_state *cxl = region->data;
+
+	if (cxl->region_vaddr) {
+		iounmap(cxl->region_vaddr);
+		cxl->region_vaddr = NULL;
+	}
+}
+
+static const struct vfio_pci_regops vfio_cxl_regops = {
+	.rw		= vfio_cxl_region_rw,
+	.mmap		= vfio_cxl_region_mmap,
+	.release	= vfio_cxl_region_release,
+};
+
+int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	u32 flags;
+	int ret;
+
+	if (!cxl)
+		return -ENODEV;
+
+	if (!cxl->region || cxl->region_vaddr)
+		return -ENODEV;
+
+	cxl->region_vaddr = ioremap_cache(cxl->region_hpa, cxl->region_size);
+	if (!cxl->region_vaddr)
+		return -ENOMEM;
+
+	flags = VFIO_REGION_INFO_FLAG_READ |
+		VFIO_REGION_INFO_FLAG_WRITE |
+		VFIO_REGION_INFO_FLAG_MMAP;
+
+	ret = vfio_pci_core_register_dev_region(vdev,
+						PCI_VENDOR_ID_CXL |
+						VFIO_REGION_TYPE_PCI_VENDOR_TYPE,
+						VFIO_REGION_SUBTYPE_CXL,
+						&vfio_cxl_regops,
+						cxl->region_size, flags,
+						cxl);
+	if (ret) {
+		iounmap(cxl->region_vaddr);
+		cxl->region_vaddr = NULL;
+		return ret;
+	}
+
+	/*
+	 * Cache the vdev->region[] index before activating the region.
+	 * vfio_pci_core_register_dev_region() placed the new entry at
+	 * vdev->region[num_regions - 1] and incremented num_regions.
+	 * vfio_cxl_zap_region_locked() uses this to avoid scanning
+	 * vdev->region[] on every FLR.
+	 */
+	cxl->dpa_region_idx = vdev->num_regions - 1;
+	WRITE_ONCE(cxl->region_active, true);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_cxl_register_cxl_region);
+
+/**
+ * vfio_cxl_unregister_cxl_region - Undo vfio_cxl_register_cxl_region()
+ * @vdev: VFIO PCI device
+ *
+ * Marks the DPA region inactive so any racing fault returns VM_FAULT_SIGBUS
+ * and resets dpa_region_idx.  Does NOT call release() or touch num_regions;
+ * vfio_pci_core_disable() will call the idempotent release() callback as
+ * normal during device close.
+ *
+ * Does NOT touch CXL subsystem state (cxl->region, cxl->cxled, cxl->cxlrd).
+ * The caller must call vfio_cxl_destroy_cxl_region() separately to release
+ * those objects.
+ */
+void vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	if (!cxl || cxl->dpa_region_idx < 0)
+		return;
+
+	WRITE_ONCE(cxl->region_active, false);
+
+	cxl->dpa_region_idx = -1;
+}
+EXPORT_SYMBOL_GPL(vfio_cxl_unregister_cxl_region);
drivers/vfio/pci/cxl/vfio_cxl_priv.h +2 -0
index 985680842a13..b870926bfb19 100644
@@ -26,9 +26,11 @@ struct vfio_pci_cxl_state {
 	resource_size_t              comp_reg_offset;
 	size_t                       comp_reg_size;
 	u32                          hdm_count;
+	int                          dpa_region_idx;
 	u16                          dvsec;
 	u8                           comp_reg_bar;
 	bool                         precommitted;
+	bool                         region_active;
 };
 
 /*
drivers/vfio/pci/vfio_pci.c +9 -0
index 0c771064c0b8..d3138badeaa6 100644
@@ -120,6 +120,15 @@ static int vfio_pci_open_device(struct vfio_device *core_vdev)
 		}
 	}
 
+	if (vdev->cxl) {
+		ret = vfio_cxl_register_cxl_region(vdev);
+		if (ret) {
+			pci_warn(pdev, "Failed to setup CXL region\n");
+			vfio_pci_core_disable(vdev);
+			return ret;
+		}
+	}
+
 	vfio_pci_core_finish_enable(vdev);
 
 	return 0;
drivers/vfio/pci/vfio_pci_core.c +11 -0
index b7364178e23d..48e0274c19aa 100644
@@ -1223,6 +1223,9 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 
 	vfio_pci_zap_and_down_write_memory_lock(vdev);
 
+	/* Zap CXL DPA region PTEs before hardware reset clears HDM state */
+	vfio_cxl_zap_region_locked(vdev);
+
 	/*
 	 * This function can be invoked while the power state is non-D0. If
 	 * pci_try_reset_function() has been called while the power state is
@@ -1236,6 +1239,14 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 
 	vfio_pci_dma_buf_move(vdev, true);
 	ret = pci_try_reset_function(vdev->pdev);
+
+	/*
+	 * Re-enable DPA region if reset succeeded; fault handler will
+	 * re-insert PFNs on next access without requiring a new mmap.
+	 */
+	if (!ret)
+		vfio_cxl_reactivate_region(vdev);
+
 	if (__vfio_pci_memory_enabled(vdev))
 		vfio_pci_dma_buf_move(vdev, false);
 	up_write(&vdev->memory_lock);
drivers/vfio/pci/vfio_pci_priv.h +13 -0
index 818d99f098bf..441b4a47637a 100644
@@ -140,6 +140,10 @@ void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev);
 int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev,
 			       resource_size_t size);
 void vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev);
+int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev);
+void vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev);
+void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev);
+void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev);
 
 #else
 
@@ -152,6 +156,15 @@ static inline int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev,
 { return 0; }
 static inline void
 vfio_cxl_destroy_cxl_region(struct vfio_pci_core_device *vdev) { }
+static inline int
+vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev)
+{ return 0; }
+static inline void
+vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev) { }
+static inline void
+vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) { }
+static inline void
+vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) { }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
 
Signature · 1 line
2.25.1
maintainer
#33 lore raw

Re: [PATCH 11/20] vfio/cxl: Expose DPA memory region to userspace with fault+zap mmap

Quoted 1 line · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

< --snip-- >

drivers/vfio/pci/cxl/vfio_cxl_core.c void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) · line 465

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +13 -0
+int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	u32 flags;
+	int ret;
+
+	if (!cxl)
+		return -ENODEV;
+
+	if (!cxl->region || cxl->region_vaddr)
+		return -ENODEV;
+
+	cxl->region_vaddr = ioremap_cache(cxl->region_hpa, cxl->region_size);

Should this be using memremap_pages() family of call rather than ioremap() like how DAX does it? CXL mem regions are not MMIO regions.

DJ

#34 lore raw

RE: [PATCH 11/20] vfio/cxl: Expose DPA memory region to userspace with fault+zap mmap

Quoted 44 lines

-----Original Message-----
From: Dave Jiang <[email protected]>
Sent: 13 March 2026 22:38
To: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]
Cc: Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 11/20] vfio/cxl: Expose DPA memory region to userspace with fault+zap mmap

External email: Use caution opening links or attachments

Quoted 1 line · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

< --snip-- >

drivers/vfio/pci/cxl/vfio_cxl_core.c void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +13 -0
+int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev) {
+     struct vfio_pci_cxl_state *cxl = vdev->cxl;
+     u32 flags;
+     int ret;
+
+     if (!cxl)
+             return -ENODEV;
+
+     if (!cxl->region || cxl->region_vaddr)
+             return -ENODEV;
+
+     cxl->region_vaddr = ioremap_cache(cxl->region_hpa,
+ cxl->region_size);

Should this be using memremap_pages() family of call rather than ioremap() like how DAX does it? CXL mem regions are not MMIO regions.

Quoted 1 line

DJ

Okay, I will check the DAX code to update this part.

#35 lore raw

PATCH 12/20 vfio/pci: Export config access helpers

In reply to [email protected]

From: Manish Honap <[email protected]>

Promote vfio_raw_config_write() and vfio_raw_config_read() to non-static so that the CXL DVSEC write handler in the next patch can call them.

---
 drivers/vfio/pci/vfio_pci_config.c | 12 ++++++------
 drivers/vfio/pci/vfio_pci_priv.h   |  8 ++++++++
 2 files changed, 14 insertions(+), 6 deletions(-)
drivers/vfio/pci/vfio_pci_config.c +6 -6
index dc4e510e6e1b..79aaf270adb2 100644
@@ -270,9 +270,9 @@ static int vfio_direct_config_read(struct vfio_pci_core_device *vdev, int pos,
 }
 
 /* Raw access skips any kind of virtualization */
-static int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos,
-				 int count, struct perm_bits *perm,
-				 int offset, __le32 val)
+int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos,
+			  int count, struct perm_bits *perm,
+			  int offset, __le32 val)
 {
 	int ret;
 
@@ -283,9 +283,9 @@ static int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos,
 	return count;
 }
 
-static int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos,
-				int count, struct perm_bits *perm,
-				int offset, __le32 *val)
+int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos,
+			 int count, struct perm_bits *perm,
+			 int offset, __le32 *val)
 {
 	int ret;
 
drivers/vfio/pci/vfio_pci_priv.h +8 -0
index 441b4a47637a..8f440f9eaa0c 100644
@@ -37,6 +37,14 @@ int vfio_pci_set_irqs_ioctl(struct vfio_pci_core_device *vdev, uint32_t flags,
 ssize_t vfio_pci_config_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			   size_t count, loff_t *ppos, bool iswrite);
 
+int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos,
+			  int count, struct perm_bits *perm,
+			  int offset, __le32 val);
+
+int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos,
+			 int count, struct perm_bits *perm,
+			 int offset, __le32 *val);
+
 ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite);
 
Signature · 1 line
2.25.1
#36 lore raw

PATCH 13/20 vfio/cxl: Introduce HDM decoder register emulation framework

In reply to [email protected]

From: Manish Honap <[email protected]>

Introduce an emulation framework to handle CXL MMIO register emulation for CXL devices passed through to a VM.

A single compact __le32 array (comp_reg_virt) covers only the HDM decoder register block (hdm_reg_size bytes, typically 256-512 bytes).

A new VFIO device region VFIO_REGION_SUBTYPE_CXL_COMP_REGS exposes this array to userspace (QEMU) as a read-write region:
- Reads return the emulated state (comp_reg_virt[])
- Writes go through the HDM register write handlers and are forwarded to hardware where appropriate

QEMU attaches a notify_change callback to this region. When the COMMIT bit is written in a decoder CTRL register the callback reads the BASE_LO/HI from the same region fd (emulated state) and maps the DPA MemoryRegion at the correct GPA in system_memory.

---
 drivers/vfio/pci/Makefile            |   2 +-
 drivers/vfio/pci/cxl/vfio_cxl_core.c |  36 ++-
 drivers/vfio/pci/cxl/vfio_cxl_emu.c  | 366 +++++++++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |  41 +++
 drivers/vfio/pci/vfio_pci_priv.h     |   7 +
 5 files changed, 450 insertions(+), 2 deletions(-)
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_emu.c
drivers/vfio/pci/Makefile +1 -1
index ecb0eacbc089..bef916495eae 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
-vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o
+vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
drivers/vfio/pci/cxl/vfio_cxl_core.c +35 -1
index 03846bd11c8a..d2401871489d 100644
@@ -45,6 +45,7 @@ static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
 	cxl = vdev->cxl;
 	cxl->dvsec = dvsec;
 	cxl->dpa_region_idx = -1;
+	cxl->comp_reg_region_idx = -1;
 
 	pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
 			     &cap_word);
@@ -124,6 +125,10 @@ static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)
 	cxl->comp_reg_offset = bar_offset;
 	cxl->comp_reg_size = CXL_COMPONENT_REG_BLOCK_SIZE;
 
+	ret = vfio_cxl_setup_virt_regs(vdev);
+	if (ret)
+		return ret;
+
 	return 0;
 }
 
@@ -281,12 +286,14 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 
 	ret = vfio_cxl_create_region_helper(vdev, SZ_256M);
 	if (ret)
-		goto failed;
+		goto regs_failed;
 
 	cxl->precommitted = true;
 
 	return;
 
+regs_failed:
+	vfio_cxl_clean_virt_regs(vdev);
 failed:
 	devm_kfree(&pdev->dev, vdev->cxl);
 	vdev->cxl = NULL;
@@ -299,6 +306,7 @@ void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
 	if (!cxl || !cxl->region)
 		return;
 
+	vfio_cxl_clean_virt_regs(vdev);
 	vfio_cxl_destroy_cxl_region(vdev);
 }
 
@@ -409,6 +417,32 @@ void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev)
 
 	if (!cxl)
 		return;
+
+	/*
+	 * Re-initialise the emulated HDM comp_reg_virt[] from hardware.
+	 * After FLR the decoder registers read as zero; mirror that in
+	 * the emulated state so QEMU sees a clean slate.
+	 */
+	vfio_cxl_reinit_comp_regs(vdev);
+
+	/*
+	 * Only re-enable the DPA mmap if the hardware has actually
+	 * re-committed decoder 0 after FLR.  Read the COMMITTED bit from the
+	 * freshly-re-snapshotted comp_reg_virt[] so we check the post-FLR
+	 * hardware state, not stale pre-reset state.
+	 *
+	 * If COMMITTED is 0 (slow firmware re-commit path), leave
+	 * region_active=false.  Guest faults will return VM_FAULT_SIGBUS
+	 * until the decoder is re-committed and the region is re-enabled.
+	 */
+	if (cxl->precommitted && cxl->comp_reg_virt) {
+		u32 ctrl = le32_to_cpu(cxl->comp_reg_virt[
+				       CXL_HDM_DECODER0_CTRL_OFFSET(0) /
+				       CXL_REG_SIZE_DWORD]);
+
+		if (ctrl & CXL_HDM_DECODER_CTRL_COMMITTED_BIT)
+			WRITE_ONCE(cxl->region_active, true);
+	}
 }
 
 static ssize_t vfio_cxl_region_rw(struct vfio_pci_core_device *core_dev,
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +366 -0

366 lines not on this page · raw message

drivers/vfio/pci/cxl/vfio_cxl_priv.h +41 -0

59 lines not on this page · raw message

drivers/vfio/pci/vfio_pci_priv.h +7 -0
index 8f440f9eaa0c..f8db9a05c033 100644
@@ -152,6 +152,8 @@ int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev);
 void vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev);
 void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev);
+int  vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev);
+void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
 
 #else
 
@@ -173,6 +175,11 @@ static inline void
 vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) { }
 static inline void
 vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) { }
+static inline int
+vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev)
+{ return 0; }
+static inline void
+vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev) { }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
 
Signature · 1 line
2.25.1
maintainer
#37 lore raw

Re: [PATCH 13/20] vfio/cxl: Introduce HDM decoder register emulation framework

Quoted 184 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Introduce an emulation framework to handle CXL MMIO register emulation for CXL devices passed through to a VM.

A single compact __le32 array (comp_reg_virt) covers only the HDM decoder register block (hdm_reg_size bytes, typically 256-512 bytes).

A new VFIO device region VFIO_REGION_SUBTYPE_CXL_COMP_REGS exposes this array to userspace (QEMU) as a read-write region:
- Reads return the emulated state (comp_reg_virt[])
- Writes go through the HDM register write handlers and are forwarded to hardware where appropriate

QEMU attaches a notify_change callback to this region. When the COMMIT bit is written in a decoder CTRL register the callback reads the BASE_LO/HI from the same region fd (emulated state) and maps the DPA MemoryRegion at the correct GPA in system_memory.

---
 drivers/vfio/pci/Makefile            |   2 +-
 drivers/vfio/pci/cxl/vfio_cxl_core.c |  36 ++-
 drivers/vfio/pci/cxl/vfio_cxl_emu.c  | 366 +++++++++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |  41 +++
 drivers/vfio/pci/vfio_pci_priv.h     |   7 +
 5 files changed, 450 insertions(+), 2 deletions(-)
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_emu.c
drivers/vfio/pci/Makefile +1 -1
index ecb0eacbc089..bef916495eae 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
-vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o
+vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
drivers/vfio/pci/cxl/vfio_cxl_core.c +35 -1
index 03846bd11c8a..d2401871489d 100644
@@ -45,6 +45,7 @@ static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
 	cxl = vdev->cxl;
 	cxl->dvsec = dvsec;
 	cxl->dpa_region_idx = -1;
+	cxl->comp_reg_region_idx = -1;
 
 	pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
 			     &cap_word);
@@ -124,6 +125,10 @@ static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)
 	cxl->comp_reg_offset = bar_offset;
 	cxl->comp_reg_size = CXL_COMPONENT_REG_BLOCK_SIZE;
 
+	ret = vfio_cxl_setup_virt_regs(vdev);
+	if (ret)
+		return ret;
+
 	return 0;
 }
 
@@ -281,12 +286,14 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 
 	ret = vfio_cxl_create_region_helper(vdev, SZ_256M);
 	if (ret)
-		goto failed;
+		goto regs_failed;
 
 	cxl->precommitted = true;
 
 	return;
 
+regs_failed:
+	vfio_cxl_clean_virt_regs(vdev);
 failed:
 	devm_kfree(&pdev->dev, vdev->cxl);
 	vdev->cxl = NULL;
@@ -299,6 +306,7 @@ void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev)
 	if (!cxl || !cxl->region)
 		return;
 
+	vfio_cxl_clean_virt_regs(vdev);
 	vfio_cxl_destroy_cxl_region(vdev);
 }
 
@@ -409,6 +417,32 @@ void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev)
 
 	if (!cxl)
 		return;
+
+	/*
+	 * Re-initialise the emulated HDM comp_reg_virt[] from hardware.
+	 * After FLR the decoder registers read as zero; mirror that in
+	 * the emulated state so QEMU sees a clean slate.
+	 */
+	vfio_cxl_reinit_comp_regs(vdev);
+
+	/*
+	 * Only re-enable the DPA mmap if the hardware has actually
+	 * re-committed decoder 0 after FLR.  Read the COMMITTED bit from the
+	 * freshly-re-snapshotted comp_reg_virt[] so we check the post-FLR
+	 * hardware state, not stale pre-reset state.
+	 *
+	 * If COMMITTED is 0 (slow firmware re-commit path), leave
+	 * region_active=false.  Guest faults will return VM_FAULT_SIGBUS
+	 * until the decoder is re-committed and the region is re-enabled.
+	 */
+	if (cxl->precommitted && cxl->comp_reg_virt) {
+		u32 ctrl = le32_to_cpu(cxl->comp_reg_virt[
+				       CXL_HDM_DECODER0_CTRL_OFFSET(0) /
+				       CXL_REG_SIZE_DWORD]);
+
+		if (ctrl & CXL_HDM_DECODER_CTRL_COMMITTED_BIT)
+			WRITE_ONCE(cxl->region_active, true);
+	}
 }
 
 static ssize_t vfio_cxl_region_rw(struct vfio_pci_core_device *core_dev,
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +54 -0
new file mode 100644
index 000000000000..d5603c80fe51
@@ -0,0 +1,366 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#include <linux/bitops.h>
+#include <linux/vfio_pci_core.h>
+
+#include "../vfio_pci_priv.h"
+#include "vfio_cxl_priv.h"
+
+/*
+ * comp_reg_virt[] layout:
+ *   Index 0..N correspond to 32-bit registers at byte offset 0..hdm_reg_size-4
+ *   within the HDM decoder capability block.
+ *
+ * Register layout within the HDM block (CXL spec 8.2.5.19):
+ *   0x00: HDM Decoder Capability
+ *   0x04: HDM Decoder Global Control
+ *   0x08: HDM Decoder Global Status
+ *   0x0c: (reserved)
+ *   For each decoder N (N=0..hdm_count-1), at base 0x10 + N*0x20:
+ *     +0x00: BASE_LO
+ *     +0x04: BASE_HI
+ *     +0x08: SIZE_LO
+ *     +0x0c: SIZE_HI
+ *     +0x10: CTRL
+ *     +0x14: TARGET_LIST_LO
+ *     +0x18: TARGET_LIST_HI
+ *     +0x1c: (reserved)
+ */
+
+static inline __le32 *hdm_reg_ptr(struct vfio_pci_cxl_state *cxl, u32 off)
+{
+	/*
+	 * off is byte offset within the HDM block; comp_reg_virt is indexed
+	 * as an array of __le32.
+	 */
+	return &cxl->comp_reg_virt[off / sizeof(__le32)];
+}
+
+static ssize_t virt_hdm_rev_reg_write(struct vfio_pci_core_device *vdev,
+				      const __le32 *val32, u64 offset, u64 size)
+{
+	/* Discard writes on reserved registers. */
+	return size;
+}
+
+static ssize_t hdm_decoder_n_lo_write(struct vfio_pci_core_device *vdev,
+				      const __le32 *val32, u64 offset, u64 size)
+{> +	u32 new_val = le32_to_cpu(*val32);
+
+	if (WARN_ON_ONCE(size != CXL_REG_SIZE_DWORD))
+		return -EINVAL;

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 64

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+> +	/* Bit [27:0] are reserved. */
+	new_val &= ~CXL_HDM_DECODER_BASE_LO_RESERVED_MASK;
+
+	*hdm_reg_ptr(vdev->cxl, offset) = cpu_to_le32(new_val);
+
+	return size;
+}
+
+static ssize_t hdm_decoder_global_ctrl_write(struct vfio_pci_core_device *vdev,
+					     const __le32 *val32, u64 offset, u64 size)

Why offset? If the dispatch function already checked and confirmed this is the offset for the global ctrl register then there's no need to pass in the offset.

Quoted 11 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +11 -0
+{
+	u32 hdm_decoder_global_cap;
+	u32 new_val = le32_to_cpu(*val32);
+
+	if (WARN_ON_ONCE(size != CXL_REG_SIZE_DWORD))
+		return -EINVAL;
+> +	/* Bit [31:2] are reserved. */
+	new_val &= ~CXL_HDM_DECODER_GLOBAL_CTRL_RESERVED_MASK;
+
+	/* Poison On Decode Error Enable bit is 0 and RO if not support. */
+	hdm_decoder_global_cap = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, 0));

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 88

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+	if (!(hdm_decoder_global_cap & CXL_HDM_CAP_POISON_ON_DECODE_ERR_BIT))
+		new_val &= ~CXL_HDM_DECODER_GLOBAL_CTRL_POISON_EN_BIT;
+
+	*hdm_reg_ptr(vdev->cxl, offset) = cpu_to_le32(new_val);
+
+	return size;
+}
+
+/*
+ * hdm_decoder_n_ctrl_write - Write handler for HDM decoder CTRL register.

If we are going to start with kdoc style comment, may as well finish the kdoc block and provide parameters and return values

Quoted 69 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +69 -0
+ *
+ * The COMMIT bit (bit 9) is the key: setting it requests the hardware to
+ * lock the decoder.  The emulated COMMITTED bit (bit 10) mirrors COMMIT
+ * immediately to allow QEMU's notify_change to detect the transition and
+ * map/unmap the DPA MemoryRegion in the guest address space.
+ *
+ * Note: the actual hardware HDM decoder programming (writing the real
+ * BASE/SIZE with host physical addresses) happens in the QEMU notify_change
+ * callback BEFORE this write reaches the hardware.  This ordering is
+ * correct because vfio_region_write() calls notify_change() first.
+ */
+static ssize_t hdm_decoder_n_ctrl_write(struct vfio_pci_core_device *vdev,
+					const __le32 *val32, u64 offset, u64 size)
+{
+	u32 hdm_decoder_global_cap;
+	u32 ro_mask = CXL_HDM_DECODER_CTRL_RO_BITS_MASK;
+	u32 rev_mask = CXL_HDM_DECODER_CTRL_RESERVED_MASK;
+	u32 new_val = le32_to_cpu(*val32);
+	u32 cur_val;
+
+	if (WARN_ON_ONCE(size != CXL_REG_SIZE_DWORD))
+		return -EINVAL;
+
+	cur_val = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, offset));
+	if (cur_val & CXL_HDM_DECODER_CTRL_COMMIT_LOCK_BIT)
+		return size;
+
+	hdm_decoder_global_cap = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, 0));
+	ro_mask |= CXL_HDM_DECODER_CTRL_DEVICE_BITS_RO;
+	rev_mask |= CXL_HDM_DECODER_CTRL_DEVICE_RESERVED;
+	if (!(hdm_decoder_global_cap & CXL_HDM_CAP_UIO_SUPPORTED_BIT))
+		rev_mask |= CXL_HDM_DECODER_CTRL_UIO_RESERVED;
+
+	new_val &= ~rev_mask;
+	cur_val &= ro_mask;
+	new_val = (new_val & ~ro_mask) | cur_val;
+
+	/*
+	 * Mirror COMMIT → COMMITTED immediately in the emulated state.
+	 * QEMU's notify_change (called before this write reaches hardware)
+	 * reads COMMITTED from the region fd to detect commit transitions.
+	 */
+	if (new_val & CXL_HDM_DECODER_CTRL_COMMIT_BIT)
+		new_val |= CXL_HDM_DECODER_CTRL_COMMITTED_BIT;
+	else
+		new_val &= ~CXL_HDM_DECODER_CTRL_COMMITTED_BIT;
+
+	*hdm_reg_ptr(vdev->cxl, offset) = cpu_to_le32(new_val);
+
+	return size;
+}
+
+/*
+ * Dispatch table for COMP_REGS region writes.	Indexed by byte offset within
+ * the HDM decoder block.  Returns the appropriate write handler.
+ *
+ * Layout:
+ *   0x00	  HDM Decoder Capability  (RO)
+ *   0x04	  HDM Global Control	  (RW with reserved masking)
+ *   0x08	  HDM Global Status	  (RO)
+ *   0x0c	  (reserved)		  (ignored)
+ *   Per decoder N, base = 0x10 + N*0x20:
+ *     base+0x00  BASE_LO  (RW, [27:0] reserved)
+ *     base+0x04  BASE_HI  (RW)
+ *     base+0x08  SIZE_LO  (RW, [27:0] reserved)
+ *     base+0x0c  SIZE_HI  (RW)
+ *     base+0x10  CTRL	   (RW, complex rules)
+ *     base+0x14  TARGET_LIST_LO  (ignored for Type-2)
+ *     base+0x18  TARGET_LIST_HI  (ignored for Type-2)

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 167

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+ *     base+0x1c  (reserved)	 (ignored)
+ */
+static ssize_t comp_regs_dispatch_write(struct vfio_pci_core_device *vdev,
+					u32 off, const __le32 *val32, u32 size)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	u32 dec_base, dec_off;
+
+	/* HDM Decoder Capability (0x00): RO */
+	if (off == 0x00)

define magic number

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 175

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +8 -0
+		return size;
+
+	/* HDM Global Control (0x04) */
+	if (off == CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET)
+		return hdm_decoder_global_ctrl_write(vdev, val32, off, size);
+
+	/* HDM Global Status (0x08): RO */
+	if (off == 0x08)

define magic number

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 183

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +8 -0
+		return size;
+
+	/* Per-decoder registers start at 0x10, stride 0x20 */
+	if (off < CXL_HDM_DECODER_FIRST_BLOCK_OFFSET)
+		return size; /* reserved gap */
+
+	dec_base = CXL_HDM_DECODER_FIRST_BLOCK_OFFSET;
+	dec_off	 = (off - dec_base) % CXL_HDM_DECODER_BLOCK_STRIDE;

Need a check here to make sure offset is within the number of supported decoders.

Quoted 225 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +183 -0
+
+	switch (dec_off) {
+	case CXL_HDM_DECODER_N_BASE_LOW_OFFSET:	 /* BASE_LO */
+	case CXL_HDM_DECODER_N_SIZE_LOW_OFFSET:	 /* SIZE_LO */
+		return hdm_decoder_n_lo_write(vdev, val32, off, size);
+	case CXL_HDM_DECODER_N_BASE_HIGH_OFFSET: /* BASE_HI */
+	case CXL_HDM_DECODER_N_SIZE_HIGH_OFFSET: /* SIZE_HI */
+		/* Full 32-bit write, no reserved bits */
+		*hdm_reg_ptr(cxl, off) = *val32;
+		return size;
+	case CXL_HDM_DECODER_N_CTRL_OFFSET:	  /* CTRL */
+		return hdm_decoder_n_ctrl_write(vdev, val32, off, size);
+	case CXL_HDM_DECODER_N_TARGET_LIST_LOW_OFFSET:
+	case CXL_HDM_DECODER_N_TARGET_LIST_HIGH_OFFSET:
+	case CXL_HDM_DECODER_N_REV_OFFSET:
+		return virt_hdm_rev_reg_write(vdev, val32, off, size);
+	default:
+		return size;
+	}
+}
+
+/*
+ * vfio_cxl_comp_regs_rw - regops rw handler for VFIO_REGION_SUBTYPE_CXL_COMP_REGS.
+ *
+ * Reads return the emulated HDM state (comp_reg_virt[]).
+ * Writes go through comp_regs_dispatch_write() for bit-field enforcement.
+ * Only 4-byte aligned 4-byte accesses are supported (hardware requirement).
+ */
+static ssize_t vfio_cxl_comp_regs_rw(struct vfio_pci_core_device *vdev,
+				     char __user *buf, size_t count,
+				     loff_t *ppos, bool iswrite)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	size_t done = 0;
+
+	if (!count)
+		return 0;
+
+	/* Clamp to region size */
+	if (pos >= cxl->hdm_reg_size)
+		return -EINVAL;
+	count = min(count, (size_t)(cxl->hdm_reg_size - pos));
+
+	while (done < count) {
+		u32 sz	 = min_t(u32, CXL_REG_SIZE_DWORD, count - done);
+		u32 off	 = pos + done;
+		__le32 v;
+
+		/* Enforce 4-byte alignment */
+		if (sz < CXL_REG_SIZE_DWORD || (off & 0x3))
+			return done ? (ssize_t)done : -EINVAL;
+
+		if (iswrite) {
+			if (copy_from_user(&v, buf + done, sizeof(v)))
+				return done ? (ssize_t)done : -EFAULT;
+			comp_regs_dispatch_write(vdev, off, &v, sizeof(v));
+		} else {
+			v = *hdm_reg_ptr(cxl, off);
+			if (copy_to_user(buf + done, &v, sizeof(v)))
+				return done ? (ssize_t)done : -EFAULT;
+		}
+		done += sizeof(v);
+	}
+
+	*ppos += done;
+	return done;
+}
+
+static void vfio_cxl_comp_regs_release(struct vfio_pci_core_device *vdev,
+				       struct vfio_pci_region *region)
+{
+	/* comp_reg_virt is freed in vfio_cxl_clean_virt_regs(), not here. */
+}
+
+static const struct vfio_pci_regops vfio_cxl_comp_regs_ops = {
+	.rw	 = vfio_cxl_comp_regs_rw,
+	.release = vfio_cxl_comp_regs_release,
+};
+
+/*
+ * vfio_cxl_setup_virt_regs - Allocate emulated HDM register state.
+ *
+ * Allocates comp_reg_virt as a compact __le32 array covering only
+ * hdm_reg_size bytes of HDM decoder registers. The initial values
+ * are read from hardware via the BAR ioremap established by the caller.
+ *
+ * DVSEC state is accessed via vdev->vconfig (see the following patch).
+ */
+int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	size_t nregs;
+
+	if (WARN_ON(!cxl->hdm_reg_size))
+		return -EINVAL;
+
+	if (pci_resource_len(vdev->pdev, cxl->comp_reg_bar) <
+	    cxl->comp_reg_offset + cxl->hdm_reg_offset + cxl->hdm_reg_size)
+		return -ENODEV;
+
+	nregs = cxl->hdm_reg_size / sizeof(__le32);
+	cxl->comp_reg_virt = kcalloc(nregs, sizeof(__le32), GFP_KERNEL);
+	if (!cxl->comp_reg_virt)
+		return -ENOMEM;
+
+	/* Establish persistent mapping; kept alive until vfio_cxl_clean_virt_regs(). */
+	cxl->hdm_iobase = ioremap(pci_resource_start(vdev->pdev, cxl->comp_reg_bar) +
+				  cxl->comp_reg_offset + cxl->hdm_reg_offset,
+				  cxl->hdm_reg_size);
+	if (!cxl->hdm_iobase) {
+		kfree(cxl->comp_reg_virt);
+		cxl->comp_reg_virt = NULL;
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
+/*
+ * Called with memory_lock write side held (from vfio_cxl_reactivate_region).
+ * Uses the pre-established hdm_iobase, no ioremap() under the lock,
+ * which would deadlock on PREEMPT_RT where ioremap() can sleep.
+ */
+void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	size_t i, nregs;
+
+	if (!cxl || !cxl->comp_reg_virt || !cxl->hdm_iobase)
+		return;
+
+	nregs = cxl->hdm_reg_size / sizeof(__le32);
+
+	for (i = 0; i < nregs; i++)
+		cxl->comp_reg_virt[i] =
+			cpu_to_le32(readl(cxl->hdm_iobase + i * sizeof(__le32)));
+}
+
+void vfio_cxl_clean_virt_regs(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	if (cxl->hdm_iobase) {
+		iounmap(cxl->hdm_iobase);
+		cxl->hdm_iobase = NULL;
+	}
+	kfree(cxl->comp_reg_virt);
+	cxl->comp_reg_virt = NULL;
+}
+
+/*
+ * vfio_cxl_register_comp_regs_region - Register the COMP_REGS device region.
+ *
+ * Exposes the emulated HDM decoder register state as a VFIO device region
+ * with type VFIO_REGION_SUBTYPE_CXL_COMP_REGS.	 QEMU attaches a
+ * notify_change callback to this region to intercept HDM COMMIT writes
+ * and map the DPA MemoryRegion at the appropriate GPA.
+ *
+ * The region is read+write only (no mmap) to ensure all accesses pass
+ * through comp_regs_dispatch_write() for proper bit-field enforcement.
+ */
+int vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	u32 flags = VFIO_REGION_INFO_FLAG_READ | VFIO_REGION_INFO_FLAG_WRITE;
+	int ret;
+
+	if (!cxl || !cxl->comp_reg_virt)
+		return -ENODEV;
+
+	ret = vfio_pci_core_register_dev_region(vdev,
+						PCI_VENDOR_ID_CXL |
+						VFIO_REGION_TYPE_PCI_VENDOR_TYPE,
+						VFIO_REGION_SUBTYPE_CXL_COMP_REGS,
+						&vfio_cxl_comp_regs_ops,
+						cxl->hdm_reg_size, flags, cxl);
+	if (!ret)
+		cxl->comp_reg_region_idx = vdev->num_regions - 1;
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_cxl_register_comp_regs_region);
drivers/vfio/pci/cxl/vfio_cxl_priv.h +26 -0
index b870926bfb19..4f2637874e9d 100644
@@ -25,14 +25,51 @@ struct vfio_pci_cxl_state {
 	size_t                       hdm_reg_size;
 	resource_size_t              comp_reg_offset;
 	size_t                       comp_reg_size;
+	__le32                      *comp_reg_virt;
+	void __iomem                *hdm_iobase;
 	u32                          hdm_count;
 	int                          dpa_region_idx;
+	int                          comp_reg_region_idx;
 	u16                          dvsec;
 	u8                           comp_reg_bar;
 	bool                         precommitted;
 	bool                         region_active;
 };
 
+/* Register access sizes */
+#define CXL_REG_SIZE_WORD  2
+#define CXL_REG_SIZE_DWORD 4
+
+/* HDM Decoder - register offsets (CXL 2.0 8.2.5.19) */
+#define CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET	  0x4
+#define CXL_HDM_DECODER_FIRST_BLOCK_OFFSET	  0x10
+#define CXL_HDM_DECODER_BLOCK_STRIDE		  0x20
+#define CXL_HDM_DECODER_N_BASE_LOW_OFFSET	  0x0
+#define CXL_HDM_DECODER_N_BASE_HIGH_OFFSET	  0x4
+#define CXL_HDM_DECODER_N_SIZE_LOW_OFFSET	  0x8
+#define CXL_HDM_DECODER_N_SIZE_HIGH_OFFSET	  0xc
+#define CXL_HDM_DECODER_N_CTRL_OFFSET		  0x10
+#define CXL_HDM_DECODER_N_TARGET_LIST_LOW_OFFSET  0x14
+#define CXL_HDM_DECODER_N_TARGET_LIST_HIGH_OFFSET 0x18
+#define CXL_HDM_DECODER_N_REV_OFFSET		  0x1c
+
+/* HDM Decoder Global Capability / Control - bit definitions */
+#define CXL_HDM_CAP_POISON_ON_DECODE_ERR_BIT BIT(10)
+#define CXL_HDM_CAP_UIO_SUPPORTED_BIT	     BIT(13)
+
+/* HDM Decoder N Control */
+#define CXL_HDM_DECODER_CTRL_COMMIT_LOCK_BIT	  BIT(8)

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 71

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +10 -0
+#define CXL_HDM_DECODER_CTRL_COMMIT_BIT		  BIT(9)
+#define CXL_HDM_DECODER_CTRL_COMMITTED_BIT	  BIT(10)
+#define CXL_HDM_DECODER_CTRL_RO_BITS_MASK	  (BIT(10) | BIT(11))
+#define CXL_HDM_DECODER_CTRL_RESERVED_MASK	  (BIT(15) | GENMASK(31, 28))
+#define CXL_HDM_DECODER_CTRL_DEVICE_BITS_RO	  BIT(12)
+#define CXL_HDM_DECODER_CTRL_DEVICE_RESERVED	  (GENMASK(19, 16) | GENMASK(23, 20))
+#define CXL_HDM_DECODER_CTRL_UIO_RESERVED	  (BIT(14) | GENMASK(27, 24))
+#define CXL_HDM_DECODER_BASE_LO_RESERVED_MASK	  GENMASK(27, 0)
+#define CXL_HDM_DECODER_GLOBAL_CTRL_RESERVED_MASK GENMASK(31, 2)
+#define CXL_HDM_DECODER_GLOBAL_CTRL_POISON_EN_BIT BIT(0)

Maybe the reg defines should go in include/cxl/regs.h? Or move shared definitions out of drivers/cxl/.

DJ

Quoted 38 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +5 -0
+
 /*
  * CXL DVSEC for CXL Devices - register offsets within the DVSEC
  * (CXL 2.0+ 8.1.3).
@@ -41,4 +78,8 @@ struct vfio_pci_cxl_state {
 #define CXL_DVSEC_CAPABILITY_OFFSET 0xa
 #define CXL_DVSEC_MEM_CAPABLE	    BIT(2)
 
+int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev);
+void vfio_cxl_clean_virt_regs(struct vfio_pci_core_device *vdev);
+void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
+
 #endif /* __LINUX_VFIO_CXL_PRIV_H */
drivers/vfio/pci/vfio_pci_priv.h +7 -0
index 8f440f9eaa0c..f8db9a05c033 100644
@@ -152,6 +152,8 @@ int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev);
 void vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev);
 void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev);
+int  vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev);
+void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
 
 #else
 
@@ -173,6 +175,11 @@ static inline void
 vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) { }
 static inline void
 vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) { }
+static inline int
+vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev)
+{ return 0; }
+static inline void
+vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev) { }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
 
#38 lore raw

RE: [PATCH 13/20] vfio/cxl: Introduce HDM decoder register emulation framework

Quoted 230 lines

-----Original Message-----
From: Dave Jiang <[email protected]>
Sent: 14 March 2026 00:36
To: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]
Cc: Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 13/20] vfio/cxl: Introduce HDM decoder register emulation framework

External email: Use caution opening links or attachments

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_create_device_state(struct

Quoted 52 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

Introduce an emulation framework to handle CXL MMIO register emulation for CXL devices passed through to a VM.

A single compact __le32 array (comp_reg_virt) covers only the HDM decoder register block (hdm_reg_size bytes, typically 256-512 bytes).

A new VFIO device region VFIO_REGION_SUBTYPE_CXL_COMP_REGS exposes this array to userspace (QEMU) as a read-write region:
- Reads return the emulated state (comp_reg_virt[])
- Writes go through the HDM register write handlers and are forwarded to hardware where appropriate

QEMU attaches a notify_change callback to this region. When the COMMIT bit is written in a decoder CTRL register the callback reads the BASE_LO/HI from the same region fd (emulated state) and maps the DPA MemoryRegion at the correct GPA in system_memory.

---
 drivers/vfio/pci/Makefile            |   2 +-
 drivers/vfio/pci/cxl/vfio_cxl_core.c |  36 ++-
drivers/vfio/pci/cxl/vfio_cxl_emu.c  | 366 +++++++++++++++++++++++++++
drivers/vfio/pci/cxl/vfio_cxl_priv.h |  41 +++
 drivers/vfio/pci/vfio_pci_priv.h     |   7 +
 5 files changed, 450 insertions(+), 2 deletions(-)  create mode

100644 drivers/vfio/pci/cxl/vfio_cxl_emu.c

drivers/vfio/pci/Makefile +0 -0
index ecb0eacbc089..bef916495eae 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o

vfio_pci_config.o

drivers/vfio/pci/Makefile +2 -1
-vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o
+vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o
+cxl/vfio_cxl_emu.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o diff --git

a/drivers/vfio/pci/cxl/vfio_cxl_core.c
b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 03846bd11c8a..d2401871489d 100644

drivers/vfio/pci/cxl/vfio_cxl_core.c +0 -0
@@ -45,6 +45,7 @@ static int vfio_cxl_create_device_state(struct

vfio_pci_core_device *vdev,

drivers/vfio/pci/cxl/vfio_cxl_core.c void vfio_pci_cxl_cleanup(struct

Quoted 36 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -0
      cxl = vdev->cxl;
      cxl->dvsec = dvsec;
      cxl->dpa_region_idx = -1;
+     cxl->comp_reg_region_idx = -1;
 
      pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
                           &cap_word); @@ -124,6 +125,10 @@ static int

vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)

      cxl->comp_reg_offset = bar_offset;
      cxl->comp_reg_size = CXL_COMPONENT_REG_BLOCK_SIZE;
drivers/vfio/pci/cxl/vfio_cxl_core.c +7 -1
+     ret = vfio_cxl_setup_virt_regs(vdev);
+     if (ret)
+             return ret;
+
      return 0;
 }
@@ -281,12 +286,14 @@ void vfio_pci_cxl_detect_and_init(struct
 vfio_pci_core_device *vdev)
 
      ret = vfio_cxl_create_region_helper(vdev, SZ_256M);
      if (ret)
-             goto failed;
+             goto regs_failed;
 
      cxl->precommitted = true;
 
      return;
 
+regs_failed:
+     vfio_cxl_clean_virt_regs(vdev);
 failed:
      devm_kfree(&pdev->dev, vdev->cxl);
      vdev->cxl = NULL;
drivers/vfio/pci/cxl/vfio_cxl_core.c +0 -0
@@ -299,6 +306,7 @@ void vfio_pci_cxl_cleanup(struct

vfio_pci_core_device *vdev)

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +2 -0
      if (!cxl || !cxl->region)
              return;
 
+     vfio_cxl_clean_virt_regs(vdev);
      vfio_cxl_destroy_cxl_region(vdev);
 }
 
@@ -409,6 +417,32 @@ void vfio_cxl_reactivate_region(struct
 vfio_pci_core_device *vdev)
 
      if (!cxl)
              return;
+

drivers/vfio/pci/cxl/vfio_cxl_core.c void vfio_cxl_reactivate_region(struct · line 431

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +10 -0
+     /*
+      * Re-initialise the emulated HDM comp_reg_virt[] from hardware.
+      * After FLR the decoder registers read as zero; mirror that in
+      * the emulated state so QEMU sees a clean slate.
+      */
+     vfio_cxl_reinit_comp_regs(vdev);
+
+     /*
+      * Only re-enable the DPA mmap if the hardware has actually
+      * re-committed decoder 0 after FLR.  Read the COMMITTED bit from

the

Quoted 29 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +15 -0
+      * freshly-re-snapshotted comp_reg_virt[] so we check the post-FLR
+      * hardware state, not stale pre-reset state.
+      *
+      * If COMMITTED is 0 (slow firmware re-commit path), leave
+      * region_active=false.  Guest faults will return VM_FAULT_SIGBUS
+      * until the decoder is re-committed and the region is re-enabled.
+      */
+     if (cxl->precommitted && cxl->comp_reg_virt) {
+             u32 ctrl = le32_to_cpu(cxl->comp_reg_virt[
+                                    CXL_HDM_DECODER0_CTRL_OFFSET(0) /
+                                    CXL_REG_SIZE_DWORD]);
+
+             if (ctrl & CXL_HDM_DECODER_CTRL_COMMITTED_BIT)
+                     WRITE_ONCE(cxl->region_active, true);
+     }
 }
 

static ssize_t vfio_cxl_region_rw(struct vfio_pci_core_device
*core_dev, diff --git a/drivers/vfio/pci/cxl/vfio_cxl_emu.c b/drivers/vfio/pci/cxl/vfio_cxl_emu.c
new file mode 100644
index 000000000000..d5603c80fe51

drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +4 -0
@@ -0,0 +1,366 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights
+reserved  */

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 14

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+
+#include <linux/bitops.h>
+#include <linux/vfio_pci_core.h>
+
+#include "../vfio_pci_priv.h"
+#include "vfio_cxl_priv.h"
+
+/*
+ * comp_reg_virt[] layout:
+ *   Index 0..N correspond to 32-bit registers at byte offset

0..hdm_reg_size-4

Quoted 12 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +12 -0
+ *   within the HDM decoder capability block.
+ *
+ * Register layout within the HDM block (CXL spec 8.2.5.19):
+ *   0x00: HDM Decoder Capability
+ *   0x04: HDM Decoder Global Control
+ *   0x08: HDM Decoder Global Status
+ *   0x0c: (reserved)
+ *   For each decoder N (N=0..hdm_count-1), at base 0x10 + N*0x20:
+ *     +0x00: BASE_LO
+ *     +0x04: BASE_HI
+ *     +0x08: SIZE_LO
+ *     +0x0c: SIZE_HI

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+ *     +0x10: CTRL
+ *     +0x14: TARGET_LIST_LO
+ *     +0x18: TARGET_LIST_HI
+ *     +0x1c: (reserved)
+ */
+
+static inline __le32 *hdm_reg_ptr(struct vfio_pci_cxl_state *cxl, u32
+off) {
+     /*
+      * off is byte offset within the HDM block; comp_reg_virt is

indexed

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +5 -0
+      * as an array of __le32.
+      */
+     return &cxl->comp_reg_virt[off / sizeof(__le32)]; }
+
+static ssize_t virt_hdm_rev_reg_write(struct vfio_pci_core_device

*vdev,

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +7 -0
+                                   const __le32 *val32, u64 offset,
+u64 size) {
+     /* Discard writes on reserved registers. */
+     return size;
+}
+
+static ssize_t hdm_decoder_n_lo_write(struct vfio_pci_core_device

*vdev,

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +4 -0
+                                   const __le32 *val32, u64 offset,
+u64 size) {> + u32 new_val = le32_to_cpu(*val32);
+
+     if (WARN_ON_ONCE(size != CXL_REG_SIZE_DWORD))

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+             return -EINVAL;
+> +  /* Bit [27:0] are reserved. */
+     new_val &= ~CXL_HDM_DECODER_BASE_LO_RESERVED_MASK;
+
+     *hdm_reg_ptr(vdev->cxl, offset) = cpu_to_le32(new_val);
+
+     return size;
+}
+
+static ssize_t hdm_decoder_global_ctrl_write(struct

vfio_pci_core_device *vdev,

Quoted 5 lines

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 64

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +2 -0
+                                          const __le32 *val32, u64
+offset, u64 size)

Why offset? If the dispatch function already checked and confirmed this is the offset for the global ctrl register then there's no need to pass in the offset.

Okay, refactored this as per suggestion.

Quoted 121 lines

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 11 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+{
+     u32 hdm_decoder_global_cap;
+     u32 new_val = le32_to_cpu(*val32);
+
+     if (WARN_ON_ONCE(size != CXL_REG_SIZE_DWORD))
+             return -EINVAL;
+> +  /* Bit [31:2] are reserved. */
+     new_val &= ~CXL_HDM_DECODER_GLOBAL_CTRL_RESERVED_MASK;
+
+     /* Poison On Decode Error Enable bit is 0 and RO if not support.

*/

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 76

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +2 -0
+     hdm_decoder_global_cap = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, 0));
+     if (!(hdm_decoder_global_cap &

CXL_HDM_CAP_POISON_ON_DECODE_ERR_BIT))

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +9 -0
+             new_val &= ~CXL_HDM_DECODER_GLOBAL_CTRL_POISON_EN_BIT;
+
+     *hdm_reg_ptr(vdev->cxl, offset) = cpu_to_le32(new_val);
+
+     return size;
+}
+
+/*
+ * hdm_decoder_n_ctrl_write - Write handler for HDM decoder CTRL

register.

If we are going to start with kdoc style comment, may as well finish the kdoc block and provide parameters and return values

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +7 -0
+ *
+ * The COMMIT bit (bit 9) is the key: setting it requests the
+hardware to
+ * lock the decoder.  The emulated COMMITTED bit (bit 10) mirrors
+COMMIT
+ * immediately to allow QEMU's notify_change to detect the transition
+and

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 102

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+ * map/unmap the DPA MemoryRegion in the guest address space.
+ *
+ * Note: the actual hardware HDM decoder programming (writing the
+real
+ * BASE/SIZE with host physical addresses) happens in the QEMU
+notify_change
+ * callback BEFORE this write reaches the hardware.  This ordering is
+ * correct because vfio_region_write() calls notify_change() first.
+ */
+static ssize_t hdm_decoder_n_ctrl_write(struct vfio_pci_core_device

*vdev,

Quoted 18 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +18 -0
+                                     const __le32 *val32, u64 offset,
+u64 size) {
+     u32 hdm_decoder_global_cap;
+     u32 ro_mask = CXL_HDM_DECODER_CTRL_RO_BITS_MASK;
+     u32 rev_mask = CXL_HDM_DECODER_CTRL_RESERVED_MASK;
+     u32 new_val = le32_to_cpu(*val32);
+     u32 cur_val;
+
+     if (WARN_ON_ONCE(size != CXL_REG_SIZE_DWORD))
+             return -EINVAL;
+
+     cur_val = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, offset));
+     if (cur_val & CXL_HDM_DECODER_CTRL_COMMIT_LOCK_BIT)
+             return size;
+
+     hdm_decoder_global_cap = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, 0));
+     ro_mask |= CXL_HDM_DECODER_CTRL_DEVICE_BITS_RO;
+     rev_mask |= CXL_HDM_DECODER_CTRL_DEVICE_RESERVED;

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+     if (!(hdm_decoder_global_cap & CXL_HDM_CAP_UIO_SUPPORTED_BIT))
+             rev_mask |= CXL_HDM_DECODER_CTRL_UIO_RESERVED;
+
+     new_val &= ~rev_mask;
+     cur_val &= ro_mask;
+     new_val = (new_val & ~ro_mask) | cur_val;
+
+     /*
+      * Mirror COMMIT → COMMITTED immediately in the emulated state.
+      * QEMU's notify_change (called before this write reaches

hardware)

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 131

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +1 -0
+      * reads COMMITTED from the region fd to detect commit

transitions.

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +13 -0
+      */
+     if (new_val & CXL_HDM_DECODER_CTRL_COMMIT_BIT)
+             new_val |= CXL_HDM_DECODER_CTRL_COMMITTED_BIT;
+     else
+             new_val &= ~CXL_HDM_DECODER_CTRL_COMMITTED_BIT;
+
+     *hdm_reg_ptr(vdev->cxl, offset) = cpu_to_le32(new_val);
+
+     return size;
+}
+
+/*
+ * Dispatch table for COMP_REGS region writes.       Indexed by byte

offset within

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +8 -0
+ * the HDM decoder block.  Returns the appropriate write handler.
+ *
+ * Layout:
+ *   0x00      HDM Decoder Capability  (RO)
+ *   0x04      HDM Global Control      (RW with reserved masking)
+ *   0x08      HDM Global Status       (RO)
+ *   0x0c      (reserved)              (ignored)
+ *   Per decoder N, base = 0x10 + N*0x20:

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+ *     base+0x00  BASE_LO  (RW, [27:0] reserved)
+ *     base+0x04  BASE_HI  (RW)
+ *     base+0x08  SIZE_LO  (RW, [27:0] reserved)
+ *     base+0x0c  SIZE_HI  (RW)
+ *     base+0x10  CTRL          (RW, complex rules)
+ *     base+0x14  TARGET_LIST_LO  (ignored for Type-2)
+ *     base+0x18  TARGET_LIST_HI  (ignored for Type-2)
+ *     base+0x1c  (reserved)  (ignored)
+ */
+static ssize_t comp_regs_dispatch_write(struct vfio_pci_core_device

*vdev,

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 167

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +7 -0
+                                     u32 off, const __le32 *val32,
+u32 size) {
+     struct vfio_pci_cxl_state *cxl = vdev->cxl;
+     u32 dec_base, dec_off;
+
+     /* HDM Decoder Capability (0x00): RO */
+     if (off == 0x00)

define magic number

Quoted 11 lines

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 175

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +9 -0
+             return size;
+
+     /* HDM Global Control (0x04) */
+     if (off == CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET)
+             return hdm_decoder_global_ctrl_write(vdev, val32, off,
+ size);
+
+     /* HDM Global Status (0x08): RO */
+     if (off == 0x08)

define magic number

Yes, removed these bare numbers and added macros for this.

Quoted 12 lines

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 183

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +8 -0
+             return size;
+
+     /* Per-decoder registers start at 0x10, stride 0x20 */
+     if (off < CXL_HDM_DECODER_FIRST_BLOCK_OFFSET)
+             return size; /* reserved gap */
+
+     dec_base = CXL_HDM_DECODER_FIRST_BLOCK_OFFSET;
+     dec_off  = (off - dec_base) % CXL_HDM_DECODER_BLOCK_STRIDE;

Need a check here to make sure offset is within the number of supported decoders.

Added a check for this verification.

Quoted 248 lines
Quoted 14 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +13 -0
+
+     switch (dec_off) {
+     case CXL_HDM_DECODER_N_BASE_LOW_OFFSET:  /* BASE_LO */
+     case CXL_HDM_DECODER_N_SIZE_LOW_OFFSET:  /* SIZE_LO */
+             return hdm_decoder_n_lo_write(vdev, val32, off, size);
+     case CXL_HDM_DECODER_N_BASE_HIGH_OFFSET: /* BASE_HI */
+     case CXL_HDM_DECODER_N_SIZE_HIGH_OFFSET: /* SIZE_HI */
+             /* Full 32-bit write, no reserved bits */
+             *hdm_reg_ptr(cxl, off) = *val32;
+             return size;
+     case CXL_HDM_DECODER_N_CTRL_OFFSET:       /* CTRL */
+             return hdm_decoder_n_ctrl_write(vdev, val32, off, size);
+     case CXL_HDM_DECODER_N_TARGET_LIST_LOW_OFFSET:

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+     case CXL_HDM_DECODER_N_TARGET_LIST_HIGH_OFFSET:
+     case CXL_HDM_DECODER_N_REV_OFFSET:
+             return virt_hdm_rev_reg_write(vdev, val32, off, size);
+     default:
+             return size;
+     }
+}
+
+/*
+ * vfio_cxl_comp_regs_rw - regops rw handler for

VFIO_REGION_SUBTYPE_CXL_COMP_REGS.

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +3 -0
+ *
+ * Reads return the emulated HDM state (comp_reg_virt[]).
+ * Writes go through comp_regs_dispatch_write() for bit-field

enforcement.

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 213

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +1 -0
+ * Only 4-byte aligned 4-byte accesses are supported (hardware

requirement).

Quoted 19 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +19 -0
+ */
+static ssize_t vfio_cxl_comp_regs_rw(struct vfio_pci_core_device *vdev,
+                                  char __user *buf, size_t count,
+                                  loff_t *ppos, bool iswrite) {
+     struct vfio_pci_cxl_state *cxl = vdev->cxl;
+     loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+     size_t done = 0;
+
+     if (!count)
+             return 0;
+
+     /* Clamp to region size */
+     if (pos >= cxl->hdm_reg_size)
+             return -EINVAL;
+     count = min(count, (size_t)(cxl->hdm_reg_size - pos));
+
+     while (done < count) {
+             u32 sz   = min_t(u32, CXL_REG_SIZE_DWORD, count - done);
+             u32 off  = pos + done;

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+             __le32 v;
+
+             /* Enforce 4-byte alignment */
+             if (sz < CXL_REG_SIZE_DWORD || (off & 0x3))
+                     return done ? (ssize_t)done : -EINVAL;
+
+             if (iswrite) {
+                     if (copy_from_user(&v, buf + done, sizeof(v)))
+                             return done ? (ssize_t)done : -EFAULT;
+                     comp_regs_dispatch_write(vdev, off, &v,

sizeof(v));

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +13 -0
+             } else {
+                     v = *hdm_reg_ptr(cxl, off);
+                     if (copy_to_user(buf + done, &v, sizeof(v)))
+                             return done ? (ssize_t)done : -EFAULT;
+             }
+             done += sizeof(v);
+     }
+
+     *ppos += done;
+     return done;
+}
+
+static void vfio_cxl_comp_regs_release(struct vfio_pci_core_device

*vdev,

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +4 -0
+                                    struct vfio_pci_region *region) {
+     /* comp_reg_virt is freed in vfio_cxl_clean_virt_regs(), not
+here. */ }
+

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+static const struct vfio_pci_regops vfio_cxl_comp_regs_ops = {
+     .rw      = vfio_cxl_comp_regs_rw,
+     .release = vfio_cxl_comp_regs_release, };
+
+/*
+ * vfio_cxl_setup_virt_regs - Allocate emulated HDM register state.
+ *
+ * Allocates comp_reg_virt as a compact __le32 array covering only
+ * hdm_reg_size bytes of HDM decoder registers. The initial values
+ * are read from hardware via the BAR ioremap established by the

caller.

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +12 -0
+ *
+ * DVSEC state is accessed via vdev->vconfig (see the following patch).
+ */
+int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev) {
+     struct vfio_pci_cxl_state *cxl = vdev->cxl;
+     size_t nregs;
+
+     if (WARN_ON(!cxl->hdm_reg_size))
+             return -EINVAL;
+
+     if (pci_resource_len(vdev->pdev, cxl->comp_reg_bar) <
+         cxl->comp_reg_offset + cxl->hdm_reg_offset + cxl-

hdm_reg_size)

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +8 -0
+             return -ENODEV;
+
+     nregs = cxl->hdm_reg_size / sizeof(__le32);
+     cxl->comp_reg_virt = kcalloc(nregs, sizeof(__le32), GFP_KERNEL);
+     if (!cxl->comp_reg_virt)
+             return -ENOMEM;
+
+     /* Establish persistent mapping; kept alive until

vfio_cxl_clean_virt_regs(). */

Quoted 6 lines

+ cxl->hdm_iobase = ioremap(pci_resource_start(vdev->pdev, cxl-
comp_reg_bar) +
+ cxl->comp_reg_offset + cxl-
hdm_reg_offset,

drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +2 -0
+                               cxl->hdm_reg_size);
+     if (!cxl->hdm_iobase) {

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+             kfree(cxl->comp_reg_virt);
+             cxl->comp_reg_virt = NULL;
+             return -ENOMEM;
+     }
+
+     return 0;
+}
+
+/*
+ * Called with memory_lock write side held (from

vfio_cxl_reactivate_region).

Quoted 20 lines

+ * Uses the pre-established hdm_iobase, no ioremap() under the lock,
+ * which would deadlock on PREEMPT_RT where ioremap() can sleep.
+ */

drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +17 -0
+void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev) {
+     struct vfio_pci_cxl_state *cxl = vdev->cxl;
+     size_t i, nregs;
+
+     if (!cxl || !cxl->comp_reg_virt || !cxl->hdm_iobase)
+             return;
+
+     nregs = cxl->hdm_reg_size / sizeof(__le32);
+
+     for (i = 0; i < nregs; i++)
+             cxl->comp_reg_virt[i] =
+                     cpu_to_le32(readl(cxl->hdm_iobase + i *
+sizeof(__le32))); }
+
+void vfio_cxl_clean_virt_regs(struct vfio_pci_core_device *vdev) {
+     struct vfio_pci_cxl_state *cxl = vdev->cxl;
+

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +10 -0
+     if (cxl->hdm_iobase) {
+             iounmap(cxl->hdm_iobase);
+             cxl->hdm_iobase = NULL;
+     }
+     kfree(cxl->comp_reg_virt);
+     cxl->comp_reg_virt = NULL;
+}
+
+/*
+ * vfio_cxl_register_comp_regs_region - Register the COMP_REGS device

region.

Quoted 2 lines

+ *
+ * Exposes the emulated HDM decoder register state as a VFIO device

region

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 13 lines

+ * with type VFIO_REGION_SUBTYPE_CXL_COMP_REGS. QEMU attaches a
+ * notify_change callback to this region to intercept HDM COMMIT

drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +11 -0
+writes
+ * and map the DPA MemoryRegion at the appropriate GPA.
+ *
+ * The region is read+write only (no mmap) to ensure all accesses
+pass
+ * through comp_regs_dispatch_write() for proper bit-field enforcement.
+ */
+int vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device
+*vdev) {
+     struct vfio_pci_cxl_state *cxl = vdev->cxl;
+     u32 flags = VFIO_REGION_INFO_FLAG_READ |

VFIO_REGION_INFO_FLAG_WRITE;

drivers/vfio/pci/cxl/vfio_cxl_emu.c line 363

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +8 -0
+     int ret;
+
+     if (!cxl || !cxl->comp_reg_virt)
+             return -ENODEV;
+
+     ret = vfio_pci_core_register_dev_region(vdev,
+                                             PCI_VENDOR_ID_CXL |
+

VFIO_REGION_TYPE_PCI_VENDOR_TYPE,

Quoted 1 line

+

VFIO_REGION_SUBTYPE_CXL_COMP_REGS,

drivers/vfio/pci/cxl/vfio_cxl_emu.c

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +2 -0
+                                             &vfio_cxl_comp_regs_ops,
+                                             cxl->hdm_reg_size, flags,

cxl);

Quoted 43 lines
drivers/vfio/pci/cxl/vfio_cxl_emu.c new file +6 -0
+     if (!ret)
+             cxl->comp_reg_region_idx = vdev->num_regions - 1;
+
+     return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_cxl_register_comp_regs_region);
drivers/vfio/pci/cxl/vfio_cxl_priv.h +20 -0
index b870926bfb19..4f2637874e9d 100644
@@ -25,14 +25,51 @@ struct vfio_pci_cxl_state {
      size_t                       hdm_reg_size;
      resource_size_t              comp_reg_offset;
      size_t                       comp_reg_size;
+     __le32                      *comp_reg_virt;
+     void __iomem                *hdm_iobase;
      u32                          hdm_count;
      int                          dpa_region_idx;
+     int                          comp_reg_region_idx;
      u16                          dvsec;
      u8                           comp_reg_bar;
      bool                         precommitted;
      bool                         region_active;
 };
 
+/* Register access sizes */
+#define CXL_REG_SIZE_WORD  2
+#define CXL_REG_SIZE_DWORD 4
+
+/* HDM Decoder - register offsets (CXL 2.0 8.2.5.19) */
+#define CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET     0x4
+#define CXL_HDM_DECODER_FIRST_BLOCK_OFFSET     0x10
+#define CXL_HDM_DECODER_BLOCK_STRIDE           0x20
+#define CXL_HDM_DECODER_N_BASE_LOW_OFFSET      0x0
+#define CXL_HDM_DECODER_N_BASE_HIGH_OFFSET     0x4
+#define CXL_HDM_DECODER_N_SIZE_LOW_OFFSET      0x8
+#define CXL_HDM_DECODER_N_SIZE_HIGH_OFFSET     0xc
+#define CXL_HDM_DECODER_N_CTRL_OFFSET                  0x10
+#define CXL_HDM_DECODER_N_TARGET_LIST_LOW_OFFSET  0x14 #define
+CXL_HDM_DECODER_N_TARGET_LIST_HIGH_OFFSET 0x18
+#define CXL_HDM_DECODER_N_REV_OFFSET           0x1c
+

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 65

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +10 -0
+/* HDM Decoder Global Capability / Control - bit definitions */
+#define CXL_HDM_CAP_POISON_ON_DECODE_ERR_BIT BIT(10)
+#define CXL_HDM_CAP_UIO_SUPPORTED_BIT             BIT(13)
+
+/* HDM Decoder N Control */
+#define CXL_HDM_DECODER_CTRL_COMMIT_LOCK_BIT   BIT(8)
+#define CXL_HDM_DECODER_CTRL_COMMIT_BIT                BIT(9)
+#define CXL_HDM_DECODER_CTRL_COMMITTED_BIT     BIT(10)
+#define CXL_HDM_DECODER_CTRL_RO_BITS_MASK      (BIT(10) | BIT(11))
+#define CXL_HDM_DECODER_CTRL_RESERVED_MASK     (BIT(15) | GENMASK(31,

28))

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 67

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +2 -0
+#define CXL_HDM_DECODER_CTRL_DEVICE_BITS_RO    BIT(12)
+#define CXL_HDM_DECODER_CTRL_DEVICE_RESERVED   (GENMASK(19, 16) |

GENMASK(23, 20))

Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 68

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_priv.h +1 -0
+#define CXL_HDM_DECODER_CTRL_UIO_RESERVED      (BIT(14) | GENMASK(27,

24))

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 71

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +3 -0
+#define CXL_HDM_DECODER_BASE_LO_RESERVED_MASK          GENMASK(27, 0)
+#define CXL_HDM_DECODER_GLOBAL_CTRL_RESERVED_MASK GENMASK(31, 2)
+#define CXL_HDM_DECODER_GLOBAL_CTRL_POISON_EN_BIT BIT(0)

Maybe the reg defines should go in include/cxl/regs.h? Or move shared definitions out of drivers/cxl/.

Added in include/uapi/cxl/cxl_regs.h

Quoted 43 lines

DJ

Quoted 40 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +1 -0
+
 /*
  * CXL DVSEC for CXL Devices - register offsets within the DVSEC
  * (CXL 2.0+ 8.1.3).
@@ -41,4 +78,8 @@ struct vfio_pci_cxl_state {  #define

CXL_DVSEC_CAPABILITY_OFFSET 0xa
#define CXL_DVSEC_MEM_CAPABLE BIT(2)

drivers/vfio/pci/cxl/vfio_cxl_priv.h +4 -0
+int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev); void
+vfio_cxl_clean_virt_regs(struct vfio_pci_core_device *vdev); void
+vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
+
 #endif /* __LINUX_VFIO_CXL_PRIV_H */
drivers/vfio/pci/vfio_pci_priv.h +0 -0
index 8f440f9eaa0c..f8db9a05c033 100644
@@ -152,6 +152,8 @@ int vfio_cxl_register_cxl_region(struct

vfio_pci_core_device *vdev); void vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev); void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev); void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev);

drivers/vfio/pci/vfio_pci_priv.h +3 -0
+int  vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device
+*vdev); void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device
+*vdev);
 
 #else
@@ -173,6 +175,11 @@ static inline void
 vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) { }

static inline void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) { }

drivers/vfio/pci/vfio_pci_priv.h +4 -0
+static inline int
+vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev)
+{ return 0; } static inline void vfio_cxl_reinit_comp_regs(struct
+vfio_pci_core_device *vdev) { }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
#39 lore raw

PATCH 14/20 vfio/cxl: Check media readiness and create CXL memdev

In reply to [email protected]

From: Manish Honap <[email protected]>

Check media readiness at probe time and create a CXL memdev for region management.

Media/range-active check is performed at probe time to keep the vfio-is-advertised-as-cxl behavior consistent. A pre-committed HDM decoder already implies media is active, so set media_ready directly instead of calling the potentially blocking cxl_await_range_active().

For memdev creation we need to determine capacity before calling devm_cxl_add_memdev(). Read the committed decoder size directly from HDM decoder hardware registers; the CXL core will see the same values when it enumerates decoders inside add_memdev.

For firmware uncommitted decoders, handling will be added in a later commit.

---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 67 +++++++++++++++++++++++++++-
 drivers/vfio/pci/cxl/vfio_cxl_emu.c  | 48 ++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_priv.h |  2 +
 3 files changed, 115 insertions(+), 2 deletions(-)
drivers/vfio/pci/cxl/vfio_cxl_core.c +65 -2
index d2401871489d..15b6c0d75d9e 100644
@@ -132,6 +132,37 @@ static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev)
 	return 0;
 }
 
+static int vfio_cxl_create_memdev(struct vfio_pci_core_device *vdev,
+				  resource_size_t capacity)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	int ret;
+
+	ret = cxl_set_capacity(&cxl->cxlds, capacity);
+	if (ret) {
+		pci_err(pdev, "Failed to set capacity: %d\n", ret);
+		return ret;
+	}
+
+	pci_dbg(pdev, "Device capacity: %llu MB (from %s)\n",
+		capacity >> 20,
+		cxl->precommitted ? "committed decoder" : "sysfs");
+	pci_dbg(pdev,
+		"vfio_cxl: creating memdev: capacity=0x%llx bytes (%llu MiB)\n",
+		(unsigned long long)capacity,
+		(unsigned long long)(capacity >> 20));
+
+	cxl->cxlmd = devm_cxl_add_memdev(&cxl->cxlds, NULL);
+	if (IS_ERR(cxl->cxlmd)) {
+		pci_err(pdev, "Failed to add CXL memdev: %ld\n",
+			PTR_ERR(cxl->cxlmd));
+		return PTR_ERR(cxl->cxlmd);
+	}
+
+	return 0;
+}
+
 int vfio_cxl_create_cxl_region(struct vfio_pci_core_device *vdev, resource_size_t size)
 {
 	struct vfio_pci_cxl_state *cxl = vdev->cxl;
@@ -250,6 +281,7 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 {
 	struct pci_dev *pdev = vdev->pdev;
 	struct vfio_pci_cxl_state *cxl;
+	resource_size_t capacity = 0;
 	u16 dvsec;
 	int ret;
 
@@ -282,13 +314,44 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 		goto failed;
 	}
 
+	cxl->cxlds.media_ready = !cxl_await_range_active(&cxl->cxlds);
+	if (!cxl->cxlds.media_ready) {
+		pci_disable_device(pdev);
+		pci_err(pdev, "CXL media not ready\n");
+		goto regs_failed;
+	}
+
+	/*
+	 * Take the single authoritative HDM decoder snapshot now that
+	 * MEM_ACTIVE is confirmed and BAR memory is still enabled.  Using
+	 * readl() per-dword ensures correct MMIO serialisation and captures
+	 * the final firmware-written values for all fields including SIZE_HIGH,
+	 * which firmware commits to the BAR at MEM_ACTIVE time.
+	 */
+	vfio_cxl_reinit_comp_regs(vdev);
+
 	pci_disable_device(pdev);
 
-	ret = vfio_cxl_create_region_helper(vdev, SZ_256M);
-	if (ret)
+	capacity = vfio_cxl_read_committed_decoder_size(vdev);
+	if (capacity == 0) {
+		/*
+		 * TODO: Add handling for devices which do not have
+		 * firmware pre-committed decoders
+		 */
+		pci_info(pdev, "Uncommitted region size must be configured via sysfs before bind\n");
 		goto regs_failed;
+	}
 
 	cxl->precommitted = true;
+	cxl->dpa_size = capacity;
+
+	ret = vfio_cxl_create_memdev(vdev, capacity);
+	if (ret)
+		goto regs_failed;
+
+	ret = vfio_cxl_create_region_helper(vdev, capacity);
+	if (ret)
+		goto regs_failed;
 
 	return;
 
drivers/vfio/pci/cxl/vfio_cxl_emu.c +48 -0

54 lines not on this page · raw message

drivers/vfio/pci/cxl/vfio_cxl_priv.h +2 -0
index 4f2637874e9d..3ef8d923a7e8 100644
@@ -26,6 +26,7 @@ struct vfio_pci_cxl_state {
 	resource_size_t              comp_reg_offset;
 	size_t                       comp_reg_size;
 	__le32                      *comp_reg_virt;
+	size_t                       dpa_size;
 	void __iomem                *hdm_iobase;
 	u32                          hdm_count;
 	int                          dpa_region_idx;
@@ -81,5 +82,6 @@ struct vfio_pci_cxl_state {
 int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev);
 void vfio_cxl_clean_virt_regs(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
+resource_size_t vfio_cxl_read_committed_decoder_size(struct vfio_pci_core_device *vdev);
 
 #endif /* __LINUX_VFIO_CXL_PRIV_H */
Signature · 1 line
2.25.1
#40 lore raw

PATCH 15/20 vfio/cxl: Introduce CXL DVSEC configuration space emulation

In reply to [email protected]

From: Manish Honap <[email protected]>

CXL devices have CXL DVSEC registers in the configuration space. Many of them affect the behaviors of the devices, e.g. enabling CXL.io/CXL.mem/CXL.cache.

However, these configurations are owned by the host and a virtualization policy should be applied when handling the access from the guest.

Introduce the emulation of CXL configuration space to handle the access of the virtual CXL configuration space from the guest.

vfio-pci-core already allocates vdev->vconfig as the authoritative virtual config space shadow. Directly use vdev->vconfig:
- DVSEC reads return data from vdev->vconfig (already populated by vfio_config_init() via vfio_ecap_init())
- DVSEC writes go through new CXL-aware write handlers that update

    vdev->vconfig in place

- The writable DVSEC registers are marked virtual in vdev->pci_config_map

---
 drivers/vfio/pci/Makefile              |   2 +-
 drivers/vfio/pci/cxl/vfio_cxl_config.c | 304 +++++++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_core.c   |   4 +
 drivers/vfio/pci/cxl/vfio_cxl_priv.h   |  38 +++-
 drivers/vfio/pci/vfio_pci.c            |  14 ++
 drivers/vfio/pci/vfio_pci_config.c     |  46 +++-
 drivers/vfio/pci/vfio_pci_priv.h       |   3 +
 include/linux/vfio_pci_core.h          |   8 +-
 8 files changed, 415 insertions(+), 4 deletions(-)
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_config.c
drivers/vfio/pci/Makefile +1 -1
index bef916495eae..7c86b7845e8f 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
-vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o
+vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o cxl/vfio_cxl_config.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +304 -0

304 lines not on this page · raw message

drivers/vfio/pci/cxl/vfio_cxl_core.c +4 -0
index 15b6c0d75d9e..e18e992800f6 100644
@@ -26,6 +26,7 @@ static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
 	struct vfio_pci_cxl_state *cxl;
 	bool cxl_mem_capable, is_cxl_type3;
 	u16 cap_word;
+	u32 hdr1;
 
 	/*
 	 * The devm allocation for the CXL state remains for the entire time
@@ -47,6 +48,9 @@ static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
 	cxl->dpa_region_idx = -1;
 	cxl->comp_reg_region_idx = -1;
 
+	pci_read_config_dword(pdev, dvsec + PCI_DVSEC_HEADER1, &hdr1);
+	cxl->dvsec_length = PCI_DVSEC_HEADER1_LEN(hdr1);
+
 	pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
 			     &cap_word);
 
drivers/vfio/pci/cxl/vfio_cxl_priv.h +37 -1
index 3ef8d923a7e8..158fe4e67f98 100644
@@ -31,6 +31,7 @@ struct vfio_pci_cxl_state {
 	u32                          hdm_count;
 	int                          dpa_region_idx;
 	int                          comp_reg_region_idx;
+	size_t                       dvsec_length;
 	u16                          dvsec;
 	u8                           comp_reg_bar;
 	bool                         precommitted;
@@ -76,9 +77,44 @@ struct vfio_pci_cxl_state {
  * (CXL 2.0+ 8.1.3).
  * Offsets are relative to the DVSEC capability base (cxl->dvsec).
  */
-#define CXL_DVSEC_CAPABILITY_OFFSET 0xa
+#define CXL_DVSEC_CAPABILITY_OFFSET       0xa
+#define CXL_DVSEC_CONTROL_OFFSET	  0xc
+#define CXL_DVSEC_STATUS_OFFSET		  0xe
+#define CXL_DVSEC_CONTROL2_OFFSET	  0x10
+#define CXL_DVSEC_STATUS2_OFFSET	  0x12
+#define CXL_DVSEC_LOCK_OFFSET		  0x14
+#define CXL_DVSEC_CAPABILITY2_OFFSET	  0x16
+#define CXL_DVSEC_RANGE1_SIZE_HIGH_OFFSET 0x18
+#define CXL_DVSEC_RANGE1_SIZE_LOW_OFFSET  0x1c
+#define CXL_DVSEC_RANGE1_BASE_HIGH_OFFSET 0x20
+#define CXL_DVSEC_RANGE1_BASE_LOW_OFFSET  0x24
+#define CXL_DVSEC_RANGE2_SIZE_HIGH_OFFSET 0x28
+#define CXL_DVSEC_RANGE2_SIZE_LOW_OFFSET  0x2c
+#define CXL_DVSEC_RANGE2_BASE_HIGH_OFFSET 0x30
+#define CXL_DVSEC_RANGE2_BASE_LOW_OFFSET  0x34
+#define CXL_DVSEC_CAPABILITY3_OFFSET	  0x38
+
 #define CXL_DVSEC_MEM_CAPABLE	    BIT(2)
 
+/* CXL Control / Status / Lock - bit definitions */
+#define CXL_CTRL_LOCK_BIT		 BIT(0)
+#define CXL_CTRL_CXL_IO_ENABLE_BIT	 BIT(1)
+#define CXL_CTRL2_INITIATE_CXL_RESET_BIT BIT(2)
+#define CXL_CAP3_VOLATILE_HDM_BIT	 BIT(3)
+#define CXL_STATUS2_RW1CS_BIT		 BIT(3)
+#define CXL_CAP3_P2P_BIT		 BIT(4)
+#define CXL_CAP2_MODIFIED_COMPLETION_BIT BIT(6)
+#define CXL_STATUS_RW1C_BIT		 BIT(14)
+#define CXL_CTRL_RESERVED_MASK		 (BIT(13) | BIT(15))
+#define CXL_CTRL_P2P_REV_MASK		 BIT(12)
+#define CXL_STATUS_RESERVED_MASK	 (GENMASK(13, 0) | BIT(15))
+#define CXL_CTRL2_RESERVED_MASK		 GENMASK(15, 6)
+#define CXL_CTRL2_HW_BITS_MASK		 (BIT(0) | BIT(1) | BIT(3))
+#define CXL_CTRL2_VOLATILE_HDM_REV_MASK	 BIT(4)
+#define CXL_CTRL2_MODIFIED_COMP_REV_MASK BIT(5)
+#define CXL_LOCK_RESERVED_MASK		 GENMASK(15, 1)
+#define CXL_BASE_LO_RESERVED_MASK	 GENMASK(27, 0)
+
 int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev);
 void vfio_cxl_clean_virt_regs(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
drivers/vfio/pci/vfio_pci.c +14 -0
index d3138badeaa6..22cf9ea831f9 100644
@@ -121,12 +121,26 @@ static int vfio_pci_open_device(struct vfio_device *core_vdev)
 	}
 
 	if (vdev->cxl) {
+		/*
+		 * pci_config_map and vconfig are valid now (allocated by
+		 * vfio_config_init() inside vfio_pci_core_enable() above).
+		 */
+		vfio_cxl_setup_dvsec_perms(vdev);
+
 		ret = vfio_cxl_register_cxl_region(vdev);
 		if (ret) {
 			pci_warn(pdev, "Failed to setup CXL region\n");
 			vfio_pci_core_disable(vdev);
 			return ret;
 		}
+
+		ret = vfio_cxl_register_comp_regs_region(vdev);
+		if (ret) {
+			pci_warn(pdev, "Failed to register COMP_REGS region\n");
+			vfio_cxl_unregister_cxl_region(vdev);
+			vfio_pci_core_disable(vdev);
+			return ret;
+		}
 	}
 
 	vfio_pci_core_finish_enable(vdev);
drivers/vfio/pci/vfio_pci_config.c +45 -1

58 lines not on this page · raw message

drivers/vfio/pci/vfio_pci_priv.h +3 -0
index f8db9a05c033..d778107fa908 100644
@@ -154,6 +154,7 @@ void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev);
 int  vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
+void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev);
 
 #else
 
@@ -180,6 +181,8 @@ vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev)
 { return 0; }
 static inline void
 vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev) { }
+static inline void
+vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev) { }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
 
include/linux/vfio_pci_core.h +7 -1
index cd8ed98a82a3..aa159d0c8da7 100644
@@ -31,7 +31,7 @@ struct p2pdma_provider;
 struct dma_buf_phys_vec;
 struct dma_buf_attachment;
 struct vfio_pci_cxl_state;
-
+struct perm_bits;
 
 struct vfio_pci_eventfd {
 	struct eventfd_ctx	*ctx;
@@ -141,6 +141,12 @@ struct vfio_pci_core_device {
 	struct list_head	ioeventfds_list;
 	struct vfio_pci_vf_token	*vf_token;
 	struct vfio_pci_cxl_state *cxl;
+	int (*dvsec_readfn)(struct vfio_pci_core_device *vdev, int pos,
+			    int count, struct perm_bits *perm,
+			    int offset, __le32 *val);
+	int (*dvsec_writefn)(struct vfio_pci_core_device *vdev, int pos,
+			     int count, struct perm_bits *perm,
+			     int offset, __le32 val);
 	struct list_head		sriov_pfs_item;
 	struct vfio_pci_core_device	*sriov_pf_core_dev;
 	struct notifier_block	nb;
Signature · 1 line
2.25.1
maintainer
#41 lore raw

Re: [PATCH 15/20] vfio/cxl: Introduce CXL DVSEC configuration space emulation

Quoted 89 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

CXL devices have CXL DVSEC registers in the configuration space. Many of them affect the behaviors of the devices, e.g. enabling CXL.io/CXL.mem/CXL.cache.

However, these configurations are owned by the host and a virtualization policy should be applied when handling the access from the guest.

Introduce the emulation of CXL configuration space to handle the access of the virtual CXL configuration space from the guest.

vfio-pci-core already allocates vdev->vconfig as the authoritative virtual config space shadow. Directly use vdev->vconfig:
- DVSEC reads return data from vdev->vconfig (already populated by vfio_config_init() via vfio_ecap_init())
- DVSEC writes go through new CXL-aware write handlers that update

    vdev->vconfig in place

- The writable DVSEC registers are marked virtual in vdev->pci_config_map

---
 drivers/vfio/pci/Makefile              |   2 +-
 drivers/vfio/pci/cxl/vfio_cxl_config.c | 304 +++++++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_core.c   |   4 +
 drivers/vfio/pci/cxl/vfio_cxl_priv.h   |  38 +++-
 drivers/vfio/pci/vfio_pci.c            |  14 ++
 drivers/vfio/pci/vfio_pci_config.c     |  46 +++-
 drivers/vfio/pci/vfio_pci_priv.h       |   3 +
 include/linux/vfio_pci_core.h          |   8 +-
 8 files changed, 415 insertions(+), 4 deletions(-)
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_config.c
drivers/vfio/pci/Makefile +1 -1
index bef916495eae..7c86b7845e8f 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
-vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o
+vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o cxl/vfio_cxl_config.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +36 -0
new file mode 100644
index 000000000000..a9560661345c
@@ -0,0 +1,304 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * CXL DVSEC configuration space emulation for vfio-pci.
+ *
+ * Integrates into the existing vfio-pci-core ecap_perms[] framework using
+ * vdev->vconfig as the sole shadow buffer for DVSEC registers.
+ *
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#include <linux/pci.h>
+#include <linux/vfio_pci_core.h>
+
+#include "../vfio_pci_priv.h"
+#include "vfio_cxl_priv.h"
+
+/* Helpers to access vdev->vconfig at a DVSEC-relative offset */
+static inline u16 dvsec_virt_read16(struct vfio_pci_core_device *vdev,
+				    u16 off)
+{
+	return le16_to_cpu(*(u16 *)(vdev->vconfig +
+				    vdev->cxl->dvsec + off));
+}
+
+static inline void dvsec_virt_write16(struct vfio_pci_core_device *vdev,
+				      u16 off, u16 val)
+{
+	*(u16 *)(vdev->vconfig + vdev->cxl->dvsec + off) = cpu_to_le16(val);
+}
+
+static inline u32 dvsec_virt_read32(struct vfio_pci_core_device *vdev,
+				    u16 off)
+{
+	return le32_to_cpu(*(u32 *)(vdev->vconfig +
+				    vdev->cxl->dvsec + off));
+}

drivers/vfio/pci/cxl/vfio_cxl_config.c line 46

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+
+static inline void dvsec_virt_write32(struct vfio_pci_core_device *vdev,
+				      u16 off, u32 val)
+{
+	*(u32 *)(vdev->vconfig + vdev->cxl->dvsec + off) = cpu_to_le32(val);
+}
+
+/* Individual DVSEC register write handlers */
+
+static void cxl_control_write(struct vfio_pci_core_device *vdev,

cxl_dvsec_control_write()

drivers/vfio/pci/cxl/vfio_cxl_config.c line 47

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+			      u16 abs_off, u16 new_val)

abs_off not needed?

drivers/vfio/pci/cxl/vfio_cxl_config.c line 60

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +13 -0
+{
+	u16 lock = dvsec_virt_read16(vdev, CXL_DVSEC_LOCK_OFFSET);
+	u16 cap3 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY3_OFFSET);
+	u16 rev_mask = CXL_CTRL_RESERVED_MASK;
+
+	if (lock & CXL_CTRL_LOCK_BIT)
+		return; /* register is locked after first write */
+
+	if (!(cap3 & CXL_CAP3_P2P_BIT))
+		rev_mask |= CXL_CTRL_P2P_REV_MASK;
+
+	new_val &= ~rev_mask;
+	new_val |= CXL_CTRL_CXL_IO_ENABLE_BIT; /* CXL.io always enabled */

Can FIELD_MODIFY() be used here?

drivers/vfio/pci/cxl/vfio_cxl_config.c line 65

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +5 -0
+
+	dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL_OFFSET, new_val);
+}
+
+static void cxl_status_write(struct vfio_pci_core_device *vdev,

cxl_dvsec_status_write()

drivers/vfio/pci/cxl/vfio_cxl_config.c line 66

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+			     u16 abs_off, u16 new_val)

abs_off not needed

drivers/vfio/pci/cxl/vfio_cxl_config.c line 77

Quoted 11 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +11 -0
+{
+	u16 cur_val = dvsec_virt_read16(vdev, CXL_DVSEC_STATUS_OFFSET);
+
+	new_val &= ~CXL_STATUS_RESERVED_MASK;
+
+	/* RW1C: writing a 1 clears the bit; writing 0 leaves it unchanged */
+	if (new_val & CXL_STATUS_RW1C_BIT)
+		new_val &= ~CXL_STATUS_RW1C_BIT;
+	else
+		new_val = (new_val & ~CXL_STATUS_RW1C_BIT) |
+			(cur_val & CXL_STATUS_RW1C_BIT);

Given there's only 1 bit we need to deal with in this register and everything else is reserved, can't we just do:

if (new_val & CXL_STATUS_RW1C_BIT)

	new_val = 0;
else
	new_val = old_val;

drivers/vfio/pci/cxl/vfio_cxl_config.c line 83

Quoted 6 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +6 -0
+
+	dvsec_virt_write16(vdev, CXL_DVSEC_STATUS_OFFSET, new_val);
+}
+
+static void cxl_control2_write(struct vfio_pci_core_device *vdev,
+			       u16 abs_off, u16 new_val)

abs_off not needed?

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +8 -0
+{
+	struct pci_dev *pdev = vdev->pdev;
+	u16 cap2 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY2_OFFSET);
+	u16 cap3 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY3_OFFSET);
+	u16 rev_mask = CXL_CTRL2_RESERVED_MASK;
+	u16 hw_bits  = CXL_CTRL2_HW_BITS_MASK;
+	bool initiate_cxl_reset = new_val & CXL_CTRL2_INITIATE_CXL_RESET_BIT;
+

drivers/vfio/pci/cxl/vfio_cxl_config.c line 101

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+	if (!(cap3 & CXL_CAP3_VOLATILE_HDM_BIT))
+		rev_mask |= CXL_CTRL2_VOLATILE_HDM_REV_MASK;
+	if (!(cap2 & CXL_CAP2_MODIFIED_COMPLETION_BIT))
+		rev_mask |= CXL_CTRL2_MODIFIED_COMP_REV_MASK;
+
+	new_val &= ~rev_mask;
+
+	/* Bits that go directly to hardware */
+	hw_bits &= new_val;
+

Bit 1 and 2 are always read 0 by hardware. Probably should clear it before writing to the virtual register?

drivers/vfio/pci/cxl/vfio_cxl_config.c line 114

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +13 -0
+	dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL2_OFFSET, new_val);
+
+	if (hw_bits)
+		pci_write_config_word(pdev, abs_off, hw_bits);
+
+	if (initiate_cxl_reset) {
+		/* TODO: invoke CXL protocol reset via cxl subsystem */
+		dev_warn(&pdev->dev, "vfio-cxl: CXL reset requested but not yet supported\n");
+	}
+}
+
+static void cxl_status2_write(struct vfio_pci_core_device *vdev,
+			      u16 abs_off, u16 new_val)

abs_off not needed

drivers/vfio/pci/cxl/vfio_cxl_config.c line 127

Quoted 13 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +13 -0
+{
+	u16 cap3 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY3_OFFSET);
+
+	/* RW1CS: write 1 to clear, but only if the capability is supported */
+	if ((cap3 & CXL_CAP3_VOLATILE_HDM_BIT) &&
+	    (new_val & CXL_STATUS2_RW1CS_BIT))
+		pci_write_config_word(vdev->pdev, abs_off,
+				      CXL_STATUS2_RW1CS_BIT);
+	/* STATUS2 is not mirrored in vconfig - reads go to hardware */
+}
+
+static void cxl_lock_write(struct vfio_pci_core_device *vdev,
+			   u16 abs_off, u16 new_val)

abs_off not needed

Quoted 17 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +17 -0
+{
+	u16 cur_val = dvsec_virt_read16(vdev, CXL_DVSEC_LOCK_OFFSET);
+
+	/* Once the LOCK bit is set it can only be cleared by conventional reset */
+	if (cur_val & CXL_CTRL_LOCK_BIT)
+		return;
+
+	new_val &= ~CXL_LOCK_RESERVED_MASK;
+	dvsec_virt_write16(vdev, CXL_DVSEC_LOCK_OFFSET, new_val);
+}
+
+static void cxl_range_base_lo_write(struct vfio_pci_core_device *vdev,
+				    u16 dvsec_off, u32 new_val)
+{
+	new_val &= ~CXL_BASE_LO_RESERVED_MASK;
+	dvsec_virt_write32(vdev, dvsec_off, new_val);
+}

drivers/vfio/pci/cxl/vfio_cxl_config.c line 154

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+
+/*
+ * vfio_cxl_dvsec_readfn - per-device DVSEC read handler.
+ *
+ * Called via vfio_pci_dvsec_dispatch_read() for devices that have registered
+ * a dvsec_readfn.  Returns shadow vconfig values for virtualized DVSEC
+ * registers (CONTROL, STATUS, CONTROL2, LOCK) so that userspace reads reflect
+ * the emulated state rather than the raw hardware value.  All other DVSEC
+ * bytes are passed through to hardware via vfio_raw_config_read().
+ */

Provide proper kdoc function header

Quoted 27 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +27 -0
+static int vfio_cxl_dvsec_readfn(struct vfio_pci_core_device *vdev,
+				 int pos, int count,
+				 struct perm_bits *perm,
+				 int offset, __le32 *val)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	u16 dvsec_off;
+
+	if (!cxl || (u16)pos < cxl->dvsec ||
+	    (u16)pos >= cxl->dvsec + cxl->dvsec_length)
+		return vfio_raw_config_read(vdev, pos, count, perm, offset, val);
+
+	dvsec_off = (u16)pos - cxl->dvsec;
+
+	switch (dvsec_off) {
+	case CXL_DVSEC_CONTROL_OFFSET:
+	case CXL_DVSEC_STATUS_OFFSET:
+	case CXL_DVSEC_CONTROL2_OFFSET:
+	case CXL_DVSEC_LOCK_OFFSET:
+		/* Return shadow vconfig value for virtualized registers */
+		memcpy(val, vdev->vconfig + pos, count);
+		return count;
+	default:
+		return vfio_raw_config_read(vdev, pos, count,
+					    perm, offset, val);
+	}> +}
+

drivers/vfio/pci/cxl/vfio_cxl_config.c line 192

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+/*
+ * vfio_cxl_dvsec_writefn - ecap_perms write handler for PCI_EXT_CAP_ID_DVSEC.
+ *
+ * Installed once into ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn by
+ * vfio_pci_init_perm_bits() when CONFIG_VFIO_CXL_CORE=y.  Applies to every
+ * device opened under vfio-pci; the vdev->cxl NULL check distinguishes CXL
+ * devices from non-CXL devices that happen to expose a DVSEC capability.
+ *
+ * @pos:    absolute byte position in config space
+ * @offset: byte offset within the capability structure

missing return value expectations

Quoted 62 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +62 -0
+ */
+static int vfio_cxl_dvsec_writefn(struct vfio_pci_core_device *vdev,
+				  int pos, int count,
+				  struct perm_bits *perm,
+				  int offset, __le32 val)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	u16 abs_off = (u16)pos;
+	u16 dvsec_off;
+	u16 wval16;
+	u32 wval32;
+
+	if (!cxl || (u16)pos < cxl->dvsec ||
+	    (u16)pos >= cxl->dvsec + cxl->dvsec_length)
+		return vfio_raw_config_write(vdev, pos, count, perm,
+					     offset, val);
+
+	pci_dbg(vdev->pdev,
+		"vfio_cxl: DVSEC write: abs=0x%04x dvsec_off=0x%04x "
+		"count=%d raw_val=0x%08x\n",
+		abs_off, abs_off - cxl->dvsec, count, le32_to_cpu(val));
+
+	dvsec_off = abs_off - cxl->dvsec;
+
+	/* Route to the appropriate per-register handler */
+	switch (dvsec_off) {
+	case CXL_DVSEC_CONTROL_OFFSET:
+		wval16 = (u16)le32_to_cpu(val);
+		cxl_control_write(vdev, abs_off, wval16);
+		break;
+	case CXL_DVSEC_STATUS_OFFSET:
+		wval16 = (u16)le32_to_cpu(val);
+		cxl_status_write(vdev, abs_off, wval16);
+		break;
+	case CXL_DVSEC_CONTROL2_OFFSET:
+		wval16 = (u16)le32_to_cpu(val);
+		cxl_control2_write(vdev, abs_off, wval16);
+		break;
+	case CXL_DVSEC_STATUS2_OFFSET:
+		wval16 = (u16)le32_to_cpu(val);
+		cxl_status2_write(vdev, abs_off, wval16);
+		break;
+	case CXL_DVSEC_LOCK_OFFSET:
+		wval16 = (u16)le32_to_cpu(val);
+		cxl_lock_write(vdev, abs_off, wval16);
+		break;
+	case CXL_DVSEC_RANGE1_BASE_HIGH_OFFSET:
+	case CXL_DVSEC_RANGE2_BASE_HIGH_OFFSET:
+		wval32 = le32_to_cpu(val);
+		dvsec_virt_write32(vdev, dvsec_off, wval32);
+		break;
+	case CXL_DVSEC_RANGE1_BASE_LOW_OFFSET:
+	case CXL_DVSEC_RANGE2_BASE_LOW_OFFSET:
+		wval32 = le32_to_cpu(val);
+		cxl_range_base_lo_write(vdev, dvsec_off, wval32);
+		break;
+	default:
+		/* RO registers: header, capability, range sizes - discard */
+		break;
+	}
+
+	return count;

drivers/vfio/pci/cxl/vfio_cxl_config.c line 264

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+}
+
+/*
+ * vfio_cxl_setup_dvsec_perms - Install per-device CXL DVSEC read/write hooks.
+ *
+ * Called once per device open after vfio_config_init() has seeded vdev->vconfig
+ * from hardware.  Registers vfio_cxl_dvsec_readfn and vfio_cxl_dvsec_writefn
+ * as the per-device DVSEC handlers.  The global dispatch functions installed
+ * in ecap_perms[PCI_EXT_CAP_ID_DVSEC] at module init call these per-device
+ * hooks so that pci_config_map bytes remain PCI_EXT_CAP_ID_DVSEC throughout.

provide proper kdoc function header

Quoted 90 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +40 -0
+ *
+ * vfio_cxl_dvsec_readfn: returns vconfig shadow for CONTROL/STATUS/CONTROL2/
+ *   LOCK; passes all other DVSEC bytes through to hardware.
+ * vfio_cxl_dvsec_writefn: enforces per-register semantics (RW1C, forced
+ *   IO_ENABLE, reserved-bit masking) and stores results in vconfig.
+ *
+ * Also forces CXL.io IO_ENABLE in the CONTROL vconfig shadow so the initial
+ * read returns 1 even before the first write.
+ */
+void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev)
+{
+	u16 ctrl = dvsec_virt_read16(vdev, CXL_DVSEC_CONTROL_OFFSET);
+
+	/*
+	 * Register per-device DVSEC read/write handlers.  The global
+	 * ecap_perms[PCI_EXT_CAP_ID_DVSEC] dispatchers will call them.
+	 *
+	 * vfio_cxl_dvsec_readfn returns vconfig shadow values for the
+	 * virtualized registers (CONTROL, STATUS, CONTROL2, LOCK) so that
+	 * reads reflect emulated state rather than raw hardware.
+	 *
+	 * vfio_cxl_dvsec_writefn enforces per-register semantics (RW1C,
+	 * forced IO_ENABLE, reserved-bit masking) and stores results in
+	 * vconfig.  Because ecap_perms[DVSEC].writefn dispatches to this
+	 * handler, the pci_config_map bytes remain as PCI_EXT_CAP_ID_DVSEC
+	 * _ no PCI_CAP_ID_INVALID_VIRT marking is needed or wanted.
+	 */
+	vdev->dvsec_readfn  = vfio_cxl_dvsec_readfn;
+	vdev->dvsec_writefn = vfio_cxl_dvsec_writefn;
+
+	/*
+	 * vconfig is seeded from hardware at open time.  Force IO_ENABLE set
+	 * in the CONTROL shadow so the initial read returns 1 even if the
+	 * hardware reset value has it cleared.	 Subsequent writes are handled
+	 * by cxl_control_write() which also forces this bit.
+	 */
+	ctrl |= CXL_CTRL_CXL_IO_ENABLE_BIT;
+	dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL_OFFSET, ctrl);
+}
+EXPORT_SYMBOL_GPL(vfio_cxl_setup_dvsec_perms);
drivers/vfio/pci/cxl/vfio_cxl_core.c +4 -0
index 15b6c0d75d9e..e18e992800f6 100644
@@ -26,6 +26,7 @@ static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
 	struct vfio_pci_cxl_state *cxl;
 	bool cxl_mem_capable, is_cxl_type3;
 	u16 cap_word;
+	u32 hdr1;
 
 	/*
 	 * The devm allocation for the CXL state remains for the entire time
@@ -47,6 +48,9 @@ static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
 	cxl->dpa_region_idx = -1;
 	cxl->comp_reg_region_idx = -1;
 
+	pci_read_config_dword(pdev, dvsec + PCI_DVSEC_HEADER1, &hdr1);
+	cxl->dvsec_length = PCI_DVSEC_HEADER1_LEN(hdr1);
+
 	pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
 			     &cap_word);
 
drivers/vfio/pci/cxl/vfio_cxl_priv.h +12 -1
index 3ef8d923a7e8..158fe4e67f98 100644
@@ -31,6 +31,7 @@ struct vfio_pci_cxl_state {
 	u32                          hdm_count;
 	int                          dpa_region_idx;
 	int                          comp_reg_region_idx;
+	size_t                       dvsec_length;
 	u16                          dvsec;
 	u8                           comp_reg_bar;
 	bool                         precommitted;
@@ -76,9 +77,44 @@ struct vfio_pci_cxl_state {
  * (CXL 2.0+ 8.1.3).
  * Offsets are relative to the DVSEC capability base (cxl->dvsec).
  */
-#define CXL_DVSEC_CAPABILITY_OFFSET 0xa
+#define CXL_DVSEC_CAPABILITY_OFFSET       0xa
+#define CXL_DVSEC_CONTROL_OFFSET	  0xc
+#define CXL_DVSEC_STATUS_OFFSET		  0xe
+#define CXL_DVSEC_CONTROL2_OFFSET	  0x10
+#define CXL_DVSEC_STATUS2_OFFSET	  0x12
+#define CXL_DVSEC_LOCK_OFFSET		  0x14
+#define CXL_DVSEC_CAPABILITY2_OFFSET	  0x16
+#define CXL_DVSEC_RANGE1_SIZE_HIGH_OFFSET 0x18
+#define CXL_DVSEC_RANGE1_SIZE_LOW_OFFSET  0x1c
+#define CXL_DVSEC_RANGE1_BASE_HIGH_OFFSET 0x20
+#define CXL_DVSEC_RANGE1_BASE_LOW_OFFSET  0x24

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 100

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +8 -0
+#define CXL_DVSEC_RANGE2_SIZE_HIGH_OFFSET 0x28
+#define CXL_DVSEC_RANGE2_SIZE_LOW_OFFSET  0x2c
+#define CXL_DVSEC_RANGE2_BASE_HIGH_OFFSET 0x30
+#define CXL_DVSEC_RANGE2_BASE_LOW_OFFSET  0x34
+#define CXL_DVSEC_CAPABILITY3_OFFSET	  0x38
+
 #define CXL_DVSEC_MEM_CAPABLE	    BIT(2)
 
+/* CXL Control / Status / Lock - bit definitions */
+#define CXL_CTRL_LOCK_BIT		 BIT(0)

CXL_CTRL_CONFIG_LOCK_BIT

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 104

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +4 -0
+#define CXL_CTRL_CXL_IO_ENABLE_BIT	 BIT(1)
+#define CXL_CTRL2_INITIATE_CXL_RESET_BIT BIT(2)
+#define CXL_CAP3_VOLATILE_HDM_BIT	 BIT(3)
+#define CXL_STATUS2_RW1CS_BIT		 BIT(3)

CXL_STATUS2_VOL_HDM_PRSV_ERR_BIT

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 107

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +3 -0
+#define CXL_CAP3_P2P_BIT		 BIT(4)
+#define CXL_CAP2_MODIFIED_COMPLETION_BIT BIT(6)
+#define CXL_STATUS_RW1C_BIT		 BIT(14)

CXL_STATUS_VIRAL_STATUS_BIT

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 116

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +9 -0
+#define CXL_CTRL_RESERVED_MASK		 (BIT(13) | BIT(15))
+#define CXL_CTRL_P2P_REV_MASK		 BIT(12)
+#define CXL_STATUS_RESERVED_MASK	 (GENMASK(13, 0) | BIT(15))
+#define CXL_CTRL2_RESERVED_MASK		 GENMASK(15, 6)
+#define CXL_CTRL2_HW_BITS_MASK		 (BIT(0) | BIT(1) | BIT(3))
+#define CXL_CTRL2_VOLATILE_HDM_REV_MASK	 BIT(4)
+#define CXL_CTRL2_MODIFIED_COMP_REV_MASK BIT(5)
+#define CXL_LOCK_RESERVED_MASK		 GENMASK(15, 1)
+#define CXL_BASE_LO_RESERVED_MASK	 GENMASK(27, 0)

Move the CXL reg offset and bit defs to a common header. Also, please group the relevant bits together per register.

DJ

Quoted 146 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +1 -0
+
 int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev);
 void vfio_cxl_clean_virt_regs(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
drivers/vfio/pci/vfio_pci.c +14 -0
index d3138badeaa6..22cf9ea831f9 100644
@@ -121,12 +121,26 @@ static int vfio_pci_open_device(struct vfio_device *core_vdev)
 	}
 
 	if (vdev->cxl) {
+		/*
+		 * pci_config_map and vconfig are valid now (allocated by
+		 * vfio_config_init() inside vfio_pci_core_enable() above).
+		 */
+		vfio_cxl_setup_dvsec_perms(vdev);
+
 		ret = vfio_cxl_register_cxl_region(vdev);
 		if (ret) {
 			pci_warn(pdev, "Failed to setup CXL region\n");
 			vfio_pci_core_disable(vdev);
 			return ret;
 		}
+
+		ret = vfio_cxl_register_comp_regs_region(vdev);
+		if (ret) {
+			pci_warn(pdev, "Failed to register COMP_REGS region\n");
+			vfio_cxl_unregister_cxl_region(vdev);
+			vfio_pci_core_disable(vdev);
+			return ret;
+		}
 	}
 
 	vfio_pci_core_finish_enable(vdev);
drivers/vfio/pci/vfio_pci_config.c +45 -1
index 79aaf270adb2..90e2c25381d6 100644
@@ -1085,6 +1085,49 @@ static int __init init_pci_ext_cap_pwr_perm(struct perm_bits *perm)
 	return 0;
 }
 
+/*
+ * vfio_pci_dvsec_dispatch_read - per-device DVSEC read dispatcher.
+ *
+ * Installed as ecap_perms[PCI_EXT_CAP_ID_DVSEC].readfn at module init.
+ * Calls vdev->dvsec_readfn when a shadow-read handler has been registered
+ * (e.g. by vfio_cxl_setup_dvsec_perms() for CXL Type-2 devices), otherwise
+ * falls through to vfio_raw_config_read for hardware pass-through.
+ *
+ * This indirection allows per-device DVSEC reads from vconfig shadow
+ * without touching the global ecap_perms[] table.
+ */
+static int vfio_pci_dvsec_dispatch_read(struct vfio_pci_core_device *vdev,
+					int pos, int count,
+					struct perm_bits *perm,
+					int offset, __le32 *val)
+{
+	if (vdev->dvsec_readfn)
+		return vdev->dvsec_readfn(vdev, pos, count, perm, offset, val);
+	return vfio_raw_config_read(vdev, pos, count, perm, offset, val);
+}
+
+/*
+ * vfio_pci_dvsec_dispatch_write - per-device DVSEC write dispatcher.
+ *
+ * Installed as ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn at module init.
+ * Calls vdev->dvsec_writefn when a handler has been registered for this
+ * device (e.g. by vfio_cxl_setup_dvsec_perms() for CXL Type-2 devices),
+ * otherwise falls through to vfio_raw_config_write so that non-CXL
+ * devices with a DVSEC capability continue to pass writes to hardware.
+ *
+ * This indirection allows per-device DVSEC handlers to be registered
+ * without touching the global ecap_perms[] table.
+ */
+static int vfio_pci_dvsec_dispatch_write(struct vfio_pci_core_device *vdev,
+					 int pos, int count,
+					 struct perm_bits *perm,
+					 int offset, __le32 val)
+{
+	if (vdev->dvsec_writefn)
+		return vdev->dvsec_writefn(vdev, pos, count, perm, offset, val);
+	return vfio_raw_config_write(vdev, pos, count, perm, offset, val);
+}
+
 /*
  * Initialize the shared permission tables
  */
@@ -1121,7 +1164,8 @@ int __init vfio_pci_init_perm_bits(void)
 	ret |= init_pci_ext_cap_err_perm(&ecap_perms[PCI_EXT_CAP_ID_ERR]);
 	ret |= init_pci_ext_cap_pwr_perm(&ecap_perms[PCI_EXT_CAP_ID_PWR]);
 	ecap_perms[PCI_EXT_CAP_ID_VNDR].writefn = vfio_raw_config_write;
-	ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn = vfio_raw_config_write;
+	ecap_perms[PCI_EXT_CAP_ID_DVSEC].readfn  = vfio_pci_dvsec_dispatch_read;
+	ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn = vfio_pci_dvsec_dispatch_write;
 
 	if (ret)
 		vfio_pci_uninit_perm_bits();
drivers/vfio/pci/vfio_pci_priv.h +3 -0
index f8db9a05c033..d778107fa908 100644
@@ -154,6 +154,7 @@ void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev);
 int  vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
+void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev);
 
 #else
 
@@ -180,6 +181,8 @@ vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev)
 { return 0; }
 static inline void
 vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev) { }
+static inline void
+vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev) { }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
 
include/linux/vfio_pci_core.h +7 -1
index cd8ed98a82a3..aa159d0c8da7 100644
@@ -31,7 +31,7 @@ struct p2pdma_provider;
 struct dma_buf_phys_vec;
 struct dma_buf_attachment;
 struct vfio_pci_cxl_state;
-
+struct perm_bits;
 
 struct vfio_pci_eventfd {
 	struct eventfd_ctx	*ctx;
@@ -141,6 +141,12 @@ struct vfio_pci_core_device {
 	struct list_head	ioeventfds_list;
 	struct vfio_pci_vf_token	*vf_token;
 	struct vfio_pci_cxl_state *cxl;
+	int (*dvsec_readfn)(struct vfio_pci_core_device *vdev, int pos,
+			    int count, struct perm_bits *perm,
+			    int offset, __le32 *val);
+	int (*dvsec_writefn)(struct vfio_pci_core_device *vdev, int pos,
+			     int count, struct perm_bits *perm,
+			     int offset, __le32 val);
 	struct list_head		sriov_pfs_item;
 	struct vfio_pci_core_device	*sriov_pf_core_dev;
 	struct notifier_block	nb;
#42 lore raw

RE: [PATCH 15/20] vfio/cxl: Introduce CXL DVSEC configuration space emulation

Quoted 119 lines

-----Original Message-----
From: Dave Jiang <[email protected]>
Sent: 14 March 2026 03:37
To: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]
Cc: Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 15/20] vfio/cxl: Introduce CXL DVSEC configuration space emulation

External email: Use caution opening links or attachments

Quoted 8 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

CXL devices have CXL DVSEC registers in the configuration space. Many of them affect the behaviors of the devices, e.g. enabling CXL.io/CXL.mem/CXL.cache.

However, these configurations are owned by the host and a virtualization policy should be applied when handling the access from

the guest.

Quoted 64 lines

Introduce the emulation of CXL configuration space to handle the access of the virtual CXL configuration space from the guest.

vfio-pci-core already allocates vdev->vconfig as the authoritative virtual config space shadow. Directly use vdev->vconfig:
- DVSEC reads return data from vdev->vconfig (already populated by vfio_config_init() via vfio_ecap_init())
- DVSEC writes go through new CXL-aware write handlers that update

    vdev->vconfig in place

- The writable DVSEC registers are marked virtual in
vdev->pci_config_map

---
 drivers/vfio/pci/Makefile              |   2 +-
 drivers/vfio/pci/cxl/vfio_cxl_config.c | 304 +++++++++++++++++++++++++
 drivers/vfio/pci/cxl/vfio_cxl_core.c   |   4 +
 drivers/vfio/pci/cxl/vfio_cxl_priv.h   |  38 +++-
 drivers/vfio/pci/vfio_pci.c            |  14 ++
 drivers/vfio/pci/vfio_pci_config.c     |  46 +++-
 drivers/vfio/pci/vfio_pci_priv.h       |   3 +
 include/linux/vfio_pci_core.h          |   8 +-
 8 files changed, 415 insertions(+), 4 deletions(-)  create mode

100644 drivers/vfio/pci/cxl/vfio_cxl_config.c

drivers/vfio/pci/Makefile +0 -0
index bef916495eae..7c86b7845e8f 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o

vfio_pci_config.o -vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o

drivers/vfio/pci/Makefile +2 -0
+vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o
+cxl/vfio_cxl_emu.o cxl/vfio_cxl_config.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o diff --git

a/drivers/vfio/pci/cxl/vfio_cxl_config.c
b/drivers/vfio/pci/cxl/vfio_cxl_config.c
new file mode 100644
index 000000000000..a9560661345c

drivers/vfio/pci/cxl/vfio_cxl_config.c new file +14 -0
@@ -0,0 +1,304 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * CXL DVSEC configuration space emulation for vfio-pci.
+ *
+ * Integrates into the existing vfio-pci-core ecap_perms[] framework
+using
+ * vdev->vconfig as the sole shadow buffer for DVSEC registers.
+ *
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights
+reserved  */
+
+#include <linux/pci.h>
+#include <linux/vfio_pci_core.h>
+

drivers/vfio/pci/cxl/vfio_cxl_config.c line 24

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+#include "../vfio_pci_priv.h"
+#include "vfio_cxl_priv.h"
+
+/* Helpers to access vdev->vconfig at a DVSEC-relative offset */
+static inline u16 dvsec_virt_read16(struct vfio_pci_core_device *vdev,
+                                 u16 off) {
+     return le16_to_cpu(*(u16 *)(vdev->vconfig +
+                                 vdev->cxl->dvsec + off)); }
+
+static inline void dvsec_virt_write16(struct vfio_pci_core_device

*vdev,

drivers/vfio/pci/cxl/vfio_cxl_config.c line 34

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+                                   u16 off, u16 val) {
+     *(u16 *)(vdev->vconfig + vdev->cxl->dvsec + off) =
+cpu_to_le16(val); }
+
+static inline u32 dvsec_virt_read32(struct vfio_pci_core_device *vdev,
+                                 u16 off) {
+     return le32_to_cpu(*(u32 *)(vdev->vconfig +
+                                 vdev->cxl->dvsec + off)); }
+
+static inline void dvsec_virt_write32(struct vfio_pci_core_device

*vdev,

Quoted 9 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 46

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +7 -0
+                                   u16 off, u32 val) {
+     *(u32 *)(vdev->vconfig + vdev->cxl->dvsec + off) =
+cpu_to_le32(val); }
+
+/* Individual DVSEC register write handlers */
+
+static void cxl_control_write(struct vfio_pci_core_device *vdev,

cxl_dvsec_control_write()

Yes, renamed.

Quoted 4 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 42

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+                           u16 abs_off, u16 new_val)

abs_off not needed?

Removed.

Quoted 17 lines
Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +4 -0
+{
+     u16 lock = dvsec_virt_read16(vdev, CXL_DVSEC_LOCK_OFFSET);
+     u16 cap3 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY3_OFFSET);
+     u16 rev_mask = CXL_CTRL_RESERVED_MASK;

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+
+     if (lock & CXL_CTRL_LOCK_BIT)
+             return; /* register is locked after first write */
+
+     if (!(cap3 & CXL_CAP3_P2P_BIT))
+             rev_mask |= CXL_CTRL_P2P_REV_MASK;
+
+     new_val &= ~rev_mask;
+     new_val |= CXL_CTRL_CXL_IO_ENABLE_BIT; /* CXL.io always enabled
+ */

Can FIELD_MODIFY() be used here?

I looked at include/linux/bitfield.h:: FIELD_MODIFY but it requires constraint of having the mask as power of 2. For rev_mask = (BIT(13) | BIT(15)) this will fail.

Quoted 10 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 65

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +4 -0
+
+     dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL_OFFSET, new_val); }
+
+static void cxl_status_write(struct vfio_pci_core_device *vdev,

cxl_dvsec_status_write()

drivers/vfio/pci/cxl/vfio_cxl_config.c line 61

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+                          u16 abs_off, u16 new_val)

abs_off not needed

Removed.

Quoted 15 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +6 -0
+{
+     u16 cur_val = dvsec_virt_read16(vdev, CXL_DVSEC_STATUS_OFFSET);
+
+     new_val &= ~CXL_STATUS_RESERVED_MASK;
+
+     /* RW1C: writing a 1 clears the bit; writing 0 leaves it unchanged

*/

drivers/vfio/pci/cxl/vfio_cxl_config.c line 77

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +5 -0
+     if (new_val & CXL_STATUS_RW1C_BIT)
+             new_val &= ~CXL_STATUS_RW1C_BIT;
+     else
+             new_val = (new_val & ~CXL_STATUS_RW1C_BIT) |
+                     (cur_val & CXL_STATUS_RW1C_BIT);
Quoted 7 lines

Given there's only 1 bit we need to deal with in this register and everything else is reserved, can't we just do:

if (new_val & CXL_STATUS_RW1C_BIT)

        new_val = 0;
else
        new_val = old_val;

Yes, updated.

Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 83

Quoted 6 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +5 -0
+
+     dvsec_virt_write16(vdev, CXL_DVSEC_STATUS_OFFSET, new_val); }
+
+static void cxl_control2_write(struct vfio_pci_core_device *vdev,
+                            u16 abs_off, u16 new_val)

abs_off not needed?

Removed.

Quoted 15 lines
Quoted 15 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +14 -0
+{
+     struct pci_dev *pdev = vdev->pdev;
+     u16 cap2 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY2_OFFSET);
+     u16 cap3 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY3_OFFSET);
+     u16 rev_mask = CXL_CTRL2_RESERVED_MASK;
+     u16 hw_bits  = CXL_CTRL2_HW_BITS_MASK;
+     bool initiate_cxl_reset = new_val &
+CXL_CTRL2_INITIATE_CXL_RESET_BIT;
+
+     if (!(cap3 & CXL_CAP3_VOLATILE_HDM_BIT))
+             rev_mask |= CXL_CTRL2_VOLATILE_HDM_REV_MASK;
+     if (!(cap2 & CXL_CAP2_MODIFIED_COMPLETION_BIT))
+             rev_mask |= CXL_CTRL2_MODIFIED_COMP_REV_MASK;
+
Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 101

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +5 -0
+     new_val &= ~rev_mask;
+
+     /* Bits that go directly to hardware */
+     hw_bits &= new_val;
+

Bit 1 and 2 are always read 0 by hardware. Probably should clear it before writing to the virtual register?

Refactored this part.

Quoted 10 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +8 -0
+     dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL2_OFFSET, new_val);
+
+     if (hw_bits)
+             pci_write_config_word(pdev, abs_off, hw_bits);
+
+     if (initiate_cxl_reset) {
+             /* TODO: invoke CXL protocol reset via cxl subsystem */
+             dev_warn(&pdev->dev, "vfio-cxl: CXL reset requested but

not yet supported\n");

Quoted 7 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 114

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +5 -0
+     }
+}
+
+static void cxl_status2_write(struct vfio_pci_core_device *vdev,
+                           u16 abs_off, u16 new_val)

abs_off not needed

Removed.

Quoted 7 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 114

Quoted 6 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +5 -0
+{
+     u16 cap3 = dvsec_virt_read16(vdev,
+CXL_DVSEC_CAPABILITY3_OFFSET);
+
+     /* RW1CS: write 1 to clear, but only if the capability is

supported */

Quoted 11 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 127

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +9 -0
+     if ((cap3 & CXL_CAP3_VOLATILE_HDM_BIT) &&
+         (new_val & CXL_STATUS2_RW1CS_BIT))
+             pci_write_config_word(vdev->pdev, abs_off,
+                                   CXL_STATUS2_RW1CS_BIT);
+     /* STATUS2 is not mirrored in vconfig - reads go to hardware */
+}
+
+static void cxl_lock_write(struct vfio_pci_core_device *vdev,
+                        u16 abs_off, u16 new_val)

abs_off not needed

Removed.

Quoted 24 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +4 -0
+{
+     u16 cur_val = dvsec_virt_read16(vdev, CXL_DVSEC_LOCK_OFFSET);
+
+     /* Once the LOCK bit is set it can only be cleared by conventional

reset */

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +5 -0
+     if (cur_val & CXL_CTRL_LOCK_BIT)
+             return;
+
+     new_val &= ~CXL_LOCK_RESERVED_MASK;
+     dvsec_virt_write16(vdev, CXL_DVSEC_LOCK_OFFSET, new_val); }

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+
+static void cxl_range_base_lo_write(struct vfio_pci_core_device *vdev,
+                                 u16 dvsec_off, u32 new_val) {
+     new_val &= ~CXL_BASE_LO_RESERVED_MASK;
+     dvsec_virt_write32(vdev, dvsec_off, new_val); }
+
+/*
+ * vfio_cxl_dvsec_readfn - per-device DVSEC read handler.
+ *
+ * Called via vfio_pci_dvsec_dispatch_read() for devices that have

registered

drivers/vfio/pci/cxl/vfio_cxl_config.c line 144

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +2 -0
+ * a dvsec_readfn.  Returns shadow vconfig values for virtualized DVSEC
+ * registers (CONTROL, STATUS, CONTROL2, LOCK) so that userspace reads
Quoted 7 lines

reflect

drivers/vfio/pci/cxl/vfio_cxl_config.c line 145

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+ * the emulated state rather than the raw hardware value.  All other

DVSEC

drivers/vfio/pci/cxl/vfio_cxl_config.c line 147

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +2 -0
+ * bytes are passed through to hardware via vfio_raw_config_read().
+ */

Provide proper kdoc function header

Updated.

Quoted 38 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 12 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +11 -0
+static int vfio_cxl_dvsec_readfn(struct vfio_pci_core_device *vdev,
+                              int pos, int count,
+                              struct perm_bits *perm,
+                              int offset, __le32 *val)
+{
+     struct vfio_pci_cxl_state *cxl = vdev->cxl;
+     u16 dvsec_off;
+
+     if (!cxl || (u16)pos < cxl->dvsec ||
+         (u16)pos >= cxl->dvsec + cxl->dvsec_length)
+             return vfio_raw_config_read(vdev, pos, count, perm,

offset, val);

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +9 -0
+
+     dvsec_off = (u16)pos - cxl->dvsec;
+
+     switch (dvsec_off) {
+     case CXL_DVSEC_CONTROL_OFFSET:
+     case CXL_DVSEC_STATUS_OFFSET:
+     case CXL_DVSEC_CONTROL2_OFFSET:
+     case CXL_DVSEC_LOCK_OFFSET:
+             /* Return shadow vconfig value for virtualized registers

*/

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +9 -0
+             memcpy(val, vdev->vconfig + pos, count);
+             return count;
+     default:
+             return vfio_raw_config_read(vdev, pos, count,
+                                         perm, offset, val);
+     }> +}
+
+/*
+ * vfio_cxl_dvsec_writefn - ecap_perms write handler for

PCI_EXT_CAP_ID_DVSEC.

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +3 -0
+ *
+ * Installed once into ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn by
+ * vfio_pci_init_perm_bits() when CONFIG_VFIO_CXL_CORE=y.  Applies to

every

drivers/vfio/pci/cxl/vfio_cxl_config.c line 180

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+ * device opened under vfio-pci; the vdev->cxl NULL check distinguishes
Quoted 8 lines

CXL

drivers/vfio/pci/cxl/vfio_cxl_config.c line 181

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+ * devices from non-CXL devices that happen to expose a DVSEC

capability.

drivers/vfio/pci/cxl/vfio_cxl_config.c line 192

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +3 -0
+ *
+ * @pos:    absolute byte position in config space
+ * @offset: byte offset within the capability structure

missing return value expectations

Updated.

Quoted 73 lines
Quoted 48 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +48 -0
+ */
+static int vfio_cxl_dvsec_writefn(struct vfio_pci_core_device *vdev,
+                               int pos, int count,
+                               struct perm_bits *perm,
+                               int offset, __le32 val)
+{
+     struct vfio_pci_cxl_state *cxl = vdev->cxl;
+     u16 abs_off = (u16)pos;
+     u16 dvsec_off;
+     u16 wval16;
+     u32 wval32;
+
+     if (!cxl || (u16)pos < cxl->dvsec ||
+         (u16)pos >= cxl->dvsec + cxl->dvsec_length)
+             return vfio_raw_config_write(vdev, pos, count, perm,
+                                          offset, val);
+
+     pci_dbg(vdev->pdev,
+             "vfio_cxl: DVSEC write: abs=0x%04x dvsec_off=0x%04x "
+             "count=%d raw_val=0x%08x\n",
+             abs_off, abs_off - cxl->dvsec, count, le32_to_cpu(val));
+
+     dvsec_off = abs_off - cxl->dvsec;
+
+     /* Route to the appropriate per-register handler */
+     switch (dvsec_off) {
+     case CXL_DVSEC_CONTROL_OFFSET:
+             wval16 = (u16)le32_to_cpu(val);
+             cxl_control_write(vdev, abs_off, wval16);
+             break;
+     case CXL_DVSEC_STATUS_OFFSET:
+             wval16 = (u16)le32_to_cpu(val);
+             cxl_status_write(vdev, abs_off, wval16);
+             break;
+     case CXL_DVSEC_CONTROL2_OFFSET:
+             wval16 = (u16)le32_to_cpu(val);
+             cxl_control2_write(vdev, abs_off, wval16);
+             break;
+     case CXL_DVSEC_STATUS2_OFFSET:
+             wval16 = (u16)le32_to_cpu(val);
+             cxl_status2_write(vdev, abs_off, wval16);
+             break;
+     case CXL_DVSEC_LOCK_OFFSET:
+             wval16 = (u16)le32_to_cpu(val);
+             cxl_lock_write(vdev, abs_off, wval16);
+             break;
+     case CXL_DVSEC_RANGE1_BASE_HIGH_OFFSET:
+     case CXL_DVSEC_RANGE2_BASE_HIGH_OFFSET:

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+             wval32 = le32_to_cpu(val);
+             dvsec_virt_write32(vdev, dvsec_off, wval32);
+             break;
+     case CXL_DVSEC_RANGE1_BASE_LOW_OFFSET:
+     case CXL_DVSEC_RANGE2_BASE_LOW_OFFSET:
+             wval32 = le32_to_cpu(val);
+             cxl_range_base_lo_write(vdev, dvsec_off, wval32);
+             break;
+     default:
+             /* RO registers: header, capability, range sizes - discard

*/

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +8 -0
+             break;
+     }
+
+     return count;
+}
+
+/*
+ * vfio_cxl_setup_dvsec_perms - Install per-device CXL DVSEC read/write

hooks.

drivers/vfio/pci/cxl/vfio_cxl_config.c line 252

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +2 -0
+ *
+ * Called once per device open after vfio_config_init() has seeded

vdev->vconfig

drivers/vfio/pci/cxl/vfio_cxl_config.c line 253

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+ * from hardware.  Registers vfio_cxl_dvsec_readfn and

vfio_cxl_dvsec_writefn

Quoted 8 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 254

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+ * as the per-device DVSEC handlers.  The global dispatch functions

installed

drivers/vfio/pci/cxl/vfio_cxl_config.c line 255

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+ * in ecap_perms[PCI_EXT_CAP_ID_DVSEC] at module init call these per-

device

drivers/vfio/pci/cxl/vfio_cxl_config.c line 256

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +1 -0
+ * hooks so that pci_config_map bytes remain PCI_EXT_CAP_ID_DVSEC

throughout.

provide proper kdoc function header

Updated.

Quoted 134 lines

drivers/vfio/pci/cxl/vfio_cxl_config.c line 258

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +2 -0
+ *
+ * vfio_cxl_dvsec_readfn: returns vconfig shadow for

CONTROL/STATUS/CONTROL2/

drivers/vfio/pci/cxl/vfio_cxl_config.c line 260

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +2 -0
+ *   LOCK; passes all other DVSEC bytes through to hardware.
+ * vfio_cxl_dvsec_writefn: enforces per-register semantics (RW1C,

forced

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +3 -0
+ *   IO_ENABLE, reserved-bit masking) and stores results in vconfig.
+ *
+ * Also forces CXL.io IO_ENABLE in the CONTROL vconfig shadow so the

initial

Quoted 8 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +8 -0
+ * read returns 1 even before the first write.
+ */
+void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev)
+{
+     u16 ctrl = dvsec_virt_read16(vdev, CXL_DVSEC_CONTROL_OFFSET);
+
+     /*
+      * Register per-device DVSEC read/write handlers.  The global

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +10 -0
+      * ecap_perms[PCI_EXT_CAP_ID_DVSEC] dispatchers will call them.
+      *
+      * vfio_cxl_dvsec_readfn returns vconfig shadow values for the
+      * virtualized registers (CONTROL, STATUS, CONTROL2, LOCK) so that
+      * reads reflect emulated state rather than raw hardware.
+      *
+      * vfio_cxl_dvsec_writefn enforces per-register semantics (RW1C,
+      * forced IO_ENABLE, reserved-bit masking) and stores results in
+      * vconfig.  Because ecap_perms[DVSEC].writefn dispatches to this
+      * handler, the pci_config_map bytes remain as

PCI_EXT_CAP_ID_DVSEC

drivers/vfio/pci/cxl/vfio_cxl_config.c

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +7 -0
+      * _ no PCI_CAP_ID_INVALID_VIRT marking is needed or wanted.
+      */
+     vdev->dvsec_readfn  = vfio_cxl_dvsec_readfn;
+     vdev->dvsec_writefn = vfio_cxl_dvsec_writefn;
+
+     /*
+      * vconfig is seeded from hardware at open time.  Force IO_ENABLE

set

drivers/vfio/pci/cxl/vfio_cxl_config.c line 290

Quoted 2 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +2 -0
+      * in the CONTROL shadow so the initial read returns 1 even if the
+      * hardware reset value has it cleared.  Subsequent writes are

handled

Review

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_config.c new file +6 -0
+      * by cxl_control_write() which also forces this bit.
+      */
+     ctrl |= CXL_CTRL_CXL_IO_ENABLE_BIT;
+     dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL_OFFSET, ctrl);
+}
+EXPORT_SYMBOL_GPL(vfio_cxl_setup_dvsec_perms);
(quoted diff) +0 -0

b/drivers/vfio/pci/cxl/vfio_cxl_core.c

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_create_device_state(struct

Quoted 4 lines

index 15b6c0d75d9e..e18e992800f6 100644

drivers/vfio/pci/cxl/vfio_cxl_core.c +0 -0
@@ -26,6 +26,7 @@ static int vfio_cxl_create_device_state(struct

vfio_pci_core_device *vdev,

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,

Quoted 7 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +1 -0
      struct vfio_pci_cxl_state *cxl;
      bool cxl_mem_capable, is_cxl_type3;
      u16 cap_word;
+     u32 hdr1;
 
      /*
       * The devm allocation for the CXL state remains for the entire

time

drivers/vfio/pci/cxl/vfio_cxl_core.c static int vfio_cxl_create_device_state(struct

Quoted 1 line
drivers/vfio/pci/cxl/vfio_cxl_core.c +0 -0
@@ -47,6 +48,9 @@ static int vfio_cxl_create_device_state(struct

vfio_pci_core_device *vdev,

Review

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_core.c +3 -0
      cxl->dpa_region_idx = -1;
      cxl->comp_reg_region_idx = -1;
 
+     pci_read_config_dword(pdev, dvsec + PCI_DVSEC_HEADER1, &hdr1);
+     cxl->dvsec_length = PCI_DVSEC_HEADER1_LEN(hdr1);
+
      pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET,
                           &cap_word);
 
(quoted diff) +0 -0

b/drivers/vfio/pci/cxl/vfio_cxl_priv.h

Quoted 27 lines

index 3ef8d923a7e8..158fe4e67f98 100644

drivers/vfio/pci/cxl/vfio_cxl_priv.h +12 -1
@@ -31,6 +31,7 @@ struct vfio_pci_cxl_state {
      u32                          hdm_count;
      int                          dpa_region_idx;
      int                          comp_reg_region_idx;
+     size_t                       dvsec_length;
      u16                          dvsec;
      u8                           comp_reg_bar;
      bool                         precommitted;
@@ -76,9 +77,44 @@ struct vfio_pci_cxl_state {
  * (CXL 2.0+ 8.1.3).
  * Offsets are relative to the DVSEC capability base (cxl->dvsec).
  */
-#define CXL_DVSEC_CAPABILITY_OFFSET 0xa
+#define CXL_DVSEC_CAPABILITY_OFFSET       0xa
+#define CXL_DVSEC_CONTROL_OFFSET       0xc
+#define CXL_DVSEC_STATUS_OFFSET                0xe
+#define CXL_DVSEC_CONTROL2_OFFSET      0x10
+#define CXL_DVSEC_STATUS2_OFFSET       0x12
+#define CXL_DVSEC_LOCK_OFFSET                  0x14
+#define CXL_DVSEC_CAPABILITY2_OFFSET   0x16
+#define CXL_DVSEC_RANGE1_SIZE_HIGH_OFFSET 0x18
+#define CXL_DVSEC_RANGE1_SIZE_LOW_OFFSET  0x1c
+#define CXL_DVSEC_RANGE1_BASE_HIGH_OFFSET 0x20
+#define CXL_DVSEC_RANGE1_BASE_LOW_OFFSET  0x24

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 100

Quoted 10 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +8 -0
+#define CXL_DVSEC_RANGE2_SIZE_HIGH_OFFSET 0x28
+#define CXL_DVSEC_RANGE2_SIZE_LOW_OFFSET  0x2c
+#define CXL_DVSEC_RANGE2_BASE_HIGH_OFFSET 0x30
+#define CXL_DVSEC_RANGE2_BASE_LOW_OFFSET  0x34
+#define CXL_DVSEC_CAPABILITY3_OFFSET   0x38
+
 #define CXL_DVSEC_MEM_CAPABLE            BIT(2)
 
+/* CXL Control / Status / Lock - bit definitions */
+#define CXL_CTRL_LOCK_BIT             BIT(0)

CXL_CTRL_CONFIG_LOCK_BIT

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 104

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +4 -0
+#define CXL_CTRL_CXL_IO_ENABLE_BIT    BIT(1)
+#define CXL_CTRL2_INITIATE_CXL_RESET_BIT BIT(2)
+#define CXL_CAP3_VOLATILE_HDM_BIT     BIT(3)
+#define CXL_STATUS2_RW1CS_BIT                 BIT(3)

CXL_STATUS2_VOL_HDM_PRSV_ERR_BIT

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 107

Quoted 3 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +3 -0
+#define CXL_CAP3_P2P_BIT              BIT(4)
+#define CXL_CAP2_MODIFIED_COMPLETION_BIT BIT(6)
+#define CXL_STATUS_RW1C_BIT           BIT(14)

CXL_STATUS_VIRAL_STATUS_BIT

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state {

Quoted 5 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +5 -0
+#define CXL_CTRL_RESERVED_MASK                (BIT(13) | BIT(15))
+#define CXL_CTRL_P2P_REV_MASK                 BIT(12)
+#define CXL_STATUS_RESERVED_MASK      (GENMASK(13, 0) | BIT(15))
+#define CXL_CTRL2_RESERVED_MASK               GENMASK(15, 6)
+#define CXL_CTRL2_HW_BITS_MASK                (BIT(0) | BIT(1) |

BIT(3))

Quoted 7 lines

drivers/vfio/pci/cxl/vfio_cxl_priv.h struct vfio_pci_cxl_state { · line 116

Quoted 4 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +4 -0
+#define CXL_CTRL2_VOLATILE_HDM_REV_MASK       BIT(4)
+#define CXL_CTRL2_MODIFIED_COMP_REV_MASK BIT(5)
+#define CXL_LOCK_RESERVED_MASK                GENMASK(15, 1)
+#define CXL_BASE_LO_RESERVED_MASK     GENMASK(27, 0)

Move the CXL reg offset and bit defs to a common header. Also, please group the relevant bits together per register.

Yes, updated as per this suggestion.

Quoted 170 lines

DJ

drivers/vfio/pci/vfio_pci.c static int vfio_pci_open_device(struct vfio_device

Quoted 9 lines
drivers/vfio/pci/cxl/vfio_cxl_priv.h +1 -0
+
 int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev);
 void vfio_cxl_clean_virt_regs(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
drivers/vfio/pci/vfio_pci.c +0 -0
index d3138badeaa6..22cf9ea831f9 100644
@@ -121,12 +121,26 @@ static int vfio_pci_open_device(struct vfio_device

*core_vdev)

drivers/vfio/pci/vfio_pci.c static int vfio_pci_open_device(struct vfio_device *core_vdev)

Quoted 6 lines
drivers/vfio/pci/vfio_pci.c +3 -0
      }
 
      if (vdev->cxl) {
+             /*
+              * pci_config_map and vconfig are valid now (allocated by
+              * vfio_config_init() inside vfio_pci_core_enable()

above).

drivers/vfio/pci/vfio_pci.c static int vfio_pci_open_device(struct vfio_device *core_vdev)

Quoted 13 lines
drivers/vfio/pci/vfio_pci.c +7 -0
+              */
+             vfio_cxl_setup_dvsec_perms(vdev);
+
              ret = vfio_cxl_register_cxl_region(vdev);
              if (ret) {
                      pci_warn(pdev, "Failed to setup CXL region\n");
                      vfio_pci_core_disable(vdev);
                      return ret;
              }
+
+             ret = vfio_cxl_register_comp_regs_region(vdev);
+             if (ret) {
+                     pci_warn(pdev, "Failed to register COMP_REGS

region\n");

Review

Quoted 8 lines
drivers/vfio/pci/vfio_pci.c +4 -0
+                     vfio_cxl_unregister_cxl_region(vdev);
+                     vfio_pci_core_disable(vdev);
+                     return ret;
+             }
      }
 
      vfio_pci_core_finish_enable(vdev);
(quoted diff) +0 -0

b/drivers/vfio/pci/vfio_pci_config.c

drivers/vfio/pci/vfio_pci_config.c static int __init

Quoted 4 lines

index 79aaf270adb2..90e2c25381d6 100644

drivers/vfio/pci/vfio_pci_config.c +0 -0
@@ -1085,6 +1085,49 @@ static int __init

init_pci_ext_cap_pwr_perm(struct perm_bits *perm)

drivers/vfio/pci/vfio_pci_config.c static int __init init_pci_ext_cap_pwr_perm(struct perm_bits *perm)

Quoted 8 lines
drivers/vfio/pci/vfio_pci_config.c +5 -0
      return 0;
 }
 
+/*
+ * vfio_pci_dvsec_dispatch_read - per-device DVSEC read dispatcher.
+ *
+ * Installed as ecap_perms[PCI_EXT_CAP_ID_DVSEC].readfn at module init.
+ * Calls vdev->dvsec_readfn when a shadow-read handler has been

registered

drivers/vfio/pci/vfio_pci_config.c static int __init · line 1093

Quoted 1 line
drivers/vfio/pci/vfio_pci_config.c +1 -0
+ * (e.g. by vfio_cxl_setup_dvsec_perms() for CXL Type-2 devices),

otherwise

drivers/vfio/pci/vfio_pci_config.c static int __init init_pci_ext_cap_pwr_perm(struct perm_bits *perm)

Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +6 -0
+ * falls through to vfio_raw_config_read for hardware pass-through.
+ *
+ * This indirection allows per-device DVSEC reads from vconfig shadow
+ * without touching the global ecap_perms[] table.
+ */
+static int vfio_pci_dvsec_dispatch_read(struct vfio_pci_core_device

*vdev,

drivers/vfio/pci/vfio_pci_config.c static int __init init_pci_ext_cap_pwr_perm(struct perm_bits *perm)

Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +6 -0
+                                     int pos, int count,
+                                     struct perm_bits *perm,
+                                     int offset, __le32 *val)
+{
+     if (vdev->dvsec_readfn)
+             return vdev->dvsec_readfn(vdev, pos, count, perm, offset,

val);

drivers/vfio/pci/vfio_pci_config.c static int __init init_pci_ext_cap_pwr_perm(struct perm_bits *perm)

Quoted 7 lines
drivers/vfio/pci/vfio_pci_config.c +7 -0
+     return vfio_raw_config_read(vdev, pos, count, perm, offset, val);
+}
+
+/*
+ * vfio_pci_dvsec_dispatch_write - per-device DVSEC write dispatcher.
+ *
+ * Installed as ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn at module

init.

drivers/vfio/pci/vfio_pci_config.c static int __init · line 1113

Quoted 1 line
drivers/vfio/pci/vfio_pci_config.c +1 -0
+ * Calls vdev->dvsec_writefn when a handler has been registered for

this

drivers/vfio/pci/vfio_pci_config.c static int __init · line 1114

Quoted 1 line
drivers/vfio/pci/vfio_pci_config.c +1 -0
+ * device (e.g. by vfio_cxl_setup_dvsec_perms() for CXL Type-2

devices),

drivers/vfio/pci/vfio_pci_config.c static int __init init_pci_ext_cap_pwr_perm(struct perm_bits *perm)

Quoted 7 lines
drivers/vfio/pci/vfio_pci_config.c +7 -0
+ * otherwise falls through to vfio_raw_config_write so that non-CXL
+ * devices with a DVSEC capability continue to pass writes to hardware.
+ *
+ * This indirection allows per-device DVSEC handlers to be registered
+ * without touching the global ecap_perms[] table.
+ */
+static int vfio_pci_dvsec_dispatch_write(struct vfio_pci_core_device

*vdev,

drivers/vfio/pci/vfio_pci_config.c static int __init init_pci_ext_cap_pwr_perm(struct perm_bits *perm)

Quoted 6 lines
drivers/vfio/pci/vfio_pci_config.c +6 -0
+                                      int pos, int count,
+                                      struct perm_bits *perm,
+                                      int offset, __le32 val)
+{
+     if (vdev->dvsec_writefn)
+             return vdev->dvsec_writefn(vdev, pos, count, perm, offset,

val);

drivers/vfio/pci/vfio_pci_config.c int __init vfio_pci_init_perm_bits(void) · line 1167

Quoted 12 lines
drivers/vfio/pci/vfio_pci_config.c +4 -1
+     return vfio_raw_config_write(vdev, pos, count, perm, offset, val);
+}
+
 /*
  * Initialize the shared permission tables
  */
@@ -1121,7 +1164,8 @@ int __init vfio_pci_init_perm_bits(void)
      ret |= init_pci_ext_cap_err_perm(&ecap_perms[PCI_EXT_CAP_ID_ERR]);
      ret |= init_pci_ext_cap_pwr_perm(&ecap_perms[PCI_EXT_CAP_ID_PWR]);
      ecap_perms[PCI_EXT_CAP_ID_VNDR].writefn = vfio_raw_config_write;
-     ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn = vfio_raw_config_write;
+     ecap_perms[PCI_EXT_CAP_ID_DVSEC].readfn  =

vfio_pci_dvsec_dispatch_read;

drivers/vfio/pci/vfio_pci_config.c int __init vfio_pci_init_perm_bits(void) · line 1168

Quoted 1 line
drivers/vfio/pci/vfio_pci_config.c +1 -0
+     ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn =

vfio_pci_dvsec_dispatch_write;

Review

Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +0 -0
      if (ret)
              vfio_pci_uninit_perm_bits();
(quoted diff) +0 -0

b/drivers/vfio/pci/vfio_pci_priv.h

drivers/vfio/pci/vfio_pci_priv.h void vfio_cxl_zap_region_locked(struct

Quoted 4 lines

index f8db9a05c033..d778107fa908 100644

drivers/vfio/pci/vfio_pci_priv.h +0 -0
@@ -154,6 +154,7 @@ void vfio_cxl_zap_region_locked(struct

vfio_pci_core_device *vdev);

drivers/vfio/pci/vfio_pci_priv.h void vfio_cxl_zap_region_locked(struct · line 155

Quoted 2 lines
drivers/vfio/pci/vfio_pci_priv.h +0 -0
 void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev);
 int  vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device

*vdev);

drivers/vfio/pci/vfio_pci_priv.h vfio_cxl_register_comp_regs_region(struct

Quoted 6 lines
drivers/vfio/pci/vfio_pci_priv.h +1 -0
 void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
+void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev);
 
 #else
 
@@ -180,6 +181,8 @@ vfio_cxl_register_comp_regs_region(struct

vfio_pci_core_device *vdev)

Review

Quoted 9 lines
drivers/vfio/pci/vfio_pci_priv.h +2 -0
 { return 0; }
 static inline void
 vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev) { }
+static inline void
+vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev) { }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
 
(quoted diff) +0 -0

b/include/linux/vfio_pci_core.h

Quoted 25 lines

index cd8ed98a82a3..aa159d0c8da7 100644

include/linux/vfio_pci_core.h +7 -1
@@ -31,7 +31,7 @@ struct p2pdma_provider;
 struct dma_buf_phys_vec;
 struct dma_buf_attachment;
 struct vfio_pci_cxl_state;
-
+struct perm_bits;
 
 struct vfio_pci_eventfd {
      struct eventfd_ctx      *ctx;
@@ -141,6 +141,12 @@ struct vfio_pci_core_device {
      struct list_head        ioeventfds_list;
      struct vfio_pci_vf_token        *vf_token;
      struct vfio_pci_cxl_state *cxl;
+     int (*dvsec_readfn)(struct vfio_pci_core_device *vdev, int pos,
+                         int count, struct perm_bits *perm,
+                         int offset, __le32 *val);
+     int (*dvsec_writefn)(struct vfio_pci_core_device *vdev, int pos,
+                          int count, struct perm_bits *perm,
+                          int offset, __le32 val);
      struct list_head                sriov_pfs_item;
      struct vfio_pci_core_device     *sriov_pf_core_dev;
      struct notifier_block   nb;
#43 lore raw

PATCH 16/20 vfio/pci: Expose CXL device and region info via VFIO ioctl

In reply to [email protected]

From: Manish Honap <[email protected]>

Expose CXL device capability information through the VFIO device info ioctl and hide the CXL component BAR from direct userspace access via the standard region info path.

Add vfio_cxl_get_info() which fills a VFIO_DEVICE_INFO_CAP_CXL capability structure with HDM register location, DPA size, commit flags, and the region indices of the two CXL VFIO device regions (DPA and COMP_REGS) so userspace does not need to scan all regions.

Add vfio_cxl_get_region_info() which intercepts BAR queries for the component register BAR and returns size=0 to hide it, directing userspace to use VFIO_REGION_SUBTYPE_CXL_COMP_REGS instead.

Hook both helpers into vfio_pci_ioctl_get_info() and vfio_pci_ioctl_get_region_info() in vfio_pci_core.c.

The CXL component register BAR contains the HDM decoder MMIO registers. Userspace must use the VFIO_REGION_SUBTYPE_CXL_COMP_REGS emulated region instead of directly mapping or reading/writing this BAR, to ensure that all accesses go through the emulation layer for correct bit-field enforcement.

Reject mmap(), barmap setup, and BAR r/w for the CXL component BAR index in vfio_pci_core_mmap(), vfio_pci_core_setup_barmap(), and vfio_pci_bar_rw() respectively.

---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 84 ++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_core.c     | 16 ++++++
 drivers/vfio/pci/vfio_pci_priv.h     | 19 +++++++
 drivers/vfio/pci/vfio_pci_rdwr.c     |  8 +++
 4 files changed, 127 insertions(+)
drivers/vfio/pci/cxl/vfio_cxl_core.c +84 -0
index e18e992800f6..bda11f99746f 100644
@@ -18,6 +18,90 @@
 
 MODULE_IMPORT_NS("CXL");
 
+u8 vfio_cxl_get_component_reg_bar(struct vfio_pci_core_device *vdev)
+{
+	return vdev->cxl->comp_reg_bar;
+}
+
+int vfio_cxl_get_region_info(struct vfio_pci_core_device *vdev,
+			     struct vfio_region_info *info,
+			     struct vfio_info_cap *caps)
+{
+	unsigned long minsz = offsetofend(struct vfio_region_info, offset);
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+
+	if (!cxl)
+		return -ENOTTY;
+
+	if (!info)
+		return -ENOTTY;
+
+	if (info->argsz < minsz)
+		return -EINVAL;
+
+	if (info->index != cxl->comp_reg_bar)
+		return -ENOTTY;
+
+	/*
+	 * Hide the component BAR for CXL. Report size 0 so userspace
+	 * uses only the VFIO_REGION_SUBTYPE_CXL_COMP_REGS device region
+	 * for BAR MMIO (HDM) emulation.
+	 */
+	info->argsz = sizeof(*info);
+	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
+	info->size = 0;
+	info->flags = 0;
+	info->cap_offset = 0;
+
+	return 0;
+}
+
+int vfio_cxl_get_info(struct vfio_pci_core_device *vdev,
+		      struct vfio_info_cap *caps)
+{
+	struct vfio_pci_cxl_state *cxl = vdev->cxl;
+	struct vfio_device_info_cap_cxl cxl_cap = {0};
+
+	if (!cxl)
+		return 0;
+
+	/*
+	 * Region indices are set at open time after
+	 * vfio_pci_core_register_dev_region() succeeds.  If either is still
+	 * -1, the device is not yet fully initialised; return EAGAIN so
+	 * userspace knows to retry rather than receiving 0xFFFFFFFF.
+	 */
+	if (cxl->dpa_region_idx < 0 || cxl->comp_reg_region_idx < 0)
+		return -EAGAIN;
+
+	/* Fill in from CXL device structure */
+	cxl_cap.header.id = VFIO_DEVICE_INFO_CAP_CXL;
+	cxl_cap.header.version = 1;
+	cxl_cap.hdm_count = cxl->hdm_count;
+	cxl_cap.hdm_regs_offset = cxl->comp_reg_offset + cxl->hdm_reg_offset;
+	cxl_cap.hdm_regs_size = cxl->hdm_reg_size;
+	cxl_cap.hdm_regs_bar_index = cxl->comp_reg_bar;
+	cxl_cap.dpa_size = cxl->dpa_size;
+
+	if (cxl->precommitted) {
+		cxl_cap.flags |= VFIO_CXL_CAP_COMMITTED |
+			VFIO_CXL_CAP_PRECOMMITTED;
+	}
+
+	/*
+         * Populate absolute VFIO region indices so userspace can query them
+         * directly with VFIO_DEVICE_GET_REGION_INFO.  Custom device regions
+         * live at VFIO_PCI_NUM_REGIONS + local_idx (see vfio_pci_core.c:999).
+         * dpa_region_idx / comp_reg_region_idx are 0-based local indices, so
+         * add VFIO_PCI_NUM_REGIONS to get the index VFIO_DEVICE_GET_REGION_INFO
+         * expects.
+         */
+        cxl_cap.dpa_region_index = VFIO_PCI_NUM_REGIONS + cxl->dpa_region_idx;
+        cxl_cap.comp_regs_region_index = VFIO_PCI_NUM_REGIONS + cxl->comp_reg_region_idx;
+
+	return vfio_info_add_capability(caps, &cxl_cap.header, sizeof(cxl_cap));
+}
+
 static int vfio_cxl_create_device_state(struct vfio_pci_core_device *vdev,
 					u16 dvsec)
 {
drivers/vfio/pci/vfio_pci_core.c +16 -0
index 48e0274c19aa..5352e7810fed 100644
@@ -989,6 +989,13 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
 	if (vdev->reset_works)
 		info.flags |= VFIO_DEVICE_FLAGS_RESET;
 
+	if (vdev->cxl) {
+		ret = vfio_cxl_get_info(vdev, &caps);
+		if (ret)
+			return ret;
+		info.flags |= VFIO_DEVICE_FLAGS_CXL;
+	}
+
 	info.num_regions = VFIO_PCI_NUM_REGIONS + vdev->num_regions;
 	info.num_irqs = VFIO_PCI_NUM_IRQS;
 
@@ -1034,6 +1041,12 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
 	struct pci_dev *pdev = vdev->pdev;
 	int i, ret;
 
+	if (vdev->cxl) {
+		ret = vfio_cxl_get_region_info(vdev, info, caps);
+		if (ret != -ENOTTY)
+			return ret;
+	}
+
 	switch (info->index) {
 	case VFIO_PCI_CONFIG_REGION_INDEX:
 		info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
@@ -1756,6 +1769,9 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
 	}
 	if (index >= VFIO_PCI_ROM_REGION_INDEX)
 		return -EINVAL;
+	/* Reject mmap of CXL component BAR; use COMP_REGS region only. */
+	if (vdev->cxl && index == vfio_cxl_get_component_reg_bar(vdev))
+		return -EINVAL;
 	if (!vdev->bar_mmap_supported[index])
 		return -EINVAL;
 
drivers/vfio/pci/vfio_pci_priv.h +19 -0
index d778107fa908..c1befe7d028d 100644
@@ -156,6 +156,13 @@ int  vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev);
 void vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev);
 void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev);
 
+int vfio_cxl_get_info(struct vfio_pci_core_device *vdev,
+		      struct vfio_info_cap *caps);
+int vfio_cxl_get_region_info(struct vfio_pci_core_device *vdev,
+			     struct vfio_region_info *info,
+			     struct vfio_info_cap *caps);
+u8 vfio_cxl_get_component_reg_bar(struct vfio_pci_core_device *vdev);
+
 #else
 
 static inline void
@@ -183,6 +190,18 @@ static inline void
 vfio_cxl_reinit_comp_regs(struct vfio_pci_core_device *vdev) { }
 static inline void
 vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev) { }
+static inline int
+vfio_cxl_get_info(struct vfio_pci_core_device *vdev,
+		  struct vfio_info_cap *caps)
+{ return -ENOTTY; }
+static inline int
+vfio_cxl_get_region_info(struct vfio_pci_core_device *vdev,
+			 struct vfio_region_info *info,
+			 struct vfio_info_cap *caps)
+{ return -ENOTTY; }
+static inline u8
+vfio_cxl_get_component_reg_bar(struct vfio_pci_core_device *vdev)
+{ return U8_MAX; }
 
 #endif /* CONFIG_VFIO_CXL_CORE */
 
drivers/vfio/pci/vfio_pci_rdwr.c +8 -0
index b38627b35c35..4f1f4882265a 100644
@@ -207,6 +207,10 @@ int vfio_pci_core_setup_barmap(struct vfio_pci_core_device *vdev, int bar)
 	if (vdev->barmap[bar])
 		return 0;
 
+	/* Do not map the CXL component BAR; use COMP_REGS region only. */
+	if (vdev->cxl && bar == vfio_cxl_get_component_reg_bar(vdev))
+		return -EINVAL;
+
 	ret = pci_request_selected_regions(pdev, 1 << bar, "vfio");
 	if (ret)
 		return ret;
@@ -236,6 +240,10 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 	ssize_t done;
 	enum vfio_pci_io_width max_width = VFIO_PCI_IO_WIDTH_8;
 
+	/* Reject BAR r/w for CXL component BAR; use COMP_REGS region only. */
+	if (vdev->cxl && bar == vfio_cxl_get_component_reg_bar(vdev))
+		return -EINVAL;
+
 	if (pci_resource_start(pdev, bar))
 		end = pci_resource_len(pdev, bar);
 	else if (bar == PCI_ROM_RESOURCE && pdev->rom && pdev->romlen)
Signature · 1 line
2.25.1
#44 lore raw

PATCH 17/20 vfio/cxl: Provide opt-out for CXL feature

In reply to [email protected]

From: Manish Honap <[email protected]>

This commit provides an opt-out mechanism to disable the CXL support from vfio module. The opt-out is provided both build time and module load time.

Build time option CONFIG_VFIO_CXL_CORE is used to enable/disable CXL support in vfio-pci module.

For runtime disabling the CXL support, use the module parameter disable_cxl. This is a per-device opt-out on the core device set by the driver before registration.

---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 4 ++++
 drivers/vfio/pci/vfio_pci.c          | 9 +++++++++
 include/linux/vfio_pci_core.h        | 1 +
 3 files changed, 14 insertions(+)
drivers/vfio/pci/cxl/vfio_cxl_core.c +4 -0
index bda11f99746f..8b42ac05a110 100644
@@ -373,6 +373,10 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev)
 	u16 dvsec;
 	int ret;
 
+	/* Honor the user opt-out decision */
+	if (vdev->disable_cxl)
+		return;
+
 	if (!pcie_is_cxl(pdev))
 		return;
 
drivers/vfio/pci/vfio_pci.c +9 -0
index 22cf9ea831f9..a6b0fb882b9f 100644
@@ -60,6 +60,12 @@ static bool disable_denylist;
 module_param(disable_denylist, bool, 0444);
 MODULE_PARM_DESC(disable_denylist, "Disable use of device denylist. Disabling the denylist allows binding to devices with known errata that may lead to exploitable stability or security issues when accessed by untrusted users.");
 
+#if IS_ENABLED(CONFIG_VFIO_CXL_CORE)
+static bool disable_cxl;
+module_param(disable_cxl, bool, 0444);
+MODULE_PARM_DESC(disable_cxl, "Disable CXL Type-2 extensions for all devices bound to vfio-pci. Variant drivers may instead set vdev->disable_cxl in their probe for per-device control without needing this parameter.");
+#endif
+
 static bool vfio_pci_dev_in_denylist(struct pci_dev *pdev)
 {
 	switch (pdev->vendor) {
@@ -189,6 +195,9 @@ static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		return PTR_ERR(vdev);
 
 	dev_set_drvdata(&pdev->dev, vdev);
+#if IS_ENABLED(CONFIG_VFIO_CXL_CORE)
+	vdev->disable_cxl = disable_cxl;
+#endif
 	vdev->pci_ops = &vfio_pci_dev_ops;
 	ret = vfio_pci_core_register_device(vdev);
 	if (ret)
include/linux/vfio_pci_core.h +1 -0
index aa159d0c8da7..48dc69df52fa 100644
@@ -130,6 +130,7 @@ struct vfio_pci_core_device {
 	bool			needs_pm_restore:1;
 	bool			pm_intx_masked:1;
 	bool			pm_runtime_engaged:1;
+	bool                    disable_cxl:1;
 	struct pci_saved_state	*pci_saved_state;
 	struct pci_saved_state	*pm_save;
 	int			ioeventfds_nr;
Signature · 1 line
2.25.1
#45 lore raw

PATCH 18/20 docs: vfio-pci: Document CXL Type-2 device passthrough

In reply to [email protected]

From: Manish Honap <[email protected]>

Add a driver-api document describing the architecture, interfaces, and operational constraints of CXL Type-2 device passthrough via vfio-pci-core.

CXL Type-2 devices (cache-coherent accelerators such as GPUs with attached device memory) present unique passthrough requirements not covered by the existing vfio-pci documentation:

- The host kernel retains ownership of the HDM decoder hardware through the CXL subsystem, so the guest cannot program decoders directly.
- Two additional VFIO device regions expose the emulated HDM register state (COMP_REGS) and the DPA memory window (DPA region) to userspace.
- DVSEC configuration space writes are intercepted and virtualized so that the guest cannot alter host-owned CXL.io / CXL.mem enable bits.
- Device reset (FLR) is coordinated through vfio_pci_ioctl_reset(): all DPA PTEs are zapped before the reset and restored afterward.

---
 Documentation/driver-api/index.rst        |   1 +
 Documentation/driver-api/vfio-pci-cxl.rst | 216 ++++++++++++++++++++++
 2 files changed, 217 insertions(+)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst
Documentation/driver-api/index.rst +1 -0
index 1833e6a0687e..7ec661846f6b 100644
@@ -47,6 +47,7 @@ of interest to most developers working on device drivers.
    vfio-mediated-device
    vfio
    vfio-pci-device-specific-driver-acceptance
+   vfio-pci-cxl
 
 Bus-level documentation
 =======================
Documentation/driver-api/vfio-pci-cxl.rst new file +216 -0

216 lines not on this page · raw message

Signature · 1 line
2.25.1
#46 lore raw

Re: [PATCH 18/20] docs: vfio-pci: Document CXL Type-2 device passthrough

Quoted 18 lines · On Thu, 12 Mar 2026 02:04:38 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

Add a driver-api document describing the architecture, interfaces, and operational constraints of CXL Type-2 device passthrough via vfio-pci-core.

CXL Type-2 devices (cache-coherent accelerators such as GPUs with attached device memory) present unique passthrough requirements not covered by the existing vfio-pci documentation:

- The host kernel retains ownership of the HDM decoder hardware through the CXL subsystem, so the guest cannot program decoders directly.
- Two additional VFIO device regions expose the emulated HDM register state (COMP_REGS) and the DPA memory window (DPA region) to userspace.
- DVSEC configuration space writes are intercepted and virtualized so that the guest cannot alter host-owned CXL.io / CXL.mem enable bits.
- Device reset (FLR) is coordinated through vfio_pci_ioctl_reset(): all DPA PTEs are zapped before the reset and restored afterward.

Quoted 1 line

Hi Manish.

Great to see this doc.

Provides a convenient place to talk about the restrictions on this current patch set and how we resolve them.

My particular interest is in the region sizing as I don't see using a locked own bios setup range as a comprehensive solution.

Shall we say, there is some awareness that the CXL spec doesn't require enough information from type 2 devices and it wasn't necessarily understood that VFIO type solutions can't rely on the
"It's an accelerator so it has a custom driver, no need for standards"

It is a gap I'd like to close. Given it's being discussed in public we can prepare a Code First proposal to either add stuff to the spec or develop some external guidance on what a device needs to do if we aren't going to need either a variant driver, or device specific handling in user space.

Quoted 19 lines
---
 Documentation/driver-api/index.rst        |   1 +
 Documentation/driver-api/vfio-pci-cxl.rst | 216 ++++++++++++++++++++++
 2 files changed, 217 insertions(+)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst
Documentation/driver-api/index.rst +0 -0
index 1833e6a0687e..7ec661846f6b 100644
 
 
 Bus-level documentation
 =======================
Documentation/driver-api/vfio-pci-cxl.rst new file +0 -0
new file mode 100644
index 000000000000..f2cbe2fdb036

Documentation/driver-api/vfio-pci-cxl.rst line 36

Quoted 9 lines
Documentation/driver-api/vfio-pci-cxl.rst new file +8 -0
 
+Device Detection
+----------------
+
+CXL Type-2 detection happens automatically when ``vfio-pci`` registers a
+device that has:
+
+1. A CXL Device DVSEC capability (PCIe DVSEC Vendor ID 0x1E98, ID 0x0000).
+2. Bit 2 (Mem_Capable) set in the CXL Capability register within that DVSEC.

FWIW to be type 2 as opposed to a type 3 non class code device (e.g. the compressed memory devices Gregory Price and others are using) you need Cache_capable as well. Might be worth making this all about CXL Type-2 and non class code Type-3.

Quoted 3 lines

+3. A PCI class code that is **not** ``0x050210`` (CXL Type-3 memory device). +4. An HDM Decoder block discoverable via the Register Locator DVSEC. +5. A pre-committed HDM decoder (BIOS/firmware programmed) with non-zero size.

This is the bit that we need to make more general. Otherwise you'll have to have a bios upgrade for every type 2 device (and no native hotplug). Note native hotplug is quite likely if anyone is switch based device pooling.

I assume that you are doing this today to get something upstream and presume it works for the type 2 device you have on the host you care about. I'm not sure there are 'general' solutions but maybe there are some heuristics or sufficient conditions for establishing the size.

Type 2 might have any of:
- Conveniently preprogrammed HDM decoders (the case you use)
- Maximum of 2 HDM decoders + the same number of Range registers. In general the problem with range registers is they are a legacy feature and there are only 2 of them whereas a real device may have many more DPA ranges. In this corner case though, is it enough to give us the necessary sizes? I think it might be but would like others familiar with the spec to confirm. (If needed I'll take this to the consortium for an 'official' view).
- A DOE and table access protocol. CDAT should give us enough info to be fairly sure what is needed.
- A CXL mailbox (maybe the version in the PCI spec now) and the spec defined commands to query what is there. Reading the intro to 8.2.10.9 Memory Device Command Sets, it's a little unclear on whether these are valid on non class code devices but I believe having the appropriate Mailbox type identifier is enough to say we expect to get them.

None of this is required though and the mailboxes are non trivial. So personally I think we should propose a new DVSEC that provides any info we need for generic passthrough. Starting with what we need to get the regions right. Until something like that is in place we will have to store this info somewhere.

There is (maybe) an alternative of doing the region allocation on demand. That is emulate the HDM decoders in QEMU (on top of the emulation here) and when settings corresponding to a region setup occur, go request one from the CXL core. The problem is we can't guarantee it will be available at that time. So we can 'guess' what to provide to the VM in terms of CXL fixed memory windows, but short of heuristics (either whole of the host offer, or divide it up based on devices present
vs what is in the VM) that is going to be prone to it not being available
later.

Where do people think this should be? We are going to end up with a device list somewhere. Could be in kernel, or in QEMU or make it an orchestrator problem (applying the 'someone else's problem' solution).

| locked after Lock register bit 0 is set. |

Quoted 13 lines
Documentation/driver-api/vfio-pci-cxl.rst new file +13 -0
+
+VMM Integration Notes
+---------------------
+
+A VMM integrating CXL Type-2 passthrough should:
+
+1. Issue ``VFIO_DEVICE_GET_INFO`` and check ``VFIO_DEVICE_FLAGS_CXL``.
+2. Walk the capability chain to find ``VFIO_DEVICE_INFO_CAP_CXL`` (id = 6).
+3. Record ``dpa_region_index``, ``comp_regs_region_index``, ``dpa_size``,
+   ``hdm_count``, ``hdm_regs_offset``, and ``hdm_regs_size``.
+4. Map the DPA region (``dpa_region_index``) with mmap() to a guest physical
+   address.  The region supports ``PROT_READ | PROT_WRITE``.
+5. Open the COMP_REGS region (``comp_regs_region_index``) and attach a

Documentation/driver-api/vfio-pci-cxl.rst line 201

Quoted 10 lines
Documentation/driver-api/vfio-pci-cxl.rst new file +10 -0
+   ``notify_change`` callback to detect COMMIT transitions.  When bit 10
+   (COMMITTED) transitions from 0 to 1 in a CTRL register read, the VMM
+   should expose the corresponding DPA range to the guest and map the
+   relevant slice of the DPA mmap.
+6. For pre-committed devices (``VFIO_CXL_CAP_PRECOMMITTED`` set) the entire
+   DPA is already mapped and the VMM need not wait for a guest COMMIT.
+7. Program the guest CXL DVSEC registers (via VFIO config space write) to
+   reflect the guest's view.  The kernel emulates all register semantics
+   including the CONFIG_LOCK one-shot latch.
+

Can you share an RFC for this flow in QEMU? Ideally also a type 2 model (there have been a few posted in the past) that would allow testing this with emulated qemu as the host, then KVM / VFIO on top of that?
If not I can probably find some time to hack something together.

Thanks,

Jonathan

maintainer
#47 lore raw

Re: [PATCH 18/20] docs: vfio-pci: Document CXL Type-2 device passthrough

Quoted 86 lines · On Fri, 13 Mar 2026 12:13:41 +0000 Jonathan Cameron…
Quoted 20 lines · On Thu, 12 Mar 2026 02:04:38 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

---
 Documentation/driver-api/index.rst        |   1 +
 Documentation/driver-api/vfio-pci-cxl.rst | 216 ++++++++++++++++++++++
 2 files changed, 217 insertions(+)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst
Documentation/driver-api/index.rst +0 -0
index 1833e6a0687e..7ec661846f6b 100644
 
 
 Bus-level documentation
 =======================
Documentation/driver-api/vfio-pci-cxl.rst new file +0 -0
new file mode 100644
index 000000000000..f2cbe2fdb036

Documentation/driver-api/vfio-pci-cxl.rst line 36

Quoted 9 lines
Documentation/driver-api/vfio-pci-cxl.rst new file +8 -0
 
+Device Detection
+----------------
+
+CXL Type-2 detection happens automatically when ``vfio-pci`` registers a
+device that has:
+
+1. A CXL Device DVSEC capability (PCIe DVSEC Vendor ID 0x1E98, ID 0x0000).
+2. Bit 2 (Mem_Capable) set in the CXL Capability register within that DVSEC.  

FWIW to be type 2 as opposed to a type 3 non class code device (e.g. the compressed memory devices Gregory Price and others are using) you need Cache_capable as well. Might be worth making this all about CXL Type-2 and non class code Type-3.

Quoted 3 lines

+3. A PCI class code that is **not** ``0x050210`` (CXL Type-3 memory device). +4. An HDM Decoder block discoverable via the Register Locator DVSEC. +5. A pre-committed HDM decoder (BIOS/firmware programmed) with non-zero size.

This is the bit that we need to make more general. Otherwise you'll have to have a bios upgrade for every type 2 device (and no native hotplug). Note native hotplug is quite likely if anyone is switch based device pooling.

I assume that you are doing this today to get something upstream and presume it works for the type 2 device you have on the host you care about. I'm not sure there are 'general' solutions but maybe there are some heuristics or sufficient conditions for establishing the size.

Type 2 might have any of:
- Conveniently preprogrammed HDM decoders (the case you use)
- Maximum of 2 HDM decoders + the same number of Range registers. In general the problem with range registers is they are a legacy feature and there are only 2 of them whereas a real device may have many more DPA ranges. In this corner case though, is it enough to give us the necessary sizes? I think it might be but would like others familiar with the spec to confirm. (If needed I'll take this to the consortium for an 'official' view).
- A DOE and table access protocol. CDAT should give us enough info to be fairly sure what is needed.
- A CXL mailbox (maybe the version in the PCI spec now) and the spec defined commands to query what is there. Reading the intro to 8.2.10.9 Memory Device Command Sets, it's a little unclear on whether these are valid on non class code devices but I believe having the appropriate Mailbox type identifier is enough to say we expect to get them.

None of this is required though and the mailboxes are non trivial. So personally I think we should propose a new DVSEC that provides any info we need for generic passthrough. Starting with what we need to get the regions right. Until something like that is in place we will have to store this info somewhere.

There is (maybe) an alternative of doing the region allocation on demand. That is emulate the HDM decoders in QEMU (on top of the emulation here) and when settings corresponding to a region setup occur, go request one from the CXL core. The problem is we can't guarantee it will be available at that time. So we can 'guess' what to provide to the VM in terms of CXL fixed memory windows, but short of heuristics (either whole of the host offer, or divide it up based on devices present
vs what is in the VM) that is going to be prone to it not being available
later.

Quoted 3 lines

Where do people think this should be? We are going to end up with a device list somewhere. Could be in kernel, or in QEMU or make it an orchestrator problem (applying the 'someone else's problem' solution).

That's the typical approach. That's what we did with resizable BARs. If we cannot guarantee allocation on demand, we need to push the policy to the device, via something that indicates the size to use, or to the orchestration, via something that allows the size to be committed out-of-band. As with REBAR, we then need to be able to restrict the guest behavior to select only the configured option.

I imagine this means for the non-pre-allocated case, we need to develop some sysfs attributes that allows that out-of-band sizing, which would then appear as a fixed, pre-allocated configuration to the guest. Thanks,

Alex

#48 lore raw

Re: [PATCH 18/20] docs: vfio-pci: Document CXL Type-2 device passthrough

Quoted 100 lines · On Tue, 17 Mar 2026 15:24:45 -0600 Alex Williamson <[email protected]> wrote:
Quoted 86 lines · On Fri, 13 Mar 2026 12:13:41 +0000 Jonathan Cameron…
Quoted 20 lines · On Thu, 12 Mar 2026 02:04:38 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

---
 Documentation/driver-api/index.rst        |   1 +
 Documentation/driver-api/vfio-pci-cxl.rst | 216 ++++++++++++++++++++++
 2 files changed, 217 insertions(+)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst
Documentation/driver-api/index.rst +0 -0
index 1833e6a0687e..7ec661846f6b 100644
 
 
 Bus-level documentation
 =======================
Documentation/driver-api/vfio-pci-cxl.rst new file +0 -0
new file mode 100644
index 000000000000..f2cbe2fdb036

Documentation/driver-api/vfio-pci-cxl.rst line 36

Quoted 9 lines
Documentation/driver-api/vfio-pci-cxl.rst new file +8 -0
 
+Device Detection
+----------------
+
+CXL Type-2 detection happens automatically when ``vfio-pci`` registers a
+device that has:
+
+1. A CXL Device DVSEC capability (PCIe DVSEC Vendor ID 0x1E98, ID 0x0000).
+2. Bit 2 (Mem_Capable) set in the CXL Capability register within that DVSEC.    

FWIW to be type 2 as opposed to a type 3 non class code device (e.g. the compressed memory devices Gregory Price and others are using) you need Cache_capable as well. Might be worth making this all about CXL Type-2 and non class code Type-3.

Quoted 3 lines

+3. A PCI class code that is **not** ``0x050210`` (CXL Type-3 memory device). +4. An HDM Decoder block discoverable via the Register Locator DVSEC. +5. A pre-committed HDM decoder (BIOS/firmware programmed) with non-zero size.

This is the bit that we need to make more general. Otherwise you'll have to have a bios upgrade for every type 2 device (and no native hotplug). Note native hotplug is quite likely if anyone is switch based device pooling.

I assume that you are doing this today to get something upstream and presume it works for the type 2 device you have on the host you care about. I'm not sure there are 'general' solutions but maybe there are some heuristics or sufficient conditions for establishing the size.

Type 2 might have any of:
- Conveniently preprogrammed HDM decoders (the case you use)
- Maximum of 2 HDM decoders + the same number of Range registers. In general the problem with range registers is they are a legacy feature and there are only 2 of them whereas a real device may have many more DPA ranges. In this corner case though, is it enough to give us the necessary sizes? I think it might be but would like others familiar with the spec to confirm. (If needed I'll take this to the consortium for an 'official' view).
- A DOE and table access protocol. CDAT should give us enough info to be fairly sure what is needed.
- A CXL mailbox (maybe the version in the PCI spec now) and the spec defined commands to query what is there. Reading the intro to 8.2.10.9 Memory Device Command Sets, it's a little unclear on whether these are valid on non class code devices but I believe having the appropriate Mailbox type identifier is enough to say we expect to get them.

None of this is required though and the mailboxes are non trivial. So personally I think we should propose a new DVSEC that provides any info we need for generic passthrough. Starting with what we need to get the regions right. Until something like that is in place we will have to store this info somewhere.

There is (maybe) an alternative of doing the region allocation on demand. That is emulate the HDM decoders in QEMU (on top of the emulation here) and when settings corresponding to a region setup occur, go request one from the CXL core. The problem is we can't guarantee it will be available at that time. So we can 'guess' what to provide to the VM in terms of CXL fixed memory windows, but short of heuristics (either whole of the host offer, or divide it up based on devices present
vs what is in the VM) that is going to be prone to it not being available
later.

Quoted 3 lines

Where do people think this should be? We are going to end up with a device list somewhere. Could be in kernel, or in QEMU or make it an orchestrator problem (applying the 'someone else's problem' solution).

That's the typical approach. That's what we did with resizable BARs. If we cannot guarantee allocation on demand, we need to push the policy to the device, via something that indicates the size to use, or to the orchestration, via something that allows the size to be committed out-of-band. As with REBAR, we then need to be able to restrict the guest behavior to select only the configured option.

Quoted 4 lines

I imagine this means for the non-pre-allocated case, we need to develop some sysfs attributes that allows that out-of-band sizing, which would then appear as a fixed, pre-allocated configuration to the guest. Thanks,

I did some reading as only vaguely familiar with how the resizeable bar stuff was done. That approach should be fairly straight forward to adapt here. Stash some config in struct pci_dev before binding vfio-pci/cxl via a sysfs interface. Given that the association with the CXL infrastructure only happens later (unlike bar config) it would then be the job of the vfio-pci/cxl driver to see what was requested and attempt to set up the CXL topology to deliver it at bind time.

Manesh, would you mind hack at small PoC on top of your existing code to see if this approach shows up any problems? I don't have anything to test against right now, though could probably hack some emulation together fairly fast. I'm thinking you'll get there faster! I'm mostly focused on this cycle stuff at the moment, and I suspect we'll be discussing this for a while yet + it has dependencies on other series that aren't in yet.

I'm not sure the PCI folk will like us stashing random stuff in their structures just because we haven't bound anything yet though so have no CXL structures to use. We should probably think about how VF CXL.mem region/sub-region assignment might work as well.

Sticking to PF (well actually just function 0) passthrough for now... For the guest, we can constrain things so there is only one right option though it will limit what topologies we can build. Basically each device passed through has it's own CXL fixed memory window, it's own host bridge, it's own root port + no switches. The sizing it sees for the CFMWS matches what we configured in the host. We could program that topology up and lock it down but that means VM BIOS nastiness so I'd leave it to the native linux code to bring it up. If anyone wants to do P2P it'll get harder to do within the spec as we will have to prevent topologies that contain foot guns like the ability to configure interleave.

This constrained approach is what we plan for the CXL class code type 3 device emulation used for DCD so we've been exploring it already. It's still possible to do annoying things like zero size decoders + skip. For now we can fail HDM decoder commits if they are particularly non sensical and we haven't handled them yet - ultimately we'll probably want to minimize what we refuse to handle as I'm sure 'other OS' may not do things the same as Linux.

P2P and the fun of single device on multiple PCI heirarchies as to be solved later. As an FYI, for bandwidth, people will be building devices that interleave memory addresses over multiple root ports. Dan reminded me of that challenge last night. See bundled ports in CXL 4.0, though this particular part related to CXL.mem is actually possible prior to that stuff for CXL.cache. Oh and don't get me started on TSP / coco challenges. I take the view they are Dan's problem for now ;)

Jonathan

Quoted 2 lines

Alex

#49 lore raw

RE: [PATCH 18/20] docs: vfio-pci: Document CXL Type-2 device passthrough

Quoted 204 lines

-----Original Message-----
From: Jonathan Cameron <[email protected]>
Sent: 19 March 2026 21:37
To: Alex Williamson <[email protected]>
Cc: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 18/20] docs: vfio-pci: Document CXL Type-2 device passthrough

External email: Use caution opening links or attachments

Quoted 30 lines · On Tue, 17 Mar 2026 15:24:45 -0600 Alex Williamson <[email protected]> wrote:
Quoted 27 lines · On Fri, 13 Mar 2026 12:13:41 +0000 Jonathan Cameron…
Quoted 23 lines · On Thu, 12 Mar 2026 02:04:38 +0530 [email protected] wrote:

From: Manish Honap <[email protected]>

---
 Documentation/driver-api/index.rst        |   1 +
 Documentation/driver-api/vfio-pci-cxl.rst | 216
++++++++++++++++++++++
 2 files changed, 217 insertions(+)  create mode 100644

Documentation/driver-api/vfio-pci-cxl.rst

Documentation/driver-api/index.rst +0 -0
index 1833e6a0687e..7ec661846f6b 100644
 
 
 Bus-level documentation
 =======================
Documentation/driver-api/vfio-pci-cxl.rst new file +0 -0
new file mode 100644
index 000000000000..f2cbe2fdb036
Quoted 7 lines
Quoted 7 lines

Documentation/driver-api/vfio-pci-cxl.rst

Quoted 7 lines
Documentation/driver-api/vfio-pci-cxl.rst new file +7 -0
+Device Detection
+----------------
+
+CXL Type-2 detection happens automatically when ``vfio-pci``
+registers a device that has:
+
+1. A CXL Device DVSEC capability (PCIe DVSEC Vendor ID 0x1E98, ID

0x0000).

Quoted 1 line
Quoted 1 line
Quoted 1 line

+2. Bit 2 (Mem_Capable) set in the CXL Capability register within

that DVSEC.

Quoted 7 lines
Quoted 7 lines

FWIW to be type 2 as opposed to a type 3 non class code device (e.g. the compressed memory devices Gregory Price and others are using) you need Cache_capable as well. Might be worth making this all about CXL Type-2 and non class code Type-3.

Quoted 1 line

+3. A PCI class code that is **not** ``0x050210`` (CXL Type-3

memory device).

Quoted 1 line
Quoted 1 line
Quoted 1 line

+4. An HDM Decoder block discoverable via the Register Locator

DVSEC.

Quoted 1 line
Quoted 1 line
Quoted 1 line

+5. A pre-committed HDM decoder (BIOS/firmware programmed) with

non-zero size.

Quoted 3 lines
Quoted 3 lines

This is the bit that we need to make more general. Otherwise you'll have to have a bios upgrade for every type 2 device (and no native

hotplug).

Quoted 9 lines
Quoted 9 lines

Note native hotplug is quite likely if anyone is switch based device pooling.

I assume that you are doing this today to get something upstream and presume it works for the type 2 device you have on the host you care about. I'm not sure there are 'general' solutions but maybe there are some heuristics or sufficient conditions for establishing the size.

Quoted 4 lines
Quoted 4 lines

Type 2 might have any of:
- Conveniently preprogrammed HDM decoders (the case you use)
- Maximum of 2 HDM decoders + the same number of Range registers. In general the problem with range registers is they are a legacy

feature

Quoted 1 line
Quoted 1 line

and there are only 2 of them whereas a real device may have many

more

Quoted 1 line
Quoted 1 line

DPA ranges. In this corner case though, is it enough to give us

the

Quoted 1 line
Quoted 1 line

necessary sizes? I think it might be but would like others

familiar

Quoted 1 line
Quoted 1 line

with the spec to confirm. (If needed I'll take this to the

consortium

Quoted 2 lines
Quoted 2 lines

for an 'official' view).
- A DOE and table access protocol. CDAT should give us enough info

to

Quoted 2 lines
Quoted 2 lines

be fairly sure what is needed.
- A CXL mailbox (maybe the version in the PCI spec now) and the spec

defined

Quoted 1 line
Quoted 1 line

commands to query what is there. Reading the intro to 8.2.10.9

Memory

Quoted 1 line
Quoted 1 line

Device Command Sets, it's a little unclear on whether these are

valid on

Quoted 1 line
Quoted 1 line

non class code devices but I believe having the appropriate

Mailbox

Quoted 9 lines
Quoted 9 lines

type identifier is enough to say we expect to get them.

None of this is required though and the mailboxes are non trivial. So personally I think we should propose a new DVSEC that provides any info we need for generic passthrough. Starting with what we need to get the regions right. Until something like that is in place we will have to store this info somewhere.

There is (maybe) an alternative of doing the region allocation on

demand.

Quoted 9 lines
Quoted 9 lines

That is emulate the HDM decoders in QEMU (on top of the emulation here) and when settings corresponding to a region setup occur, go request one from the CXL core. The problem is we can't guarantee it will be available at that time. So we can 'guess' what to provide to the VM in terms of CXL fixed memory windows, but short of heuristics (either whole of the host offer, or divide it up based on devices present vs what is in the VM) that is going to be prone to it not being available later.

Quoted 3 lines
Quoted 3 lines

Where do people think this should be? We are going to end up with a device list somewhere. Could be in kernel, or in QEMU or make it an orchestrator problem (applying the 'someone else's problem'

solution).

Quoted 11 lines

That's the typical approach. That's what we did with resizable BARs. If we cannot guarantee allocation on demand, we need to push the policy to the device, via something that indicates the size to use, or to the orchestration, via something that allows the size to be committed out-of-band. As with REBAR, we then need to be able to restrict the guest behavior to select only the configured option.

I imagine this means for the non-pre-allocated case, we need to develop some sysfs attributes that allows that out-of-band sizing, which would then appear as a fixed, pre-allocated configuration to the

guest.

Quoted 1 line

Thanks,

I did some reading as only vaguely familiar with how the resizeable bar stuff was done. That approach should be fairly straight forward to adapt here.
Stash some config in struct pci_dev before binding vfio-pci/cxl via a sysfs interface. Given that the association with the CXL infrastructure only happens later (unlike bar config) it would then be the job of the vfio-pci/cxl driver to see what was requested and attempt to set up the CXL topology to deliver it at bind time.

Manesh, would you mind hack at small PoC on top of your existing code to see if this approach shows up any problems? I don't have anything to test against right now, though could probably hack some emulation together fairly fast.
I'm thinking you'll get there faster! I'm mostly focused on this cycle stuff at the moment, and I suspect we'll be discussing this for a while yet + it has dependencies on other series that aren't in yet.

I'm not sure the PCI folk will like us stashing random stuff in their structures just because we haven't bound anything yet though so have no CXL structures to use. We should probably think about how VF CXL.mem region/sub-region assignment might work as well.

Sticking to PF (well actually just function 0) passthrough for now... For the guest, we can constrain things so there is only one right option though it will limit what topologies we can build. Basically each device passed through has it's own CXL fixed memory window, it's own host bridge, it's own root port + no switches. The sizing it sees for the CFMWS matches what we configured in the host. We could program that topology up and lock it down but that means VM BIOS nastiness so I'd leave it to the native linux code to bring it up. If anyone wants to do P2P it'll get harder to do within the spec as we will have to prevent topologies that contain foot guns like the ability to configure interleave.

This constrained approach is what we plan for the CXL class code type 3 device emulation used for DCD so we've been exploring it already. It's still possible to do annoying things like zero size decoders + skip. For now we can fail HDM decoder commits if they are particularly non sensical and we haven't handled them yet - ultimately we'll probably want to minimize what we refuse to handle as I'm sure 'other OS' may not do things the same as Linux.

P2P and the fun of single device on multiple PCI heirarchies as to be solved later. As an FYI, for bandwidth, people will be building devices that interleave memory addresses over multiple root ports. Dan reminded me of that challenge last night. See bundled ports in CXL 4.0, though this particular part related to CXL.mem is actually possible prior to that stuff for CXL.cache. Oh and don't get me started on TSP / coco challenges.
I take the view they are Dan's problem for now ;)

Quoted 2 lines

Jonathan

Hello Alex, Jonathan,

I will shortly provide the updated vfio-cxl patch v2 and the QEMU RFC as per suggestions mentioned here.

Quoted 3 lines
Quoted 3 lines

Alex

#50 lore raw

PATCH 19/20 selftests/vfio: Add CXL Type-2 passthrough tests

In reply to [email protected]

From: Manish Honap <[email protected]>

Add a selftest suite exercising the CXL Type-2 device passthrough interfaces introduced by the vfio-cxl patch series.

The tests are designed to run on a host with a CXL Type-2 device bound to vfio-pci and CONFIG_VFIO_CXL_CORE=y. They verify the kernel ABI without requiring a running QEMU instance or a guest OS.

Test cases:

cxl_test_device_info:
Open the VFIO device. Call VFIO_DEVICE_GET_INFO and verify:
- VFIO_DEVICE_FLAGS_CXL is set in info.flags
- VFIO_DEVICE_FLAGS_PCI is also set (CXL implies PCI)
- info.num_regions > VFIO_PCI_NUM_REGIONS (CXL adds extra regions)
Walk the capability chain and locate VFIO_DEVICE_INFO_CAP_CXL. Verify:
- hdm_count >= 1
- hdm_regs_bar_index <= 5
- hdm_regs_size >= 0x20 * hdm_count (minimum: one decoder slot)
- dpa_size > 0 (pre-committed decoder present)
- dpa_region_index and comp_regs_region_index within bounds

cxl_test_component_bar_hidden:
Query VFIO_DEVICE_GET_REGION_INFO for the BAR index reported by hdm_regs_bar_index. Verify info.size == 0, confirming the host has hidden the component BAR from direct userspace access.

cxl_test_comp_regs_region:
Query VFIO_DEVICE_GET_REGION_INFO for comp_regs_region_index. Verify:
- flags has READ and WRITE set, mmap NOT set
- size == hdm_regs_size
Open the region fd and read 4 bytes at offset 0 (HDM Decoder Cap). Verify the read succeeds and returns a non-zero value. Attempt a misaligned read (3-byte or offset 1) _ verify EINVAL. Attempt a 4-byte write to offset 0 (RO register) _ verify it silently succeeds (write to RO discards without error per design). Write a known pattern to HDM Decoder 0 BASE_LO (offset 0x10) and read back _ verify the written value (with reserved bits cleared) is returned.

cxl_test_dpa_region:
Query VFIO_DEVICE_GET_REGION_INFO for dpa_region_index. Verify:
- flags has READ, WRITE, MMAP set
- size == dpa_size (consistent with device info)
mmap() the full DPA region (MAP_SHARED). Verify mmap() succeeds. Write a test pattern to offset 0 of the mapping (triggers first page fault / PFN insertion). Read back and verify the value. munmap() the region. Verify no crash.

cxl_test_bar_mmap_rejected:
Attempt to mmap() the component BAR directly via the standard VFIO_PCI_BAR*_REGION_INDEX path. Verify EINVAL is returned.

cxl_test_bar_read_rejected:
Attempt to read() the component BAR region fd. Verify EINVAL.

cxl_test_disable_cxl_param:
(Requires root + module reload capability)
Reload vfio-pci with disable_cxl=1. Rebind the device. Call VFIO_DEVICE_GET_INFO and verify VFIO_DEVICE_FLAGS_CXL is NOT set and num_regions == VFIO_PCI_NUM_REGIONS (no extra CXL regions). Reload vfio-pci without the parameter and rebind to restore state.

---
 tools/testing/selftests/vfio/Makefile         |   1 +
 .../selftests/vfio/vfio_cxl_type2_test.c      | 816 ++++++++++++++++++
 2 files changed, 817 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/vfio_cxl_type2_test.c
tools/testing/selftests/vfio/Makefile +1 -0
index 3c796ca99a50..2cac98302609 100644
@@ -4,6 +4,7 @@ TEST_GEN_PROGS += vfio_iommufd_setup_test
 TEST_GEN_PROGS += vfio_pci_device_test
 TEST_GEN_PROGS += vfio_pci_device_init_perf_test
 TEST_GEN_PROGS += vfio_pci_driver_test
+TEST_GEN_PROGS += vfio_cxl_type2_test
 
 TEST_FILES += scripts/cleanup.sh
 TEST_FILES += scripts/lib.sh
tools/testing/selftests/vfio/vfio_cxl_type2_test.c new file +816 -0

816 lines not on this page · raw message

Signature · 1 line
2.25.1
#51 lore raw

PATCH 20/20 selftests/vfio: Fix VLA initialisation in vfio_pci_irq_set()

In reply to [email protected]

From: Manish Honap <[email protected]>

C does not permit initialiser expressions on variable-length arrays. vfio_pci_irq_set() declared

u8 buf[sizeof(struct vfio_irq_set) + sizeof(int) * count] = {};

where count is a function parameter, making buf a VLA. GCC rejects this with "variable-sized object may not be initialized".

Replace the initialiser with an explicit memset() immediately after the declaration.

Fixes 19faf6fd969c2 ("vfio: selftests: Add a helper library for VFIO selftests")
---
 tools/testing/selftests/vfio/lib/vfio_pci_device.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/vfio/lib/vfio_pci_device.c +3 -1
index fac4c0ecadef..3258e814f450 100644
@@ -26,8 +26,10 @@
 static void vfio_pci_irq_set(struct vfio_pci_device *device,
 			     u32 index, u32 vector, u32 count, int *fds)
 {
-	u8 buf[sizeof(struct vfio_irq_set) + sizeof(int) * count] = {};
+	u8 buf[sizeof(struct vfio_irq_set) + sizeof(int) * count];
 	struct vfio_irq_set *irq = (void *)&buf;
+
+	memset(buf, 0, sizeof(buf));
 	int *irq_fds = (void *)&irq->data;
 
 	irq->argsz = sizeof(buf);
Signature · 1 line
2.25.1
maintainer
#52 lore raw

Re: [PATCH 20/20] selftests/vfio: Fix VLA initialisation in vfio_pci_irq_set()

Quoted 7 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

C does not permit initialiser expressions on variable-length arrays. vfio_pci_irq_set() declared

u8 buf[sizeof(struct vfio_irq_set) + sizeof(int) * count] = {};

Quoted 7 lines

where count is a function parameter, making buf a VLA. GCC rejects this with "variable-sized object may not be initialized".

Replace the initialiser with an explicit memset() immediately after the declaration.

Fixes 19faf6fd969c2 ("vfio: selftests: Add a helper library for VFIO selftests")

Should this fix be split out from the series and sent ahead? Does not seem to be tied to the current implementation.

DJ

Quoted 21 lines
---
 tools/testing/selftests/vfio/lib/vfio_pci_device.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/vfio/lib/vfio_pci_device.c +3 -1
index fac4c0ecadef..3258e814f450 100644
@@ -26,8 +26,10 @@
 static void vfio_pci_irq_set(struct vfio_pci_device *device,
 			     u32 index, u32 vector, u32 count, int *fds)
 {
-	u8 buf[sizeof(struct vfio_irq_set) + sizeof(int) * count] = {};
+	u8 buf[sizeof(struct vfio_irq_set) + sizeof(int) * count];
 	struct vfio_irq_set *irq = (void *)&buf;
+
+	memset(buf, 0, sizeof(buf));
 	int *irq_fds = (void *)&irq->data;
 
 	irq->argsz = sizeof(buf);
#53 lore raw

RE: [PATCH 20/20] selftests/vfio: Fix VLA initialisation in vfio_pci_irq_set()

Quoted 34 lines

-----Original Message-----
From: Dave Jiang <[email protected]>
Sent: 14 March 2026 03:54
To: Manish Honap <[email protected]>; Aniket Agashe <[email protected]>; Ankit Agrawal <[email protected]>; Alex Williamson <[email protected]>; Vikram Sethi <[email protected]>; Jason Gunthorpe <[email protected]>; Matt Ochs <[email protected]>; Shameer Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yishai Hadas <[email protected]>; [email protected]
Cc: Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 20/20] selftests/vfio: Fix VLA initialisation in vfio_pci_irq_set()

External email: Use caution opening links or attachments

Quoted 10 lines · On 3/11/26 1:34 PM, [email protected] wrote:

From: Manish Honap <[email protected]>

C does not permit initialiser expressions on variable-length arrays. vfio_pci_irq_set() declared

u8 buf[sizeof(struct vfio_irq_set) + sizeof(int) * count] = {};

where count is a function parameter, making buf a VLA. GCC rejects this with "variable-sized object may not be initialized".

Quoted 8 lines
Quoted 5 lines

Replace the initialiser with an explicit memset() immediately after the declaration.

Fixes 19faf6fd969c2 ("vfio: selftests: Add a helper library for VFIO

selftests")

Should this fix be split out from the series and sent ahead? Does not seem to be tied to the current implementation.

Yes, sent this for review to David Matlack and linux-kselftest

Quoted 25 lines

DJ

Quoted 22 lines
---
 tools/testing/selftests/vfio/lib/vfio_pci_device.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/vfio/lib/vfio_pci_device.c +3 -1
index fac4c0ecadef..3258e814f450 100644
@@ -26,8 +26,10 @@
 static void vfio_pci_irq_set(struct vfio_pci_device *device,
                           u32 index, u32 vector, u32 count, int *fds)
 {
-     u8 buf[sizeof(struct vfio_irq_set) + sizeof(int) * count] = {};
+     u8 buf[sizeof(struct vfio_irq_set) + sizeof(int) * count];
      struct vfio_irq_set *irq = (void *)&buf;
+
+     memset(buf, 0, sizeof(buf));
      int *irq_fds = (void *)&irq->data;
 
      irq->argsz = sizeof(buf);