Skip to contentSkip to conversation

hw/cxl: Passthrough HDM decoder emulation

PATCH In review Jonathan Cameron 2 participants 4 messages Feb 27, 2023

lore
Share & tools
lore
#0 lore raw

v2 cover hw/cxl: Passthrough HDM decoder emulation

v2:
- Rebase and pick up tags.
- State prereq patche sets more clearly.

Mostly sending out again because some of the precursors have been updated and to fix a typo in a tag given on v1.

Until now, testing using CXL has relied up always using two root ports below a host bridge, to work around a current assumption in the Linux kernel support that, in the single root port case, the implementation will use the allowed passthrough decoder implementation choice. If that choice is made all accesses are routed from the host bridge to the single root port that is present. Effectively we have a pass through decoder (it is called that in the kernel driver).

This patch series implements that functionality and makes it the default See patch 2 for a discussion of why I think we can make this change without backwards compatibility issues (basically if it didn't work before who are we breaking by making it work?)

Whilst this limitation has been known since the initial QEMU patch postings / kernel CXL region support, Fan Ni ran into it recently reminding me that we should solve it.

Note that if you enable hdm_for_passthrough and use a configuration that would otherwise get a passthrough decoder, the linux kernel will currently fail to set it up correctly. That's a bug / missing feature in Linux not an issue with the emulation.

Based on series "[PATCH v4 00/10] hw/cxl: CXL emulation cleanups and minor fixes for upstream" Based on series "[PATCH v6 0/8] hw/cxl: RAS error emulation and injection"

Based on: Message-Id: [email protected] Based on: Message-Id: [email protected]

Jonathan Cameron (2):
hw/pci: Add pcie_count_ds_port() and pcie_find_port_first() helpers hw/pxb-cxl: Support passthrough HDM Decoders unless overridden

 hw/cxl/cxl-host.c                   | 31 ++++++++++++--------
 hw/pci-bridge/pci_expander_bridge.c | 44 +++++++++++++++++++++++++----
 hw/pci/pcie_port.c                  | 38 +++++++++++++++++++++++++
 include/hw/cxl/cxl.h                |  1 +
 include/hw/cxl/cxl_component.h      |  1 +
 include/hw/pci/pci_bridge.h         |  1 +
 include/hw/pci/pcie_port.h          |  2 ++
 7 files changed, 101 insertions(+), 17 deletions(-)
Signature · 1 line
2.37.2
#1 lore raw

PATCH v2 1/2 hw/pci: Add pcie_count_ds_port() and pcie_find_port_first() helpers

These two helpers enable host bridges to operate differently depending on the number of downstream ports, in particular if there is only a single port.

Useful for CXL where HDM address decoders are allowed to be implicit in the host bridge if there is only a single root port.

---
 hw/pci/pcie_port.c         | 38 ++++++++++++++++++++++++++++++++++++++
 include/hw/pci/pcie_port.h |  2 ++
 2 files changed, 40 insertions(+)
hw/pci/pcie_port.c +38 -0
index 65a397ad23..1e9c485e5e 100644
@@ -161,6 +161,44 @@ PCIDevice *pcie_find_port_by_pn(PCIBus *bus, uint8_t pn)
     return NULL;
 }
 
+/* Find first port in devfn number order */
+PCIDevice *pcie_find_port_first(PCIBus *bus)
+{
+    int devfn;
+
+    for (devfn = 0; devfn < ARRAY_SIZE(bus->devices); devfn++) {
+        PCIDevice *d = bus->devices[devfn];
+
+        if (!d || !pci_is_express(d) || !d->exp.exp_cap) {
+            continue;
+        }
+
+        if (object_dynamic_cast(OBJECT(d), TYPE_PCIE_PORT)) {
+            return d;
+        }
+    }
+
+    return NULL;
+}
+
+int pcie_count_ds_ports(PCIBus *bus)
+{
+    int dsp_count = 0;
+    int devfn;
+
+    for (devfn = 0; devfn < ARRAY_SIZE(bus->devices); devfn++) {
+        PCIDevice *d = bus->devices[devfn];
+
+        if (!d || !pci_is_express(d) || !d->exp.exp_cap) {
+            continue;
+        }
+        if (object_dynamic_cast(OBJECT(d), TYPE_PCIE_PORT)) {
+            dsp_count++;
+        }
+    }
+    return dsp_count;
+}
+
 static const TypeInfo pcie_port_type_info = {
     .name = TYPE_PCIE_PORT,
     .parent = TYPE_PCI_BRIDGE,
include/hw/pci/pcie_port.h +2 -0
index 6c40e3733f..90e6cf45b8 100644
@@ -41,6 +41,8 @@ struct PCIEPort {
 void pcie_port_init_reg(PCIDevice *d);
 
 PCIDevice *pcie_find_port_by_pn(PCIBus *bus, uint8_t pn);
+PCIDevice *pcie_find_port_first(PCIBus *bus);
+int pcie_count_ds_ports(PCIBus *bus);
 
 #define TYPE_PCIE_SLOT "pcie-slot"
 OBJECT_DECLARE_SIMPLE_TYPE(PCIESlot, PCIE_SLOT)
Signature · 1 line
2.37.2
#2 lore raw

PATCH v2 2/2 hw/pxb-cxl: Support passthrough HDM Decoders unless overridden

In reply to Jonathan Cameron

The CXL r3.0 specification allows for there to be no HDM decoders on CXL Host Bridges if they have only a single root port. Instead, all accesses directed to the host bridge (as specified in CXL Fixed Memory Windows) are assumed to be routed to the single root port.

Linux currently assumes this implementation choice. So to simplify testing, make QEMU emulation also default to no HDM decoders under these particular circumstances, but provide a hdm_for_passthrough boolean option to have HDM decoders as previously.

Technically this is breaking backwards compatibility, but given the only known software stack used with the QEMU emulation is the Linux kernel and this configuration did not work before this change, there are unlikely to be any complaints that it now works. The option is retained to allow testing of software that does allow for these HDM decoders to exist, once someone writes it.

--
v2: Pick up and fix typo in tag from Fan Ni

---
 hw/cxl/cxl-host.c                   | 31 ++++++++++++--------
 hw/pci-bridge/pci_expander_bridge.c | 44 +++++++++++++++++++++++++----
 include/hw/cxl/cxl.h                |  1 +
 include/hw/cxl/cxl_component.h      |  1 +
 include/hw/pci/pci_bridge.h         |  1 +
 5 files changed, 61 insertions(+), 17 deletions(-)
hw/cxl/cxl-host.c +19 -12
index 3c1ec8732a..6e923ceeaf 100644
@@ -146,21 +146,28 @@ static PCIDevice *cxl_cfmws_find_device(CXLFixedWindow *fw, hwaddr addr)
         return NULL;
     }
 
-    hb_cstate = cxl_get_hb_cstate(hb);
-    if (!hb_cstate) {
-        return NULL;
-    }
+    if (cxl_get_hb_passthrough(hb)) {
+        rp = pcie_find_port_first(hb->bus);
+        if (!rp) {
+            return NULL;
+        }
+    } else {
+        hb_cstate = cxl_get_hb_cstate(hb);
+        if (!hb_cstate) {
+            return NULL;
+        }
 
-    cache_mem = hb_cstate->crb.cache_mem_registers;
+        cache_mem = hb_cstate->crb.cache_mem_registers;
 
-    target_found = cxl_hdm_find_target(cache_mem, addr, &target);
-    if (!target_found) {
-        return NULL;
-    }
+        target_found = cxl_hdm_find_target(cache_mem, addr, &target);
+        if (!target_found) {
+            return NULL;
+        }
 
-    rp = pcie_find_port_by_pn(hb->bus, target);
-    if (!rp) {
-        return NULL;
+        rp = pcie_find_port_by_pn(hb->bus, target);
+        if (!rp) {
+            return NULL;
+        }
     }
 
     d = pci_bridge_get_sec_bus(PCI_BRIDGE(rp))->devices[0];
hw/pci-bridge/pci_expander_bridge.c +39 -5

87 lines not on this page · raw message

include/hw/cxl/cxl.h +1 -0
index b161be59b7..b2cffbb364 100644
@@ -49,6 +49,7 @@ struct CXLHost {
     PCIHostState parent_obj;
 
     CXLComponentState cxl_cstate;
+    bool passthrough;
 };
 
 #define TYPE_PXB_CXL_HOST "pxb-cxl-host"
include/hw/cxl/cxl_component.h +1 -0
index ec4203b83f..42c7e581a7 100644
@@ -247,6 +247,7 @@ static inline hwaddr cxl_decode_ig(int ig)
 }
 
 CXLComponentState *cxl_get_hb_cstate(PCIHostState *hb);
+bool cxl_get_hb_passthrough(PCIHostState *hb);
 
 void cxl_doe_cdat_init(CXLComponentState *cxl_cstate, Error **errp);
 void cxl_doe_cdat_release(CXLComponentState *cxl_cstate);
include/hw/pci/pci_bridge.h +1 -0
index 63a7521567..81a058bb2c 100644
@@ -92,6 +92,7 @@ struct PXBDev {
     uint8_t bus_nr;
     uint16_t numa_node;
     bool bypass_iommu;
+    bool hdm_for_passthrough;
     struct cxl_dev {
         CXLHost *cxl_host_bridge; /* Pointer to a CXLHost */
     } cxl;
Signature · 1 line
2.37.2
#3 lore raw
In reply to Jonathan Cameron
Quoted 39 lines · Jonathan Cameron

v2:
- Rebase and pick up tags.
- State prereq patche sets more clearly.

Mostly sending out again because some of the precursors have been updated and to fix a typo in a tag given on v1.

Until now, testing using CXL has relied up always using two root ports below a host bridge, to work around a current assumption in the Linux kernel support that, in the single root port case, the implementation will use the allowed passthrough decoder implementation choice. If that choice is made all accesses are routed from the host bridge to the single root port that is present. Effectively we have a pass through decoder (it is called that in the kernel driver).

This patch series implements that functionality and makes it the default See patch 2 for a discussion of why I think we can make this change without backwards compatibility issues (basically if it didn't work before who are we breaking by making it work?)

Whilst this limitation has been known since the initial QEMU patch postings / kernel CXL region support, Fan Ni ran into it recently reminding me that we should solve it.

Note that if you enable hdm_for_passthrough and use a configuration that would otherwise get a passthrough decoder, the linux kernel will currently fail to set it up correctly. That's a bug / missing feature in Linux not an issue with the emulation.

Based on series "[PATCH v4 00/10] hw/cxl: CXL emulation cleanups and minor fixes for upstream" Based on series "[PATCH v6 0/8] hw/cxl: RAS error emulation and injection"

Based on: Message-Id: [email protected] Based on: Message-Id: [email protected]

Jonathan Cameron (2):
hw/pci: Add pcie_count_ds_port() and pcie_find_port_first() helpers hw/pxb-cxl: Support passthrough HDM Decoders unless overridden

Quoted 13 lines
 hw/cxl/cxl-host.c                   | 31 ++++++++++++--------
 hw/pci-bridge/pci_expander_bridge.c | 44 +++++++++++++++++++++++++----
 hw/pci/pcie_port.c                  | 38 +++++++++++++++++++++++++
 include/hw/cxl/cxl.h                |  1 +
 include/hw/cxl/cxl_component.h      |  1 +
 include/hw/pci/pci_bridge.h         |  1 +
 include/hw/pci/pcie_port.h          |  2 ++
 7 files changed, 101 insertions(+), 17 deletions(-)
Signature · 1 line
2.37.2

The code looks good for me.

Applied the two patch series mentioned above before applying this patch series on ToT, it is clean.
Tested to crete region, create namespace and convert to system RAM and use it with simple command (numactl --membind=1 htop), no issue related to the patch has been found.

Also tried with the latest volatile patch (Message-ID:
[email protected]), creating volatile region and converting to system RAM (automatically) and online and test it with the same command as for pmem above works fine.

Fan