hw/cxl: Passthrough HDM decoder emulation
cover hw/cxl: Passthrough HDM decoder emulation
Changes since RFC: (Thanks to Fan Ni)
- Fix trivial whitespace and long line issues.
Until now, testing using CXL has relied up always using two root ports below a host bridge, to work around a current assumption in the Linux kernel support that, in the single root port case, the implementation will use the allowed passthrough decoder implementation choice. If that choice is made all accesses are routed from the host bridge to the single root port that is present. Effectively we have a pass through decoder (it is called that in the kernel driver).
This patch series implements that functionality and makes it the default See patch 2 for a discussion of why I think we can make this change without backwards compatibility issues (basically if it didn't work before who are we breaking by making it work?)
Whilst this limitation has been known since the initial QEMU patch postings / kernel CXL region support, Fan Ni Ran into it recently reminding me that we should solve it.
Based on top of:
https://lore.kernel.org/linux-cxl/[email protected]/
[PATCH v2 0/7] hw/cxl: RAS error emulation and injection
which is in turn on top of:
https://lore.kernel.org/all/[email protected]/
[PATCH v2 0/8] hw/cxl: CXL emulation cleanups and minor fixes for upstream
Jonathan Cameron (2):
hw/pci: Add pcie_count_ds_port() and pcie_find_port_first() helpers hw/pxb-cxl: Support passthrough HDM Decoders unless overridden
hw/cxl/cxl-host.c | 31 ++++++++++++-------- hw/pci-bridge/pci_expander_bridge.c | 44 +++++++++++++++++++++++++---- hw/pci/pcie_port.c | 38 +++++++++++++++++++++++++ include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_component.h | 1 + include/hw/pci/pci_bridge.h | 1 + include/hw/pci/pcie_port.h | 2 ++ 7 files changed, 101 insertions(+), 17 deletions(-)
Signature · 1 line
2.37.2
PATCH 1/2 hw/pci: Add pcie_count_ds_port() and pcie_find_port_first() helpers
These two helpers enable host bridges to operate differently depending on the number of downstream ports, in particular if there is only a single port.
Useful for CXL where HDM address decoders are allowed to be implicit in the host bridge if there is only a single root port.
--- hw/pci/pcie_port.c | 38 ++++++++++++++++++++++++++++++++++++++ include/hw/pci/pcie_port.h | 2 ++ 2 files changed, 40 insertions(+)
hw/pci/pcie_port.c +38 -0
@@ -161,6 +161,44 @@ PCIDevice *pcie_find_port_by_pn(PCIBus *bus, uint8_t pn)return NULL ; } +/* Find first port in devfn number order */ +PCIDevice *pcie_find_port_first (PCIBus *bus) +{ +int devfn; + +for (devfn =0 ; devfn <ARRAY_SIZE (bus->devices); devfn++) { +PCIDevice *d = bus->devices[devfn]; + +if (!d || !pci_is_express (d) || !d->exp.exp_cap) { +continue ; + } + +if (object_dynamic_cast (OBJECT (d),TYPE_PCIE_PORT )) { +return d; + } + } + +return NULL ; +} + +int pcie_count_ds_ports (PCIBus *bus) +{ +int dsp_count =0 ; +int devfn; + +for (devfn =0 ; devfn <ARRAY_SIZE (bus->devices); devfn++) { +PCIDevice *d = bus->devices[devfn]; + +if (!d || !pci_is_express (d) || !d->exp.exp_cap) { +continue ; + } +if (object_dynamic_cast (OBJECT (d),TYPE_PCIE_PORT )) { + dsp_count++; + } + } +return dsp_count; +} +static const TypeInfo pcie_port_type_info = { .name =TYPE_PCIE_PORT , .parent =TYPE_PCI_BRIDGE ,
include/hw/pci/pcie_port.h +2 -0
@@ -41,6 +41,8 @@ struct PCIEPort {void pcie_port_init_reg (PCIDevice *d);PCIDevice *pcie_find_port_by_pn (PCIBus *bus,uint8_t pn); +PCIDevice *pcie_find_port_first (PCIBus *bus); +int pcie_count_ds_ports (PCIBus *bus);#define TYPE_PCIE_SLOT "pcie-slot"OBJECT_DECLARE_SIMPLE_TYPE (PCIESlot,PCIE_SLOT )
Signature · 1 line
2.37.2
PATCH 2/2 hw/pxb-cxl: Support passthrough HDM Decoders unless overridden
In reply to Jonathan CameronThe CXL r3.0 specification allows for there to be no HDM decoders on CXL Host Bridges if they have only a single root port. Instead, all accesses directed to the host bridge (as specified in CXL Fixed Memory Windows) are assumed to be routed to the single root port.
Linux currently assumes this implementation choice. So to simplify testing, make QEMU emulation also default to no HDM decoders under these particular circumstances, but provide a hdm_for_passthrough boolean option to have HDM decoders as previously.
Technically this is breaking backwards compatibility, but given the only known software stack used with the QEMU emulation is the Linux kernel and this configuration did not work before this change, there are unlikely to be any complaints that it now works. The option is retained to allow testing of software that does allow for these HDM decoders to exist, once someone writes it.
--- hw/cxl/cxl-host.c | 31 ++++++++++++-------- hw/pci-bridge/pci_expander_bridge.c | 44 +++++++++++++++++++++++++---- include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_component.h | 1 + include/hw/pci/pci_bridge.h | 1 + 5 files changed, 61 insertions(+), 17 deletions(-)
hw/cxl/cxl-host.c +19 -12
@@ -146,21 +146,28 @@ static PCIDevice *cxl_cfmws_find_device(CXLFixedWindow *fw, hwaddr addr)return NULL ; } - hb_cstate =cxl_get_hb_cstate (hb); -if (!hb_cstate) { -return NULL ; - } +if (cxl_get_hb_passthrough (hb)) { + rp =pcie_find_port_first (hb->bus); +if (!rp) { +return NULL ; + } + }else { + hb_cstate =cxl_get_hb_cstate (hb); +if (!hb_cstate) { +return NULL ; + } - cache_mem = hb_cstate->crb.cache_mem_registers; + cache_mem = hb_cstate->crb.cache_mem_registers; - target_found =cxl_hdm_find_target (cache_mem, addr, &target); -if (!target_found) { -return NULL ; - } + target_found =cxl_hdm_find_target (cache_mem, addr, &target); +if (!target_found) { +return NULL ; + } - rp =pcie_find_port_by_pn (hb->bus, target); -if (!rp) { -return NULL ; + rp =pcie_find_port_by_pn (hb->bus, target); +if (!rp) { +return NULL ; + } } d =pci_bridge_get_sec_bus (PCI_BRIDGE (rp))->devices[0 ];
hw/pci-bridge/pci_expander_bridge.c +39 -5
87 lines not on this page · raw message
include/hw/cxl/cxl.h +1 -0
@@ -49,6 +49,7 @@ struct CXLHost {PCIHostState parent_obj;CXLComponentState cxl_cstate; +bool passthrough; };#define TYPE_PXB_CXL_HOST "pxb-cxl-host"
include/hw/cxl/cxl_component.h +1 -0
@@ -249,6 +249,7 @@ static inline hwaddr cxl_decode_ig(int ig) }CXLComponentState *cxl_get_hb_cstate (PCIHostState *hb); +bool cxl_get_hb_passthrough (PCIHostState *hb);void cxl_doe_cdat_init (CXLComponentState *cxl_cstate,Error **errp);void cxl_doe_cdat_release (CXLComponentState *cxl_cstate);
include/hw/pci/pci_bridge.h +1 -0
@@ -92,6 +92,7 @@ struct PXBDev {uint8_t bus_nr;uint16_t numa_node;bool bypass_iommu; +bool hdm_for_passthrough;struct cxl_dev {CXLHost *cxl_host_bridge;/* Pointer to a CXLHost */ } cxl;
Signature · 1 line
2.37.2
Re: [PATCH 2/2] hw/pxb-cxl: Support passthrough HDM Decoders unless overridden
Quoted 137 lines · Jonathan Cameron
The CXL r3.0 specification allows for there to be no HDM decoders on CXL Host Bridges if they have only a single root port. Instead, all accesses directed to the host bridge (as specified in CXL Fixed Memory Windows) are assumed to be routed to the single root port.
Linux currently assumes this implementation choice. So to simplify testing, make QEMU emulation also default to no HDM decoders under these particular circumstances, but provide a
hdm_for_passthroughboolean option to have HDM decoders as previously.Technically this is breaking backwards compatibility, but given the only known software stack used with the QEMU emulation is the Linux kernel and this configuration did not work before this change, there are unlikely to be any complaints that it now works. The option is retained to allow testing of software that does allow for these HDM decoders to exist, once someone writes it.
Reported-by Fan Ni <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-host.c | 31 ++++++++++++-------- hw/pci-bridge/pci_expander_bridge.c | 44 +++++++++++++++++++++++++---- include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_component.h | 1 + include/hw/pci/pci_bridge.h | 1 + 5 files changed, 61 insertions(+), 17 deletions(-)hw/cxl/cxl-host.c +19
-12@@ -146,21 +146,28 @@ static PCIDevice *cxl_cfmws_find_device(CXLFixedWindow *fw, hwaddr addr)return NULL ; } - hb_cstate =cxl_get_hb_cstate (hb); -if (!hb_cstate) { -return NULL ; - } +if (cxl_get_hb_passthrough (hb)) { + rp =pcie_find_port_first (hb->bus); +if (!rp) { +return NULL ; + } + }else { + hb_cstate =cxl_get_hb_cstate (hb); +if (!hb_cstate) { +return NULL ; + } - cache_mem = hb_cstate->crb.cache_mem_registers; + cache_mem = hb_cstate->crb.cache_mem_registers; - target_found =cxl_hdm_find_target (cache_mem, addr, &target); -if (!target_found) { -return NULL ; - } + target_found =cxl_hdm_find_target (cache_mem, addr, &target); +if (!target_found) { +return NULL ; + } - rp =pcie_find_port_by_pn (hb->bus, target); -if (!rp) { -return NULL ; + rp =pcie_find_port_by_pn (hb->bus, target); +if (!rp) { +return NULL ; + } } d =pci_bridge_get_sec_bus (PCI_BRIDGE (rp))->devices[0 ];hw/pci-bridge/pci_expander_bridge.c +27
-2@@ -15,6 +15,7 @@#include "hw/pci/pci.h" #include "hw/pci/pci_bus.h" #include "hw/pci/pci_host.h" +#include "hw/pci/pcie_port.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h" #include "hw/pci-bridge/pci_expander_bridge.h" @@ -79,6 +80,13 @@ CXLComponentState *cxl_get_hb_cstate(PCIHostState *hb)return &host->cxl_cstate; } +bool cxl_get_hb_passthrough (PCIHostState *hb) +{ +CXLHost *host =PXB_CXL_HOST (hb); + +return host->passthrough; +} +static int pxb_bus_num (PCIBus *bus) {PXBDev *pxb =convert_to_pxb (bus->parent_dev);@@ -289,15 +297,32 @@ static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin)return pin -PCI_SLOT (pxb->devfn); } -static void pxb_dev_reset (DeviceState *dev) +static void pxb_cxl_dev_reset (DeviceState *dev) {CXLHost *cxl =PXB_CXL_DEV (dev)->cxl.cxl_host_bridge;CXLComponentState *cxl_cstate = &cxl->cxl_cstate; +PCIHostState *hb =PCI_HOST_BRIDGE (cxl);uint32_t *reg_state = cxl_cstate->crb.cache_mem_registers;uint32_t *write_msk = cxl_cstate->crb.cache_mem_regs_write_mask; +int dsp_count =0 ;cxl_component_register_init_common (reg_state, write_msk,CXL2_ROOT_PORT ); -ARRAY_FIELD_DP32 (reg_state,CXL_HDM_DECODER_CAPABILITY ,TARGET_COUNT ,8 ); +/* +* The CXL specification allows for host bridges with no HDM decoders +* if they only have a single root port. +*/ +if (!PXB_DEV (dev)->hdm_for_passthrough) { + dsp_count =pcie_count_ds_ports (hb->bus); + } +/* Initial reset will have 0 dsp so wait until > 0 */ +if (dsp_count ==1 ) { + cxl->passthrough = true; +/* Set Capability ID in header to NONE */ +ARRAY_FIELD_DP32 (reg_state,CXL_HDM_CAPABILITY_HEADER ,ID ,0 ); + }else { +ARRAY_FIELD_DP32 (reg_state,CXL_HDM_DECODER_CAPABILITY ,TARGET_COUNT , +8 ); + } }static gint pxb_compare (gconstpointer a,gconstpointer b)@@ -481,9 +506,18 @@ static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp) }pxb_dev_realize_common (dev,CXL , errp);
hw/pci-bridge/pci_expander_bridge.c static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp) · line 517
Quoted 10 lines
hw/pci-bridge/pci_expander_bridge.c +7
-1-pxb_dev_reset (DEVICE (dev)); +pxb_cxl_dev_reset (DEVICE (dev)); } +static Property pxb_cxl_dev_properties[] = { +/* Note: 0 is not a legal PXB bus number. */ +DEFINE_PROP_UINT8 ("bus_nr" , PXBDev, bus_nr,0 ), +DEFINE_PROP_UINT16 ("numa_node" , PXBDev, numa_node,NUMA_NODE_UNASSIGNED ), +DEFINE_PROP_BOOL ("bypass_iommu" , PXBDev, bypass_iommu, false), +DEFINE_PROP_BOOL ("hdm_for_passthrough" , PXBDev, hdm_for_passthrough, false),
when setting hdm_for_passthrough to true at the qemu command line, we will see the segfault issue as before. I think this is expected as it is the logic in cxl_cfmws_find_device. Wondering if there will be following fixes to handle the case when hdm_for_passthrough is true.
Quoted 61 lines
hw/pci-bridge/pci_expander_bridge.c +5
-2+ DEFINE_PROP_END_OF_LIST(), +}; +static void pxb_cxl_dev_class_init (ObjectClass *klass,void *data) {DeviceClass *dc =DEVICE_CLASS (klass);@@ -497,12 +531,12 @@ static void pxb_cxl_dev_class_init(ObjectClass *klass, void *data)*/ dc->desc ="CXL Host Bridge" ; -device_class_set_props (dc, pxb_dev_properties); +device_class_set_props (dc, pxb_cxl_dev_properties);set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories);/* Host bridges aren't hotpluggable. FIXME: spec reference */ dc->hotpluggable = false; - dc->reset = pxb_dev_reset; + dc->reset = pxb_cxl_dev_reset; }static const TypeInfo pxb_cxl_dev_info = {include/hw/cxl/cxl.h +1
-0@@ -49,6 +49,7 @@ struct CXLHost {PCIHostState parent_obj;CXLComponentState cxl_cstate; +bool passthrough; };#define TYPE_PXB_CXL_HOST "pxb-cxl-host"include/hw/cxl/cxl_component.h +1
-0@@ -249,6 +249,7 @@ static inline hwaddr cxl_decode_ig(int ig) }CXLComponentState *cxl_get_hb_cstate (PCIHostState *hb); +bool cxl_get_hb_passthrough (PCIHostState *hb);void cxl_doe_cdat_init (CXLComponentState *cxl_cstate,Error **errp);void cxl_doe_cdat_release (CXLComponentState *cxl_cstate);include/hw/pci/pci_bridge.h +1
-0@@ -92,6 +92,7 @@ struct PXBDev {uint8_t bus_nr;uint16_t numa_node;bool bypass_iommu; +bool hdm_for_passthrough;struct cxl_dev {CXLHost *cxl_host_bridge;/* Pointer to a CXLHost */ } cxl;Signature · 1 line
2.37.2
Re: [PATCH 2/2] hw/pxb-cxl: Support passthrough HDM Decoders unless overridden
Quoted 33 lines · On Thu, 26 Jan 2023 21:57:35 +0000 Fan Ni <[email protected]> wrote:
Quoted 21 lines · Jonathan Cameron
The CXL r3.0 specification allows for there to be no HDM decoders on CXL Host Bridges if they have only a single root port. Instead, all accesses directed to the host bridge (as specified in CXL Fixed Memory Windows) are assumed to be routed to the single root port.
Linux currently assumes this implementation choice. So to simplify testing, make QEMU emulation also default to no HDM decoders under these particular circumstances, but provide a
hdm_for_passthroughboolean option to have HDM decoders as previously.Technically this is breaking backwards compatibility, but given the only known software stack used with the QEMU emulation is the Linux kernel and this configuration did not work before this change, there are unlikely to be any complaints that it now works. The option is retained to allow testing of software that does allow for these HDM decoders to exist, once someone writes it.
Reported-by Fan Ni <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>pxb_dev_realize_common(dev, CXL, errp);hw/pci-bridge/pci_expander_bridge.c static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp) · line 517
Quoted 10 lines
hw/pci-bridge/pci_expander_bridge.c +7
-1-pxb_dev_reset (DEVICE (dev)); +pxb_cxl_dev_reset (DEVICE (dev)); } +static Property pxb_cxl_dev_properties[] = { +/* Note: 0 is not a legal PXB bus number. */ +DEFINE_PROP_UINT8 ("bus_nr" , PXBDev, bus_nr,0 ), +DEFINE_PROP_UINT16 ("numa_node" , PXBDev, numa_node,NUMA_NODE_UNASSIGNED ), +DEFINE_PROP_BOOL ("bypass_iommu" , PXBDev, bypass_iommu, false), +DEFINE_PROP_BOOL ("hdm_for_passthrough" , PXBDev, hdm_for_passthrough, false),
Quoted 4 lines
when setting
hdm_for_passthroughto true at the qemu command line, we will see the segfault issue as before. I think this is expected as it is the logic incxl_cfmws_find_device. Wondering if there will be following fixes to handle the case whenhdm_for_passthroughis true.
Absolutely, I'd expect a kernel fix for that case, but it's probably not high priority for anyone given we don't yet have any hardware that does that (as far as I know anyway!)
I wanted to keep the control here to make that easy to test when we do have the fix in place.
Jonathan
Fan Ni
Tested-byRe: [PATCH 2/2] hw/pxb-cxl: Support passthrough HDM Decoders unless overridden
Quoted 41 lines · Jonathan Cameron
Quoted 33 lines · On Thu, 26 Jan 2023 21:57:35 +0000 Fan Ni <[email protected]> wrote:
Quoted 21 lines · Jonathan Cameron
The CXL r3.0 specification allows for there to be no HDM decoders on CXL Host Bridges if they have only a single root port. Instead, all accesses directed to the host bridge (as specified in CXL Fixed Memory Windows) are assumed to be routed to the single root port.
Linux currently assumes this implementation choice. So to simplify testing, make QEMU emulation also default to no HDM decoders under these particular circumstances, but provide a
hdm_for_passthroughboolean option to have HDM decoders as previously.Technically this is breaking backwards compatibility, but given the only known software stack used with the QEMU emulation is the Linux kernel and this configuration did not work before this change, there are unlikely to be any complaints that it now works. The option is retained to allow testing of software that does allow for these HDM decoders to exist, once someone writes it.
Reported-by Fan Ni <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>pxb_dev_realize_common(dev, CXL, errp);hw/pci-bridge/pci_expander_bridge.c static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp) · line 517
Quoted 10 lines
hw/pci-bridge/pci_expander_bridge.c +7
-1-pxb_dev_reset (DEVICE (dev)); +pxb_cxl_dev_reset (DEVICE (dev)); } +static Property pxb_cxl_dev_properties[] = { +/* Note: 0 is not a legal PXB bus number. */ +DEFINE_PROP_UINT8 ("bus_nr" , PXBDev, bus_nr,0 ), +DEFINE_PROP_UINT16 ("numa_node" , PXBDev, numa_node,NUMA_NODE_UNASSIGNED ), +DEFINE_PROP_BOOL ("bypass_iommu" , PXBDev, bypass_iommu, false), +DEFINE_PROP_BOOL ("hdm_for_passthrough" , PXBDev, hdm_for_passthrough, false),Quoted 4 lines
when setting
hdm_for_passthroughto true at the qemu command line, we will see the segfault issue as before. I think this is expected as it is the logic incxl_cfmws_find_device. Wondering if there will be following fixes to handle the case whenhdm_for_passthroughis true.
Quoted 8 lines
Absolutely, I'd expect a kernel fix for that case, but it's probably not high priority for anyone given we don't yet have any hardware that does that (as far as I know anyway!)
I wanted to keep the control here to make that easy to test when we do have the fix in place.
Jonathan
Make sense!
Re: [PATCH 2/2] hw/pxb-cxl: Support passthrough HDM Decoders unless overridden
Quoted 47 lines · On Fri, 27 Jan 2023 17:02:36 +0000 Fan Ni <[email protected]> wrote:
Quoted 45 lines · Jonathan Cameron
Quoted 33 lines · On Thu, 26 Jan 2023 21:57:35 +0000 Fan Ni <[email protected]> wrote:
Quoted 21 lines · Jonathan Cameron
The CXL r3.0 specification allows for there to be no HDM decoders on CXL Host Bridges if they have only a single root port. Instead, all accesses directed to the host bridge (as specified in CXL Fixed Memory Windows) are assumed to be routed to the single root port.
Linux currently assumes this implementation choice. So to simplify testing, make QEMU emulation also default to no HDM decoders under these particular circumstances, but provide a
hdm_for_passthroughboolean option to have HDM decoders as previously.Technically this is breaking backwards compatibility, but given the only known software stack used with the QEMU emulation is the Linux kernel and this configuration did not work before this change, there are unlikely to be any complaints that it now works. The option is retained to allow testing of software that does allow for these HDM decoders to exist, once someone writes it.
Reported-by Fan Ni <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>pxb_dev_realize_common(dev, CXL, errp);hw/pci-bridge/pci_expander_bridge.c static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp) · line 517
Quoted 10 lines
hw/pci-bridge/pci_expander_bridge.c +7
-1-pxb_dev_reset (DEVICE (dev)); +pxb_cxl_dev_reset (DEVICE (dev)); } +static Property pxb_cxl_dev_properties[] = { +/* Note: 0 is not a legal PXB bus number. */ +DEFINE_PROP_UINT8 ("bus_nr" , PXBDev, bus_nr,0 ), +DEFINE_PROP_UINT16 ("numa_node" , PXBDev, numa_node,NUMA_NODE_UNASSIGNED ), +DEFINE_PROP_BOOL ("bypass_iommu" , PXBDev, bypass_iommu, false), +DEFINE_PROP_BOOL ("hdm_for_passthrough" , PXBDev, hdm_for_passthrough, false),Quoted 4 lines
when setting
hdm_for_passthroughto true at the qemu command line, we will see the segfault issue as before. I think this is expected as it is the logic incxl_cfmws_find_device. Wondering if there will be following fixes to handle the case whenhdm_for_passthroughis true.Absolutely, I'd expect a kernel fix for that case, but it's probably not high priority for anyone given we don't yet have any hardware that does that (as far as I know anyway!)
Quoted 7 lines
Quoted 3 lines
I wanted to keep the control here to make that easy to test when we do have the fix in place.
JonathanMake sense!
Reviwed-by Fan Ni <[email protected]>Tested-by Fan Ni <[email protected]>
Thanks!
Typo in Reviewed-by If this version gets picked up hopefully Michael can fix whilst applying. If I do a v2 for other feedback I'll fix it.
Jonathan