hw/pcie: Multi-root support for Q35
v2 cover hw/pcie: Multi-root support for Q35
The PXB host bridge provides a way to have multiple PCI hierarchies (PCI root buses). This series is porting the PXB device to Q35 machines by:
- removing the internal pci-bridge
- make the PXB bus PCIe for Q35, while it remains PCI for i440fx.
This approach works because the Root Complexes are exposed to guest as regular (legacy) opaque PCI host bridges.
Tested on Fedora and Windows guests with both Root Ports and PCIe Switches.
v1 -> v2:
Addressed Gerd Hoffmann comments:
- Added x-enable-internal-bridge compat property to keep the PCI bridge for older machine to avoid breaking migration.
Thanks,
Marcel
Marcel Apfelbaum (4):
hw/pxb: remove the built-in pci bridge
hw/acpi: merge pxb adjacent memory/IO ranges
hw/pc: query both q35 and i440fx bus
hw/pxb: add support for PCIe
hw/i386/acpi-build.c | 126 +++++++++++++++++++++--------------- hw/i386/pc.c | 2 +- hw/pci-bridge/pci_expander_bridge.c | 45 ++++++++++--- hw/pci-host/q35.c | 8 +++ include/hw/compat.h | 4 ++ include/hw/i386/pc.h | 9 +++ 6 files changed, 133 insertions(+), 61 deletions(-)
Signature · 1 line
2.1.0
PATCH v2 1/4 hw/pxb: remove the built-in pci bridge
As part of porting the pxb device to Q35 remove the internal pci-2-pci bridge. The only way to hot-pug devices on the extra PCI root buses is by adding a pci-2-pci to the pxb before the firmware assign the IO/mem resources.
Keep the internal bridge for existing machines.
--- hw/pci-bridge/pci_expander_bridge.c | 27 +++++++++++++++++++-------- include/hw/compat.h | 4 ++++ 2 files changed, 23 insertions(+), 8 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +19 -8
@@ -34,6 +34,9 @@ typedef struct PXBBus {#define TYPE_PXB_DEVICE "pxb"#define PXB_DEV (obj) OBJECT_CHECK(PXBDev, (obj), TYPE_PXB_DEVICE) +#define PXB_FLAG_ENABLE_BRIDGE_BIT 0 +#define PXB_FLAG_ENABLE_BRIDGE (1 << PXB_FLAG_ENABLE_BRIDGE_BIT) + typedef struct PXBDev { /*< private >*/ PCIDevice parent_obj;
@@ -41,6 +44,7 @@ typedef struct PXBDev {uint8_t bus_nr;uint16_t numa_node; +uint32_t flags; } PXBDev;static GList *pxb_dev_list;
@@ -196,7 +200,7 @@ static gint pxb_compare(gconstpointer a, gconstpointer b)static int pxb_dev_initfn (PCIDevice *dev) {PXBDev *pxb =PXB_DEV (dev); -DeviceState *ds, *bds; +DeviceState *ds, *bds =NULL ;PCIBus *bus;const char *dev_name =NULL ;
@@ -211,18 +215,21 @@ static int pxb_dev_initfn(PCIDevice *dev) } ds =qdev_create (NULL ,TYPE_PXB_HOST ); - bus =pci_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS ); +if (pxb->flags &PXB_FLAG_ENABLE_BRIDGE ) { + bus =pci_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS ); + bds =qdev_create (BUS (bus),"pci-bridge" ); + bds->id = dev_name; +qdev_prop_set_uint8 (bds,PCI_BRIDGE_DEV_PROP_CHASSIS_NR , pxb->bus_nr); +qdev_prop_set_bit (bds,PCI_BRIDGE_DEV_PROP_SHPC , false); + }else { + bus =pci_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_BUS ); + } bus->parent_dev = dev; bus->address_space_mem = dev->bus->address_space_mem; bus->address_space_io = dev->bus->address_space_io; bus->map_irq = pxb_map_irq_fn; - bds =qdev_create (BUS (bus),"pci-bridge" ); - bds->id = dev_name; -qdev_prop_set_uint8 (bds,PCI_BRIDGE_DEV_PROP_CHASSIS_NR , pxb->bus_nr); -qdev_prop_set_bit (bds,PCI_BRIDGE_DEV_PROP_SHPC , false); -PCI_HOST_BRIDGE (ds)->bus = bus;if (pxb_register_bus (dev, bus)) {
@@ -230,7 +237,9 @@ static int pxb_dev_initfn(PCIDevice *dev) }qdev_init_nofail (ds); -qdev_init_nofail (bds); +if (bds) { +qdev_init_nofail (bds); + }pci_word_test_and_set_mask (dev->config +PCI_STATUS ,PCI_STATUS_66MHZ |PCI_STATUS_FAST_BACK );
@@ -251,6 +260,8 @@ static Property pxb_dev_properties[] = {/* Note: 0 is not a legal a PXB bus number. */ DEFINE_PROP_UINT8 ("bus_nr" , PXBDev, bus_nr,0 ),DEFINE_PROP_UINT16 ("numa_node" , PXBDev, numa_node,NUMA_NODE_UNASSIGNED ), +DEFINE_PROP_BIT ("x-enable-internal-bridge" , PXBDev, flags, +PXB_FLAG_ENABLE_BRIDGE_BIT , false), DEFINE_PROP_END_OF_LIST(), };
include/hw/compat.h +4 -0
@@ -10,6 +10,10 @@ .driver = "virtio-pci",\ .property = "disable-pcie",\ .value = "on",\ + },{\ + .driver = "pxb",\ + .property = "x-enable-internal-bridge",\ + .value = "on",\ },#define HW_COMPAT_2_3 \
Signature · 1 line
2.1.0
PATCH v2 2/4 hw/acpi: merge pxb adjacent memory/IO ranges
In reply to Marcel ApfelbaumSince the PXB has no longer a built-in PCI bridge, the ACPI will include IO/MEM ranges per device. Try to merge adjacent resources to reduce the ACPI tables length.
--- hw/i386/acpi-build.c | 123 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 74 insertions(+), 49 deletions(-)
hw/i386/acpi-build.c +74 -49
@@ -762,16 +762,59 @@ static void crs_replace_with_free_ranges(GPtrArray *ranges,g_ptr_array_free (free_ranges, false); } +/* +* crs_range_merge - merges adjacent ranges in the given array. +* Array elements are deleted and replaced with the merged ranges. +*/ +static void crs_range_merge (GPtrArray *range) +{ +GPtrArray *tmp =g_ptr_array_new_with_free_func (crs_range_free); +CrsRangeEntry *entry; +uint64_t range_base, range_limit; +int i; + +if (!range->len) { +return ; + } + +g_ptr_array_sort (range, crs_range_compare); + + entry =g_ptr_array_index (range,0 ); + range_base = entry->base; + range_limit = entry->limit; +for (i =1 ; i < range->len; i++) { + entry =g_ptr_array_index (range, i); +if (entry->base -1 == range_limit) { + range_limit = entry->limit; + }else { +crs_range_insert (tmp, range_base, range_limit); + range_base = entry->base; + range_limit = entry->limit; + } + } +crs_range_insert (tmp, range_base, range_limit); + +g_ptr_array_set_size (range,0 ); +for (i =0 ; i < tmp->len; i++) { + entry =g_ptr_array_index (tmp, i); +crs_range_insert (range, entry->base, entry->limit); + } +g_ptr_array_free (tmp, true); +} +static Aml *build_crs (PCIHostState *host,GPtrArray *io_ranges,GPtrArray *mem_ranges) {Aml *crs =aml_resource_template (); +GPtrArray *host_io_ranges =g_ptr_array_new_with_free_func (crs_range_free); +GPtrArray *host_mem_ranges =g_ptr_array_new_with_free_func (crs_range_free); +CrsRangeEntry *entry;uint8_t max_bus =pci_bus_num (host->bus);uint8_t type;int devfn; +int i;for (devfn =0 ; devfn <ARRAY_SIZE (host->bus->devices); devfn++) { -int i;uint64_t range_base, range_limit;PCIDevice *dev = host->bus->devices[devfn];
@@ -794,26 +837,9 @@ static Aml *build_crs(PCIHostState *host, }if (r->type &PCI_BASE_ADDRESS_SPACE_IO ) { -aml_append (crs, -aml_word_io (AML_MIN_FIXED ,AML_MAX_FIXED , -AML_POS_DECODE ,AML_ENTIRE_RANGE , -0 , - range_base, - range_limit, -0 , - range_limit - range_base +1 )); -crs_range_insert (io_ranges, range_base, range_limit); +crs_range_insert (host_io_ranges, range_base, range_limit); }else {/* "memory" */ -aml_append (crs, -aml_dword_memory (AML_POS_DECODE ,AML_MIN_FIXED , -AML_MAX_FIXED ,AML_NON_CACHEABLE , -AML_READ_WRITE , -0 , - range_base, - range_limit, -0 , - range_limit - range_base +1 )); -crs_range_insert (mem_ranges, range_base, range_limit); +crs_range_insert (host_mem_ranges, range_base, range_limit); } }
@@ -832,15 +858,7 @@ static Aml *build_crs(PCIHostState *host,* that do not support multiple root buses */ if (range_base && range_base <= range_limit) { -aml_append (crs, -aml_word_io (AML_MIN_FIXED ,AML_MAX_FIXED , -AML_POS_DECODE ,AML_ENTIRE_RANGE , -0 , - range_base, - range_limit, -0 , - range_limit - range_base +1 )); -crs_range_insert (io_ranges, range_base, range_limit); +crs_range_insert (host_io_ranges, range_base, range_limit); } range_base =
@@ -853,16 +871,7 @@ static Aml *build_crs(PCIHostState *host,* that do not support multiple root buses */ if (range_base && range_base <= range_limit) { -aml_append (crs, -aml_dword_memory (AML_POS_DECODE ,AML_MIN_FIXED , -AML_MAX_FIXED ,AML_NON_CACHEABLE , -AML_READ_WRITE , -0 , - range_base, - range_limit, -0 , - range_limit - range_base +1 )); -crs_range_insert (mem_ranges, range_base, range_limit); +crs_range_insert (host_mem_ranges, range_base, range_limit); } range_base =
@@ -875,20 +884,36 @@ static Aml *build_crs(PCIHostState *host,* that do not support multiple root buses */ if (range_base && range_base <= range_limit) { -aml_append (crs, -aml_dword_memory (AML_POS_DECODE ,AML_MIN_FIXED , -AML_MAX_FIXED ,AML_NON_CACHEABLE , -AML_READ_WRITE , -0 , - range_base, - range_limit, -0 , - range_limit - range_base +1 )); -crs_range_insert (mem_ranges, range_base, range_limit); +crs_range_insert (host_mem_ranges, range_base, range_limit); } } } +crs_range_merge (host_io_ranges); +for (i =0 ; i < host_io_ranges->len; i++) { + entry =g_ptr_array_index (host_io_ranges, i); +aml_append (crs, +aml_word_io (AML_MIN_FIXED ,AML_MAX_FIXED , +AML_POS_DECODE ,AML_ENTIRE_RANGE , +0 , entry->base, entry->limit,0 , + entry->limit - entry->base +1 )); +crs_range_insert (io_ranges, entry->base, entry->limit); + } +g_ptr_array_free (host_io_ranges, true); + +crs_range_merge (host_mem_ranges); +for (i =0 ; i < host_mem_ranges->len; i++) { + entry =g_ptr_array_index (host_mem_ranges, i); +aml_append (crs, +aml_dword_memory (AML_POS_DECODE ,AML_MIN_FIXED , +AML_MAX_FIXED ,AML_NON_CACHEABLE , +AML_READ_WRITE , +0 , entry->base, entry->limit,0 , + entry->limit - entry->base +1 )); +crs_range_insert (mem_ranges, entry->base, entry->limit); + } +g_ptr_array_free (host_mem_ranges, true); + aml_append(crs, aml_word_bus_number(AML_MIN_FIXED, AML_MAX_FIXED, AML_POS_DECODE, 0,
Signature · 1 line
2.1.0
PATCH v2 3/4 hw/pc: query both q35 and i440fx bus
In reply to Marcel ApfelbaumLook for pxb devices on both i386 machines.
--- hw/i386/acpi-build.c | 3 +-- hw/i386/pc.c | 2 +- hw/pci-host/q35.c | 8 ++++++++ include/hw/i386/pc.h | 9 +++++++++ 4 files changed, 19 insertions(+), 3 deletions(-)
hw/i386/acpi-build.c +1 -2
@@ -950,8 +950,7 @@ build_ssdt(GArray *table_data, GArray *linker,/* Reserve space for header */ acpi_data_push (ssdt->buf,sizeof (AcpiTableHeader)); -/* Extra PCI root buses are implemented only for i440fx */ - bus =find_i440fx (); + bus =find_pc ();if (bus) { QLIST_FOREACH(bus, &bus->child, sibling) {uint8_t bus_num =pci_bus_num (bus);
hw/i386/pc.c +1 -1
@@ -1174,7 +1174,7 @@ void pc_guest_info_machine_done(Notifier *notifier, void *data)PcGuestInfoState *guest_info_state =container_of (notifier, PcGuestInfoState, machine_done); -PCIBus *bus =find_i440fx (); +PCIBus *bus =find_pc ();if (bus) {int extra_hosts =0 ;
hw/pci-host/q35.c +8 -0
@@ -511,6 +511,14 @@ static void mch_realize(PCIDevice *d, Error **errp) } } +PCIBus *find_q35 (void ) +{ +PCIHostState *s =OBJECT_CHECK (PCIHostState, +object_resolve_path ("/machine/q35" ,NULL ), +TYPE_PCI_HOST_BRIDGE ); +return s ? s->bus :NULL ; +} +uint64_t mch_mcfg_base (void ) {bool ambiguous;
include/hw/i386/pc.h +9 -0
@@ -242,6 +242,15 @@ PCIBus *i440fx_init(const char *host_type, const char *pci_type, MemoryRegion *ram_memory);PCIBus *find_i440fx (void ); +PCIBus *find_q35 (void ); + +static inline PCIBus *find_pc (void ) +{ +PCIBus *bus =find_i440fx (); + +return bus ? bus :find_q35 (); +} +/* piix4.c */ extern PCIDevice *piix4_dev;int piix4_init (PCIBus *bus,ISABus **isa_bus,int devfn);
Signature · 1 line
2.1.0
Re: [Qemu-devel] [PATCH V2 3/4] hw/pc: query both q35 and i440fx bus
Quoted 3 lines · Marcel Apfelbaum
Look for pxb devices on both i386 machines.
Signed-off-by Marcel Apfelbaum <[email protected]>
[...]
hw/pci-host/q35.c static void mch_realize(PCIDevice *d, Error **errp) · line 518
Quoted 13 lines
hw/pci-host/q35.c +5
-0@@ -511,6 +511,14 @@ static void mch_realize(PCIDevice *d, Error **errp) } } +PCIBus *find_q35 (void ) +{ +PCIHostState *s =OBJECT_CHECK (PCIHostState, +object_resolve_path ("/machine/q35" ,NULL ), +TYPE_PCI_HOST_BRIDGE );
Why not use the PCI_HOST_BRIDGE macro?
Quoted 11 lines
hw/pci-host/q35.c +3
-0+return s ? s->bus :NULL ; +} +uint64_t mch_mcfg_base (void ) {bool ambiguous;include/hw/i386/pc.h +0
-0
include/hw/i386/pc.h PCIBus *i440fx_init(const char *host_type, const char *pci_type, · line 251
Quoted 10 lines
include/hw/i386/pc.h +7
-0@@ -242,6 +242,15 @@ PCIBus *i440fx_init(const char *host_type, const char *pci_type, MemoryRegion *ram_memory);PCIBus *find_i440fx (void ); +PCIBus *find_q35 (void ); + +static inline PCIBus *find_pc (void ) +{ +PCIBus *bus =find_i440fx (); + +return bus ? bus :find_q35 ();
Have you considered a /machine/pci link, or (in case you don't want to add something new to the QOM tree) a simple PCMachineState::pci_bus struct field?
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH V2 3/4] hw/pc: query both q35 and i440fx bus
Quoted 13 lines · Eduardo Habkost
Quoted 3 lines · Marcel Apfelbaum
Look for pxb devices on both i386 machines.
Signed-off-by Marcel Apfelbaum <[email protected]>[...]
Quoted 8 lines
hw/pci-host/q35.c +0
-0@@ -511,6 +511,14 @@ static void mch_realize(PCIDevice *d, Error **errp) } }
Quoted 6 lines
hw/pci-host/q35.c static void mch_realize(PCIDevice *d, Error **errp) · line 518
Quoted 5 lines
hw/pci-host/q35.c +5
-0+PCIBus *find_q35 (void ) +{ +PCIHostState *s =OBJECT_CHECK (PCIHostState, +object_resolve_path ("/machine/q35" ,NULL ), +TYPE_PCI_HOST_BRIDGE );
Hi Eduardo,
Thanks for the review.
Quoted 1 line
Why not use the
PCI_HOST_BRIDGEmacro?
Do you mean ?
PCIHostState *s = PCI_HOST_BRIDGE(object_resolve_path("/machine/q35", NULL));I can use it, thanks -- I was doing the usual copy/paste of the i440fx counterpart :)
Quoted 14 lines
Quoted 14 lines
hw/pci-host/q35.c +3
-0+return s ? s->bus :NULL ; +} +uint64_t mch_mcfg_base (void ) {bool ambiguous;include/hw/i386/pc.h +0
-0@@ -242,6 +242,15 @@ PCIBus *i440fx_init(const char *host_type, const char *pci_type, MemoryRegion *ram_memory);
Quoted 12 lines
include/hw/i386/pc.h PCIBus *i440fx_init(const char *host_type, const char *pci_type, · line 251
Quoted 8 lines
include/hw/i386/pc.h +7
-0PCIBus *find_i440fx (void ); +PCIBus *find_q35 (void ); + +static inline PCIBus *find_pc (void ) +{ +PCIBus *bus =find_i440fx (); + +return bus ? bus :find_q35 ();Have you considered a /machine/pci link, or (in case you don't want to add something new to the QOM tree) a simple
PCMachineState::pci_busstruct field?
This is a nice idea, I like the PC machine state, I'll look into it.
Thanks,
Marcel
Quoted 1 line
PATCH v2 4/4 hw/pxb: add support for PCIe
In reply to Marcel ApfelbaumThe PXB internal bus type is created in conformance with the primary root bus (bus 0).
--- hw/pci-bridge/pci_expander_bridge.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +18 -2
@@ -23,6 +23,9 @@#define TYPE_PXB_BUS "pxb-bus"#define PXB_BUS (obj) OBJECT_CHECK(PXBBus, (obj), TYPE_PXB_BUS) +#define TYPE_PXB_PCIE_BUS "pxb-pcie-bus" +#define PXB_PCIE_BUS (obj) OBJECT_CHECK(PXBBus, (obj), TYPE_PXB_PCIE_BUS) + typedef struct PXBBus { /*< private >*/ PCIBus parent_obj;
@@ -86,10 +89,18 @@ static const TypeInfo pxb_bus_info = { .class_init = pxb_bus_class_init, }; +static const TypeInfo pxb_pcie_bus_info = { + .name =TYPE_PXB_PCIE_BUS , + .parent =TYPE_PCIE_BUS , + .instance_size =sizeof (PXBBus), + .class_init = pxb_bus_class_init, +}; +static const char *pxb_host_root_bus_path (PCIHostState *host_bridge,PCIBus *rootbus) { -PXBBus *bus =PXB_BUS (rootbus); +PXBBus *bus =pci_bus_is_express (rootbus) ? +PXB_PCIE_BUS (rootbus) :PXB_BUS (rootbus);snprintf (bus->bus_path,8 ,"0000:%02x" ,pxb_bus_num (rootbus));return bus->bus_path;
@@ -222,7 +233,11 @@ static int pxb_dev_initfn(PCIDevice *dev)qdev_prop_set_uint8 (bds,PCI_BRIDGE_DEV_PROP_CHASSIS_NR , pxb->bus_nr);qdev_prop_set_bit (bds,PCI_BRIDGE_DEV_PROP_SHPC , false); }else { - bus =pci_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_BUS ); +if (pci_bus_is_express (dev->bus)) { + bus =pci_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_PCIE_BUS ); + }else { + bus =pci_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_BUS ); + } } bus->parent_dev = dev;
@@ -290,6 +305,7 @@ static const TypeInfo pxb_dev_info = {static void pxb_register_types (void ) {type_register_static (&pxb_bus_info); +type_register_static (&pxb_pcie_bus_info);type_register_static (&pxb_host_info);type_register_static (&pxb_dev_info); }
Signature · 1 line
2.1.0
Quoted 3 lines · Marcel Apfelbaum
Addressed Gerd Hoffmann comments:
- Added x-enable-internal-bridge compat property to keep the PCI bridge for older machine to avoid breaking migration.
This will break PXB command lines written for "-M pc" in QEMU <= 2.5. Technically there's no prohibition against doing so, but we've never done it; I'm not sure why we should start doing so now.
Can we have two devices, on for a PCI expander bridge and one for a PCIe expander bridge?
Paolo
Quoted 10 lines · Paolo Bonzini
Quoted 3 lines · Marcel Apfelbaum
Addressed Gerd Hoffmann comments:
- Added x-enable-internal-bridge compat property to keep the PCI bridge for older machine to avoid breaking migration.This will break PXB command lines written for "-M pc" in QEMU <= 2.5. Technically there's no prohibition against doing so, but we've never done it; I'm not sure why we should start doing so now.
Hi Paolo,
Can you please explain how does it break the command line?
For QEMU < 2.5 we automatically have the x-enable-internal-bridge compat property "on" so the command line remains the same.
For QEMU >= 2.5 is "off" by default and the bridge disappear, without command line changes.
Quoted 3 lines
Can we have two devices, on for a PCI expander bridge and one for a PCIe expander bridge?
Gerd also asked for this, OK, I am going for it.
Thanks,
Marcel
Quoted 3 lines
Paolo
Quoted 14 lines · Marcel Apfelbaum
Quoted 10 lines · Paolo Bonzini
Quoted 3 lines · Marcel Apfelbaum
Addressed Gerd Hoffmann comments:
- Added x-enable-internal-bridge compat property to keep the PCI bridge for older machine to avoid breaking migration.This will break PXB command lines written for "-M pc" in QEMU <= 2.5. Technically there's no prohibition against doing so, but we've never done it; I'm not sure why we should start doing so now.
Hi Paolo,
Quoted 8 lines
Can you please explain how does it break the command line?
For QEMU < 2.5 we automatically have the x-enable-internal-bridge compat property "on"
so the command line remains the same.For QEMU >= 2.5 is "off" by default and the bridge disappear, without command line changes.
If you were using "-M pc", and you expected "-device" to place things under the PCI-to-PCI bridge, that command line will not work anymore I think.
Management tools can work around it. They can always set the new property to off---which by the way means it should not be "x" prefixed---and assume the PCI-to-PCI bridge is there if the property doesn't exist. But it's inconvenient, and we try to avoid lockstep changes to QEMU and management.
Paolo
Quoted 12 lines
Quoted 2 lines
Can we have two devices, on for a PCI expander bridge and one for a PCIe expander bridge?
Gerd also asked for this, OK, I am going for it.
Thanks,
MarcelQuoted 3 lines
Paolo
Quoted 23 lines · Paolo Bonzini
Quoted 20 lines · Marcel Apfelbaum
Quoted 10 lines · Paolo Bonzini
Quoted 3 lines · Marcel Apfelbaum
Addressed Gerd Hoffmann comments:
- Added x-enable-internal-bridge compat property to keep the PCI bridge for older machine to avoid breaking migration.This will break PXB command lines written for "-M pc" in QEMU <= 2.5. Technically there's no prohibition against doing so, but we've never done it; I'm not sure why we should start doing so now.
Hi Paolo,
Can you please explain how does it break the command line?
For QEMU < 2.5 we automatically have the x-enable-internal-bridge compat property "on"
so the command line remains the same.
Quoted 6 lines
Quoted 2 lines
For QEMU >= 2.5 is "off" by default and the bridge disappear, without command line changes.
If you were using "-M pc", and you expected "-device" to place things under the PCI-to-PCI bridge, that command line will not work anymore I think.
Actually the command line would work. The devices will be placed on the root bus instead of the pci bridge bus. The trick here is that we name the bus the devices will be attached to by the pxb id.
- When we have the bridge -> the pxb id will be used for the secondary bus
- when don't have the bridge -> the pxb id wil be used for the root bus.
Both times the command line remains -device pxb,id=my_root -device e1000,bus=my_root
Quoted 7 lines
Management tools can work around it. They can always set the new property to off---which by the way means it should not be "x" prefixed---and assume the PCI-to-PCI bridge is there if the property doesn't exist. But it's inconvenient, and we try to avoid lockstep changes to QEMU and management.
I understand, I'll implement this as a new device "pxb-pcie"
Thanks,
Marcel
Quoted 14 lines
Paolo
Quoted 12 lines
Quoted 2 lines
Can we have two devices, on for a PCI expander bridge and one for a PCIe expander bridge?
Gerd also asked for this, OK, I am going for it.
Thanks,
MarcelQuoted 3 lines
Paolo
Quoted 9 lines · Marcel Apfelbaum
Actually the command line would work. The devices will be placed on the root bus
instead of the pci bridge bus. The trick here is that we name the bus the devices
will be attached to by the pxb id.
- When we have the bridge -> the pxb id will be used for the secondary bus
- when don't have the bridge -> the pxb id wil be used for the root bus.
Both times the command line remains -device pxb,id=my_root -device e1000,bus=my_root
What would you lose? Hotplug?
Paolo
Quoted 13 lines · Paolo Bonzini
Quoted 9 lines · Marcel Apfelbaum
Actually the command line would work. The devices will be placed on the root bus
instead of the pci bridge bus. The trick here is that we name the bus the devices
will be attached to by the pxb id.
- When we have the bridge -> the pxb id will be used for the secondary bus
- when don't have the bridge -> the pxb id wil be used for the root bus.
Both times the command line remains -device pxb,id=my_root -device e1000,bus=my_root
Quoted 2 lines
What would you lose? Hotplug?
Without the bridge? Yes.
However the user can add it manually the pci-bridge and have it anyway.
I wanted to get rid of the internal pci-bridge as a default, and this is why pxb and pxb-pcie are he same device now (except bus type)
Quoted 2 lines
Paolo
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
Paolo
Quoted 2 lines
I wanted to get rid of the internal pci-bridge as a default, and this is why pxb and pxb-pcie are he same device now (except bus type)
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5) or on the root bus itself (QEMU >= 2.5).
I know is guest visible, but the guest will see one of them depending on the machine type.
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it, but because the implementation is *exactly* the same I think we should gain more by maintaining one device.
Thanks,
Marcel
Quoted 4 lines
Paolo
Quoted 2 lines
I wanted to get rid of the internal pci-bridge as a default, and this is why pxb and pxb-pcie are he same device now (except bus type)
Quoted 24 lines · Marcel Apfelbaum
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5) or on the root bus itself (QEMU >= 2.5).I know is guest visible, but the guest will see one of them depending on the machine type.
Quoted 7 lines
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it, but because the implementation is *exactly* the same I think we should gain more by maintaining one device.
Thanks,
Marcel
Yes, I think you want a new "pci-extender" device which is just the extender. Then existing pxb will create both it and the bridge behind it. Maybe creating pxb which is extender+bridge was a mistake, I don't know, but we shipped it in QEMU so we support it.
Quoted 5 lines
Quoted 5 lines
Paolo
Quoted 2 lines
I wanted to get rid of the internal pci-bridge as a default, and this is why pxb and pxb-pcie are he same device now (except bus type)
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Marcel Apfelbaum
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5) or on the root bus itself (QEMU >= 2.5).I know is guest visible, but the guest will see one of them depending on the machine type.
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it, but because the implementation is *exactly* the same I think we should gain more by maintaining one device.
Quoted 7 lines
Quoted 2 lines
Thanks,
MarcelYes, I think you want a new "pci-extender" device which is just the extender. Then existing pxb will create both it and the bridge behind it. Maybe creating pxb which is extender+bridge was a mistake, I don't know, but we shipped it in QEMU so we support it.
OK, but this device will be both pci/pcie, depending on the machine type right? No need to split it too?
Thanks,
Marcel
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Paolo
Quoted 2 lines
I wanted to get rid of the internal pci-bridge as a default, and this is why pxb and pxb-pcie are he same device now (except bus type)
Quoted 39 lines · Marcel Apfelbaum
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Marcel Apfelbaum
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5) or on the root bus itself (QEMU >= 2.5).I know is guest visible, but the guest will see one of them depending on the machine type.
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it, but because the implementation is *exactly* the same I think we should gain more by maintaining one device.
Quoted 7 lines
Quoted 2 lines
Thanks,
MarcelYes, I think you want a new "pci-extender" device which is just the extender. Then existing pxb will create both it and the bridge behind it. Maybe creating pxb which is extender+bridge was a mistake, I don't know, but we shipped it in QEMU so we support it.
Quoted 5 lines
OK, but this device will be both pci/pcie, depending on the machine type right? No need to split it too?
Thanks,
Marcel
It's ok to have a single device but I don't like tying it to a machine type, much.
It's always integrated within a RC, right? Can you go by the type of the parent device?
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Paolo
Quoted 2 lines
I wanted to get rid of the internal pci-bridge as a default, and this is why pxb and pxb-pcie are he same device now (except bus type)
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Marcel Apfelbaum
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Marcel Apfelbaum
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5) or on the root bus itself (QEMU >= 2.5).I know is guest visible, but the guest will see one of them depending on the machine type.
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it, but because the implementation is *exactly* the same I think we should gain more by maintaining one device.
Quoted 7 lines
Quoted 2 lines
Thanks,
MarcelYes, I think you want a new "pci-extender" device which is just the extender. Then existing pxb will create both it and the bridge behind it. Maybe creating pxb which is extender+bridge was a mistake, I don't know, but we shipped it in QEMU so we support it.
OK, but this device will be both pci/pcie, depending on the machine type right? No need to split it too?
Quoted 7 lines
Quoted 2 lines
Thanks,
MarcelIt's ok to have a single device but I don't like tying it to a machine type, much.
It's always integrated within a RC, right? Can you go by the type of the parent device?
Yes and this is what I am doing, looking for the bus the pxb is on and not for the machine type.
I'll start working on it,
Thanks,
Marcel
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Paolo
Quoted 2 lines
I wanted to get rid of the internal pci-bridge as a default, and this is why pxb and pxb-pcie are he same device now (except bus type)
Quoted 9 lines · Marcel Apfelbaum
Quoted 5 lines
Yes, I think you want a new "pci-extender" device which is just the extender.
Then existing pxb will create both it and the bridge behind it. Maybe creating pxb which is extender+bridge was a mistake, I don't know, but we shipped it in QEMU so we support it.OK, but this device will be both pci/pcie, depending on the machine type right?
No need to split it too?
No, I think there's no need to split it. It can be a property.
Paolo
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Marcel Apfelbaum
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5) or on the root bus itself (QEMU >= 2.5).I know is guest visible, but the guest will see one of them depending on the machine type.
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it, but because the implementation is *exactly* the same I think we should gain more by maintaining one device.
Quoted 7 lines
Quoted 2 lines
Thanks,
MarcelYes, I think you want a new "pci-extender" device which is just the extender. Then existing pxb will create both it and the bridge behind it. Maybe creating pxb which is extender+bridge was a mistake, I don't know, but we shipped it in QEMU so we support it.
At least from libvirt's POV, pxb support hasn't yet been added, so even though qemu has shipped it, libvirt (and thus ovirt and openstack at least) hasn't used it, and from the sounds of this discussion, it probably won't; we'll just directly support the new model for 440fx and q35.
Quoted 22 lines · Marcel Apfelbaum
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5) or on the root bus itself (QEMU >= 2.5).
Quoted 7 lines
I know is guest visible, but the guest will see one of them depending on the machine type.
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it,
but because the implementation is *exactly* the same I think we should gain more by maintaining one device.
I have no opinion on two devices vs. one device + property in this particular case, I just want to interject that I'd expect the difference in maintaince to be negligible.
A second device basically takes a copy of the TypeInfo with some (trivial) init function to make it different. Might be a few more lines of code than adding a property, but in complexity, it's a wash.
In case you plan to get rid of the old variant: with two devices, you deprecate and later delete the old device. With device + property, you deprecate setting the property, and later delete it. The former might be a bit easier to document.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I wanted to get rid of the internal pci-bridge as a default, and this is why pxb and pxb-pcie are he same device now (except bus type)
Quoted 36 lines · Markus Armbruster
Quoted 22 lines · Marcel Apfelbaum
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5) or on the root bus itself (QEMU >= 2.5).Quoted 7 lines
I know is guest visible, but the guest will see one of them depending on the machine type.
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it,
but because the implementation is *exactly* the same I think we should gain more by maintaining one device.I have no opinion on two devices vs. one device + property in this particular case, I just want to interject that I'd expect the difference in maintaince to be negligible.
Quoted 8 lines
A second device basically takes a copy of the TypeInfo with some (trivial) init function to make it different. Might be a few more lines of code than adding a property, but in complexity, it's a wash.
In case you plan to get rid of the old variant: with two devices, you deprecate and later delete the old device. With device + property, you deprecate setting the property, and later delete it. The former might be a bit easier to document.
Hi Markus,
Thank you for the review.
Following all the comments I have received, I am going to:
1. Leave the old device (pxb) as is and mark it as deprecated. Maybe you can point me on the cleanest way to mark a device as deprecated?
2. Create a new device (pci-expander) that will behave as a:
- PCI root, if the machine's bus 0 is legacy PCI.
- PCI express root (Root Complex), if machine's bus 0 is PCI Express.
This way I will not need two device and not even a property for (pci/pcie). We don't really need an extra PCI root on a PCIe machine and vice versa.
By the way, following the same concept I converted the virtio devices to PCIe devices. I didn't want a new set of devices.
Thanks,
Marcel
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I wanted to get rid of the internal pci-bridge as a default, and this is why pxb and pxb-pcie are he same device now (except bus type)
Quoted 48 lines · Marcel Apfelbaum
Quoted 38 lines · Markus Armbruster
Quoted 23 lines · Marcel Apfelbaum
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5)
or on the root bus itself (QEMU >= 2.5).Quoted 8 lines
I know is guest visible, but the guest will see one of them depending on the machine type.
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it,
but because the implementation is *exactly* the same I think we should gain more
by maintaining one device.I have no opinion on two devices vs. one device + property in this particular case, I just want to interject that I'd expect the difference in maintaince to be negligible.
Quoted 8 lines
A second device basically takes a copy of the TypeInfo with some (trivial) init function to make it different. Might be a few more lines of code than adding a property, but in complexity, it's a wash.
In case you plan to get rid of the old variant: with two devices, you deprecate and later delete the old device. With device + property, you deprecate setting the property, and later delete it. The former might be a bit easier to document.
Quoted 7 lines
Hi Markus,
Thank you for the review.Following all the comments I have received, I am going to:
1. Leave the old device (pxb) as is and mark it as deprecated. Maybe you can point me on the cleanest way to mark a device as deprecated?
First, we need to decide whether they're deprecated or legacy.
Deprecated means their use is discouraged and they may go away eventually. Users are advised to switch to the replacement.
Legacy means we keep them around just for compatibility. Existing uses are just fine, but new use is discouraged.
We usually remove deprecated interfaces from help and documentation. Sometimes we mark them deprecated for a while before we remove them.
Legacy interfaces stay documented, but should be clearly marked.
Right now, -device help shows all pluggable devices, i.e. the devices that don't have cannot_instantiate_with_device_add_yet set. In other words, there's no way to suppress deprecated devices, yet.
Two ways to mark them:
1. Set DeviceClass member desc to a string containing "(deprecated)" or "(legacy)", respectively.
2. Create a new DeviceCategory, and move the device there.
I prefer 2. You can easily turn it into a way to suppress deprecated
devices: just skip the "deprecated" category.
Quoted 10 lines
2. Create a new device (pci-expander) that will behave as a:
- PCI root, if the machine's bus 0 is legacy PCI.
- PCI express root (Root Complex), if machine's bus 0 is PCI Express.This way I will not need two device and not even a property for (pci/pcie). We don't really need an extra PCI root on a PCIe machine and vice versa.
By the way, following the same concept I converted the virtio devices to PCIe devices.
I didn't want a new set of devices.
Trust your judgement :)
[...]
Quoted 77 lines · Markus Armbruster
Quoted 48 lines · Marcel Apfelbaum
Quoted 38 lines · Markus Armbruster
Quoted 23 lines · Marcel Apfelbaum
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Marcel Apfelbaum
Quoted 1 line
What would you lose? Hotplug?
Without the bridge? Yes. However the user can add it manually the pci-bridge and have it anyway.
Ok, I guess that's more or less acceptable. It's still ugly however, to the point that I wonder if we should rename the device and call the old one a failed experiment.
I guess we can rename the pxb to extra-root or something, but in this way will have a deprecated/duplicated device to support and kill in the future.
Why not use the compat property as it is?
Again, the command line *remains* the same, the difference is where the devices associated with the pxb will land: on the secondary bus (for QEMU < 2.5)
or on the root bus itself (QEMU >= 2.5).Quoted 8 lines
I know is guest visible, but the guest will see one of them depending on the machine type.
Regarding the splitting of pxb into 2 devices (pci/pcie), I have nothing against it,
but because the implementation is *exactly* the same I think we should gain more
by maintaining one device.I have no opinion on two devices vs. one device + property in this particular case, I just want to interject that I'd expect the difference in maintaince to be negligible.
Quoted 8 lines
A second device basically takes a copy of the TypeInfo with some (trivial) init function to make it different. Might be a few more lines of code than adding a property, but in complexity, it's a wash.
In case you plan to get rid of the old variant: with two devices, you deprecate and later delete the old device. With device + property, you deprecate setting the property, and later delete it. The former might be a bit easier to document.
Quoted 7 lines
Hi Markus,
Thank you for the review.Following all the comments I have received, I am going to:
1. Leave the old device (pxb) as is and mark it as deprecated. Maybe you can point me on the cleanest way to mark a device as deprecated?
First, we need to decide whether they're deprecated or legacy.
Deprecated means their use is discouraged and they may go away eventually. Users are advised to switch to the replacement.
Legacy means we keep them around just for compatibility. Existing uses are just fine, but new use is discouraged.
We usually remove deprecated interfaces from help and documentation. Sometimes we mark them deprecated for a while before we remove them.
Legacy interfaces stay documented, but should be clearly marked.
Right now, -device help shows all pluggable devices, i.e. the devices that don't have
cannot_instantiate_with_device_add_yetset. In other words, there's no way to suppress deprecated devices, yet.Two ways to mark them:
Quoted 7 lines
1. Set DeviceClass member desc to a string containing "(deprecated)" or "(legacy)", respectively.
2. Create a new DeviceCategory, and move the device there.
I prefer 2. You can easily turn it into a way to suppress deprecated
devices: just skip the "deprecated" category.
Thanks Markus for the detailed information!
Quoted 5 lines
Quoted 5 lines
2. Create a new device (pci-expander) that will behave as a:
- PCI root, if the machine's bus 0 is legacy PCI.
- PCI express root (Root Complex), if machine's bus 0 is PCI Express.
Quoted 8 lines
Quoted 6 lines
This way I will not need two device and not even a property for (pci/pcie). We don't really need an extra PCI root on a PCIe machine and vice versa.
By the way, following the same concept I converted the virtio devices to PCIe devices.
I didn't want a new set of devices.Trust your judgement :)
Re-thinking this whole thing, I have a new plan :)
We have the pxb device which is working just fine. Having the internal pci-bridge by default is a good idea (hot-plug support, smaller ACPI tables).
Adding a new device, say "root-complex", that re-uses some of the pxb code is the best way to go, and it will be easier to maintain when those devices will diverge.
Thanks,
Marcel
Quoted 3 lines
[...]