s390x/pci: PCI bridge plugging fixes
cover s390x/pci: PCI bridge plugging fixes
Hotplugging of PCI bridges is right now pretty much broken. Coldplugging and hotplugging will assign wrong primary bus numbers in some scenarios.
I base my knowledge on how this is supposed to work on
http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.html
I did a couple of tests, building whole hierarchies of bridges, both hot and coldplugged. "info pci" as well as the Linux guests showed what I was expecting.
David Hildenbrand (2):
s390x/pci: Fix primary bus number for PCI bridges s390x/pci: Fix hotplugging of PCI bridges
hw/s390x/s390-pci-bus.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-)
Signature · 1 line
2.17.2
PATCH 1/2 s390x/pci: Fix primary bus number for PCI bridges
The primary bus number corresponds always to the bus number of the bus the bridge is attached to.
Right now, if we have two bridges attached to the same bus (e.g. root bus) this is however not the case. Fix assignment.
While at it
- Add a comment why we have to reassign durign every reset (which I found to be supprising)
- Drop setting the PCI_SUBORDINATE_BUS temporarily to 0xff. As we are setting it via a DFS and not via a BFS (as discussed e.g. in [1]), this is not necessary. The last number when we return is the highest number.
Please note that hotplugging of bridges is in general still broken, will be fixed next.
[1] http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.html
--- hw/s390x/s390-pci-bus.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
hw/s390x/s390-pci-bus.c +8 -5
@@ -862,7 +862,8 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,qbus_set_hotplug_handler (bus,DEVICE (s), errp);if (dev->hotplugged) { -pci_default_write_config (pdev,PCI_PRIMARY_BUS , s->bus_no,1 ); +pci_default_write_config (pdev,PCI_PRIMARY_BUS , +pci_dev_bus_num (pdev),1 ); s->bus_no +=1 ;pci_default_write_config (pdev,PCI_SECONDARY_BUS , s->bus_no,1 ); do {
@@ -1016,8 +1017,6 @@ static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, void *opaque) {S390pciState *s = opaque; -unsigned int primary = s->bus_no; -unsigned int subordinate =0xff ;PCIBus *sec_bus =NULL ; if ((pci_default_read_config(pdev, PCI_HEADER_TYPE, 1) !=
@@ -1026,7 +1025,7 @@ static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, } (s->bus_no)++; -pci_default_write_config (pdev,PCI_PRIMARY_BUS , primary,1 ); +pci_default_write_config (pdev,PCI_PRIMARY_BUS ,pci_dev_bus_num (pdev),1 );pci_default_write_config (pdev,PCI_SECONDARY_BUS , s->bus_no,1 );pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , s->bus_no,1 );
@@ -1035,7 +1034,7 @@ static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev,return ; } -pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , subordinate,1 ); +/* Assign numbers to all child bridges. The last is the highest number. */ pci_for_each_device (sec_bus,pci_bus_num (sec_bus), s390_pci_enumerate_bridge, s);pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , s->bus_no,1 );
@@ -1046,6 +1045,10 @@ static void s390_pcihost_reset(DeviceState *dev)S390pciState *s =S390_PCI_HOST_BRIDGE (dev);PCIBus *bus = s->parent_obj.bus; +/* +* When resetting a PCI bridge, the assigned numbers are set to 0. So +* on every system reset, we also have to reassign numbers. +*/ s->bus_no =0 ;pci_for_each_device (bus,pci_bus_num (bus), s390_pci_enumerate_bridge, s); }
Signature · 1 line
2.17.2
Re: [Qemu-devel] [PATCH v1 1/2] s390x/pci: Fix primary bus number for PCI bridges
Quoted 8 lines · David Hildenbrand
The primary bus number corresponds always to the bus number of the bus the bridge is attached to.
Right now, if we have two bridges attached to the same bus (e.g. root bus) this is however not the case. Fix assignment.
While at it
- Add a comment why we have to reassign durign every reset (which I
s/durign/during/
Quoted 1 line
found to be supprising)
s/supprising/surprising/
Quoted 4 lines
- Drop setting the
PCI_SUBORDINATE_BUStemporarily to0xff. As we are setting it via a DFS and not via a BFS (as discussed e.g. in [1]), this is not necessary. The last number when we return is the highest number.
I think that explanation is slightly wrong / misleading. It's not about DFS vs. BFS, it's about guest code vs. QEMU code. If you do a bridge setup from the guest side, you've got to set the subordinate bus number to 0xff to make sure that the bridge forwards all config space accesses to the attached devices while you're scanning the devices that are attached to the bridge.
But this code is not running in the guest, it is running in QEMU, and so it can access the config space of the attached devices directly via pci_default_write_config() - the write do not need to pass the parent bridge in this case, and thus the subordinate bus number in the bridge is ignored for the config space write access.
Quoted 9 lines
Please note that hotplugging of bridges is in general still broken, will be fixed next.
[1] http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.html
Signed-off-by David Hildenbrand <[email protected]>--- hw/s390x/s390-pci-bus.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
With the commit message fixed:
Re: [Qemu-devel] [PATCH v1 1/2] s390x/pci: Fix primary bus number for PCI bridges
Quoted 21 lines · Thomas Huth
Quoted 8 lines · David Hildenbrand
The primary bus number corresponds always to the bus number of the bus the bridge is attached to.
Right now, if we have two bridges attached to the same bus (e.g. root bus) this is however not the case. Fix assignment.
While at it
- Add a comment why we have to reassign durign every reset (which Is/durign/during/
Quoted 1 line
found to be supprising)
s/supprising/surprising/
Quoted 4 lines
- Drop setting the
PCI_SUBORDINATE_BUStemporarily to0xff. As we are setting it via a DFS and not via a BFS (as discussed e.g. in [1]), this is not necessary. The last number when we return is the highest number.
Quoted 11 lines
I think that explanation is slightly wrong / misleading. It's not about DFS vs. BFS, it's about guest code vs. QEMU code. If you do a bridge setup from the guest side, you've got to set the subordinate bus number to
0xffto make sure that the bridge forwards all config space accesses to the attached devices while you're scanning the devices that are attached to the bridge.
But this code is not running in the guest, it is running in QEMU, and so it can access the config space of the attached devices directly viapci_default_write_config()- the write do not need to pass the parent bridge in this case, and thus the subordinate bus number in the bridge is ignored for the config space write access.
Indeed, I phrased that better in the spapr/pci patch I sent, What about this:
"
The primary bus number corresponds always to the bus number of the bus the bridge is attached to.
Right now, if we have two bridges attached to the same bus (e.g. root bus) this is however not the case. The first bridge will have primary bus 0, the second bridge primary bus 1, which is wrong. Fix the assignment.
While at it, drop setting the PCI_SUBORDINATE_BUS temporarily to 0xff. Setting it temporarily to that value (as discussed e.g. in [1]), is only relevant for a running system that probes the buses. The value is effectively unused for us just doing a DFS.
Also add a comment why we have to reassign during every reset (which I found to be surprising.
Please note that hotplugging of bridges is in general still broken, will be fixed next.
"
Quoted 4 lines
Quoted 4 lines
Please note that hotplugging of bridges is in general still broken, will be fixed next.
Quoted 8 lines
Quoted 6 lines
[1] http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.html
Signed-off-by David Hildenbrand <[email protected]>--- hw/s390x/s390-pci-bus.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)With the commit message fixed:
@Conny can you fix up when applying if there are no other comments?
Quoted 3 lines
Reviewed-by Thomas Huth <[email protected]>
Thanks Thomas!
Signature · 3 lines
Thanks, David / dhildenb
Re: [Qemu-devel] [PATCH v1 1/2] s390x/pci: Fix primary bus number for PCI bridges
Quoted 63 lines · On Wed, 23 Jan 2019 11:32:48 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 21 lines · Thomas Huth
Quoted 8 lines · David Hildenbrand
The primary bus number corresponds always to the bus number of the bus the bridge is attached to.
Right now, if we have two bridges attached to the same bus (e.g. root bus) this is however not the case. Fix assignment.
While at it
- Add a comment why we have to reassign durign every reset (which Is/durign/during/
Quoted 1 line
found to be supprising)
s/supprising/surprising/
Quoted 4 lines
- Drop setting the
PCI_SUBORDINATE_BUStemporarily to0xff. As we are setting it via a DFS and not via a BFS (as discussed e.g. in [1]), this is not necessary. The last number when we return is the highest number.Quoted 11 lines
I think that explanation is slightly wrong / misleading. It's not about DFS vs. BFS, it's about guest code vs. QEMU code. If you do a bridge setup from the guest side, you've got to set the subordinate bus number to
0xffto make sure that the bridge forwards all config space accesses to the attached devices while you're scanning the devices that are attached to the bridge.
But this code is not running in the guest, it is running in QEMU, and so it can access the config space of the attached devices directly viapci_default_write_config()- the write do not need to pass the parent bridge in this case, and thus the subordinate bus number in the bridge is ignored for the config space write access.Indeed, I phrased that better in the spapr/pci patch I sent, What about this:
"
The primary bus number corresponds always to the bus number of the bus the bridge is attached to.Right now, if we have two bridges attached to the same bus (e.g. root bus) this is however not the case. The first bridge will have primary bus 0, the second bridge primary bus 1, which is wrong. Fix the assignment.
While at it, drop setting the
PCI_SUBORDINATE_BUStemporarily to0xff. Setting it temporarily to that value (as discussed e.g. in [1]), is only relevant for a running system that probes the buses. The value is effectively unused for us just doing a DFS.Also add a comment why we have to reassign during every reset (which I found to be surprising.
Please note that hotplugging of bridges is in general still broken, will be fixed next.
"Quoted 6 lines
Quoted 6 lines
Please note that hotplugging of bridges is in general still broken, will be fixed next.
[1] http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.html
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Signed-off-by David Hildenbrand <[email protected]>--- hw/s390x/s390-pci-bus.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)With the commit message fixed:
@Conny can you fix up when applying if there are no other comments?
Sure, can do. Waiting for zpci maintainer ack :)
Quoted 8 lines
Quoted 4 lines
Reviewed-by Thomas Huth <[email protected]>Thanks Thomas!
Re: [Qemu-devel] [PATCH v1 1/2] s390x/pci: Fix primary bus number for PCI bridges
In reply to David HildenbrandQuoted 41 lines · David Hildenbrand
Quoted 21 lines · Thomas Huth
Quoted 8 lines · David Hildenbrand
The primary bus number corresponds always to the bus number of the bus the bridge is attached to.
Right now, if we have two bridges attached to the same bus (e.g. root bus) this is however not the case. Fix assignment.
While at it
- Add a comment why we have to reassign durign every reset (which Is/durign/during/
Quoted 1 line
found to be supprising)
s/supprising/surprising/
Quoted 4 lines
- Drop setting the
PCI_SUBORDINATE_BUStemporarily to0xff. As we are setting it via a DFS and not via a BFS (as discussed e.g. in [1]), this is not necessary. The last number when we return is the highest number.Quoted 11 lines
I think that explanation is slightly wrong / misleading. It's not about DFS vs. BFS, it's about guest code vs. QEMU code. If you do a bridge setup from the guest side, you've got to set the subordinate bus number to
0xffto make sure that the bridge forwards all config space accesses to the attached devices while you're scanning the devices that are attached to the bridge.
But this code is not running in the guest, it is running in QEMU, and so it can access the config space of the attached devices directly viapci_default_write_config()- the write do not need to pass the parent bridge in this case, and thus the subordinate bus number in the bridge is ignored for the config space write access.Indeed, I phrased that better in the spapr/pci patch I sent, What about this:
"
The primary bus number corresponds always to the bus number of the bus the bridge is attached to.
Quoted 8 lines
Right now, if we have two bridges attached to the same bus (e.g. root bus) this is however not the case. The first bridge will have primary bus 0, the second bridge primary bus 1, which is wrong. Fix the assignment.
While at it, drop setting the
PCI_SUBORDINATE_BUStemporarily to0xff. Setting it temporarily to that value (as discussed e.g. in [1]), is only relevant for a running system that probes the buses. The value is effectively unused for us just doing a DFS.
I'd rather replace the last sentence with:
The value is not necessary in QEMU code since we can access the config space of the devices directly here, without the need to pass the config space write requests through the superordinate bridges.
Apart from that, text looks fine to me now.
Thomas
PATCH 2/2 s390x/pci: Fix hotplugging of PCI bridges
In reply to David HildenbrandWhen hotplugging a PCI bridge right now to the root port, we resolve pci_get_bus(pdev)->parent_dev, which results in a SEGFAULT. Hotplugging really only works right now when hotplugging to another bridge.
Instead, we have to properly check if we are already at the root.
Let's cleanup the code while at it a bit and factor out updating the subordiante bus number into a separate function. The check for "old_nr < nr" is right now not strictly necessary, but makes it more obvious what is actually going on.
--- hw/s390x/s390-pci-bus.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)
hw/s390x/s390-pci-bus.c +19 -9
@@ -843,6 +843,21 @@ static void s390_pcihost_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } } +static void s390_pci_update_subordinate (PCIDevice *dev,uint32_t nr) +{ +uint32_t old_nr; + +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); +while (!pci_bus_is_root (pci_get_bus (dev))) { + dev =pci_get_bus (dev)->parent_dev; + + old_nr =pci_default_read_config (dev,PCI_SUBORDINATE_BUS ,1 ); +if (old_nr < nr) { +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); + } + } +} +static void s390_pcihost_plug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {
@@ -851,26 +866,21 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,S390PCIBusDevice *pbdev =NULL ;if (object_dynamic_cast (OBJECT (dev),TYPE_PCI_BRIDGE )) { -BusState *bus;PCIBridge *pb =PCI_BRIDGE (dev); -PCIDevice *pdev =PCI_DEVICE (dev); + pdev =PCI_DEVICE (dev);pci_bridge_map_irq (pb, dev->id, s390_pci_map_irq);pci_setup_iommu (&pb->sec_bus, s390_pci_dma_iommu, s); - bus =BUS (&pb->sec_bus); -qbus_set_hotplug_handler (bus,DEVICE (s), errp); +qbus_set_hotplug_handler (BUS (&pb->sec_bus),DEVICE (s), errp);if (dev->hotplugged) {pci_default_write_config (pdev,PCI_PRIMARY_BUS ,pci_dev_bus_num (pdev),1 ); s->bus_no +=1 ;pci_default_write_config (pdev,PCI_SECONDARY_BUS , s->bus_no,1 ); -do { - pdev =pci_get_bus (pdev)->parent_dev; -pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , - s->bus_no,1 ); - }while (pci_get_bus (pdev) &&pci_dev_bus_num (pdev)); + +s390_pci_update_subordinate (pdev, s->bus_no); } }else if (object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { pdev =PCI_DEVICE (dev);
Signature · 1 line
2.17.2
Re: [Qemu-devel] [PATCH v1 2/2] s390x/pci: Fix hotplugging of PCI bridges
Quoted 43 lines · David Hildenbrand
When hotplugging a PCI bridge right now to the root port, we resolve pci_get_bus(pdev)->parent_dev, which results in a SEGFAULT. Hotplugging really only works right now when hotplugging to another bridge.
Instead, we have to properly check if we are already at the root.
Let's cleanup the code while at it a bit and factor out updating the subordiante bus number into a separate function. The check for "
old_nr< nr" is right now not strictly necessary, but makes it more obvious what is actually going on.Signed-off-by David Hildenbrand <[email protected]>--- hw/s390x/s390-pci-bus.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)hw/s390x/s390-pci-bus.c +15
-0@@ -843,6 +843,21 @@ static void s390_pcihost_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } } +static void s390_pci_update_subordinate (PCIDevice *dev,uint32_t nr) +{ +uint32_t old_nr; + +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); +while (!pci_bus_is_root (pci_get_bus (dev))) { + dev =pci_get_bus (dev)->parent_dev; + + old_nr =pci_default_read_config (dev,PCI_SUBORDINATE_BUS ,1 ); +if (old_nr < nr) { +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); + } + } +} +static void s390_pcihost_plug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 871
Quoted 8 lines
hw/s390x/s390-pci-bus.c +1
-2@@ -851,26 +866,21 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,S390PCIBusDevice *pbdev =NULL ;if (object_dynamic_cast (OBJECT (dev),TYPE_PCI_BRIDGE )) { -BusState *bus;PCIBridge *pb =PCI_BRIDGE (dev); -PCIDevice *pdev =PCI_DEVICE (dev); + pdev =PCI_DEVICE (dev);
Why not keep the direct assignment?
Quoted 9 lines
hw/s390x/s390-pci-bus.c +1
-2pci_bridge_map_irq (pb, dev->id, s390_pci_map_irq);pci_setup_iommu (&pb->sec_bus, s390_pci_dma_iommu, s); - bus =BUS (&pb->sec_bus); -qbus_set_hotplug_handler (bus,DEVICE (s), errp); +qbus_set_hotplug_handler (BUS (&pb->sec_bus),DEVICE (s), errp);if (dev->hotplugged) { pci_default_write_config(pdev, PCI_PRIMARY_BUS,
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 883
Quoted 10 lines
hw/s390x/s390-pci-bus.c +2
-5pci_dev_bus_num(pdev), 1); s->bus_no +=1 ;pci_default_write_config (pdev,PCI_SECONDARY_BUS , s->bus_no,1 ); -do { - pdev =pci_get_bus (pdev)->parent_dev; -pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , - s->bus_no,1 ); - }while (pci_get_bus (pdev) &&pci_dev_bus_num (pdev)); + +s390_pci_update_subordinate (pdev, s->bus_no);
While your changes look OK at a first glance, and certainly fix the crash, I wonder whether this is the right thing to do here at all. Why does the hypervisor set up the all the bridge registers here? I'd rather expect that this is the job of the guest after it detects a hot-plugged device?
Also I'm not sure whether the numbering is right in every case. Let's use the example from the URL that you've mentioned in the cover letter
(http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.html step 4):
Assume that you hot-plug another bridge "5" to "Bridge 2". If I get the code right, the setup will now look like this:
CPU
|
--------------+-------------- Bus 0
|
| Pri=0
BR1 Sec=1
| Sub=5
|
------+-------+----------+--- Bus 1
| |
| Pri=1 | Pri=1
BR3 Sec=3 BR2 Sec=2
| Sub=4 | Sub=5
| |
--+---+--- Bus 3 --+-+- Bus 2
| |
| Pri=3 | Pri=2
BR4 Sec=4 BR5 Sec=5
| Sub=4 | Sub=5
| |
--+---- Bus 4 --+-- Bus 5Requests to bus 3 and 4 are now also routed through to bus 2 and 5. That sounds wrong to me. Or is this how hot-plugging of PCI bridges is supposed to work??
Thomas
Re: [Qemu-devel] [PATCH v1 2/2] s390x/pci: Fix hotplugging of PCI bridges
Quoted 46 lines · Thomas Huth
Quoted 45 lines · David Hildenbrand
When hotplugging a PCI bridge right now to the root port, we resolve pci_get_bus(pdev)->parent_dev, which results in a SEGFAULT. Hotplugging really only works right now when hotplugging to another bridge.
Instead, we have to properly check if we are already at the root.
Let's cleanup the code while at it a bit and factor out updating the subordiante bus number into a separate function. The check for "
old_nr< nr" is right now not strictly necessary, but makes it more obvious what is actually going on.Signed-off-by David Hildenbrand <[email protected]>--- hw/s390x/s390-pci-bus.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)hw/s390x/s390-pci-bus.c +15
-0@@ -843,6 +843,21 @@ static void s390_pcihost_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } } +static void s390_pci_update_subordinate (PCIDevice *dev,uint32_t nr) +{ +uint32_t old_nr; + +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); +while (!pci_bus_is_root (pci_get_bus (dev))) { + dev =pci_get_bus (dev)->parent_dev; + + old_nr =pci_default_read_config (dev,PCI_SUBORDINATE_BUS ,1 ); +if (old_nr < nr) { +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); + } + } +} +static void s390_pcihost_plug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {@@ -851,26 +866,21 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,S390PCIBusDevice *pbdev =NULL ;
Quoted 8 lines
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 871
Quoted 6 lines
hw/s390x/s390-pci-bus.c +1
-2if (object_dynamic_cast (OBJECT (dev),TYPE_PCI_BRIDGE )) { -BusState *bus;PCIBridge *pb =PCI_BRIDGE (dev); -PCIDevice *pdev =PCI_DEVICE (dev); + pdev =PCI_DEVICE (dev);Why not keep the direct assignment?
We have the same local variable in that function already. I'm just reusing it.
Quoted 22 lines
Quoted 10 lines
hw/s390x/s390-pci-bus.c +1
-2pci_bridge_map_irq (pb, dev->id, s390_pci_map_irq);pci_setup_iommu (&pb->sec_bus, s390_pci_dma_iommu, s); - bus =BUS (&pb->sec_bus); -qbus_set_hotplug_handler (bus,DEVICE (s), errp); +qbus_set_hotplug_handler (BUS (&pb->sec_bus),DEVICE (s), errp);if (dev->hotplugged) { pci_default_write_config(pdev, PCI_PRIMARY_BUS,hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 883
Quoted 10 lines
hw/s390x/s390-pci-bus.c +2
-5pci_dev_bus_num(pdev), 1); s->bus_no +=1 ;pci_default_write_config (pdev,PCI_SECONDARY_BUS , s->bus_no,1 ); -do { - pdev =pci_get_bus (pdev)->parent_dev; -pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , - s->bus_no,1 ); - }while (pci_get_bus (pdev) &&pci_dev_bus_num (pdev)); + +s390_pci_update_subordinate (pdev, s->bus_no);
Quoted 5 lines
While your changes look OK at a first glance, and certainly fix the crash, I wonder whether this is the right thing to do here at all. Why does the hypervisor set up the all the bridge registers here? I'd rather expect that this is the job of the guest after it detects a hot-plugged device?
I honestly have no idea who is responsible for that. I can see that spapr does not seem to handle hotplugging the way s390x does. I was hoping that Colin maybe know why we have to do that on s390x this way. At least this patch does not change the behavior but only fixes it.
Quoted 7 lines
Also I'm not sure whether the numbering is right in every case. Let's use the example from the URL that you've mentioned in the cover letter
(http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.html step 4):Assume that you hot-plug another bridge "5" to "Bridge 2". If I get the code right, the setup will now look like this:
Yes, this is what I would expect! This implies that any bus in the hierarchy lies between Sec and Sub. But that there might be multiple candidates, which feels wrong as you correctly say.
Quoted 23 lines
CPU
|
--------------+-------------- Bus 0
|
| Pri=0
BR1 Sec=1
| Sub=5
|
------+-------+----------+--- Bus 1
| |
| Pri=1 | Pri=1
BR3 Sec=3 BR2 Sec=2
| Sub=4 | Sub=5
| |
--+---+--- Bus 3 --+-+- Bus 2
| |
| Pri=3 | Pri=2
BR4 Sec=4 BR5 Sec=5
| Sub=4 | Sub=5
| |
--+---- Bus 4 --+-- Bus 5Quoted 4 lines
Requests to bus 3 and 4 are now also routed through to bus 2 and 5. That sounds wrong to me. Or is this how hot-plugging of PCI bridges is supposed to work??
Learning more about PCI
Quoting from [2]
"PCIe enumeration is generally done twice. First, your BIOS (UEFI or otherwise) will do it, to figure out who's present and how much memory they need. This data can then be passed on to the host OS who can take it as-is, but Linux and Windows often perform their own enumeration procedure ..."
"The custom software part comes in during this enumeration process and that is you must reserve ahead of time PCI Bus numbers, and memory segments for potential future devices -- this is sometimes called 'bus padding'. This avoids the need to re-enumerate the bus in the future which can often not be done without disruption to the system ..."
So if I understand correctly, the BIOS/Firmware/QEMU only provides the initial state. After that, the guest is responsible to manage it, especially to manage hotplug.
However, I wonder if it would also be up to BIOS/Firmware/QEMU to do this bus padding or if it is completely on the OS side. And if hotplugged devices simply have all 0's.
What about special cases where hotplug happends after BIOS enumerated but before the guest started?
[2]
https://electronics.stackexchange.com/questions/208767/does-pcie-hotplug-actually-work-in-practice
Signature · 3 lines
Thanks, David / dhildenb
Thomas Huth
Reviewed-byRe: [Qemu-devel] [PATCH v1 2/2] s390x/pci: Fix hotplugging of PCI bridges
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Thomas Huth
Quoted 50 lines · David Hildenbrand
When hotplugging a PCI bridge right now to the root port, we resolve pci_get_bus(pdev)->parent_dev, which results in a SEGFAULT. Hotplugging really only works right now when hotplugging to another bridge.
Instead, we have to properly check if we are already at the root.
Let's cleanup the code while at it a bit and factor out updating the subordiante bus number into a separate function. The check for "
old_nr< nr" is right now not strictly necessary, but makes it more obvious what is actually going on.Signed-off-by David Hildenbrand <[email protected]>--- hw/s390x/s390-pci-bus.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)hw/s390x/s390-pci-bus.c +15
-2@@ -843,6 +843,21 @@ static void s390_pcihost_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } } +static void s390_pci_update_subordinate (PCIDevice *dev,uint32_t nr) +{ +uint32_t old_nr; + +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); +while (!pci_bus_is_root (pci_get_bus (dev))) { + dev =pci_get_bus (dev)->parent_dev; + + old_nr =pci_default_read_config (dev,PCI_SUBORDINATE_BUS ,1 ); +if (old_nr < nr) { +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); + } + } +} +static void s390_pcihost_plug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {@@ -851,26 +866,21 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,S390PCIBusDevice *pbdev =NULL ;if (object_dynamic_cast (OBJECT (dev),TYPE_PCI_BRIDGE )) { -BusState *bus;PCIBridge *pb =PCI_BRIDGE (dev); -PCIDevice *pdev =PCI_DEVICE (dev);
Quoted 6 lines
Quoted 3 lines
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 871
Quoted 1 line
hw/s390x/s390-pci-bus.c +1
-0+ pdev =PCI_DEVICE (dev);Why not keep the direct assignment?
We have the same local variable in that function already. I'm just reusing it.
D'oh, right. Please disregard my comment.
Quoted 28 lines
Quoted 22 lines
Quoted 10 lines
hw/s390x/s390-pci-bus.c +1
-2pci_bridge_map_irq (pb, dev->id, s390_pci_map_irq);pci_setup_iommu (&pb->sec_bus, s390_pci_dma_iommu, s); - bus =BUS (&pb->sec_bus); -qbus_set_hotplug_handler (bus,DEVICE (s), errp); +qbus_set_hotplug_handler (BUS (&pb->sec_bus),DEVICE (s), errp);if (dev->hotplugged) { pci_default_write_config(pdev, PCI_PRIMARY_BUS,hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 883
Quoted 10 lines
hw/s390x/s390-pci-bus.c +2
-5pci_dev_bus_num(pdev), 1); s->bus_no +=1 ;pci_default_write_config (pdev,PCI_SECONDARY_BUS , s->bus_no,1 ); -do { - pdev =pci_get_bus (pdev)->parent_dev; -pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , - s->bus_no,1 ); - }while (pci_get_bus (pdev) &&pci_dev_bus_num (pdev)); + +s390_pci_update_subordinate (pdev, s->bus_no);Quoted 5 lines
While your changes look OK at a first glance, and certainly fix the crash, I wonder whether this is the right thing to do here at all. Why does the hypervisor set up the all the bridge registers here? I'd rather expect that this is the job of the guest after it detects a hot-plugged device?
Quoted 4 lines
I honestly have no idea who is responsible for that. I can see that spapr does not seem to handle hotplugging the way s390x does. I was hoping that Colin maybe know why we have to do that on s390x this way. At least this patch does not change the behavior but only fixes it.
Right, you're patch certainly fixes a crash, and does not make it any worse than it was before, so:
Quoted 56 lines
Quoted 7 lines
Also I'm not sure whether the numbering is right in every case. Let's use the example from the URL that you've mentioned in the cover letter
(http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.html step 4):Assume that you hot-plug another bridge "5" to "Bridge 2". If I get the code right, the setup will now look like this:
Yes, this is what I would expect! This implies that any bus in the hierarchy lies between Sec and Sub. But that there might be multiple candidates, which feels wrong as you correctly say.
Quoted 23 lines
CPU | --------------+-------------- Bus 0 | | Pri=0 BR1 Sec=1 | Sub=5 | ------+-------+----------+--- Bus 1 | | | Pri=1 | Pri=1 BR3 Sec=3 BR2 Sec=2 | Sub=4 | Sub=5 | | --+---+--- Bus 3 --+-+- Bus 2 | | | Pri=3 | Pri=2 BR4 Sec=4 BR5 Sec=5 | Sub=4 | Sub=5 | | --+---- Bus 4 --+-- Bus 5Quoted 4 lines
Requests to bus 3 and 4 are now also routed through to bus 2 and 5. That sounds wrong to me. Or is this how hot-plugging of PCI bridges is supposed to work??
Learning more about PCI
Quoting from [2]
"PCIe enumeration is generally done twice. First, your BIOS (UEFI or otherwise) will do it, to figure out who's present and how much memory they need. This data can then be passed on to the host OS who can take it as-is, but Linux and Windows often perform their own enumeration procedure ..."
"The custom software part comes in during this enumeration process and that is you must reserve ahead of time PCI Bus numbers, and memory segments for potential future devices -- this is sometimes called 'bus padding'. This avoids the need to re-enumerate the bus in the future which can often not be done without disruption to the system ..."
Quoted 7 lines
So if I understand correctly, the BIOS/Firmware/QEMU only provides the initial state. After that, the guest is responsible to manage it, especially to manage hotplug.
However, I wonder if it would also be up to BIOS/Firmware/QEMU to do this bus padding or if it is completely on the OS side. And if hotplugged devices simply have all 0's.
Anyway, the current behavior sounds pretty wrong to me. I am sure that we can not simply traverse the bridge hierarchy up to the top and change the subordinate bus setting everywhere without informing the guest about that change. What if the guest OS has cached the values from the bridges? It then might use the wrong value for future calculations...
Quoted 2 lines
What about special cases where hotplug happends after BIOS enumerated but before the guest started?
I can't comment on s390x, but IIRC, on POWER, there is a hotplug interrupt which keeps being pending until the guest OS has initialized it's interrupt handler. Then it is up to the guest OS to set up the PCI device, if I remember clearly.
Thomas
Re: [Qemu-devel] [PATCH v1 2/2] s390x/pci: Fix hotplugging of PCI bridges
Quoted 164 lines · Thomas Huth
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Thomas Huth
Quoted 50 lines · David Hildenbrand
When hotplugging a PCI bridge right now to the root port, we resolve pci_get_bus(pdev)->parent_dev, which results in a SEGFAULT. Hotplugging really only works right now when hotplugging to another bridge.
Instead, we have to properly check if we are already at the root.
Let's cleanup the code while at it a bit and factor out updating the subordiante bus number into a separate function. The check for "
old_nr< nr" is right now not strictly necessary, but makes it more obvious what is actually going on.Signed-off-by David Hildenbrand <[email protected]>--- hw/s390x/s390-pci-bus.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)hw/s390x/s390-pci-bus.c +15
-2@@ -843,6 +843,21 @@ static void s390_pcihost_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } } +static void s390_pci_update_subordinate (PCIDevice *dev,uint32_t nr) +{ +uint32_t old_nr; + +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); +while (!pci_bus_is_root (pci_get_bus (dev))) { + dev =pci_get_bus (dev)->parent_dev; + + old_nr =pci_default_read_config (dev,PCI_SUBORDINATE_BUS ,1 ); +if (old_nr < nr) { +pci_default_write_config (dev,PCI_SUBORDINATE_BUS , nr,1 ); + } + } +} +static void s390_pcihost_plug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {@@ -851,26 +866,21 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,S390PCIBusDevice *pbdev =NULL ;if (object_dynamic_cast (OBJECT (dev),TYPE_PCI_BRIDGE )) { -BusState *bus;PCIBridge *pb =PCI_BRIDGE (dev); -PCIDevice *pdev =PCI_DEVICE (dev);Quoted 6 lines
Quoted 3 lines
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 871
Quoted 1 line
hw/s390x/s390-pci-bus.c +1
-0+ pdev =PCI_DEVICE (dev);Why not keep the direct assignment?
We have the same local variable in that function already. I'm just reusing it.
D'oh, right. Please disregard my comment.
Quoted 28 lines
Quoted 22 lines
Quoted 10 lines
hw/s390x/s390-pci-bus.c +1
-2pci_bridge_map_irq (pb, dev->id, s390_pci_map_irq);pci_setup_iommu (&pb->sec_bus, s390_pci_dma_iommu, s); - bus =BUS (&pb->sec_bus); -qbus_set_hotplug_handler (bus,DEVICE (s), errp); +qbus_set_hotplug_handler (BUS (&pb->sec_bus),DEVICE (s), errp);if (dev->hotplugged) { pci_default_write_config(pdev, PCI_PRIMARY_BUS,hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 883
Quoted 10 lines
hw/s390x/s390-pci-bus.c +2
-5pci_dev_bus_num(pdev), 1); s->bus_no +=1 ;pci_default_write_config (pdev,PCI_SECONDARY_BUS , s->bus_no,1 ); -do { - pdev =pci_get_bus (pdev)->parent_dev; -pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , - s->bus_no,1 ); - }while (pci_get_bus (pdev) &&pci_dev_bus_num (pdev)); + +s390_pci_update_subordinate (pdev, s->bus_no);Quoted 5 lines
While your changes look OK at a first glance, and certainly fix the crash, I wonder whether this is the right thing to do here at all. Why does the hypervisor set up the all the bridge registers here? I'd rather expect that this is the job of the guest after it detects a hot-plugged device?
Quoted 4 lines
I honestly have no idea who is responsible for that. I can see that spapr does not seem to handle hotplugging the way s390x does. I was hoping that Colin maybe know why we have to do that on s390x this way. At least this patch does not change the behavior but only fixes it.
Right, you're patch certainly fixes a crash, and does not make it any worse than it was before, so:
Reviewed-by Thomas Huth <[email protected]>Quoted 56 lines
Quoted 7 lines
Also I'm not sure whether the numbering is right in every case. Let's use the example from the URL that you've mentioned in the cover letter
(http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.html step 4):Assume that you hot-plug another bridge "5" to "Bridge 2". If I get the code right, the setup will now look like this:
Yes, this is what I would expect! This implies that any bus in the hierarchy lies between Sec and Sub. But that there might be multiple candidates, which feels wrong as you correctly say.
Quoted 23 lines
CPU | --------------+-------------- Bus 0 | | Pri=0 BR1 Sec=1 | Sub=5 | ------+-------+----------+--- Bus 1 | | | Pri=1 | Pri=1 BR3 Sec=3 BR2 Sec=2 | Sub=4 | Sub=5 | | --+---+--- Bus 3 --+-+- Bus 2 | | | Pri=3 | Pri=2 BR4 Sec=4 BR5 Sec=5 | Sub=4 | Sub=5 | | --+---- Bus 4 --+-- Bus 5Quoted 4 lines
Requests to bus 3 and 4 are now also routed through to bus 2 and 5. That sounds wrong to me. Or is this how hot-plugging of PCI bridges is supposed to work??
Learning more about PCI
Quoting from [2]
"PCIe enumeration is generally done twice. First, your BIOS (UEFI or otherwise) will do it, to figure out who's present and how much memory they need. This data can then be passed on to the host OS who can take it as-is, but Linux and Windows often perform their own enumeration procedure ..."
"The custom software part comes in during this enumeration process and that is you must reserve ahead of time PCI Bus numbers, and memory segments for potential future devices -- this is sometimes called 'bus padding'. This avoids the need to re-enumerate the bus in the future which can often not be done without disruption to the system ..."
Quoted 7 lines
So if I understand correctly, the BIOS/Firmware/QEMU only provides the initial state. After that, the guest is responsible to manage it, especially to manage hotplug.
However, I wonder if it would also be up to BIOS/Firmware/QEMU to do this bus padding or if it is completely on the OS side. And if hotplugged devices simply have all 0's.
Quoted 5 lines
Anyway, the current behavior sounds pretty wrong to me. I am sure that we can not simply traverse the bridge hierarchy up to the top and change the subordinate bus setting everywhere without informing the guest about that change. What if the guest OS has cached the values from the bridges? It then might use the wrong value for future calculations...
Yes, it also sounds wrong to me. Maybe this is what Firmware manages on s390x - maybe they reserve bus numbers. Or maybe hotplug has to really be handled by the guests. Only IBM people can clarify that.
Thanks!
Signature · 3 lines
Thanks, David / dhildenb
Quoted 10 lines · On Tue, 22 Jan 2019 13:51:31 +0100 David Hildenbrand <[email protected]> wrote:
Hotplugging of PCI bridges is right now pretty much broken. Coldplugging and hotplugging will assign wrong primary bus numbers in some scenarios.
I base my knowledge on how this is supposed to work on
http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.htmlI did a couple of tests, building whole hierarchies of bridges, both hot and coldplugged. "info pci" as well as the Linux guests showed what I was expecting.
Quoted 7 lines
David Hildenbrand (2):
s390x/pci: Fix primary bus number for PCI bridges s390x/pci: Fix hotplugging of PCI bridgeshw/s390x/s390-pci-bus.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-)
I'll leave the actual review of this to folks familiar with how zPCI is supposed to work :)
Does the guest actually see anything of this? Consistency is good, and not crashing even better, but I think all of the topology is invisible on the guest side anyway...
Quoted 21 lines · Cornelia Huck
Quoted 10 lines · On Tue, 22 Jan 2019 13:51:31 +0100 David Hildenbrand <[email protected]> wrote:
Hotplugging of PCI bridges is right now pretty much broken. Coldplugging and hotplugging will assign wrong primary bus numbers in some scenarios.
I base my knowledge on how this is supposed to work on
http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.htmlI did a couple of tests, building whole hierarchies of bridges, both hot and coldplugged. "info pci" as well as the Linux guests showed what I was expecting.
Quoted 7 lines
David Hildenbrand (2):
s390x/pci: Fix primary bus number for PCI bridges s390x/pci: Fix hotplugging of PCI bridgeshw/s390x/s390-pci-bus.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-)
Quoted 7 lines
I'll leave the actual review of this to folks familiar with how zPCI is supposed to work :)
Does the guest actually see anything of this? Consistency is good, and not crashing even better, but I think all of the topology is invisible on the guest side anyway...
I am no PCI expert, but I think the guest is able to read/write these numbers via the configuration space. As far as I can see, on x86 the BIOS builds the topology. On Power/s390x this job is delegated to firmware / QEMU.
There are quite some numbers in QEMU relying on these numbers to be
correct: E.g. pci_secondary_bus_in_range()
If the guest relies on the topology to be correct, things can go wrong. I have no idea how Linux guests actually use the topology.
Also, the output of "info pci" will be wrong.
Anyhow, this is the right thing to do, but I agree that Patch #1 might not be as critical as patch #2.
Signature · 3 lines
Thanks, David / dhildenb
Quoted 35 lines · On Wed, 23 Jan 2019 09:02:34 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 21 lines · Cornelia Huck
Quoted 10 lines · On Tue, 22 Jan 2019 13:51:31 +0100 David Hildenbrand <[email protected]> wrote:
Hotplugging of PCI bridges is right now pretty much broken. Coldplugging and hotplugging will assign wrong primary bus numbers in some scenarios.
I base my knowledge on how this is supposed to work on
http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.htmlI did a couple of tests, building whole hierarchies of bridges, both hot and coldplugged. "info pci" as well as the Linux guests showed what I was expecting.
Quoted 7 lines
David Hildenbrand (2):
s390x/pci: Fix primary bus number for PCI bridges s390x/pci: Fix hotplugging of PCI bridgeshw/s390x/s390-pci-bus.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-)Quoted 7 lines
I'll leave the actual review of this to folks familiar with how zPCI is supposed to work :)
Does the guest actually see anything of this? Consistency is good, and not crashing even better, but I think all of the topology is invisible on the guest side anyway...
I am no PCI expert, but I think the guest is able to read/write these numbers via the configuration space. As far as I can see, on x86 the BIOS builds the topology. On Power/s390x this job is delegated to firmware / QEMU.
Quoted 5 lines
There are quite some numbers in QEMU relying on these numbers to be
correct: E.g.pci_secondary_bus_in_range()If the guest relies on the topology to be correct, things can go wrong. I have no idea how Linux guests actually use the topology.
Ok, if this shows up in the config space, it is guest visible. I was thinking mostly about the zPCI enumeration, which results in a flat layout.
Quoted 5 lines
Also, the output of "info pci" will be wrong.
Anyhow, this is the right thing to do, but I agree that Patch #1 might not be as critical as patch #2.
I'm certainly not arguing against inclusion :)
Quoted 10 lines · On Tue, 22 Jan 2019 13:51:31 +0100 David Hildenbrand <[email protected]> wrote:
Hotplugging of PCI bridges is right now pretty much broken. Coldplugging and hotplugging will assign wrong primary bus numbers in some scenarios.
I base my knowledge on how this is supposed to work on
http://www.science.unitn.it/~fiorella/guidelinux/tlk/node76.htmlI did a couple of tests, building whole hierarchies of bridges, both hot and coldplugged. "info pci" as well as the Linux guests showed what I was expecting.
Quoted 7 lines
David Hildenbrand (2):
s390x/pci: Fix primary bus number for PCI bridges s390x/pci: Fix hotplugging of PCI bridgeshw/s390x/s390-pci-bus.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-)
So, if I understand the discussion correctly:
- Patch 1 is fine, just needs a description tweak.
- Patch 2 fixes a crash, and certainly makes things better. However, current handling seems to be at least a bit odd, if not broken; something to be looked into later.
I'd like to queue these, can I get an ack/feedback from IBM?