pcie-root-port: let it has higher migrate priority
PATCH 1/1 pcie-root-port: let it has higher migrate priority
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.
So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
---
Marcel & all,
I think it's possible that we need similar thing for other bridge-like devices, but I'm not that familiar. Would you help confirm? Thanks,
--- hw/pci-bridge/gen_pcie_root_port.c | 1 + include/migration/vmstate.h | 1 + 2 files changed, 2 insertions(+)
hw/pci-bridge/gen_pcie_root_port.c +1 -0
@@ -101,6 +101,7 @@ static void gen_rp_realize(DeviceState *dev, Error **errp)static const VMStateDescription vmstate_rp_dev = { .name ="pcie-root-port" , + .priority =MIG_PRI_PCIE_ROOT_PORT , .version_id =1 , .minimum_version_id =1 , .post_load = pcie_cap_slot_post_load,
include/migration/vmstate.h +1 -0
@@ -148,6 +148,7 @@ enum VMStateFlags { typedef enum { MIG_PRI_DEFAULT = 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_PCIE_ROOT_PORT, /* Must happen before IOMMU */ MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX,
Signature · 1 line
2.14.3
Hi Peter,
Quoted 32 lines · Peter Xu
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
CC Alex Williamson <[email protected]>CC Marcel Apfelbaum <[email protected]>CC Michael S. Tsirkin <[email protected]>CC Dr. David Alan Gilbert <[email protected]>CC Juan Quintela <[email protected]>CC Laurent Vivier <[email protected]>Reported-by Maxime Coquelin <[email protected]>Signed-off-by Peter Xu <[email protected]>
Quoted 5 lines
---
Marcel & all,I think it's possible that we need similar thing for other bridge-like devices, but I'm not that familiar. Would you help confirm? Thanks,
Is a pity we don't have a way to mark the migration priority in a base class. Dave, maybe we do have a way?
In the meantime you would need to add it also to:
- ioh3420 (Intel root port)
- xio3130_downstream (Intel switch downstream port)
- xio3130_upstream (The counterpart of the above, you want the whole switch to be migrated before loading the IOMMU device state)
- pcie_pci_bridge (for pci devices)
- pci-pci bridge (if for some reason you have one attached to the pcie_pci_brdge)
- i82801b11 (dmi-pci bridge, we want to deprecate it bu is there for now)
Thanks,
Marcel
Quoted 29 lines
hw/pci-bridge/gen_pcie_root_port.c | 1 + include/migration/vmstate.h | 1 + 2 files changed, 2 insertions(+)hw/pci-bridge/gen_pcie_root_port.c +1
-0@@ -101,6 +101,7 @@ static void gen_rp_realize(DeviceState *dev, Error **errp)static const VMStateDescription vmstate_rp_dev = { .name ="pcie-root-port" , + .priority =MIG_PRI_PCIE_ROOT_PORT , .version_id =1 , .minimum_version_id =1 , .post_load = pcie_cap_slot_post_load,include/migration/vmstate.h +1
-0@@ -148,6 +148,7 @@ enum VMStateFlags { typedef enum { MIG_PRI_DEFAULT = 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_PCIE_ROOT_PORT, /* Must happen before IOMMU */ MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX,
Quoted 53 lines · Marcel Apfelbaum
Hi Peter,
Quoted 32 lines · Peter Xu
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
CC Alex Williamson <[email protected]>CC Marcel Apfelbaum <[email protected]>CC Michael S. Tsirkin <[email protected]>CC Dr. David Alan Gilbert <[email protected]>CC Juan Quintela <[email protected]>CC Laurent Vivier <[email protected]>Reported-by Maxime Coquelin <[email protected]>Signed-off-by Peter Xu <[email protected]>Quoted 5 lines
---
Marcel & all,I think it's possible that we need similar thing for other bridge-like devices, but I'm not that familiar. Would you help confirm? Thanks,
Is a pity we don't have a way to mark the migration priority in a base class. Dave, maybe we do have a way?
In the meantime you would need to add it also to:
- ioh3420 (Intel root port)
-xio3130_downstream(Intel switch downstream port)
-xio3130_upstream(The counterpart of the above, you want the whole switch to be migrated before loading the IOMMU device state)
-pcie_pci_bridge(for pci devices)
- pci-pci bridge (if for some reason you have one attached to thepcie_pci_brdge)
- i82801b11 (dmi-pci bridge, we want to deprecate it bu is there for now)
Quoted 2 lines
Thanks,
Marcel
It's kind of strange that we need to set the priority manually. Can't migration figure it out itself? I think bus must always be migrated before the devices behind it ...
Quoted 29 lines
Quoted 29 lines
hw/pci-bridge/gen_pcie_root_port.c | 1 + include/migration/vmstate.h | 1 + 2 files changed, 2 insertions(+)hw/pci-bridge/gen_pcie_root_port.c +1
-0@@ -101,6 +101,7 @@ static void gen_rp_realize(DeviceState *dev, Error **errp)static const VMStateDescription vmstate_rp_dev = { .name ="pcie-root-port" , + .priority =MIG_PRI_PCIE_ROOT_PORT , .version_id =1 , .minimum_version_id =1 , .post_load = pcie_cap_slot_post_load,include/migration/vmstate.h +1
-0@@ -148,6 +148,7 @@ enum VMStateFlags { typedef enum { MIG_PRI_DEFAULT = 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_PCIE_ROOT_PORT, /* Must happen before IOMMU */ MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX,
Quoted 54 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 53 lines · Marcel Apfelbaum
Hi Peter,
Quoted 32 lines · Peter Xu
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
CC Alex Williamson <[email protected]>CC Marcel Apfelbaum <[email protected]>CC Michael S. Tsirkin <[email protected]>CC Dr. David Alan Gilbert <[email protected]>CC Juan Quintela <[email protected]>CC Laurent Vivier <[email protected]>Reported-by Maxime Coquelin <[email protected]>Signed-off-by Peter Xu <[email protected]>Quoted 5 lines
---
Marcel & all,I think it's possible that we need similar thing for other bridge-like devices, but I'm not that familiar. Would you help confirm? Thanks,
Is a pity we don't have a way to mark the migration priority in a base class. Dave, maybe we do have a way?
In the meantime you would need to add it also to:
- ioh3420 (Intel root port)
-xio3130_downstream(Intel switch downstream port)
-xio3130_upstream(The counterpart of the above, you want the whole switch to be migrated before loading the IOMMU device state)
-pcie_pci_bridge(for pci devices)
- pci-pci bridge (if for some reason you have one attached to thepcie_pci_brdge)
- i82801b11 (dmi-pci bridge, we want to deprecate it bu is there for now)
Quoted 8 lines
Quoted 2 lines
Thanks,
MarcelIt's kind of strange that we need to set the priority manually. Can't migration figure it out itself?
I think bus
must always be migrated before the devices behind it ...
I think that's true; but:
a) Is the iommu a child of any of the PCI busses?
b) does anything ensure that the bridge that's a parent of a bus gets migrated before the bus it provides?
c) What happens with more htan one root port?
Dave
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
hw/pci-bridge/gen_pcie_root_port.c | 1 + include/migration/vmstate.h | 1 + 2 files changed, 2 insertions(+)hw/pci-bridge/gen_pcie_root_port.c +1
-0@@ -101,6 +101,7 @@ static void gen_rp_realize(DeviceState *dev, Error **errp)static const VMStateDescription vmstate_rp_dev = { .name ="pcie-root-port" , + .priority =MIG_PRI_PCIE_ROOT_PORT , .version_id =1 , .minimum_version_id =1 , .post_load = pcie_cap_slot_post_load,include/migration/vmstate.h +0
-0@@ -148,6 +148,7 @@ enum VMStateFlags {
Quoted 8 lines
Quoted 8 lines
include/migration/vmstate.h enum VMStateFlags { · line 154
Quoted 8 lines
include/migration/vmstate.h +1
-0typedef enum { MIG_PRI_DEFAULT = 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_PCIE_ROOT_PORT, /* Must happen before IOMMU */ MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX,
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 58 lines · Dr. David Alan Gilbert
Quoted 57 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 55 lines · Marcel Apfelbaum
Hi Peter,
Quoted 32 lines · Peter Xu
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
CC Alex Williamson <[email protected]>CC Marcel Apfelbaum <[email protected]>CC Michael S. Tsirkin <[email protected]>CC Dr. David Alan Gilbert <[email protected]>CC Juan Quintela <[email protected]>CC Laurent Vivier <[email protected]>Reported-by Maxime Coquelin <[email protected]>Signed-off-by Peter Xu <[email protected]>Quoted 5 lines
---
Marcel & all,I think it's possible that we need similar thing for other bridge-like devices, but I'm not that familiar. Would you help confirm? Thanks,
Is a pity we don't have a way to mark the migration priority in a base class. Dave, maybe we do have a way?
In the meantime you would need to add it also to:
- ioh3420 (Intel root port)
-xio3130_downstream(Intel switch downstream port)
-xio3130_upstream(The counterpart of the above, you want the whole switch to be migrated before loading the IOMMU device state)
-pcie_pci_bridge(for pci devices)
- pci-pci bridge (if for some reason you have one attached to thepcie_pci_brdge)
- i82801b11 (dmi-pci bridge, we want to deprecate it bu is there for now)Thanks,
Marcel
Quoted 8 lines
Quoted 5 lines
It's kind of strange that we need to set the priority manually. Can't migration figure it out itself?
I think bus
must always be migrated before the devices behind it ...I think that's true; but:
a) Is the iommu a child of any of the PCI busses?
No, is a sysbus device.
Quoted 2 lines
b) does anything ensure that the bridge that's a parent of a bus gets migrated before the bus it provides?
I think this was Michael's question :)
Quoted 2 lines
c) What happens with more htan one root port?
Root ports can't be nested, anyway, I suppose the migration should follow the bus numbering order.
The question now is what happens if the migration is happening before the guest firmware finishes assigning numbers to buses...
Still, QEMU has enough information to decide the right ordering, the question is if the current migration mechanism has some ordering or the only one is the "VMStateFlags".
Thanks,
Marcel
Quoted 35 lines
Dave
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
hw/pci-bridge/gen_pcie_root_port.c | 1 + include/migration/vmstate.h | 1 + 2 files changed, 2 insertions(+)hw/pci-bridge/gen_pcie_root_port.c +1
-0@@ -101,6 +101,7 @@ static void gen_rp_realize(DeviceState *dev, Error **errp)static const VMStateDescription vmstate_rp_dev = { .name ="pcie-root-port" , + .priority =MIG_PRI_PCIE_ROOT_PORT , .version_id =1 , .minimum_version_id =1 , .post_load = pcie_cap_slot_post_load,include/migration/vmstate.h +0
-0@@ -148,6 +148,7 @@ enum VMStateFlags {Quoted 8 lines
Quoted 8 lines
include/migration/vmstate.h enum VMStateFlags { · line 154
Quoted 8 lines
include/migration/vmstate.h +1
-0typedef enum { MIG_PRI_DEFAULT = 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_PCIE_ROOT_PORT, /* Must happen before IOMMU */ MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX,--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 62 lines · * Marcel Apfelbaum ([email protected]) wrote:
Quoted 61 lines · Dr. David Alan Gilbert
Quoted 59 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 53 lines · Marcel Apfelbaum
Hi Peter,
Quoted 32 lines · Peter Xu
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
CC Alex Williamson <[email protected]>CC Marcel Apfelbaum <[email protected]>CC Michael S. Tsirkin <[email protected]>CC Dr. David Alan Gilbert <[email protected]>CC Juan Quintela <[email protected]>CC Laurent Vivier <[email protected]>Reported-by Maxime Coquelin <[email protected]>Signed-off-by Peter Xu <[email protected]>Quoted 5 lines
---
Marcel & all,I think it's possible that we need similar thing for other bridge-like devices, but I'm not that familiar. Would you help confirm? Thanks,
Is a pity we don't have a way to mark the migration priority in a base class. Dave, maybe we do have a way?
In the meantime you would need to add it also to:
- ioh3420 (Intel root port)
-xio3130_downstream(Intel switch downstream port)
-xio3130_upstream(The counterpart of the above, you want the whole switch to be migrated before loading the IOMMU device state)
-pcie_pci_bridge(for pci devices)
- pci-pci bridge (if for some reason you have one attached to thepcie_pci_brdge)
- i82801b11 (dmi-pci bridge, we want to deprecate it bu is there for now)Quoted 2 lines
Thanks,
MarcelIt's kind of strange that we need to set the priority manually. Can't migration figure it out itself?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I think bus
must always be migrated before the devices behind it ...I think that's true; but:
a) Is the iommu a child of any of the PCI busses?No, is a sysbus device.
OK, so even if we were arguing about whether PCI busses got migrated in order, that doesn't help Peter's case, because there's nothing that orders the iommu relative to the PCI.
Quoted 11 lines
Quoted 2 lines
b) does anything ensure that the bridge that's a parent of a bus gets migrated before the bus it provides?
I think this was Michael's question :)
Quoted 2 lines
c) What happens with more htan one root port?
Root ports can't be nested, anyway, I suppose the migration should follow the bus numbering order.
Quoted 6 lines
The question now is what happens if the migration is happening before the guest firmware finishes assigning numbers to buses...
Still, QEMU has enough information to decide the right ordering, the question is if the current migration mechanism has some ordering or the only one is the "VMStateFlags".
It's ordered on two things:
a) The priority field that Peter is using
b) The order of registration of the device.
(b) is of course dangerously unstable.
Dave
Quoted 39 lines
Thanks,
MarcelQuoted 35 lines
Dave
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
hw/pci-bridge/gen_pcie_root_port.c | 1 + include/migration/vmstate.h | 1 + 2 files changed, 2 insertions(+)hw/pci-bridge/gen_pcie_root_port.c +1
-0@@ -101,6 +101,7 @@ static void gen_rp_realize(DeviceState *dev, Error **errp)static const VMStateDescription vmstate_rp_dev = { .name ="pcie-root-port" , + .priority =MIG_PRI_PCIE_ROOT_PORT , .version_id =1 , .minimum_version_id =1 , .post_load = pcie_cap_slot_post_load,include/migration/vmstate.h +0
-0@@ -148,6 +148,7 @@ enum VMStateFlags {Quoted 8 lines
Quoted 8 lines
include/migration/vmstate.h enum VMStateFlags { · line 154
Quoted 8 lines
include/migration/vmstate.h +1
-0typedef enum { MIG_PRI_DEFAULT = 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_PCIE_ROOT_PORT, /* Must happen before IOMMU */ MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX,--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
On Thu, Feb 01, 2018 at 10:01:31PM +0200, Marcel Apfelbaum wrote:
[...]
Quoted 2 lines
Root ports can't be nested, anyway, I suppose the migration should follow the bus numbering order.
Could I ask whether this is a must? And if yes, why?
Quoted 3 lines
The question now is what happens if the migration is happening before the guest firmware finishes assigning numbers to buses...
Do you mean that vIOMMU may fetch wrong context entries too?
Note that as long as vIOMMU DMAR is off globally, vIOMMU will not fetch context entries at all. So IMHO this problem should not happen during the firmware execution time (assuming that the firmware should not enable vIOMMU at all).
Thanks,
Signature · 1 line
Peter Xu
Quoted 9 lines · Peter Xu
On Thu, Feb 01, 2018 at 10:01:31PM +0200, Marcel Apfelbaum wrote:
[...]
Quoted 2 lines
Root ports can't be nested, anyway, I suppose the migration should follow the bus numbering order.
Could I ask whether this is a must? And if yes, why?
Not sure. The above will ensure that if a device needs some parent/bus info at load time, the information will be valid.
But if it worked until now, maybe most of the devices do not need that.
Quoted 6 lines
Quoted 3 lines
The question now is what happens if the migration is happening before the guest firmware finishes assigning numbers to buses...
Do you mean that vIOMMU may fetch wrong context entries too?
No, only that the bus number will not be available at load time. In this case is OK since the firmware will continue to assign bus numbers at target side.
Thanks,
Marcel
Quoted 7 lines
Note that as long as vIOMMU DMAR is off globally, vIOMMU will not fetch context entries at all. So IMHO this problem should not happen during the firmware execution time (assuming that the firmware should not enable vIOMMU at all).
Thanks,
Quoted 38 lines · * Marcel Apfelbaum ([email protected]) wrote:
Hi Peter,
Quoted 35 lines · Peter Xu
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
CC Alex Williamson <[email protected]>CC Marcel Apfelbaum <[email protected]>CC Michael S. Tsirkin <[email protected]>CC Dr. David Alan Gilbert <[email protected]>CC Juan Quintela <[email protected]>CC Laurent Vivier <[email protected]>Reported-by Maxime Coquelin <[email protected]>Signed-off-by Peter Xu <[email protected]>---
Marcel & all,
Quoted 5 lines
Quoted 2 lines
I think it's possible that we need similar thing for other bridge-like devices, but I'm not that familiar. Would you help confirm? Thanks,
Is a pity we don't have a way to mark the migration priority in a base class. Dave, maybe we do have a way?
Not that I'm aware of; the 'priority' field is associated with the VMSD and it's not really connected to the class hierarchy at all.
Dave
Quoted 33 lines
In the meantime you would need to add it also to:
- ioh3420 (Intel root port)
-xio3130_downstream(Intel switch downstream port)
-xio3130_upstream(The counterpart of the above, you want the whole switch to be migrated before loading the IOMMU device state)
-pcie_pci_bridge(for pci devices)
- pci-pci bridge (if for some reason you have one attached to thepcie_pci_brdge)
- i82801b11 (dmi-pci bridge, we want to deprecate it bu is there for now)Thanks,
MarcelQuoted 21 lines
hw/pci-bridge/gen_pcie_root_port.c | 1 + include/migration/vmstate.h | 1 + 2 files changed, 2 insertions(+)hw/pci-bridge/gen_pcie_root_port.c +1
-0@@ -101,6 +101,7 @@ static void gen_rp_realize(DeviceState *dev, Error **errp)static const VMStateDescription vmstate_rp_dev = { .name ="pcie-root-port" , + .priority =MIG_PRI_PCIE_ROOT_PORT , .version_id =1 , .minimum_version_id =1 , .post_load = pcie_cap_slot_post_load,include/migration/vmstate.h +0
-0@@ -148,6 +148,7 @@ enum VMStateFlags {
Quoted 9 lines
include/migration/vmstate.h enum VMStateFlags { · line 154
Quoted 8 lines
include/migration/vmstate.h +1
-0typedef enum { MIG_PRI_DEFAULT = 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_PCIE_ROOT_PORT, /* Must happen before IOMMU */ MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX,
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 44 lines · Marcel Apfelbaum
Hi Peter,
Quoted 32 lines · Peter Xu
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
CC Alex Williamson <[email protected]>CC Marcel Apfelbaum <[email protected]>CC Michael S. Tsirkin <[email protected]>CC Dr. David Alan Gilbert <[email protected]>CC Juan Quintela <[email protected]>CC Laurent Vivier <[email protected]>Reported-by Maxime Coquelin <[email protected]>Signed-off-by Peter Xu <[email protected]>Quoted 5 lines
---
Marcel & all,I think it's possible that we need similar thing for other bridge-like devices, but I'm not that familiar. Would you help confirm? Thanks,
Is a pity we don't have a way to mark the migration priority in a base class. Dave, maybe we do have a way?
Quoted 8 lines
In the meantime you would need to add it also to:
- ioh3420 (Intel root port)
-xio3130_downstream(Intel switch downstream port)
-xio3130_upstream(The counterpart of the above, you want the whole switch to be migrated before loading the IOMMU device state)
-pcie_pci_bridge(for pci devices)
- pci-pci bridge (if for some reason you have one attached to thepcie_pci_brdge)
- i82801b11 (dmi-pci bridge, we want to deprecate it bu is there for now)
I'll see whether there is any better way to do this instead of duplicating, but I'm not really sure about it.
Anyway, this list is helpful. Thanks Marcel.
Signature · 1 line
Peter Xu
Quoted 17 lines · * Peter Xu ([email protected]) wrote:
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.
Quoted 5 lines
So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
Two questions (partially overlapping with what I replied to Michaels):
a) What happens with multiple IOMMUs?
b) What happens with multiple root ports?
c) How correct is this ordering on different implementations
(e.g. ARM/Power/etc)
Dave
Quoted 47 lines
CC Alex Williamson <[email protected]>CC Marcel Apfelbaum <[email protected]>CC Michael S. Tsirkin <[email protected]>CC Dr. David Alan Gilbert <[email protected]>CC Juan Quintela <[email protected]>CC Laurent Vivier <[email protected]>Reported-by Maxime Coquelin <[email protected]>Signed-off-by Peter Xu <[email protected]>---
Marcel & all,I think it's possible that we need similar thing for other bridge-like devices, but I'm not that familiar. Would you help confirm? Thanks,
--- hw/pci-bridge/gen_pcie_root_port.c | 1 + include/migration/vmstate.h | 1 + 2 files changed, 2 insertions(+)hw/pci-bridge/gen_pcie_root_port.c +1
-0@@ -101,6 +101,7 @@ static void gen_rp_realize(DeviceState *dev, Error **errp)static const VMStateDescription vmstate_rp_dev = { .name ="pcie-root-port" , + .priority =MIG_PRI_PCIE_ROOT_PORT , .version_id =1 , .minimum_version_id =1 , .post_load = pcie_cap_slot_post_load,include/migration/vmstate.h +1
-0@@ -148,6 +148,7 @@ enum VMStateFlags { typedef enum { MIG_PRI_DEFAULT = 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_PCIE_ROOT_PORT, /* Must happen before IOMMU */ MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX,Signature · 1 line
2.14.3
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 18 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Peter Xu ([email protected]) wrote:
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.
Quoted 8 lines
Quoted 5 lines
So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
Two questions (partially overlapping with what I replied to Michaels):
a) What happens with multiple IOMMUs?
If there are more IOMMUs, then the patch will let all the vIOMMUs be migrated after pcie root ports.
But a more true answer is that: I don't really know. :)
Because I even don't know how multiple vIOMMUs will coop with each other, especially nested. In nested case, maybe there will be dependency between vIOMMUs, but I'll avoid thinking about that until we support more than one vIOMMUs.
Quoted 1 line
b) What happens with multiple root ports?
Same answer as previous one: all of them will be migrated before any vIOMMUs.
Note that IMHO we don't care which pcie root port is migrated first - IMHO they should not depend on each other, but Marcel may correct me.
Quoted 2 lines
c) How correct is this ordering on different implementations
(e.g. ARM/Power/etc)
Currently it won't affect since Intel IOMMU is the only user for MIG_PRI_IOMMU. After SMMU is merged it may affect (if it uses this bit), but IMHO it's fine too as long as pcie root ports won't depend on anything related to SMMU.
Thanks,
Signature · 1 line
Peter Xu
Quoted 28 lines · Peter Xu
Quoted 18 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Peter Xu ([email protected]) wrote:
In the past, we prioritized IOMMU migration so that we have such a priority order:
IOMMU > PCI Devices
When migrating a guest with both vIOMMU and pcie-root-port, we'll always migrate vIOMMU first, since pcie-root-port will be seen to have the same priority of general PCI devices.
That's problematic.
The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during
post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up.Quoted 8 lines
Quoted 5 lines
So let's boost the PCIe root ports to be even with higher priority:
PCIe Root Port > IOMMU > PCI Devices
A smoke test shows that this patch fixes bug 1538953.
Two questions (partially overlapping with what I replied to Michaels):
a) What happens with multiple IOMMUs?
Quoted 7 lines
If there are more IOMMUs, then the patch will let all the vIOMMUs be migrated after pcie root ports.
But a more true answer is that: I don't really know. :)
Because I even don't know how multiple vIOMMUs will coop with each other, especially nested.
I am not aware of "nested" IOMMUs. Multiple IOMMUs work together by dividing the bus ranges, when each of them declares in the corresponding ACPI table the bus/device/range is in charge of.
However there was a kernel bug some time ago preventing several IOMMUs to work together, I am not sure the problem is solved yet.
In nested case, maybe there will be
Quoted 11 lines
dependency between vIOMMUs, but I'll avoid thinking about that until we support more than one vIOMMUs.
Quoted 1 line
b) What happens with multiple root ports?
Same answer as previous one: all of them will be migrated before any vIOMMUs.
Note that IMHO we don't care which pcie root port is migrated first - IMHO they should not depend on each other, but Marcel may correct me.
Right, each Root Port is independent from each other.
Thanks,
Marcel
Quoted 10 lines
Quoted 2 lines
c) How correct is this ordering on different implementations
(e.g. ARM/Power/etc)Currently it won't affect since Intel IOMMU is the only user for
MIG_PRI_IOMMU. After SMMU is merged it may affect (if it uses this bit), but IMHO it's fine too as long as pcie root ports won't depend on anything related to SMMU.Thanks,