SR-IOV PF reset and QEMU VFs VFIO passthrough
Hello,
I may have soon the PF driver of an SR-IOV card to code and make work with QEMU/KVM so I have the following questions.
In an AMD64 setup where QEMU use VFIO to passthrough the VFs of an SR-IOV card to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?
I read that pci_reset_function would call pci_restore_state restoring the SR-IOV configuration after the reset of the PF.
The ways the hardware work means that the VFs would disappear and reappear in a short lapse of time.
Will these events be handled by the kernel pci hotplug code ?
Given that the PF driver restore the PF config space after the reset will /sys files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or will it goes smoothly ?
Best regards
Benoît Canet
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines · Benoît Canet
Hello,
I may have soon the PF driver of an SR-IOV card to code and make work with QEMU/KVM so I have the following questions.
In an AMD64 setup where QEMU use VFIO to passthrough the VFs of an SR-IOV card to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?
I read that
pci_reset_functionwould callpci_restore_staterestoring the SR-IOV configuration after the reset of the PF.
Quoted 8 lines
The ways the hardware work means that the VFs would disappear and reappear in a short lapse of time.
Will these events be handled by the kernel pci hotplug code ?
Given that the PF driver restore the PF config space after the reset will /sys files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or will it goes smoothly ?
On an Intel 82576 SR-IOV NIC, a FLR of the PF does not cause the VFs to be removed. It's not clear to me that they continue working across the reset, but there is no hotplug. If there was a hotplug, vfio-pci won't release the device while it's in use, so the hotplug would be blocked until the devices becomes unused, such as from the VM being shutdown. Thanks,
Alex
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Thanks a lot for the answer.
Best regards
Benoît
Quoted 33 lines · Alex Williamson
Quoted 12 lines · Benoît Canet
Hello,
I may have soon the PF driver of an SR-IOV card to code and make work with QEMU/KVM so I have the following questions.
In an AMD64 setup where QEMU use VFIO to passthrough the VFs of an SR-IOV card to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?
I read that
pci_reset_functionwould callpci_restore_staterestoring the SR-IOV configuration after the reset of the PF.Quoted 8 lines
The ways the hardware work means that the VFs would disappear and reappear in a short lapse of time.
Will these events be handled by the kernel pci hotplug code ?
Given that the PF driver restore the PF config space after the reset will /sys files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or will it goes smoothly ?
On an Intel 82576 SR-IOV NIC, a FLR of the PF does not cause the VFs to be removed. It's not clear to me that they continue working across the reset, but there is no hotplug. If there was a hotplug, vfio-pci won't release the device while it's in use, so the hotplug would be blocked until the devices becomes unused, such as from the VM being shutdown. Thanks,
Alex
Quoted 10 lines · Benoît Canet
Hello,
I may have soon the PF driver of an SR-IOV card to code and make work with QEMU/KVM so I have the following questions.
In an AMD64 setup where QEMU use VFIO to passthrough the VFs of an SR-IOV card to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?
the reset occurs long before the device is passed to the guest.
Quoted 3 lines
I read that
pci_reset_functionwould callpci_restore_staterestoring the SR-IOV configuration after the reset of the PF.
correct.
Quoted 3 lines
The ways the hardware work means that the VFs would disappear and reappear in a short lapse of time.
Not sure your definitiion of 'disappear'. If you mean: if I had another thread poking at the device, the device would appear to be removed, then come back (if os poking hasn't crashed from the device's lack of response).
If you mean the VF gets entirely removed from the PCI tree, then no. A pci reset != hot unplug/plug. The device remains in the device tree.
Quoted 2 lines
Will these events be handled by the kernel pci hotplug code ?
'these events' ??? -- which events....
FLR is currently done by libvirt & qemu/vfio to ensure assigned devices are quiesced as they are switched from host->guest domain, and guest->(back-to-)host domain.
Quoted 1 line
Given that the PF driver restore the PF config space after the reset will /sys
The PF driver doesn't do the config space restore -- it's done in PCI core code.
Quoted 1 line
files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or
As stated above, the devices don't disappear from the device tree, so they don't get removed/added to the /sys(/bus/pci/...) files.
Quoted 2 lines
will it goes smoothly ?
it goes smoothly today.... :-/
Quoted 4 lines
Best regards
Benoît Canet
Quoted 4 lines
Quoted 3 lines
to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?
the reset occurs long before the device is passed to the guest.
I was asking this because the PF driver should reset the PF while the VF are used by VFIO/QEMU when the PF doesn't respond anymore.
Quoted 8 lines
The PF driver doesn't do the config space restore -- it's done in PCI core code.
Quoted 1 line
files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or
As stated above, the devices don't disappear from the device tree, so they don't get removed/added to the /sys(/bus/pci/...) files.
Quoted 2 lines
will it goes smoothly ?
it goes smoothly today.... :-/
Happy to read that thanks for the answer.
Best regards
Benoît Canet
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Benoît Canet
Quoted 4 lines
Quoted 3 lines
to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?
the reset occurs long before the device is passed to the guest.
I was asking this because the PF driver should reset the PF while the VF are used by VFIO/QEMU when the PF doesn't respond anymore.
What your VF does while your PF is being reset is PF (& VF) dependent. A 'good design' would not impact the VF operation, other than to stall it until the PF completed reset. My experience, though, is that the PF has to be brought up to some level of functionality to share the physical resources with the VFs.
Quoted 18 lines
Quoted 8 lines
The PF driver doesn't do the config space restore -- it's done in PCI core code.
Quoted 1 line
files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or
As stated above, the devices don't disappear from the device tree, so they don't get removed/added to the /sys(/bus/pci/...) files.
Quoted 2 lines
will it goes smoothly ?
it goes smoothly today.... :-/
Happy to read that thanks for the answer.
Best regards
Benoît Canet
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines
Quoted 3 lines
I was asking this because the PF driver should reset the PF while the VF are used by VFIO/QEMU when the PF doesn't respond anymore.
What your VF does while your PF is being reset is PF (& VF) dependent. A 'good design' would not impact the VF operation, other than to stall it until the PF completed reset. My experience, though, is that the PF has to be brought up to some level of functionality to share the physical resources with the VFs.
When the PF does an FLR the hardware go back to its default state, the SR-IOV configuration is gone and the VFs disappears from the bus.
Then the restore state function of the kernel reset code would bring the SR-IOV PF configuration back.
The hardware also have a privately owned SR-IOV related configuration in the PF configuration space. This configuration is used to configure the VFs resources. (memory)
Best regards
Benoît Canet
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Benoît Canet
Quoted 7 lines
Quoted 3 lines
I was asking this because the PF driver should reset the PF while the VF are used by VFIO/QEMU when the PF doesn't respond anymore.
What your VF does while your PF is being reset is PF (& VF) dependent. A 'good design' would not impact the VF operation, other than to stall it until the PF completed reset. My experience, though, is that the PF has to be brought up to some level of functionality to share the physical resources with the VFs.
Quoted 5 lines
When the PF does an FLR the hardware go back to its default state, the SR-IOV configuration is gone and the VFs disappears from the bus.
Then the restore state function of the kernel reset code would bring the SR-IOV PF configuration back.
Ok, now you're a bit mis-led here.
The configuration header for SRIOV is _not_ put back.
Only the std, PCI config header section is put back in place, along with msi(x), pm-caps.
If the hw wipes out all VF state setup (which it should, IMO), all VF configuration will be lost in the hw...
*but*, the PCI core will still think the VFs exist (not hot-unplugged, no more than PF was); trying to setup the VFs again, will fail (or worse).
Quoted 4 lines
The hardware also have a privately owned SR-IOV related configuration in the PF configuration space. This configuration is used to configure the VFs resources. (memory)
Per the SRIOV spec, yes, but that's in PCIe ext cfg space. That area of the PCI configuration is not saved or restored by dev-reset.
Quoted 7 lines
Best regards
Benoît Canet
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines
Quoted 5 lines
When the PF does an FLR the hardware go back to its default state, the SR-IOV configuration is gone and the VFs disappears from the bus.
Then the restore state function of the kernel reset code would bring the SR-IOV PF configuration back.
Quoted 8 lines
Ok, now you're a bit mis-led here.
The configuration header for SRIOV is _not_ put back.
Only the std, PCI config header section is put back in place, along with msi(x), pm-caps.
If the hw wipes out all VF state setup (which it should, IMO), all VF configuration will be lost in the hw...
*but*, the PCI core will still think the VFs exist (not hot-unplugged, no more than PF was); trying to setup the VFs again, will fail (or worse).
I read the following code on a not so hold kernel.
-----------
int pci_reset_function(struct pci_dev *dev)
{
Quoted 14 lines
.......int rc;
.......rc =
pci_dev_reset(dev, 1);
.......if (rc)
.......>.......return rc;.......pci_save_state(dev);
......./*
....... * both INTx and MSI are disabled after the Interrupt Disable bit ....... * is set and the Bus Master bit is cleared.
....... */
.......pci_write_config_word(dev,PCI_COMMAND,PCI_COMMAND_INTX_DISABLE);
Quoted 5 lines
.......rc =
pci_dev_reset(dev, 0);.......pci_restore_state(dev);
.......return rc;
}EXPORT_SYMBOL_GPL(pci_reset_function);
-----------
and
-----------
/**
* pci_restore_state - Restore the saved state of a PCI device
* @dev: - PCI device that we're dealing with
*/
void pci_restore_state(struct pci_dev *dev)
{
Quoted 7 lines
.......if (!dev->state_saved)
.......>.......return;......./* PCI Express register must be restored first */ .......pci_restore_pcie_state(dev); .......pci_restore_ats_state(dev);
Quoted 7 lines
.......pci_restore_config_space(dev);
.......pci_restore_pcix_state(dev);
.......pci_restore_msi_state(dev);
.......pci_restore_iov_state(dev);.......dev->state_saved = false;
}
-----------
with pci_restore_iov_state calling sriov_restore_state:
-----------
static void sriov_restore_state(struct pci_dev *dev) {
Quoted 8 lines
.......int i;
.......u16 ctrl;
.......structpci_sriov*iov =dev->sriov;.......pci_read_config_word(dev,
iov->pos+PCI_SRIOV_CTRL, &ctrl); .......if (ctrl &PCI_SRIOV_CTRL_VFE) .......>.......return;
Quoted 8 lines
.......for (i =
PCI_IOV_RESOURCES; i <=PCI_IOV_RESOURCE_END; i++) .......>.......pci_update_resource(dev, i);.......pci_write_config_dword(dev,
iov->pos+PCI_SRIOV_SYS_PGSIZE,iov->pgsz); .......pci_write_config_word(dev,iov->pos+PCI_SRIOV_NUM_VF,iov->num_VFs); .......pci_write_config_word(dev,iov->pos+PCI_SRIOV_CTRL,iov->ctrl); .......if (iov->ctrl&PCI_SRIOV_CTRL_VFE) .......>.......msleep(100);
}
--------
The sriov_restore_state looked like if it does the right thing but maybe I missread the code.
Quoted 7 lines
Quoted 5 lines
The hardware also have a privately owned SR-IOV related configuration in the PF configuration space. This configuration is used to configure the VFs resources. (memory)
Per the SRIOV spec, yes, but that's in PCIe ext cfg space. That area of the PCI configuration is not saved or restored by dev-reset.
Can a callback be added so PF driver can restore this state ?
Best regards
Benoît Canet
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 89 lines · Benoît Canet
Quoted 5 lines
Quoted 5 lines
When the PF does an FLR the hardware go back to its default state, the SR-IOV configuration is gone and the VFs disappears from the bus.
Then the restore state function of the kernel reset code would bring the SR-IOV PF configuration back.Quoted 8 lines
Ok, now you're a bit mis-led here.
The configuration header for SRIOV is _not_ put back.
Only the std, PCI config header section is put back in place, along with msi(x), pm-caps.
If the hw wipes out all VF state setup (which it should, IMO), all VF configuration will be lost in the hw...
*but*, the PCI core will still think the VFs exist (not hot-unplugged, no more than PF was); trying to setup the VFs again, will fail (or worse).I read the following code on a not so hold kernel.
-----------
intpci_reset_function(struct pci_dev *dev)
{Quoted 14 lines
.......int rc;
.......rc =
pci_dev_reset(dev, 1);
.......if (rc)
.......>.......return rc;.......pci_save_state(dev);
......./*
....... * both INTx and MSI are disabled after the Interrupt Disable bit ....... * is set and the Bus Master bit is cleared.
....... */
.......pci_write_config_word(dev,PCI_COMMAND,PCI_COMMAND_INTX_DISABLE);Quoted 5 lines
.......rc =
pci_dev_reset(dev, 0);.......pci_restore_state(dev);
.......return rc;
}
EXPORT_SYMBOL_GPL(pci_reset_function);
-----------and
-----------
/**
*pci_restore_state- Restore the saved state of a PCI device
* @dev: - PCI device that we're dealing with
*/
voidpci_restore_state(struct pci_dev *dev)
{Quoted 7 lines
.......if (!dev->state_saved)
.......>.......return;......./* PCI Express register must be restored first */ .......pci_restore_pcie_state(dev); .......pci_restore_ats_state(dev);
Quoted 7 lines
.......pci_restore_config_space(dev);
.......pci_restore_pcix_state(dev);
.......pci_restore_msi_state(dev);
.......pci_restore_iov_state(dev);.......dev->state_saved = false;
}
-----------with
pci_restore_iov_statecallingsriov_restore_state:-----------
static voidsriov_restore_state(struct pci_dev *dev){Quoted 8 lines
.......int i;
.......u16 ctrl;
.......structpci_sriov*iov =dev->sriov;.......pci_read_config_word(dev,
iov->pos+ PCI_SRIOV_CTRL,&ctrl); .......if (ctrl&PCI_SRIOV_CTRL_VFE) .......>.......return;Quoted 8 lines
.......for (i =
PCI_IOV_RESOURCES; i<=PCI_IOV_RESOURCE_END; i++) .......>.......pci_update_resource(dev, i);.......pci_write_config_dword(dev,
iov->pos+PCI_SRIOV_SYS_PGSIZE,iov->pgsz); .......pci_write_config_word(dev,iov->pos+PCI_SRIOV_NUM_VF,iov->num_VFs); .......pci_write_config_word(dev,iov->pos+PCI_SRIOV_CTRL,iov->ctrl); .......if (iov->ctrl&PCI_SRIOV_CTRL_VFE) .......>.......msleep(100);
Quoted 6 lines
}
--------The
sriov_restore_statelooked like if it does the right thing but maybe I missread the code.
/my bad; I forgot about the save|restore_iov_state calls.... doh!
Now it gets down to how well your hw (& driver) works after the reset is done...
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
The hardware also have a privately owned SR-IOV related configuration in the PF configuration space. This configuration is used to configure the VFs resources. (memory)
Per the SRIOV spec, yes, but that's in PCIe ext cfg space. That area of the PCI configuration is not saved or restored by dev-reset.
Can a callback be added so PF driver can restore this state ?
As you pointed out, no need to, unless it's a device-specific, PCIe cap structure. the SRIOV caps are re-instated, as you showed above...
Quoted 7 lines
Best regards
Benoît Canet
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Per the SRIOV spec, yes, but that's in PCIe ext cfg space. That area of the PCI configuration is not saved or restored by dev-reset.
Can a callback be added so PF driver can restore this state ?
As you pointed out, no need to, unless it's a device-specific, PCIe cap structure. the SRIOV caps are re-instated, as you showed above...
I think this is a device specific structure.
It would be annoying only in case of a PF FLR initiated via /sys or something external to the PF driver because the PF driver could restore this area after it initiate an FLR.
Best regards
Benoît Canet