Skip to contentSkip to conversation

SR-IOV PF reset and QEMU VFs VFIO passthrough

DISCUSSION Benoît Canet 5 participants 11 messages Jun 1, 2013 – Jun 4, 2013 7 messages not in the archive PCI

lore
Share & tools
lore
#0 lore raw

Hello,

I may have soon the PF driver of an SR-IOV card to code and make work with QEMU/KVM so I have the following questions.

In an AMD64 setup where QEMU use VFIO to passthrough the VFs of an SR-IOV card to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?

I read that pci_reset_function would call pci_restore_state restoring the SR-IOV configuration after the reset of the PF.

The ways the hardware work means that the VFs would disappear and reappear in a short lapse of time.

Will these events be handled by the kernel pci hotplug code ?

Given that the PF driver restore the PF config space after the reset will /sys files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or will it goes smoothly ?

Best regards

Benoît Canet

#2 lore raw
In reply to a message not in the archive
Quoted 12 lines · Benoît Canet

Hello,

I may have soon the PF driver of an SR-IOV card to code and make work with QEMU/KVM so I have the following questions.

In an AMD64 setup where QEMU use VFIO to passthrough the VFs of an SR-IOV card to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?

I read that pci_reset_function would call pci_restore_state restoring the SR-IOV configuration after the reset of the PF.

Quoted 8 lines

The ways the hardware work means that the VFs would disappear and reappear in a short lapse of time.

Will these events be handled by the kernel pci hotplug code ?

Given that the PF driver restore the PF config space after the reset will /sys files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or will it goes smoothly ?

On an Intel 82576 SR-IOV NIC, a FLR of the PF does not cause the VFs to be removed. It's not clear to me that they continue working across the reset, but there is no hotplug. If there was a hotplug, vfio-pci won't release the device while it's in use, so the hotplug would be blocked until the devices becomes unused, such as from the VM being shutdown. Thanks,

Alex

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#3 lore raw

Thanks a lot for the answer.

Best regards

Benoît

Quoted 33 lines · Alex Williamson
Quoted 12 lines · Benoît Canet

Hello,

I may have soon the PF driver of an SR-IOV card to code and make work with QEMU/KVM so I have the following questions.

In an AMD64 setup where QEMU use VFIO to passthrough the VFs of an SR-IOV card to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?

I read that pci_reset_function would call pci_restore_state restoring the SR-IOV configuration after the reset of the PF.

Quoted 8 lines

The ways the hardware work means that the VFs would disappear and reappear in a short lapse of time.

Will these events be handled by the kernel pci hotplug code ?

Given that the PF driver restore the PF config space after the reset will /sys files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or will it goes smoothly ?

On an Intel 82576 SR-IOV NIC, a FLR of the PF does not cause the VFs to be removed. It's not clear to me that they continue working across the reset, but there is no hotplug. If there was a hotplug, vfio-pci won't release the device while it's in use, so the hotplug would be blocked until the devices becomes unused, such as from the VM being shutdown. Thanks,

Alex

#4 lore raw
In reply to Benoît Canet
Quoted 10 lines · Benoît Canet

Hello,

I may have soon the PF driver of an SR-IOV card to code and make work with QEMU/KVM so I have the following questions.

In an AMD64 setup where QEMU use VFIO to passthrough the VFs of an SR-IOV card to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?

the reset occurs long before the device is passed to the guest.

Quoted 3 lines

I read that pci_reset_function would call pci_restore_state restoring the SR-IOV configuration after the reset of the PF.

correct.

Quoted 3 lines

The ways the hardware work means that the VFs would disappear and reappear in a short lapse of time.

Not sure your definitiion of 'disappear'. If you mean: if I had another thread poking at the device, the device would appear to be removed, then come back (if os poking hasn't crashed from the device's lack of response).
If you mean the VF gets entirely removed from the PCI tree, then no. A pci reset != hot unplug/plug. The device remains in the device tree.

Quoted 2 lines

Will these events be handled by the kernel pci hotplug code ?

'these events' ??? -- which events....
FLR is currently done by libvirt & qemu/vfio to ensure assigned devices are quiesced as they are switched from host->guest domain, and guest->(back-to-)host domain.

Quoted 1 line

Given that the PF driver restore the PF config space after the reset will /sys

The PF driver doesn't do the config space restore -- it's done in PCI core code.

Quoted 1 line

files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or

As stated above, the devices don't disappear from the device tree, so they don't get removed/added to the /sys(/bus/pci/...) files.

Quoted 2 lines

will it goes smoothly ?

it goes smoothly today.... :-/

Quoted 4 lines

Best regards

Benoît Canet

#5 lore raw
Quoted 4 lines
Quoted 3 lines

to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?

the reset occurs long before the device is passed to the guest.

I was asking this because the PF driver should reset the PF while the VF are used by VFIO/QEMU when the PF doesn't respond anymore.

Quoted 8 lines

The PF driver doesn't do the config space restore -- it's done in PCI core code.

Quoted 1 line

files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or

As stated above, the devices don't disappear from the device tree, so they don't get removed/added to the /sys(/bus/pci/...) files.

Quoted 2 lines

will it goes smoothly ?

it goes smoothly today.... :-/

Happy to read that thanks for the answer.

Best regards

Benoît Canet

#7 lore raw
In reply to a message not in the archive
Quoted 8 lines · Benoît Canet
Quoted 4 lines
Quoted 3 lines

to a guest will the consequences of a PF FLR be handled fine by QEMU and the guest ?

the reset occurs long before the device is passed to the guest.

I was asking this because the PF driver should reset the PF while the VF are used by VFIO/QEMU when the PF doesn't respond anymore.

What your VF does while your PF is being reset is PF (& VF) dependent. A 'good design' would not impact the VF operation, other than to stall it until the PF completed reset. My experience, though, is that the PF has to be brought up to some level of functionality to share the physical resources with the VFs.

Quoted 18 lines
Quoted 8 lines

The PF driver doesn't do the config space restore -- it's done in PCI core code.

Quoted 1 line

files used by QEMU disappear and reappear messing the QEMU VFIO passthrough or

As stated above, the devices don't disappear from the device tree, so they don't get removed/added to the /sys(/bus/pci/...) files.

Quoted 2 lines

will it goes smoothly ?

it goes smoothly today.... :-/

Happy to read that thanks for the answer.

Best regards

Benoît Canet
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
In reply to a message not in the archive
Quoted 7 lines
Quoted 3 lines

I was asking this because the PF driver should reset the PF while the VF are used by VFIO/QEMU when the PF doesn't respond anymore.

What your VF does while your PF is being reset is PF (& VF) dependent. A 'good design' would not impact the VF operation, other than to stall it until the PF completed reset. My experience, though, is that the PF has to be brought up to some level of functionality to share the physical resources with the VFs.

When the PF does an FLR the hardware go back to its default state, the SR-IOV configuration is gone and the VFs disappears from the bus.
Then the restore state function of the kernel reset code would bring the SR-IOV PF configuration back.

The hardware also have a privately owned SR-IOV related configuration in the PF configuration space. This configuration is used to configure the VFs resources. (memory)

Best regards

Benoît Canet

#11 lore raw
In reply to a message not in the archive
Quoted 8 lines · Benoît Canet
Quoted 7 lines
Quoted 3 lines

I was asking this because the PF driver should reset the PF while the VF are used by VFIO/QEMU when the PF doesn't respond anymore.

What your VF does while your PF is being reset is PF (& VF) dependent. A 'good design' would not impact the VF operation, other than to stall it until the PF completed reset. My experience, though, is that the PF has to be brought up to some level of functionality to share the physical resources with the VFs.

Quoted 5 lines

When the PF does an FLR the hardware go back to its default state, the SR-IOV configuration is gone and the VFs disappears from the bus.
Then the restore state function of the kernel reset code would bring the SR-IOV PF configuration back.

Ok, now you're a bit mis-led here.
The configuration header for SRIOV is _not_ put back.
Only the std, PCI config header section is put back in place, along with msi(x), pm-caps.
If the hw wipes out all VF state setup (which it should, IMO), all VF configuration will be lost in the hw...
*but*, the PCI core will still think the VFs exist (not hot-unplugged, no more than PF was); trying to setup the VFs again, will fail (or worse).

Quoted 4 lines

The hardware also have a privately owned SR-IOV related configuration in the PF configuration space. This configuration is used to configure the VFs resources. (memory)

Per the SRIOV spec, yes, but that's in PCIe ext cfg space. That area of the PCI configuration is not saved or restored by dev-reset.

Quoted 7 lines

Best regards

Benoît Canet
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#13 lore raw
In reply to a message not in the archive
Quoted 5 lines
Quoted 5 lines

When the PF does an FLR the hardware go back to its default state, the SR-IOV configuration is gone and the VFs disappears from the bus.
Then the restore state function of the kernel reset code would bring the SR-IOV PF configuration back.

Quoted 8 lines

Ok, now you're a bit mis-led here.
The configuration header for SRIOV is _not_ put back.
Only the std, PCI config header section is put back in place, along with msi(x), pm-caps.
If the hw wipes out all VF state setup (which it should, IMO), all VF configuration will be lost in the hw...
*but*, the PCI core will still think the VFs exist (not hot-unplugged, no more than PF was); trying to setup the VFs again, will fail (or worse).

I read the following code on a not so hold kernel.

-----------
int pci_reset_function(struct pci_dev *dev)
{

Quoted 14 lines

.......int rc;

.......rc = pci_dev_reset(dev, 1);
.......if (rc)
.......>.......return rc;

.......pci_save_state(dev);

......./*
....... * both INTx and MSI are disabled after the Interrupt Disable bit ....... * is set and the Bus Master bit is cleared.
....... */
.......pci_write_config_word(dev, PCI_COMMAND, PCI_COMMAND_INTX_DISABLE);

Quoted 5 lines

.......rc = pci_dev_reset(dev, 0);

.......pci_restore_state(dev);

.......return rc;

}
EXPORT_SYMBOL_GPL(pci_reset_function);
-----------

and

-----------
/**
* pci_restore_state - Restore the saved state of a PCI device
* @dev: - PCI device that we're dealing with
*/
void pci_restore_state(struct pci_dev *dev)
{

Quoted 7 lines

.......if (!dev->state_saved)
.......>.......return;

......./* PCI Express register must be restored first */ .......pci_restore_pcie_state(dev); .......pci_restore_ats_state(dev);

Quoted 7 lines

.......pci_restore_config_space(dev);

.......pci_restore_pcix_state(dev);
.......pci_restore_msi_state(dev);
.......pci_restore_iov_state(dev);

.......dev->state_saved = false;

}
-----------

with pci_restore_iov_state calling sriov_restore_state:

-----------
static void sriov_restore_state(struct pci_dev *dev) {

Quoted 8 lines

.......int i;
.......u16 ctrl;
.......struct pci_sriov *iov = dev->sriov;

.......pci_read_config_word(dev, iov->pos + PCI_SRIOV_CTRL, &ctrl); .......if (ctrl & PCI_SRIOV_CTRL_VFE) .......>.......return;

Quoted 8 lines

.......for (i = PCI_IOV_RESOURCES; i <= PCI_IOV_RESOURCE_END; i++) .......>.......pci_update_resource(dev, i);

.......pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz); .......pci_write_config_word(dev, iov->pos + PCI_SRIOV_NUM_VF, iov->num_VFs); .......pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); .......if (iov->ctrl & PCI_SRIOV_CTRL_VFE) .......>.......msleep(100);

}
--------

The sriov_restore_state looked like if it does the right thing but maybe I missread the code.

Quoted 7 lines
Quoted 5 lines

The hardware also have a privately owned SR-IOV related configuration in the PF configuration space. This configuration is used to configure the VFs resources. (memory)

Per the SRIOV spec, yes, but that's in PCIe ext cfg space. That area of the PCI configuration is not saved or restored by dev-reset.

Can a callback be added so PF driver can restore this state ?

Best regards

Benoît Canet

#15 lore raw
In reply to a message not in the archive
Quoted 89 lines · Benoît Canet
Quoted 5 lines
Quoted 5 lines

When the PF does an FLR the hardware go back to its default state, the SR-IOV configuration is gone and the VFs disappears from the bus.
Then the restore state function of the kernel reset code would bring the SR-IOV PF configuration back.

Quoted 8 lines

Ok, now you're a bit mis-led here.
The configuration header for SRIOV is _not_ put back.
Only the std, PCI config header section is put back in place, along with msi(x), pm-caps.
If the hw wipes out all VF state setup (which it should, IMO), all VF configuration will be lost in the hw...
*but*, the PCI core will still think the VFs exist (not hot-unplugged, no more than PF was); trying to setup the VFs again, will fail (or worse).

I read the following code on a not so hold kernel.

-----------
int pci_reset_function(struct pci_dev *dev)
{

Quoted 14 lines

.......int rc;

.......rc = pci_dev_reset(dev, 1);
.......if (rc)
.......>.......return rc;

.......pci_save_state(dev);

......./*
....... * both INTx and MSI are disabled after the Interrupt Disable bit ....... * is set and the Bus Master bit is cleared.
....... */
.......pci_write_config_word(dev, PCI_COMMAND, PCI_COMMAND_INTX_DISABLE);

Quoted 5 lines

.......rc = pci_dev_reset(dev, 0);

.......pci_restore_state(dev);

.......return rc;

}
EXPORT_SYMBOL_GPL(pci_reset_function);
-----------

and

-----------
/**
* pci_restore_state - Restore the saved state of a PCI device
* @dev: - PCI device that we're dealing with
*/
void pci_restore_state(struct pci_dev *dev)
{

Quoted 7 lines

.......if (!dev->state_saved)
.......>.......return;

......./* PCI Express register must be restored first */ .......pci_restore_pcie_state(dev); .......pci_restore_ats_state(dev);

Quoted 7 lines

.......pci_restore_config_space(dev);

.......pci_restore_pcix_state(dev);
.......pci_restore_msi_state(dev);
.......pci_restore_iov_state(dev);

.......dev->state_saved = false;

}
-----------

with pci_restore_iov_state calling sriov_restore_state:

-----------
static void sriov_restore_state(struct pci_dev *dev) {

Quoted 8 lines

.......int i;
.......u16 ctrl;
.......struct pci_sriov *iov = dev->sriov;

.......pci_read_config_word(dev, iov->pos + PCI_SRIOV_CTRL,&ctrl); .......if (ctrl& PCI_SRIOV_CTRL_VFE) .......>.......return;

Quoted 8 lines

.......for (i = PCI_IOV_RESOURCES; i<= PCI_IOV_RESOURCE_END; i++) .......>.......pci_update_resource(dev, i);

.......pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz); .......pci_write_config_word(dev, iov->pos + PCI_SRIOV_NUM_VF, iov->num_VFs); .......pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); .......if (iov->ctrl& PCI_SRIOV_CTRL_VFE) .......>.......msleep(100);

Quoted 6 lines

}
--------

The sriov_restore_state looked like if it does the right thing but maybe I missread the code.

/my bad; I forgot about the save|restore_iov_state calls.... doh!

Now it gets down to how well your hw (& driver) works after the reset is done...

Quoted 10 lines
Quoted 7 lines
Quoted 5 lines

The hardware also have a privately owned SR-IOV related configuration in the PF configuration space. This configuration is used to configure the VFs resources. (memory)

Per the SRIOV spec, yes, but that's in PCIe ext cfg space. That area of the PCI configuration is not saved or restored by dev-reset.

Can a callback be added so PF driver can restore this state ?

As you pointed out, no need to, unless it's a device-specific, PCIe cap structure. the SRIOV caps are re-instated, as you showed above...

Quoted 7 lines

Best regards

Benoît Canet
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#17 lore raw
In reply to a message not in the archive
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Per the SRIOV spec, yes, but that's in PCIe ext cfg space. That area of the PCI configuration is not saved or restored by dev-reset.

Can a callback be added so PF driver can restore this state ?

As you pointed out, no need to, unless it's a device-specific, PCIe cap structure. the SRIOV caps are re-instated, as you showed above...

I think this is a device specific structure.

It would be annoying only in case of a PF FLR initiated via /sys or something external to the PF driver because the PF driver could restore this area after it initiate an FLR.

Best regards

Benoît Canet