MSI for Virtio PCI transport
Hi,
We are looking into adapting Virtio spec for configurable physical PCIe endpoint devices to expose Virtio devices to the host machine connected over PCIe. This allows us to use the existing frontend drivers on the host machine, thus minimizing the development efforts. This idea is not new as some vendors like NVidia have already released customized PCIe devices exposing Virtio devices to the host machines. But we are working on making the configurable PCIe devices running Linux kernel to expose Virtio devices using the PCI Endpoint (EP) subsystem.
Below is the simplistic represenation of the idea with virt-net as an example. But this could be extended to any supported Virtio devices:
HOST ENDPOINT
+-----------------------------+ +-----------------------------+
| | | |
| Linux Kernel | | Linux Kernel |
| | | |
| | | +------------------+ |
| | | | | |
| | | | Modem | |
| | | | | |
| | | +---------|--------+ |
| | | | |
| +------------------+ | | +---------|--------+ |
| | | | | | | |
| | Virt-net | | | | Virtio EPF | |
| | | | | | | |
| +---------|--------+ | | +---------|--------+ |
| | | | | |
| +---------|--------+ | | +---------|--------+ |
| | | | | | | |
| | Virtio PCI | | | | PCI EP Subsystem | |
| | | | | | | |
| +---------|--------+ | | +---------|--------+ |
| SW | | | SW | |
----------------|-------------- ----------------|--------------
| HW | | | HW | |
| +---------|--------+ | | +---------|--------+ |
| | | | | | | |
| | PCIe RC | | | | PCIe EP | |
| | | | | | | |
+-----+---------|--------+----+ +-----+---------|--------+----+
| |
| |
| |
| PCIe |
----------------------------------------- While doing so, we faced an issue due to lack of MSI support defined in Virtio spec for PCI transport. Currently, the PCI transport (starting from 0.9.5) has only defined INTx (legacy) and MSI-X interrupts for the device to send notifications to the guest. While it works well for the hypervisor to guest communcation, when a physical PCIe device is used as a Virtio device, lack of MSI support is hurting the performance (when there is no MSI-X).
Most of the physical PCIe endpoint devices support MSI interrupts over MSI-X for simplicity and with Virtio not supporting MSI, falling back to legacy INTx interrupts is affecting the performance.
First of all, INTx requires the PCIe devices to send two MSG TLPs (Assert/Deassert) to emulate level triggered interrupt on the host. And there could be some delay between assert and deassert messages to make sure that the host recognizes it as an interrupt (level trigger). Also, the INTx interrupts are limited to 1 per function, so all the notifications from device has to share this single interrupt (INTA).
On the other hand, MSI requires only one MWr TLP from the device to host and since it is a posted write, there is no delay involved. Also, a single PCIe function can use upto 32 MSIs, thus making it possible to use one MSI vector per virtqueue (32 is more than enough for most of the usecases).
So my question is, why does the Virtio spec not supporting MSI? If there are no major blocker in supporting MSI, could we propose adding MSI to the Virtio spec?
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Hi,
Quoted 45 lines
From: Manivannan Sadhasivam <[email protected]>
Sent: Monday, June 24, 2024 9:50 PMHi,
We are looking into adapting Virtio spec for configurable physical PCIe endpoint devices to expose Virtio devices to the host machine connected over PCIe. This allows us to use the existing frontend drivers on the host machine, thus minimizing the development efforts. This idea is not new as some vendors like NVidia have already released customized PCIe devices exposing Virtio devices to the host machines. But we are working on making the configurable PCIe devices running Linux kernel to expose Virtio devices using the PCI Endpoint (EP) subsystem.
Below is the simplistic represenation of the idea with virt-net as an example. But this could be extended to any supported Virtio devices:
HOST ENDPOINT
+-----------------------------+ +-----------------------------+ | | | | | Linux Kernel | | Linux Kernel | | | | | | | | +------------------+ | | | | | | | | | | | Modem | | | | | | | | | | | +---------|--------+ | | | | | | | +------------------+ | | +---------|--------+ | | | | | | | | | | | Virt-net | | | | Virtio EPF | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | Virtio PCI | | | | PCI EP Subsystem | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | SW | | | SW | | ----------------|-------------- ----------------|-------------- | HW | | | HW | | | +---------|--------+ | | +---------|--------+ | | | | | | | | |
Quoted 9 lines
| | PCIe RC | | | | PCIe EP | |
| | | | | | | |
+-----+---------|--------+----+ +-----+---------|--------+----+
| |
| |
| |
| PCIe |
-----------------------------------------Can you please explain what is PCIe EP subsystem is?
I assume, it is a subsystem to somehow configure the PCIe EP HW instances? If yes, it is not connected to any PCIe RC in your diagram.
So how does the MSI help in this case?
Quoted 7 lines
While doing so, we faced an issue due to lack of MSI support defined in Virtio spec for PCI transport. Currently, the PCI transport (starting from 0.9.5) has only defined INTx (legacy) and MSI-X interrupts for the device to send notifications to the guest. While it works well for the hypervisor to guest communcation, when a physical PCIe device is used as a Virtio device, lack of MSI support is hurting the performance (when there is no MSI-X).
I am familiar with the scale issue of MSI-X, which is better for MSI (relative to MSI-X). What prevents implementing the MSI-X?
Quoted 1 line
Most of the physical PCIe endpoint devices support MSI interrupts over MSI-
I am not sure if this is true. :)
But not a concern either.
Quoted 10 lines
X for simplicity and with Virtio not supporting MSI, falling back to legacy INTx interrupts is affecting the performance.
First of all, INTx requires the PCIe devices to send two MSG TLPs (Assert/Deassert) to emulate level triggered interrupt on the host. And there could be some delay between assert and deassert messages to make sure that the host recognizes it as an interrupt (level trigger). Also, the INTx interrupts are limited to 1 per function, so all the notifications from device has to share this single interrupt (INTA).
Yes, INTx deprecation is in my list but didn’t get their yet.
Quoted 9 lines
On the other hand, MSI requires only one MWr TLP from the device to host and since it is a posted write, there is no delay involved. Also, a single PCIe function can use upto 32 MSIs, thus making it possible to use one MSI vector per virtqueue (32 is more than enough for most of the usecases).
So my question is, why does the Virtio spec not supporting MSI? If there are no major blocker in supporting MSI, could we propose adding MSI to the Virtio spec?
MSI addition is good for virtio for small scale devices of 1 to 32.
PCIe EP may support MSI-X and MSI both the capabilities and sw can give preference to MSI when the need is <= 32 vectors.
Though I don’t see it anyway related to PCIe EP configuration in your diagram. In other words, PCI EP subsystem can still work with MSI-X.
Can you please elaborate it?
Quoted 4 lines
- Mani
--
மணிவண்ணன் சதாசிவம்
Quoted 56 lines · Parav Pandit
Hi,
Quoted 45 lines
From: Manivannan Sadhasivam <[email protected]>
Sent: Monday, June 24, 2024 9:50 PMHi,
We are looking into adapting Virtio spec for configurable physical PCIe endpoint devices to expose Virtio devices to the host machine connected over PCIe. This allows us to use the existing frontend drivers on the host machine, thus minimizing the development efforts. This idea is not new as some vendors like NVidia have already released customized PCIe devices exposing Virtio devices to the host machines. But we are working on making the configurable PCIe devices running Linux kernel to expose Virtio devices using the PCI Endpoint (EP) subsystem.
Below is the simplistic represenation of the idea with virt-net as an example. But this could be extended to any supported Virtio devices:
HOST ENDPOINT
+-----------------------------+ +-----------------------------+ | | | | | Linux Kernel | | Linux Kernel | | | | | | | | +------------------+ | | | | | | | | | | | Modem | | | | | | | | | | | +---------|--------+ | | | | | | | +------------------+ | | +---------|--------+ | | | | | | | | | | | Virt-net | | | | Virtio EPF | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | Virtio PCI | | | | PCI EP Subsystem | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | SW | | | SW | | ----------------|-------------- ----------------|-------------- | HW | | | HW | | | +---------|--------+ | | +---------|--------+ | | | | | | | | |Quoted 9 lines
| | PCIe RC | | | | PCIe EP | | | | | | | | | | +-----+---------|--------+----+ +-----+---------|--------+----+ | | | | | | | PCIe | -----------------------------------------
Quoted 4 lines
Can you please explain what is PCIe EP subsystem is?
I assume, it is a subsystem to somehow configure the PCIe EP HW instances? If yes, it is not connected to any PCIe RC in your diagram.
PCIe EP subsystem is a Linux kernel framework to configure the PCIe EP IP inside an SoC/device. Here 'Endpoint' is a separate SoC/device that is running Linux kernel and uses PCIe EP subsystem in kernel [1] to configure the PCIe EP IP based on product usecase like GPU card, NVMe, Modem, WLAN etc...
[1] https://docs.kernel.org/PCI/endpoint/pci-endpoint.html
Quoted 2 lines
So how does the MSI help in this case?
I think you are missing the point that 'Endpoint' is a separate SoC/device that is connected to a host machine over PCIe. Just like how you would connect a PCIe based GPU card to a Desktop PC. Only difference is, most of the PCIe cards will run on a proprietary firmware supplied by the vendor, but here the firmware itself can be built by the user and configurable. And this is where Virtio is going to be exposed.
Quoted 11 lines
Quoted 8 lines
While doing so, we faced an issue due to lack of MSI support defined in Virtio spec for PCI transport. Currently, the PCI transport (starting from 0.9.5) has only defined INTx (legacy) and MSI-X interrupts for the device to send notifications to the guest. While it works well for the hypervisor to guest communcation, when a physical PCIe device is used as a Virtio device, lack of MSI support is hurting the performance (when there is no MSI-X).
I am familiar with the scale issue of MSI-X, which is better for MSI (relative to MSI-X). What prevents implementing the MSI-X?
As I said, most of the devices I'm aware doesn't support MSI-X in hardware itself (I mean in the PCIe EP IP inside the SoC/device). For simple usecases like WLAN, modem, MSI-X is not really required.
Quoted 4 lines
Quoted 1 line
Most of the physical PCIe endpoint devices support MSI interrupts over MSI-
I am not sure if this is true. :)
But not a concern either.
It really depends on the usecase I would say.
Quoted 17 lines
Quoted 10 lines
X for simplicity and with Virtio not supporting MSI, falling back to legacy INTx interrupts is affecting the performance.
First of all, INTx requires the PCIe devices to send two MSG TLPs (Assert/Deassert) to emulate level triggered interrupt on the host. And there could be some delay between assert and deassert messages to make sure that the host recognizes it as an interrupt (level trigger). Also, the INTx interrupts are limited to 1 per function, so all the notifications from device has to share this single interrupt (INTA).
Yes, INTx deprecation is in my list but didn’t get their yet.
Quoted 5 lines
On the other hand, MSI requires only one MWr TLP from the device to host and since it is a posted write, there is no delay involved. Also, a single PCIe function can use upto 32 MSIs, thus making it possible to use one MSI vector per virtqueue (32 is more than enough for most of the usecases).
Quoted 7 lines
Quoted 4 lines
So my question is, why does the Virtio spec not supporting MSI? If there are no major blocker in supporting MSI, could we propose adding MSI to the Virtio spec?
MSI addition is good for virtio for small scale devices of 1 to 32.
PCIe EP may support MSI-X and MSI both the capabilities and sw can give preference to MSI when the need is <= 32 vectors.
PCIe specification only mandates the devices to support either MSI or MSI-X.
Reference: PCIe spec r5.0, sec 6.1.4:
"All PCI Express device Functions that are capable of generating interrupts must support MSI or MSI-X or both."
So MSI-X is clearly an optional feature which simple devices tend to ignore. But if both are supported, then obviously Virtio will make use of MSI-X, but that's not the case here.
Quoted 4 lines
Though I don’t see it anyway related to PCIe EP configuration in your diagram. In other words, PCI EP subsystem can still work with MSI-X.
Can you please elaborate it?
I hope the above info clarifies. If not, please let me know.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 68 lines
From: Manivannan Sadhasivam <[email protected]>
Sent: Tuesday, June 25, 2024 11:14 AMQuoted 15 lines · Parav Pandit
Hi,
Quoted 13 lines
From: Manivannan Sadhasivam <[email protected]>
Sent: Monday, June 24, 2024 9:50 PMHi,
We are looking into adapting Virtio spec for configurable physical PCIe endpoint devices to expose Virtio devices to the host machine connected over PCIe. This allows us to use the existing frontend drivers on the host machine, thus minimizing the development efforts. This idea is not new as some vendors like NVidia have already released customized PCIe devices exposing Virtio devices to the host machines. But we are working on making the configurable PCIe devices running Linux kernel to expose Virtio devices using the PCI
Endpoint (EP) subsystem.
Quoted 2 lines
Quoted 2 lines
Below is the simplistic represenation of the idea with virt-net as an
example.
Quoted 41 lines
Quoted 32 lines
But this could be extended to any supported Virtio devices:
HOST ENDPOINT
+-----------------------------+ +-----------------------------+ +-----------------------------+ | | | | | Linux Kernel | | Linux Kernel | | | | | | | | +------------------+ | | | | | | | | | | | Modem | | | | | | | | | | | +---------|--------+ | | | | | | | +------------------+ | | +---------|--------+ | | | | | | | | | | | Virt-net | | | | Virtio EPF | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | Virtio PCI | | | | PCI EP Subsystem | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | SW | | | SW | | ----------------|-------------- ----------------|-------------- | HW | | | HW | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | PCIe RC | | | | PCIe EP | |Quoted 9 lines
| | | | | | | | +-----+---------|--------+----+ +-----+---------|--------+----+ +-----+---------|--------+----+ | | | | | | | PCIe | -----------------------------------------Quoted 2 lines
Can you please explain what is PCIe EP subsystem is? I assume, it is a subsystem to somehow configure the PCIe EP HW
Quoted 9 lines
instances?
Quoted 2 lines
If yes, it is not connected to any PCIe RC in your diagram.
PCIe EP subsystem is a Linux kernel framework to configure the PCIe EP IP inside an SoC/device. Here 'Endpoint' is a separate SoC/device that is running Linux kernel and uses PCIe EP subsystem in kernel [1] to configure the PCIe EP IP based on product usecase like GPU card, NVMe, Modem, WLAN etc...
I understood the PCI EP subsystem.
I didn’t follow it in context of virtio device as you have "virtio EPF".
Quoted 5 lines
Quoted 2 lines
So how does the MSI help in this case?
I think you are missing the point that 'Endpoint' is a separate SoC/device that is connected to a host machine over PCIe.
Understood.
Quoted 1 line
Just like how you would connect a PCIe based GPU card to a Desktop PC.
Also understood.
Quoted 5 lines
Only difference is, most of the PCIe
cards will run on a proprietary firmware supplied by the vendor, but here the firmware itself can be built by the user and configurable. And this is where Virtio is going to be exposed.
This part I read few times, but not understanding.
A PCI EP can be a virtio or nvme or any device.
A PCI controller driver in Linux can call devm_pci_epc_create() and implement virtio specific configuration headers. Don’t see this anyway related to MSI-X.
A PCI controller driver may operate a non virtio SoC device, right?
Are you trying to create a new kind of virtio device that is actually bind to PCI controller driver? and if so, it likely needs a new device id as this is the control point of PCIe EP device.
and idea is to consume MSI vectors by this PCI controller driver like updated virtio PCI driver , it looks fine to me.
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
While doing so, we faced an issue due to lack of MSI support defined in Virtio spec for PCI transport. Currently, the PCI transport (starting from 0.9.5) has only defined INTx (legacy) and MSI-X interrupts for the device to send notifications to the guest. While it works well for the hypervisor to guest communcation, when a physical PCIe device is used as a Virtio device, lack of MSI support is
hurting the performance (when there is no MSI-X).
Quoted 2 lines
Quoted 1 line
I am familiar with the scale issue of MSI-X, which is better for MSI (relative
to MSI-X).
Quoted 2 lines
What prevents implementing the MSI-X?
As I said, most of the devices I'm aware doesn't support MSI-X in hardware itself (I mean in the PCIe EP IP inside the SoC/device). For simple usecases like WLAN, modem, MSI-X is not really required.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Most of the physical PCIe endpoint devices support MSI interrupts over MSI-
I am not sure if this is true. :)
But not a concern either.It really depends on the usecase I would say.
Right. And also depends from which side you see it.
i.e. to see PCIe EP device from RC side or from Linux PCIe EP subsystem side.
A PCIe EP device does not support MSI-X is confusing term to say,
Because from a host (server) root complex point of view, which is seeing virtio or nvme PCI device, it is a PCIe EP device.
Therefore, for simplicity, just say,
A virtio PCI device does not support MSI interrupt mode.
And it is useful to support it, that is optimized then MSI-X at lower scale.
Quoted 28 lines
Quoted 9 lines
Quoted 9 lines
X for simplicity and with Virtio not supporting MSI, falling back to legacy INTx interrupts is affecting the performance.
First of all, INTx requires the PCIe devices to send two MSG TLPs (Assert/Deassert) to emulate level triggered interrupt on the host. And there could be some delay between assert and deassert messages to make sure that the host recognizes it as an interrupt (level trigger). Also, the INTx interrupts are limited to 1 per function, so all the notifications from device has to share this single interrupt
(INTA).
Quoted 7 lines
Quoted 1 line
Yes, INTx deprecation is in my list but didn’t get their yet.
Quoted 4 lines
On the other hand, MSI requires only one MWr TLP from the device to host and since it is a posted write, there is no delay involved. Also, a single PCIe function can use upto 32 MSIs, thus making it possible to use one MSI vector per virtqueue (32 is more than enough for
most of the usecases).
Quoted 7 lines
Quoted 5 lines
So my question is, why does the Virtio spec not supporting MSI? If there are no major blocker in supporting MSI, could we propose adding MSI to the Virtio spec?
MSI addition is good for virtio for small scale devices of 1 to 32. PCIe EP may support MSI-X and MSI both the capabilities and sw can give
preference to MSI when the need is <= 32 vectors.
Quoted 1 line
Quoted 7 lines
PCIe specification only mandates the devices to support either MSI or MSI-X.
Reference: PCIe spec r5.0, sec 6.1.4:
"All PCI Express device Functions that are capable of generating interrupts must support MSI or MSI-X or both."
I am referring to the last word "both".
Quoted 4 lines
So MSI-X is clearly an optional feature which simple devices tend to ignore. But if both are supported, then obviously Virtio will make use of MSI-X, but that's not the case here.
If both are supported, and required scale by the driver is <=32, driver can choose MSI due to its lightweight nature. Why do you say "obviously virtio will make use of MSI-X?" do you mean current code or future code?
Quoted 8 lines
Quoted 1 line
Though I don’t see it anyway related to PCIe EP configuration in your
diagram.
Quoted 3 lines
In other words, PCI EP subsystem can still work with MSI-X. Can you please elaborate it?
I hope the above info clarifies. If not, please let me know.
The only part that is not clear to me is, the PCIe EP controller driver is attaching to virtio device or some vendor specific SoC platform device?
Quoted 5 lines · Parav Pandit
Quoted 4 lines
So MSI-X is clearly an optional feature which simple devices tend to ignore. But if both are supported, then obviously Virtio will make use of MSI-X, but that's not the case here.
If both are supported, and required scale by the driver is <=32, driver can choose MSI due to its lightweight nature.
Unlikely, MSI vectors are tricky to mask and this is a problem for interrupt balancing. So MSIX is better for performance even if the # of vectors is low.
Signature · 1 line
MST
Quoted 13 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 25, 2024 1:25 PM
To: Parav Pandit <[email protected]>
Cc: Manivannan Sadhasivam <[email protected]>; virtio- [email protected]; [email protected]
Subject: Re: MSI for Virtio PCI transportQuoted 4 lines · Parav Pandit
Quoted 4 lines
So MSI-X is clearly an optional feature which simple devices tend to ignore. But if both are supported, then obviously Virtio will make use of MSI-X, but that's not the case here.
Quoted 5 lines
Quoted 1 line
If both are supported, and required scale by the driver is <=32, driver can
choose MSI due to its lightweight nature.
Unlikely, MSI vectors are tricky to mask and this is a problem for interrupt balancing. So MSIX is better for performance even if the # of vectors is low.
Masking to my knowledge is not used by MSIX.
Didn't follow how MSIX helps with performance.
The benefit of MSI is it does not need to store addr+data pair per vector.
Quoted 3 lines
--
MST
Quoted 13 lines · Parav Pandit
Quoted 13 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 25, 2024 1:25 PM
To: Parav Pandit <[email protected]>
Cc: Manivannan Sadhasivam <[email protected]>; virtio- [email protected]; [email protected]
Subject: Re: MSI for Virtio PCI transportQuoted 4 lines · Parav Pandit
Quoted 4 lines
So MSI-X is clearly an optional feature which simple devices tend to ignore. But if both are supported, then obviously Virtio will make use of MSI-X, but that's not the case here.
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
If both are supported, and required scale by the driver is <=32, driver can
choose MSI due to its lightweight nature.
Unlikely, MSI vectors are tricky to mask and this is a problem for interrupt balancing. So MSIX is better for performance even if the # of vectors is low.
Masking to my knowledge is not used by MSIX.
There's a mask bit per vector, yes.
Quoted 1 line
Didn't follow how MSIX helps with performance.
Linux uses mask/change/unmask to balance interrupts between CPUs. *That* is important for performance.
Quoted 1 line
The benefit of MSI is it does not need to store addr+data pair per vector.
The addr/data thing wasn't invented just to make hardware costs go up.
Quoted 3 lines
Quoted 3 lines
--
MST
Quoted 26 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 25, 2024 1:40 PMQuoted 10 lines · Parav Pandit
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 25, 2024 1:25 PM
To: Parav Pandit <[email protected]>
Cc: Manivannan Sadhasivam <[email protected]>; virtio- [email protected]; [email protected]
Subject: Re: MSI for Virtio PCI transportQuoted 1 line · Parav Pandit
Quoted 1 line
So MSI-X is clearly an optional feature which simple devices tend to
ignore.
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
But if both are supported, then obviously Virtio will make use of MSI-X, but that's not the case here.
If both are supported, and required scale by the driver is <=32, driver can
choose MSI due to its lightweight nature.
Unlikely, MSI vectors are tricky to mask and this is a problem for interrupt balancing. So MSIX is better for performance even if the # of
vectors is low.
Quoted 8 lines
Quoted 1 line
Masking to my knowledge is not used by MSIX.
There's a mask bit per vector, yes.
Quoted 1 line
Didn't follow how MSIX helps with performance.
Linux uses mask/change/unmask to balance interrupts between CPUs. *That* is important for performance.
Ah, I see, didn't know this.
Frequently reprogramming the mask in device is expensive and also requires synchronization to not miss the interrupt.
In case if you have pointer to the Linux code please share.
A while back, I recollect seeing reprogramming done by the IOAPIC at the interrupt table level within the cpu (without involving the device) to forward to different cpu.
Quoted 6 lines
Quoted 3 lines
The benefit of MSI is it does not need to store addr+data pair per vector.
The addr/data thing wasn't invented just to make hardware costs go up.
I am aware of it. :)
I will let other non virtio forums finish their ongoing optimization work in this area to reduce hardware costs.
Quoted 30 lines · Parav Pandit
Quoted 30 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 25, 2024 1:40 PMQuoted 10 lines · Parav Pandit
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 25, 2024 1:25 PM
To: Parav Pandit <[email protected]>
Cc: Manivannan Sadhasivam <[email protected]>; virtio- [email protected]; [email protected]
Subject: Re: MSI for Virtio PCI transportQuoted 1 line · Parav Pandit
Quoted 1 line
So MSI-X is clearly an optional feature which simple devices tend to
ignore.
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
But if both are supported, then obviously Virtio will make use of MSI-X, but that's not the case here.
If both are supported, and required scale by the driver is <=32, driver can
choose MSI due to its lightweight nature.
Unlikely, MSI vectors are tricky to mask and this is a problem for interrupt balancing. So MSIX is better for performance even if the # of
vectors is low.
Quoted 1 line
Masking to my knowledge is not used by MSIX.
There's a mask bit per vector, yes.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Didn't follow how MSIX helps with performance.
Linux uses mask/change/unmask to balance interrupts between CPUs. *That* is important for performance.
Ah, I see, didn't know this.
Frequently reprogramming the mask in device is expensive and also requires synchronization to not miss the interrupt. In case if you have pointer to the Linux code please share.
static inline void pci_msix_mask(struct msi_desc *desc)
{
desc->pci.msix_ctrl |= PCI_MSIX_ENTRY_CTRL_MASKBIT;
pci_msix_write_vector_ctrl(desc, desc->pci.msix_ctrl);
/* Flush write to device */
readl(desc->pci.mask_base);
}and
static inline void pci_write_msg_msix(struct msi_desc *desc, struct msi_msg *msg)
{
void __iomem *base = pci_msix_desc_addr(desc);
u32 ctrl = desc->pci.msix_ctrl;
bool unmasked = !(ctrl & PCI_MSIX_ENTRY_CTRL_MASKBIT);
if (desc->pci.msi_attrib.is_virtual)
return;
/*
* The specification mandates that the entry is masked
* when the message is modified:
*
* "If software changes the Address or Data value of an
* entry while the entry is unmasked, the result is
* undefined."
*/
if (unmasked)
pci_msix_write_vector_ctrl(desc, ctrl | PCI_MSIX_ENTRY_CTRL_MASKBIT);
writel(msg->address_lo, base + PCI_MSIX_ENTRY_LOWER_ADDR);
writel(msg->address_hi, base + PCI_MSIX_ENTRY_UPPER_ADDR);
writel(msg->data, base + PCI_MSIX_ENTRY_DATA);
if (unmasked)
pci_msix_write_vector_ctrl(desc, ctrl);
/* Ensure that the writes are visible in the device */
readl(base + PCI_MSIX_ENTRY_DATA);
}both do this.
Quoted 1 line
A while back, I recollect seeing reprogramming done by the IOAPIC at the interrupt table level within the cpu (without involving the device) to forward to different cpu.
AFAIK MSI goes to lapic not to ioapic. I might be confused though.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The benefit of MSI is it does not need to store addr+data pair per vector.
The addr/data thing wasn't invented just to make hardware costs go up.
I am aware of it. :)
I will let other non virtio forums finish their ongoing optimization work in this area to reduce hardware costs.
Absolutely.
Signature · 1 line
MST
Quoted 96 lines · Parav Pandit
Quoted 68 lines
From: Manivannan Sadhasivam <[email protected]>
Sent: Tuesday, June 25, 2024 11:14 AMQuoted 15 lines · Parav Pandit
Hi,
Quoted 13 lines
From: Manivannan Sadhasivam <[email protected]>
Sent: Monday, June 24, 2024 9:50 PMHi,
We are looking into adapting Virtio spec for configurable physical PCIe endpoint devices to expose Virtio devices to the host machine connected over PCIe. This allows us to use the existing frontend drivers on the host machine, thus minimizing the development efforts. This idea is not new as some vendors like NVidia have already released customized PCIe devices exposing Virtio devices to the host machines. But we are working on making the configurable PCIe devices running Linux kernel to expose Virtio devices using the PCI
Endpoint (EP) subsystem.
Quoted 2 lines
Quoted 2 lines
Below is the simplistic represenation of the idea with virt-net as an
example.
Quoted 41 lines
Quoted 32 lines
But this could be extended to any supported Virtio devices:
HOST ENDPOINT
+-----------------------------+ +-----------------------------+ +-----------------------------+ | | | | | Linux Kernel | | Linux Kernel | | | | | | | | +------------------+ | | | | | | | | | | | Modem | | | | | | | | | | | +---------|--------+ | | | | | | | +------------------+ | | +---------|--------+ | | | | | | | | | | | Virt-net | | | | Virtio EPF | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | Virtio PCI | | | | PCI EP Subsystem | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | SW | | | SW | | ----------------|-------------- ----------------|-------------- | HW | | | HW | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | PCIe RC | | | | PCIe EP | |Quoted 9 lines
| | | | | | | | +-----+---------|--------+----+ +-----+---------|--------+----+ +-----+---------|--------+----+ | | | | | | | PCIe | -----------------------------------------Quoted 2 lines
Can you please explain what is PCIe EP subsystem is? I assume, it is a subsystem to somehow configure the PCIe EP HW
Quoted 9 lines
instances?
Quoted 2 lines
If yes, it is not connected to any PCIe RC in your diagram.
PCIe EP subsystem is a Linux kernel framework to configure the PCIe EP IP inside an SoC/device. Here 'Endpoint' is a separate SoC/device that is running Linux kernel and uses PCIe EP subsystem in kernel [1] to configure the PCIe EP IP based on product usecase like GPU card, NVMe, Modem, WLAN etc...
I understood the PCI EP subsystem.
I didn’t follow it in context of virtio device as you have "virtio EPF".Quoted 5 lines
Quoted 2 lines
So how does the MSI help in this case?
I think you are missing the point that 'Endpoint' is a separate SoC/device that is connected to a host machine over PCIe.
Understood.
Quoted 1 line
Just like how you would connect a PCIe based GPU card to a Desktop PC.
Also understood.
Quoted 5 lines
Only difference is, most of the PCIe
cards will run on a proprietary firmware supplied by the vendor, but here the firmware itself can be built by the user and configurable. And this is where Virtio is going to be exposed.
Quoted 4 lines
This part I read few times, but not understanding.
A PCI EP can be a virtio or nvme or any device.
A PCI controller driver in Linux can calldevm_pci_epc_create()and implement virtio specific configuration headers.
Right. Although, there is one more component called EPF (Endpoint Function) driver that implements the header for the device. I didn't go in detail because I thought that would mislead the discussion. But if you want to know more about PCI Endpoint subsystem, please take a look at my ELEC presentation:
https://elinux.org/images/3/3a/PCI_Endpoint_drivers_in_Linux_kernel_and_How_to_write_one_.pdf
Quoted 1 line
Don’t see this anyway related to MSI-X.
MSI/MSI-X is a PCIe endpoint controller (hardware) capability. Based on that, the PCI EP subsystem will expose those functionalities to the host. But if the underlying hardware is not supporing MSI-X, then it won't be exposed.
Quoted 2 lines
A PCI controller driver may operate a non virtio SoC device, right?
If you go over my presentation, then you will get the internals of EP subsystem. According to that, a PCIe endpoint device may expose it as any kind of device to the host. That behavior is defined by the EPF driver. Currently, mainline Linux kernel supports a test driver, MHI (Qcom specific), NTB function drivers. And if Virtio is supported, then there would be a common Virtio driver together with an usecase specific driver like virt-net, virt-blk etc...
Quoted 3 lines
Are you trying to create a new kind of virtio device that is actually bind to PCI controller driver? and if so, it likely needs a new device id as this is the control point of PCIe EP device.
No new Virtio device. We are just trying to expose transitional Virtio devices like virt-net, virt-blk, virt-console etc... The idea is to just expose these devices and make use of the existing frontend drivers on the host machine.
Just like how a hypervisor would expose Virtio devices to the guest. Here, hypervisor is replaced by the PCIe endpoint device running Linux kernel.
Quoted 2 lines
and idea is to consume MSI vectors by this PCI controller driver like updated virtio PCI driver , it looks fine to me.
Yeah, since our devices are supporting MSI only, we want the host side Virtio stack to make use of it.
Quoted 29 lines
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
While doing so, we faced an issue due to lack of MSI support defined in Virtio spec for PCI transport. Currently, the PCI transport (starting from 0.9.5) has only defined INTx (legacy) and MSI-X interrupts for the device to send notifications to the guest. While it works well for the hypervisor to guest communcation, when a physical PCIe device is used as a Virtio device, lack of MSI support is
hurting the performance (when there is no MSI-X).
Quoted 2 lines
Quoted 1 line
I am familiar with the scale issue of MSI-X, which is better for MSI (relative
to MSI-X).
Quoted 2 lines
What prevents implementing the MSI-X?
As I said, most of the devices I'm aware doesn't support MSI-X in hardware itself (I mean in the PCIe EP IP inside the SoC/device). For simple usecases like WLAN, modem, MSI-X is not really required.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Most of the physical PCIe endpoint devices support MSI interrupts over MSI-
I am not sure if this is true. :)
But not a concern either.It really depends on the usecase I would say.
Right. And also depends from which side you see it.
i.e. to see PCIe EP device from RC side or from Linux PCIe EP subsystem side.
Quoted 8 lines
A PCIe EP device does not support MSI-X is confusing term to say,
Because from a host (server) root complex point of view, which is seeing virtio or nvme PCI device, it is a PCIe EP device.Therefore, for simplicity, just say,
A virtio PCI device does not support MSI interrupt mode.
And it is useful to support it, that is optimized then MSI-X at lower scale.
Ok, I now get what you are saying. I was describing more from a PCIe endpoint device point of view.
Quoted 28 lines
Quoted 28 lines
Quoted 9 lines
Quoted 9 lines
X for simplicity and with Virtio not supporting MSI, falling back to legacy INTx interrupts is affecting the performance.
First of all, INTx requires the PCIe devices to send two MSG TLPs (Assert/Deassert) to emulate level triggered interrupt on the host. And there could be some delay between assert and deassert messages to make sure that the host recognizes it as an interrupt (level trigger). Also, the INTx interrupts are limited to 1 per function, so all the notifications from device has to share this single interrupt
(INTA).
Quoted 7 lines
Quoted 1 line
Yes, INTx deprecation is in my list but didn’t get their yet.
Quoted 4 lines
On the other hand, MSI requires only one MWr TLP from the device to host and since it is a posted write, there is no delay involved. Also, a single PCIe function can use upto 32 MSIs, thus making it possible to use one MSI vector per virtqueue (32 is more than enough for
most of the usecases).
Quoted 7 lines
Quoted 5 lines
So my question is, why does the Virtio spec not supporting MSI? If there are no major blocker in supporting MSI, could we propose adding MSI to the Virtio spec?
MSI addition is good for virtio for small scale devices of 1 to 32. PCIe EP may support MSI-X and MSI both the capabilities and sw can give
preference to MSI when the need is <= 32 vectors.
Quoted 1 line
Quoted 9 lines
Quoted 7 lines
PCIe specification only mandates the devices to support either MSI or MSI-X.
Reference: PCIe spec r5.0, sec 6.1.4:
"All PCI Express device Functions that are capable of generating interrupts must support MSI or MSI-X or both."
I am referring to the last word "both".
But there is an 'or' before 'both', that's what I'm trying to highlight. Because, there is no necessity for a device to support MSI-X. But current Virtio Linux driver expects the device to support MSI-X, otherwise falls back to INTx.
Quoted 8 lines
Quoted 5 lines
So MSI-X is clearly an optional feature which simple devices tend to ignore. But if both are supported, then obviously Virtio will make use of MSI-X, but that's not the case here.
If both are supported, and required scale by the driver is <=32, driver can choose MSI due to its lightweight nature. Why do you say "obviously virtio will make use of MSI-X?" do you mean current code or future code?
Current code:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/virtio/virtio_pci_common.c#n102
The API vp_request_msix_vectors() just requests MSI-X using the flag PCI_IRQ_MSIX. Because of this, even if the device supports MSI, it won't be used by Virtio, hence falling back to legacy INTx.
Quoted 10 lines
Quoted 8 lines
Quoted 1 line
Though I don’t see it anyway related to PCIe EP configuration in your
diagram.
Quoted 3 lines
In other words, PCI EP subsystem can still work with MSI-X. Can you please elaborate it?
I hope the above info clarifies. If not, please let me know.
The only part that is not clear to me is, the PCIe EP controller driver is attaching to virtio device or some vendor specific SoC platform device?
I think above justification will clarify this.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 106 lines
From: Manivannan Sadhasivam <[email protected]>
Sent: Tuesday, June 25, 2024 2:41 PMQuoted 70 lines · Parav Pandit
Quoted 70 lines
From: Manivannan Sadhasivam <[email protected]>
Sent: Tuesday, June 25, 2024 11:14 AMQuoted 16 lines · Parav Pandit
Hi,
Quoted 14 lines
From: Manivannan Sadhasivam <[email protected]>
Sent: Monday, June 24, 2024 9:50 PMHi,
We are looking into adapting Virtio spec for configurable physical PCIe endpoint devices to expose Virtio devices to the host machine connected over PCIe. This allows us to use the existing frontend drivers on the host machine, thus minimizing the development efforts. This idea is not new as some vendors like NVidia have already released customized PCIe devices exposing Virtio devices to the host machines. But we are working on making the configurable PCIe devices running Linux kernel to expose Virtio devices using the PCI
Endpoint (EP) subsystem.
Quoted 3 lines
Quoted 3 lines
Below is the simplistic represenation of the idea with virt-net as an
example.
Quoted 43 lines
Quoted 32 lines
But this could be extended to any supported Virtio devices:
HOST ENDPOINT
+-----------------------------+ +-----------------------------+ +-----------------------------+ | | | | | Linux Kernel | | Linux Kernel | | | | | | | | +------------------+ | | | | | | | | | | | Modem | | | | | | | | | | | +---------|--------+ | | | | | | | +------------------+ | | +---------|--------+ | | | | | | | | | | | Virt-net | | | | Virtio EPF | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | Virtio PCI | | | | PCI EP Subsystem | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | SW | | | SW | | ----------------|-------------- ----------------|-------------- | HW | | | HW | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | PCIe RC | | | | PCIe EP | |Quoted 9 lines
| | | | | | | | +-----+---------|--------+----+ +-----+---------|--------+----+ +-----+---------|--------+----+ | | | | | | | PCIe | -----------------------------------------Can you please explain what is PCIe EP subsystem is? I assume, it is a subsystem to somehow configure the PCIe EP HW
Quoted 8 lines
Quoted 8 lines
instances?
Quoted 2 lines
If yes, it is not connected to any PCIe RC in your diagram.
PCIe EP subsystem is a Linux kernel framework to configure the PCIe EP IP inside an SoC/device. Here 'Endpoint' is a separate SoC/device that is running Linux kernel and uses PCIe EP subsystem in kernel
[1] to configure the PCIe EP IP based on product usecase like GPU card,NVMe, Modem, WLAN etc...
Quoted 22 lines
Quoted 1 line
I understood the PCI EP subsystem.
I didn’t follow it in context of virtio device as you have "virtio EPF".Quoted 5 lines
Quoted 2 lines
So how does the MSI help in this case?
I think you are missing the point that 'Endpoint' is a separate SoC/device that is connected to a host machine over PCIe.
Understood.
Quoted 1 line
Just like how you would connect a PCIe based GPU card to a Desktop PC.
Also understood.
Quoted 5 lines
Only difference is, most of the PCIe cards will run on a proprietary firmware supplied by the vendor, but here the firmware itself can be built by the user and configurable. And this is where Virtio is going to be exposed.
This part I read few times, but not understanding.
Quoted 10 lines
Quoted 2 lines
A PCI EP can be a virtio or nvme or any device.
A PCI controller driver in Linux can calldevm_pci_epc_create()andimplement virtio specific configuration headers.
Right. Although, there is one more component called EPF (Endpoint Function) driver that implements the header for the device. I didn't go in detail because I thought that would mislead the discussion. But if you want to know more about PCI Endpoint subsystem, please take a look at my ELEC presentation:
Yes. bringing PCI EPF to this discussion was confusing.
But I am familiar with building composable PCI EPs for virtio and others. MSI has no relation to how one composes PCIe EPs.
Quoted 5 lines
Quoted 1 line
Don’t see this anyway related to MSI-X.
MSI/MSI-X is a PCIe endpoint controller (hardware) capability. Based on that, the PCI EP subsystem will expose those functionalities to the host. But if the underlying hardware is not supporing MSI-X, then it won't be exposed.
Yes.
Quoted 4 lines
Quoted 3 lines
A PCI controller driver may operate a non virtio SoC device, right?
Quoted 8 lines
If you go over my presentation, then you will get the internals of EP subsystem. According to that, a PCIe endpoint device may expose it as any kind of device to the host. That behavior is defined by the EPF driver. Currently, mainline Linux kernel supports a test driver, MHI (Qcom specific), NTB function drivers. And if Virtio is supported, then there would be a common Virtio driver together with an usecase specific driver like virt-net, virt- blk etc...
Yes. I am clear.
Quoted 9 lines
Quoted 1 line
Are you trying to create a new kind of virtio device that is actually bind to PCI
controller driver?
Quoted 1 line
and if so, it likely needs a new device id as this is the control point of PCIe EP
device.
Quoted 1 line
No new Virtio device. We are just trying to expose transitional Virtio devices like virt-net, virt-blk, virt-console etc... The idea is to just expose these devices and make use of the existing frontend drivers on the host machine.
Right. I went one step further to see if you are exploring "struct device" in the devm_pci_epc_create() to be also a virtio device or not. I understood now, that is not your goal.
You are just replacing hypervisor sw with the semi hw to compose the hardware PCIe EPs.
Quoted 11 lines
Just like how a hypervisor would expose Virtio devices to the guest. Here, hypervisor is replaced by the PCIe endpoint device running Linux kernel.
Quoted 1 line
and idea is to consume MSI vectors by this PCI controller driver like updated
virtio PCI driver , it looks fine to me.
Quoted 1 line
Yeah, since our devices are supporting MSI only, we want the host side Virtio stack to make use of it.
Yeah, make sense.
Quoted 84 lines
Quoted 26 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
While doing so, we faced an issue due to lack of MSI support defined in Virtio spec for PCI transport. Currently, the PCI transport (starting from 0.9.5) has only defined INTx (legacy) and MSI-X interrupts for the device to send notifications to the guest. While it works well for the hypervisor to guest communcation, when a physical PCIe device is used as a Virtio device, lack of MSI support is
hurting the performance (when there is no MSI-X).
Quoted 3 lines
Quoted 1 line
I am familiar with the scale issue of MSI-X, which is better for MSI (relative
to MSI-X).
Quoted 2 lines
What prevents implementing the MSI-X?
As I said, most of the devices I'm aware doesn't support MSI-X in hardware itself (I mean in the PCIe EP IP inside the SoC/device). For simple usecases like WLAN, modem, MSI-X is not really required.
Quoted 5 lines
Quoted 2 lines
Most of the physical PCIe endpoint devices support MSI interrupts over MSI-
I am not sure if this is true. :)
But not a concern either.Quoted 7 lines
Quoted 2 lines
It really depends on the usecase I would say.
Right. And also depends from which side you see it.
i.e. to see PCIe EP device from RC side or from Linux PCIe EP subsystem side.A PCIe EP device does not support MSI-X is confusing term to say, Because from a host (server) root complex point of view, which is seeing
virtio or nvme PCI device, it is a PCIe EP device.
Quoted 6 lines
Therefore, for simplicity, just say,
A virtio PCI device does not support MSI interrupt mode.
And it is useful to support it, that is optimized then MSI-X at lower scale.Ok, I now get what you are saying. I was describing more from a PCIe endpoint device point of view.
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
Quoted 11 lines
X for simplicity and with Virtio not supporting MSI, falling back to legacy INTx interrupts is affecting the performance.
First of all, INTx requires the PCIe devices to send two MSG TLPs
(Assert/Deassert) to emulate level triggered interrupt on the host. And there could be some delay between assert and deassert messages to make sure that the host recognizes it as an interrupt (level trigger). Also, the INTx interrupts are limited to 1 per function, so all the notifications from device has to share this single interrupt(INTA).
Quoted 8 lines
Quoted 1 line
Yes, INTx deprecation is in my list but didn’t get their yet.
Quoted 5 lines
On the other hand, MSI requires only one MWr TLP from the device to host and since it is a posted write, there is no delay involved. Also, a single PCIe function can use upto 32 MSIs, thus making it possible to use one MSI vector per virtqueue (32 is more than enough for
most of the usecases).
Quoted 5 lines
Quoted 5 lines
So my question is, why does the Virtio spec not supporting MSI? If there are no major blocker in supporting MSI, could we propose adding MSI to the Virtio spec?
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
MSI addition is good for virtio for small scale devices of 1 to 32. PCIe EP may support MSI-X and MSI both the capabilities and sw can give
preference to MSI when the need is <= 32 vectors.
Quoted 1 line
PCIe specification only mandates the devices to support either MSI or MSI-
X.
Quoted 6 lines
Quoted 6 lines
Reference: PCIe spec r5.0, sec 6.1.4:
"All PCI Express device Functions that are capable of generating interrupts must support MSI or MSI-X or both."
Quoted 7 lines
Quoted 2 lines
I am referring to the last word "both".
But there is an 'or' before 'both', that's what I'm trying to highlight. Because, there is no necessity for a device to support MSI-X. But current Virtio Linux driver expects the device to support MSI-X, otherwise falls back to INTx.
Yes. we are aligned.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
So MSI-X is clearly an optional feature which simple devices tend to ignore. But if both are supported, then obviously Virtio will make use of MSI-X, but that's not the case here.
Quoted 7 lines
Quoted 1 line
If both are supported, and required scale by the driver is <=32, driver can
choose MSI due to its lightweight nature.
Quoted 1 line
Why do you say "obviously virtio will make use of MSI-X?" do you mean
current code or future code?
Quoted 1 line
Current code:
..
Ok.
Whatever PCIe EPs you make with MSI only capability will work, once the virtio spec and driver will support MSI.
The basic motivation to add MSI support to the virtio PCI transport is to support light weight composable hw PCI devices, which are more performant than INTx and light weight than MSI-X. Looks good to me. I am ignoring masking bits limitation of the MSI for now.
Thanks.
Quoted 24 lines
The API
vp_request_msix_vectors()just requests MSI-X using the flagPCI_IRQ_MSIX. Because of this, even if the device supports MSI, it won't be used by Virtio, hence falling back to legacy INTx.Quoted 11 lines
Quoted 9 lines
Quoted 2 lines
Though I don’t see it anyway related to PCIe EP configuration in your
diagram.
Quoted 3 lines
In other words, PCI EP subsystem can still work with MSI-X. Can you please elaborate it?
I hope the above info clarifies. If not, please let me know.
The only part that is not clear to me is, the PCIe EP controller driver is
attaching to virtio device or some vendor specific SoC platform device?
I think above justification will clarify this.
- Mani
--
மணிவண்ணன் சதாசிவம்
Quoted 74 lines · Manivannan Sadhasivam
Hi,
We are looking into adapting Virtio spec for configurable physical PCIe endpoint devices to expose Virtio devices to the host machine connected over PCIe. This allows us to use the existing frontend drivers on the host machine, thus minimizing the development efforts. This idea is not new as some vendors like NVidia have already released customized PCIe devices exposing Virtio devices to the host machines. But we are working on making the configurable PCIe devices running Linux kernel to expose Virtio devices using the PCI Endpoint (EP) subsystem.
Below is the simplistic represenation of the idea with virt-net as an example. But this could be extended to any supported Virtio devices:
HOST ENDPOINT
+-----------------------------+ +-----------------------------+ | | | | | Linux Kernel | | Linux Kernel | | | | | | | | +------------------+ | | | | | | | | | | | Modem | | | | | | | | | | | +---------|--------+ | | | | | | | +------------------+ | | +---------|--------+ | | | | | | | | | | | Virt-net | | | | Virtio EPF | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | Virtio PCI | | | | PCI EP Subsystem | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | SW | | | SW | | ----------------|-------------- ----------------|-------------- | HW | | | HW | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | PCIe RC | | | | PCIe EP | | | | | | | | | | +-----+---------|--------+----+ +-----+---------|--------+----+ | | | | | | | PCIe | -----------------------------------------While doing so, we faced an issue due to lack of MSI support defined in Virtio spec for PCI transport. Currently, the PCI transport (starting from 0.9.5) has only defined INTx (legacy) and MSI-X interrupts for the device to send notifications to the guest. While it works well for the hypervisor to guest communcation, when a physical PCIe device is used as a Virtio device, lack of MSI support is hurting the performance (when there is no MSI-X).
Most of the physical PCIe endpoint devices support MSI interrupts over MSI-X for simplicity and with Virtio not supporting MSI, falling back to legacy INTx interrupts is affecting the performance.
First of all, INTx requires the PCIe devices to send two MSG TLPs (Assert/Deassert) to emulate level triggered interrupt on the host. And there could be some delay between assert and deassert messages to make sure that the host recognizes it as an interrupt (level trigger). Also, the INTx interrupts are limited to 1 per function, so all the notifications from device has to share this single interrupt (INTA).
On the other hand, MSI requires only one MWr TLP from the device to host and since it is a posted write, there is no delay involved. Also, a single PCIe function can use upto 32 MSIs, thus making it possible to use one MSI vector per virtqueue (32 is more than enough for most of the usecases).
Quoted 7 lines
So my question is, why does the Virtio spec not supporting MSI? If there are no major blocker in supporting MSI, could we propose adding MSI to the Virtio spec?
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Yes, it's possible to add - however, you also said EP requires more changes from virtio. So maybe we need "virtio over EP" then. Let's try to figure out the full list of issues, to see which makes more sense.
Quoted 78 lines · Michael S. Tsirkin
Quoted 77 lines · Manivannan Sadhasivam
Hi,
We are looking into adapting Virtio spec for configurable physical PCIe endpoint devices to expose Virtio devices to the host machine connected over PCIe. This allows us to use the existing frontend drivers on the host machine, thus minimizing the development efforts. This idea is not new as some vendors like NVidia have already released customized PCIe devices exposing Virtio devices to the host machines. But we are working on making the configurable PCIe devices running Linux kernel to expose Virtio devices using the PCI Endpoint (EP) subsystem.
Below is the simplistic represenation of the idea with virt-net as an example. But this could be extended to any supported Virtio devices:
HOST ENDPOINT
+-----------------------------+ +-----------------------------+ | | | | | Linux Kernel | | Linux Kernel | | | | | | | | +------------------+ | | | | | | | | | | | Modem | | | | | | | | | | | +---------|--------+ | | | | | | | +------------------+ | | +---------|--------+ | | | | | | | | | | | Virt-net | | | | Virtio EPF | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | Virtio PCI | | | | PCI EP Subsystem | | | | | | | | | | | +---------|--------+ | | +---------|--------+ | | SW | | | SW | | ----------------|-------------- ----------------|-------------- | HW | | | HW | | | +---------|--------+ | | +---------|--------+ | | | | | | | | | | | PCIe RC | | | | PCIe EP | | | | | | | | | | +-----+---------|--------+----+ +-----+---------|--------+----+ | | | | | | | PCIe | -----------------------------------------While doing so, we faced an issue due to lack of MSI support defined in Virtio spec for PCI transport. Currently, the PCI transport (starting from 0.9.5) has only defined INTx (legacy) and MSI-X interrupts for the device to send notifications to the guest. While it works well for the hypervisor to guest communcation, when a physical PCIe device is used as a Virtio device, lack of MSI support is hurting the performance (when there is no MSI-X).
Most of the physical PCIe endpoint devices support MSI interrupts over MSI-X for simplicity and with Virtio not supporting MSI, falling back to legacy INTx interrupts is affecting the performance.
First of all, INTx requires the PCIe devices to send two MSG TLPs (Assert/Deassert) to emulate level triggered interrupt on the host. And there could be some delay between assert and deassert messages to make sure that the host recognizes it as an interrupt (level trigger). Also, the INTx interrupts are limited to 1 per function, so all the notifications from device has to share this single interrupt (INTA).
On the other hand, MSI requires only one MWr TLP from the device to host and since it is a posted write, there is no delay involved. Also, a single PCIe function can use upto 32 MSIs, thus making it possible to use one MSI vector per virtqueue (32 is more than enough for most of the usecases).
So my question is, why does the Virtio spec not supporting MSI? If there are no major blocker in supporting MSI, could we propose adding MSI to the Virtio spec?
Quoted 7 lines
Quoted 4 lines
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்Yes, it's possible to add - however, you also said EP requires more changes from virtio. So maybe we need "virtio over EP" then.
I don't think we need a separate 'Virtio over EP'. EP uses PCI transport, so 'Virtio over PCI' is fine as it is. Just because 'Virtio over PCI' was designed based on virtual PCI devices exposed by the hypervisor, the real world limitations were not taken into account. And that's what we are trying to add.
Quoted 3 lines
Let's try to figure out the full list of issues, to see which makes more sense.
Sure. But each one would need a separate discussion, that's why I started this thread for MSI. Let me check with Shunsuke and come up with an exhaustive list.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்