virtio PCI on KVM without IO BARs
Subsystems (2)
Michael S. Tsirkin
maintainerOK we talked about this a while ago, here's
a summary and some proposals:
At the moment, virtio PCI uses IO BARs for all accesses.
The reason for IO use is the cost of different VM exit types of transactions and their emulation on KVM on x86 (it would be trivial to use memory BARs on non x86 platforms
if they don't have PIO).
Example benchmark (cycles per transaction):
(io access) outw 1737 (memory access) movw 4341
for comparison:
(hypercall access): vmcall 1566
(pv memory access) movw_fast 1817 (*explanation what this is below)
This creates a problem if we want to make virtio devices proper PCI express devices with native hotplug support. This is because each hotpluggable PCI express device always has a PCI express port (port per device),
where each port is represented by a PCI to PCI bridge. In turn, a PCI to PCI bridge claims a 4Kbyte aligned range of IO addresses. This means that we can have at most 15 such devices, this is a nasty limitation.
Another problem with PIO is support for physical virtio devices, and nested virt: KVM currently programs all PIO accesses to cause vm exit, so using this device in a VM will be slow.
So we really want to stop using IO BARs completely if at all possible, but looking at the table above, switching to memory BAR and movw for notifications will not work well.
Possible solutions:
1. hypercall instead of PIO
basically add a hypercall that gets an MMIO address/data and does an MMIO write for us.
We'll want some capability in the device to let guest know
this is what it should do.
Pros: even faster than PIO
Cons: this won't help nested or assigned devices (won't hurt
them either as it will be conditional on the capability above).
Cons: need host kernel support, which then has to be maintained
forever, even if intel speeds up MMIO exits.
2. pv memory access
There are two reasons that memory access is slower:
- one is that it's handled as an EPT misconfiguration error so handled by cpu slow path
- one is that we need to decode the x86 instruction in software, to calculate address/data for the access.
We could agree that guests would use a specific instruction for virtio accesses, and fast-path it specifically. This is the pv memory access option above.
Pros: helps assigned devices and nested virt
Pros: easy to drop if hardware support is there
Cons: a bit slower than IO
Cons: need host kernel support
3. hypervisor assigned IO address
qemu can reserve IO addresses and assign to virtio devices. 2 bytes per device (for notification and ISR access) will be enough. So we can reserve 4K and this gets us 2000 devices. From KVM perspective, nothing changes.
We'll want some capability in the device to let guest know this is what it should do, and pass the io address. One way to reserve the addresses is by using the bridge.
Pros: no need for host kernel support
Pros: regular PIO so fast
Cons: does not help assigned devices, breaks nested virt
Simply counting pros/cons, option 3 seems best. It's also the easiest to implement.
Comments?
Signature · 1 line
MST
Quoted 3 lines · Michael S. Tsirkin
Another problem with PIO is support for physical virtio devices, and nested virt: KVM currently programs all PIO accesses to cause vm exit, so using this device in a VM will be slow.
Not answering your question, but support for programming direct PIO access into KVM's I/O bitmap would be feasible. Such feature may have some value for assigned devices that use PIO more heavily. They cause lengthy user-space exists so far.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Quoted 3 lines · Michael S. Tsirkin
Another problem with PIO is support for physical virtio devices, and nested virt: KVM currently programs all PIO accesses to cause vm exit, so using this device in a VM will be slow.
Not answering your question, but support for programming direct PIO access into KVM's I/O bitmap would be feasible. Such feature may have some value for assigned devices that use PIO more heavily. They cause lengthy user-space exists so far.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Quoted 71 lines · Michael S. Tsirkin
OK we talked about this a while ago, here's
a summary and some proposals:
At the moment, virtio PCI uses IO BARs for all accesses.The reason for IO use is the cost of different VM exit types of transactions and their emulation on KVM on x86 (it would be trivial to use memory BARs on non x86 platforms
if they don't have PIO).
Example benchmark (cycles per transaction):(io access) outw 1737 (memory access) movw 4341for comparison:
(hypercall access): vmcall 1566
(pv memory access)movw_fast1817 (*explanation what this is below)This creates a problem if we want to make virtio devices proper PCI express devices with native hotplug support. This is because each hotpluggable PCI express device always has a PCI express port (port per device),
where each port is represented by a PCI to PCI bridge. In turn, a PCI to PCI bridge claims a 4Kbyte aligned range of IO addresses. This means that we can have at most 15 such devices, this is a nasty limitation.Another problem with PIO is support for physical virtio devices, and nested virt: KVM currently programs all PIO accesses to cause vm exit, so using this device in a VM will be slow.
So we really want to stop using IO BARs completely if at all possible, but looking at the table above, switching to memory BAR and movw for notifications will not work well.
Possible solutions:
1. hypercall instead of PIO
basically add a hypercall that gets an MMIO address/data and does an MMIO write for us.
We'll want some capability in the device to let guest know
this is what it should do.
Pros: even faster than PIO
Cons: this won't help nested or assigned devices (won't hurt
them either as it will be conditional on the capability above).
Cons: need host kernel support, which then has to be maintained
forever, even if intel speeds up MMIO exits.2. pv memory access
There are two reasons that memory access is slower:
- one is that it's handled as an EPT misconfiguration error so handled by cpu slow path
- one is that we need to decode the x86 instruction in software, to calculate address/data for the access.We could agree that guests would use a specific instruction for virtio accesses, and fast-path it specifically. This is the pv memory access option above.
Pros: helps assigned devices and nested virt
Pros: easy to drop if hardware support is there
Cons: a bit slower than IO
Cons: need host kernel support3. hypervisor assigned IO address
qemu can reserve IO addresses and assign to virtio devices. 2 bytes per device (for notification and ISR access) will be enough. So we can reserve 4K and this gets us 2000 devices. From KVM perspective, nothing changes.
We'll want some capability in the device to let guest know this is what it should do, and pass the io address. One way to reserve the addresses is by using the bridge.
Pros: no need for host kernel support
Pros: regular PIO so fast
Cons: does not help assigned devices, breaks nested virt
Quoted 2 lines
Simply counting pros/cons, option 3 seems best. It's also the easiest to implement.
Agree.
Quoted 13 lines · Michael S. Tsirkin
3. hypervisor assigned IO address
qemu can reserve IO addresses and assign to virtio devices. 2 bytes per device (for notification and ISR access) will be enough. So we can reserve 4K and this gets us 2000 devices. From KVM perspective, nothing changes.
We'll want some capability in the device to let guest know this is what it should do, and pass the io address. One way to reserve the addresses is by using the bridge.
Pros: no need for host kernel support
Pros: regular PIO so fast
Cons: does not help assigned devices, breaks nested virt
Quoted 3 lines
Simply counting pros/cons, option 3 seems best. It's also the easiest to implement.
The problem here is the 4K I/O window for IO device BARs in bridges. Why not simply add a (possibly proprietary) capability to the PCI bridge to allow a much narrower window? That fits much more nicely into the device resource assignment on the guest side, and could even be implemented on a real hardware device -- we can offer it to the PCI-SIG for standardization, even.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 18 lines · H. Peter Anvin
Quoted 13 lines · Michael S. Tsirkin
3. hypervisor assigned IO address
qemu can reserve IO addresses and assign to virtio devices. 2 bytes per device (for notification and ISR access) will be enough. So we can reserve 4K and this gets us 2000 devices. From KVM perspective, nothing changes.
We'll want some capability in the device to let guest know this is what it should do, and pass the io address. One way to reserve the addresses is by using the bridge.
Pros: no need for host kernel support
Pros: regular PIO so fast
Cons: does not help assigned devices, breaks nested virtQuoted 3 lines
Simply counting pros/cons, option 3 seems best. It's also the easiest to implement.
Quoted 7 lines
The problem here is the 4K I/O window for IO device BARs in bridges. Why not simply add a (possibly proprietary) capability to the PCI bridge to allow a much narrower window? That fits much more nicely into the device resource assignment on the guest side, and could even be implemented on a real hardware device -- we can offer it to the PCI-SIG for standardization, even.
Just a correction: I'm of course not talking about BARs but of the bridge windows. The BARs are not a problem; an I/O BAR can cover as little as four bytes.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Michael S. Tsirkin
maintainerQuoted 27 lines · H. Peter Anvin
Quoted 18 lines · H. Peter Anvin
Quoted 13 lines · Michael S. Tsirkin
3. hypervisor assigned IO address
qemu can reserve IO addresses and assign to virtio devices. 2 bytes per device (for notification and ISR access) will be enough. So we can reserve 4K and this gets us 2000 devices. From KVM perspective, nothing changes.
We'll want some capability in the device to let guest know this is what it should do, and pass the io address. One way to reserve the addresses is by using the bridge.
Pros: no need for host kernel support
Pros: regular PIO so fast
Cons: does not help assigned devices, breaks nested virtQuoted 3 lines
Simply counting pros/cons, option 3 seems best. It's also the easiest to implement.
Quoted 7 lines
The problem here is the 4K I/O window for IO device BARs in bridges. Why not simply add a (possibly proprietary) capability to the PCI bridge to allow a much narrower window? That fits much more nicely into the device resource assignment on the guest side, and could even be implemented on a real hardware device -- we can offer it to the PCI-SIG for standardization, even.
Quoted 5 lines
Just a correction: I'm of course not talking about BARs but of the bridge windows. The BARs are not a problem; an I/O BAR can cover as little as four bytes.
-hpa
Right. Though even with better granularify bridge windows would still be a (smaller) problem causing fragmentation.
If we were to extend the PCI spec I would go for a bridge without windows at all: a bridge can snoop on configuration transactions and responses programming devices behind it and build a full map of address to device mappings.
In partucular, this would be a good fit for an uplink bridge in a PCI express switch, which is integrated with downlink bridges on the same silicon, so bridge windows do nothing but add overhead.
Quoted 3 lines
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 4 lines · Michael S. Tsirkin
Right. Though even with better granularify bridge windows would still be a (smaller) problem causing fragmentation.
Quoted 9 lines
If we were to extend the PCI spec I would go for a bridge without windows at all: a bridge can snoop on configuration transactions and responses programming devices behind it and build a full map of address to device mappings.
In partucular, this would be a good fit for an uplink bridge in a PCI express switch, which is integrated with downlink bridges on the same silicon, so bridge windows do nothing but add overhead.
True, but the real problem is that the downlink (type 1 header) is typically on a different piece of silicon than the device BAR (type 0 header).
I am not sure that a snooping-based system will work and not be prohibitive in its hardware cost on an actual hardware system. I suspect it would decay into needing a large RAM array in every bridge.
-hpa
Michael S. Tsirkin
maintainerQuoted 19 lines · H. Peter Anvin
Quoted 4 lines · Michael S. Tsirkin
Right. Though even with better granularify bridge windows would still be a (smaller) problem causing fragmentation.
Quoted 9 lines
If we were to extend the PCI spec I would go for a bridge without windows at all: a bridge can snoop on configuration transactions and responses programming devices behind it and build a full map of address to device mappings.
In partucular, this would be a good fit for an uplink bridge in a PCI express switch, which is integrated with downlink bridges on the same silicon, so bridge windows do nothing but add overhead.
True, but the real problem is that the downlink (type 1 header) is typically on a different piece of silicon than the device BAR (type 0 header).
Quoted 5 lines
I am not sure that a snooping-based system will work and not be prohibitive in its hardware cost on an actual hardware system. I suspect it would decay into needing a large RAM array in every bridge.
-hpa
This might be more difficult if you allow arbitrary number of functions behind a bridge. However, this is not a problem for integrated functions (e.g. on the same silicon or board).
Bridge could specify which slots host integrated functions and do snooping for these, and have regular window rules apply to open slots.
Signature · 1 line
MST
Quoted 71 lines · Michael S. Tsirkin
OK we talked about this a while ago, here's
a summary and some proposals:
At the moment, virtio PCI uses IO BARs for all accesses.The reason for IO use is the cost of different VM exit types of transactions and their emulation on KVM on x86 (it would be trivial to use memory BARs on non x86 platforms
if they don't have PIO).
Example benchmark (cycles per transaction):(io access) outw 1737 (memory access) movw 4341for comparison:
(hypercall access): vmcall 1566
(pv memory access)movw_fast1817 (*explanation what this is below)This creates a problem if we want to make virtio devices proper PCI express devices with native hotplug support. This is because each hotpluggable PCI express device always has a PCI express port (port per device),
where each port is represented by a PCI to PCI bridge. In turn, a PCI to PCI bridge claims a 4Kbyte aligned range of IO addresses. This means that we can have at most 15 such devices, this is a nasty limitation.Another problem with PIO is support for physical virtio devices, and nested virt: KVM currently programs all PIO accesses to cause vm exit, so using this device in a VM will be slow.
So we really want to stop using IO BARs completely if at all possible, but looking at the table above, switching to memory BAR and movw for notifications will not work well.
Possible solutions:
1. hypercall instead of PIO
basically add a hypercall that gets an MMIO address/data and does an MMIO write for us.
We'll want some capability in the device to let guest know
this is what it should do.
Pros: even faster than PIO
Cons: this won't help nested or assigned devices (won't hurt
them either as it will be conditional on the capability above).
Cons: need host kernel support, which then has to be maintained
forever, even if intel speeds up MMIO exits.2. pv memory access
There are two reasons that memory access is slower:
- one is that it's handled as an EPT misconfiguration error so handled by cpu slow path
- one is that we need to decode the x86 instruction in software, to calculate address/data for the access.We could agree that guests would use a specific instruction for virtio accesses, and fast-path it specifically. This is the pv memory access option above.
Pros: helps assigned devices and nested virt
Pros: easy to drop if hardware support is there
Cons: a bit slower than IO
Cons: need host kernel support3. hypervisor assigned IO address
qemu can reserve IO addresses and assign to virtio devices. 2 bytes per device (for notification and ISR access) will be enough. So we can reserve 4K and this gets us 2000 devices.
From KVM perspective, nothing changes.
We'll want some capability in the device to let guest know this is what it should do, and pass the io address. One way to reserve the addresses is by using the bridge.
Pros: no need for host kernel support
Pros: regular PIO so fast
Cons: does not help assigned devices, breaks nested virt
Quoted 5 lines
Simply counting pros/cons, option 3 seems best. It's also the easiest to implement.
Comments?
apologies for late response...
It seems that solution 1 would be the best option for the following reasons:
a) (nearly?) every virt technology out there (xen, kvm, vmware, hyperv) has pv drivers in the major OS's using virt (Windows, Linux), so having a hypercall table searched, initialized and used for fast virtio register access is trivially simple to do.
b) the support can be added with whatever pvdriver set is provided w/o impacting OS core support.
c) it's architecture neutral, or can be made architecture neutral. e.g., inb/outb & PCI ioport support is very different btwn x86 & non-x86. A hypercall interface would not have that dependency/difference.
d) it doesn't require new OS support in std/core areas for
new standard(s), as another thread proposed; this kind of approach has a long, time delay to get defined & implemented across OS's. In contrast, a hypercall defined interface can be indep. of standards bodies, and if built into a pvdriver core, can change &/or adapt rapidly, and have additional i/f mechanisms for version-levels, which enables cross-hypervisor(version) migration.
e) the hypercall can be extended to do pv-specific hot add/remove, eliminating dependencies on emulation support of ACPI-hp or PCIe-hp, and simply(?) track core interfaces for hot-plug of (class) devices.
f) For migration, hypercall interfaces could be extended for better/faster migration as well (suspend/resume pv device).
my (late) 5 cents (I'll admit it was more than 2 cents)... Don
Quoted 6 lines · Don Dutile
c) it's architecture neutral, or can be made architecture neutral. e.g., inb/outb & PCI ioport support is very different btwn x86 &
non-x86.
A hypercall interface would not have that dependency/difference.
You are joking, right? Hypercalls are if anything *more* architecture and OS dependent.
-hpa