AlacrityVM guest drivers

This will generally be used for hypervisors to publish any host-side virtual devices up to a guest. The guest will have the opportunity to consume any devices present on the vbus-proxy as if they were platform devices, similar to existing buses like PCI.

Signed-off-by Gregory Haskins <[email protected]>

---

 MAINTAINERS                 |    6 ++
 arch/x86/Kconfig            |    2 +
 drivers/Makefile            |    1 
 drivers/vbus/Kconfig        |   14 ++++
 drivers/vbus/Makefile       |    3 +
 drivers/vbus/bus-proxy.c    |  152 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/vbus_driver.h |   73 +++++++++++++++++++++
 7 files changed, 251 insertions(+), 0 deletions(-)
 create mode 100644 drivers/vbus/Kconfig
 create mode 100644 drivers/vbus/Makefile
 create mode 100644 drivers/vbus/bus-proxy.c
 create mode 100644 include/linux/vbus_driver.h

Quoted 8 lines · Gregory Haskins

This will generally be used for hypervisors to publish any host-side virtual devices up to a guest. The guest will have the opportunity to consume any devices present on the vbus-proxy as if they were platform devices, similar to existing buses like PCI.

Signed-off-by Gregory Haskins <[email protected]>

---

Quoted 13 lines

 MAINTAINERS                 |    6 ++
 arch/x86/Kconfig            |    2 +
 drivers/Makefile            |    1 
 drivers/vbus/Kconfig        |   14 ++++
 drivers/vbus/Makefile       |    3 +
 drivers/vbus/bus-proxy.c    |  152 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/vbus_driver.h |   73 +++++++++++++++++++++
 7 files changed, 251 insertions(+), 0 deletions(-)
 create mode 100644 drivers/vbus/Kconfig
 create mode 100644 drivers/vbus/Makefile
 create mode 100644 drivers/vbus/bus-proxy.c
 create mode 100644 include/linux/vbus_driver.h

Is there a consensus on this with the KVM folks? (i've added the KVM list to the Cc:)

Ingo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 25 lines · Ingo Molnar

Quoted 8 lines · Gregory Haskins

This will generally be used for hypervisors to publish any host-side virtual devices up to a guest. The guest will have the opportunity to consume any devices present on the vbus-proxy as if they were platform devices, similar to existing buses like PCI.

Signed-off-by Gregory Haskins <[email protected]>

---

Quoted 13 lines

 MAINTAINERS                 |    6 ++
 arch/x86/Kconfig            |    2 +
 drivers/Makefile            |    1  drivers/vbus/Kconfig        |   
14 ++++
 drivers/vbus/Makefile       |    3 +
 drivers/vbus/bus-proxy.c    |  152 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/vbus_driver.h |   73 +++++++++++++++++++++
 7 files changed, 251 insertions(+), 0 deletions(-)
 create mode 100644 drivers/vbus/Kconfig
 create mode 100644 drivers/vbus/Makefile
 create mode 100644 drivers/vbus/bus-proxy.c
 create mode 100644 include/linux/vbus_driver.h

Quoted 2 lines

Is there a consensus on this with the KVM folks? (i've added the KVM list to the Cc:)

I'll let Avi comment about it from a KVM perspective but from a QEMU perspective, I don't think we want to support two paravirtual IO frameworks. I'd like to see them converge. Since there's an install base of guests today with virtio drivers, there really ought to be a compelling reason to change the virtio ABI in a non-backwards compatible way. This means convergence really ought to be adding features to virtio.

On paper, I don't think vbus really has any features over virtio. vbus does things in different ways (paravirtual bus vs. pci for discovery) but I think we're happy with how virtio does things today.

I think the reason vbus gets better performance for networking today is that vbus' backends are in the kernel while virtio's backends are currently in userspace. Since Michael has a functioning in-kernel backend for virtio-net now, I suspect we're weeks (maybe days) away from performance results. My expectation is that vhost + virtio-net will be as good as venet + vbus. If that's the case, then I don't see any reason to adopt vbus unless Greg things there are other compelling features over virtio.

Quoted 6 lines

This will generally be used for hypervisors to publish any host-side virtual devices up to a guest. The guest will have the opportunity to consume any devices present on the vbus-proxy as if they were platform devices, similar to existing buses like PCI.

Is there a consensus on this with the KVM folks? (i've added the KVM list to the Cc:)

Quoted 8 lines · Gregory Haskins

This will generally be used for hypervisors to publish any host-side virtual devices up to a guest. The guest will have the opportunity to consume any devices present on the vbus-proxy as if they were platform devices, similar to existing buses like PCI.

Signed-off-by Gregory Haskins <[email protected]>

---

Quoted 12 lines

 MAINTAINERS                 |    6 ++
 arch/x86/Kconfig            |    2 +
 drivers/Makefile            |    1 
 drivers/vbus/Kconfig        |   14 ++++
 drivers/vbus/Makefile       |    3 +
 drivers/vbus/bus-proxy.c    |  152 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/vbus_driver.h |   73 +++++++++++++++++++++
 7 files changed, 251 insertions(+), 0 deletions(-)
 create mode 100644 drivers/vbus/Kconfig
 create mode 100644 drivers/vbus/Makefile
 create mode 100644 drivers/vbus/bus-proxy.c
 create mode 100644 include/linux/vbus_driver.h

Is there a consensus on this with the KVM folks? (i've added the KVM list to the Cc:)

Quoted 2 lines

I think the reason vbus gets better performance for networking today is that vbus' backends are in the kernel while virtio's backends are currently in userspace. Since Michael has a functioning in-kernel backend for virtio-net now, I suspect we're weeks (maybe days) away from performance results. My expectation is that vhost + virtio-net will be as good as venet + vbus. If that's the case, then I don't see any reason to adopt vbus unless Greg things there are other compelling features over virtio.

Keeping virtio's backend in user-space was rather stupid IMHO.

Quoted 26 lines · Ingo Molnar

Quoted 8 lines · Gregory Haskins

This will generally be used for hypervisors to publish any host-side virtual devices up to a guest. The guest will have the opportunity to consume any devices present on the vbus-proxy as if they were platform devices, similar to existing buses like PCI.

Signed-off-by Gregory Haskins <[email protected]>

---

Quoted 14 lines

 MAINTAINERS                 |    6 ++
 arch/x86/Kconfig            |    2 +
 drivers/Makefile            |    1  drivers/vbus/Kconfig        |  
14 ++++
 drivers/vbus/Makefile       |    3 +
 drivers/vbus/bus-proxy.c    |  152
+++++++++++++++++++++++++++++++++++++++++++
 include/linux/vbus_driver.h |   73 +++++++++++++++++++++
 7 files changed, 251 insertions(+), 0 deletions(-)
 create mode 100644 drivers/vbus/Kconfig
 create mode 100644 drivers/vbus/Makefile
 create mode 100644 drivers/vbus/bus-proxy.c
 create mode 100644 include/linux/vbus_driver.h

Quoted 3 lines

Is there a consensus on this with the KVM folks? (i've added the KVM list to the Cc:)

I'll let Avi comment about it from a KVM perspective but from a QEMU perspective, I don't think we want to support two paravirtual IO frameworks. I'd like to see them converge. Since there's an install base of guests today with virtio drivers, there really ought to be a compelling reason to change the virtio ABI in a non-backwards compatible way.

This means convergence really ought to be adding features to virtio.

On paper, I don't think vbus really has any features over virtio.

vbus
does things in different ways (paravirtual bus vs. pci for discovery) but I think we're happy with how virtio does things today.

I think the reason vbus gets better performance for networking today is that vbus' backends are in the kernel while virtio's backends are currently in userspace.

Since Michael has a functioning in-kernel
backend for virtio-net now, I suspect we're weeks (maybe days) away from performance results. My expectation is that vhost + virtio-net will be as good as venet + vbus.

If that's the case, then I don't see any
reason to adopt vbus unless Greg things there are other compelling features over virtio.

Quoted 9 lines · Ingo Molnar

Quoted 6 lines

This will generally be used for hypervisors to publish any host-side virtual devices up to a guest. The guest will have the opportunity to consume any devices present on the vbus-proxy as if they were platform devices, similar to existing buses like PCI.

Is there a consensus on this with the KVM folks? (i've added the KVM list to the Cc:)

My opinion is that this is a duplication of effort and we'd be better off if everyone contributed to enhancing virtio, which already has widely deployed guest drivers and non-Linux guest support.

It may have merit if it is proven that it is technically superior to virtio (and I don't mean some benchmark in some point in time; I mean design wise). So far I haven't seen any indications that it is.

Quoted 10 lines · Ingo Molnar

Quoted 9 lines

I think the reason vbus gets better performance for networking today is that vbus' backends are in the kernel while virtio's backends are currently in userspace. Since Michael has a functioning in-kernel backend for virtio-net now, I suspect we're weeks (maybe days) away from performance results. My expectation is that vhost + virtio-net will be as good as venet + vbus. If that's the case, then I don't see any reason to adopt vbus unless Greg things there are other compelling features over virtio.

Quoted 1 line

Keeping virtio's backend in user-space was rather stupid IMHO.

I don't think it's quite so clear.

Quoted 3 lines

Having the _option_ to piggyback to user-space (for flexibility, extensibility, etc.) is OK, but not having kernel acceleration is bad.

Quoted 23 lines · Ingo Molnar

Quoted 8 lines · Gregory Haskins

This will generally be used for hypervisors to publish any host-side virtual devices up to a guest. The guest will have the opportunity to consume any devices present on the vbus-proxy as if they were platform devices, similar to existing buses like PCI.

Signed-off-by Gregory Haskins <[email protected]>

---

Quoted 12 lines

 MAINTAINERS                 |    6 ++
 arch/x86/Kconfig            |    2 +
 drivers/Makefile            |    1 
 drivers/vbus/Kconfig        |   14 ++++
 drivers/vbus/Makefile       |    3 +
 drivers/vbus/bus-proxy.c    |  152 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/vbus_driver.h |   73 +++++++++++++++++++++
 7 files changed, 251 insertions(+), 0 deletions(-)
 create mode 100644 drivers/vbus/Kconfig
 create mode 100644 drivers/vbus/Makefile
 create mode 100644 drivers/vbus/bus-proxy.c
 create mode 100644 include/linux/vbus_driver.h

Quoted 4 lines

Is there a consensus on this with the KVM folks? (i've added the KVM list to the Cc:)

Hi Ingo,

Avi can correct me if I am wrong, but the agreement that he and I came to a few months ago was something to the effect of:

kvm will be neutral towards various external IO subsystems, and instead provide various hooks (see irqfd, ioeventfd) to permit these IO subsystems to interface with kvm.

AlacrityVM is one of the first projects to take advantage of that interface. AlacrityVM is kvm-core + vbus-core + vbus-kvm-connector + vbus-enhanced qemu + guest drivers. This thread is part of the guest-drivers portion. Note that it is specific to alacrityvm, not kvm, which is why the kvm list was not included in the conversation (also an agreement with Avi:
http://lkml.org/lkml/2009/8/6/231).

Note: No one has ever proposed to change the virtio-ABI. In fact, this thread in question doesn't even touch virtio, and even the patches that I have previously posted to add virtio-capability do it in a backwards compatible way

Case in point: Take an upstream kernel and you can modprobe the vbus-pcibridge in and virtio devices will work over that transport unmodified.

Quoted 5 lines

vbus
does things in different ways (paravirtual bus vs. pci for discovery) but I think we're happy with how virtio does things today.

Thats fine. KVM can stick with virtio-pci if it wants. AlacrityVM will support virtio-pci and vbus (with possible convergence with virtio-vbus). If at some point KVM thinks vbus is interesting, I will gladly work with getting it integrated into upstream KVM as well. Until then, they can happily coexist without issue between the two projects.

Quoted 4 lines

I think the reason vbus gets better performance for networking today is that vbus' backends are in the kernel while virtio's backends are currently in userspace.

Well, with all due respect, you also said initially when I announced vbus that in-kernel doesn't matter, and tried to make virtio-net run as fast as venet from userspace ;) Given that we never saw those userspace patches from you that in fact equaled my performance, I assume you were wrong about that statement.

Perhaps you were wrong about other things too?

Quoted 5 lines

Since Michael has a functioning in-kernel
backend for virtio-net now, I suspect we're weeks (maybe days) away from performance results. My expectation is that vhost + virtio-net will be as good as venet + vbus.

This is not entirely impossible, at least for certain simple benchmarks like singleton throughput and latency.

But if you think that this
somehow invalidates vbus as a concept, you have missed the point entirely.

vbus is about creating a flexible (e.g. cross hypervisor, and even physical system or userspace application) in-kernel IO containers with linux. The "guest" interface represents what I believe to be the ideal interface for ease of use, yet maximum performance for software-to-software interaction.

venet was originally crafted just to validate the approach and test the vbus interface. It ended up being so much faster that virtio-net, that people in the vbus community started coding against its ABI.

OTOH, Michael's patch is purely targeted at improving virtio-net on kvm, and its likewise constrained by various limitations of that decision (such as its reliance of the PCI model, and the kvm memory scheme). The tradeoff is that his approach will work in all existing virtio-net kvm guests, and is probably significantly less code since he can re-use the qemu PCI bus model.

Conversely, I am not afraid of requiring a new driver to optimize the general PV interface. In the long term, this will reduce the amount of reimplementing the same code over and over, reduce system overhead, and it adds new features not previously available (for instance, coalescing and prioritizing interrupts).

Quoted 4 lines

If that's the case, then I don't see any
reason to adopt vbus unless Greg things there are other compelling features over virtio.

Aside from the fact that this is another confusion of the vbus/virtio relationship...yes, of course there are compelling features (IMHO) or I wouldn't be expending effort ;) They are at least compelling enough to put in AlacrityVM. If upstream KVM doesn't want them, that's KVMs decision and I am fine with that. Simply never apply my qemu patches to qemu-kvm.git, and KVM will be blissfully unaware if vbus is present. I do hope that I can convince the KVM community otherwise, however. :)

Quoted 4 lines

My opinion is that this is a duplication of effort and we'd be better off if everyone contributed to enhancing virtio, which already has widely deployed guest drivers and non-Linux guest support.

Quoted 5 lines

It may have merit if it is proven that it is technically superior to virtio (and I don't mean some benchmark in some point in time; I mean design wise). So far I haven't seen any indications that it is.

The design is very different, so hopefully I can start to convince you why it might be interesting.

Quoted 36 lines · Gregory Haskins

Quoted 22 lines · Ingo Molnar

Quoted 8 lines · Gregory Haskins

This will generally be used for hypervisors to publish any host-side virtual devices up to a guest. The guest will have the opportunity to consume any devices present on the vbus-proxy as if they were platform devices, similar to existing buses like PCI.

Signed-off-by Gregory Haskins <[email protected]>

---

Quoted 12 lines

 MAINTAINERS                 |    6 ++
 arch/x86/Kconfig            |    2 +
 drivers/Makefile            |    1 
 drivers/vbus/Kconfig        |   14 ++++
 drivers/vbus/Makefile       |    3 +
 drivers/vbus/bus-proxy.c    |  152 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/vbus_driver.h |   73 +++++++++++++++++++++
 7 files changed, 251 insertions(+), 0 deletions(-)
 create mode 100644 drivers/vbus/Kconfig
 create mode 100644 drivers/vbus/Makefile
 create mode 100644 drivers/vbus/bus-proxy.c
 create mode 100644 include/linux/vbus_driver.h

Quoted 4 lines

Is there a consensus on this with the KVM folks? (i've added the KVM list to the Cc:)

Hi Ingo,

Avi can correct me if I am wrong, but the agreement that he and I came to a few months ago was something to the effect of:

kvm will be neutral towards various external IO subsystems, and instead provide various hooks (see irqfd, ioeventfd) to permit these IO subsystems to interface with kvm.

Quoted 7 lines

AlacrityVM is one of the first projects to take advantage of that interface. AlacrityVM is kvm-core + vbus-core + vbus-kvm-connector + vbus-enhanced qemu + guest drivers. This thread is part of the guest-drivers portion. Note that it is specific to alacrityvm, not kvm, which is why the kvm list was not included in the conversation (also an agreement with Avi:
http://lkml.org/lkml/2009/8/6/231).

Well my own opinion is that the fracturing of the Linux internal driver space into diverging pieces of duplicate functionality (absent compelling technical reasons) is harmful.

I don't have any technical objections to vbus/venet (I had in the past re interrupts but I believe you've addressed them), and it appears to perform very well. However I still think we should address virtio's shortcomings (as Michael is doing) rather than create a competitor. We have enough external competition, we don't need in-tree competitors.

Hi Ingo,

1) First off, let me state that I have made every effort to propose this as a solution to integrate with KVM, the most recent of which is April:

   http://lkml.org/lkml/2009/4/21/408

If you read through the various vbus related threads on LKML/KVM posted this year, I think you will see that I made numerous polite offerings to work with people on finding a common solution here, including Michael.

In the end, Michael decided that go a different route using some of the ideas proposed in vbus + venet-tap to create vhost-net. This is fine, and I respect his decision. But do not try to pin "fracturing" on me, because I tried everything to avoid it. :)

Quoted 10 lines · Gregory Haskins

Quoted 9 lines

My opinion is that this is a duplication of effort and we'd be better off if everyone contributed to enhancing virtio, which already has widely deployed guest drivers and non-Linux guest support.

It may have merit if it is proven that it is technically superior to virtio (and I don't mean some benchmark in some point in time; I mean design wise). So far I haven't seen any indications that it is.

Quoted 3 lines

The design is very different, so hopefully I can start to convince you why it might be interesting.

We've been through this before I believe. If you can point out specific differences that make venet outperform virtio-net I'll be glad to hear (and steal) them though.

Hi Ingo,

1) First off, let me state that I have made every effort to propose this as a solution to integrate with KVM, the most recent of which is April:

http://lkml.org/lkml/2009/4/21/408

If you read through the various vbus related threads on LKML/KVM posted this year, I think you will see that I made numerous polite offerings to work with people on finding a common solution here, including Michael.

In the end, Michael decided that go a different route using some of the ideas proposed in vbus + venet-tap to create vhost-net. This is fine, and I respect his decision. But do not try to pin "fracturing" on me, because I tried everything to avoid it. :)

Since I still disagree with the fundamental approach of how KVM IO works,

Prior to my effort, KVM was humming along at the status quo and I came along with a closer eye and almost doubled the throughput and cut latency by 78%. Given an apparent disagreement with aspects of my approach, Michael went off and created a counter example that was motivated by my performance findings.

Therefore, even if Avi ultimately accepts Michaels vhost approach instead of mine, Linux as a hypervisor platform has been significantly _improved_ by a little friendly competition, not somehow damaged by it.

4) Lastly, these patches are almost entirely just stand alone Linux drivers that do not affect KVM if KVM doesn't wish to acknowledge them. Its just like any of the other numerous drivers that are accepted
upstream into Linux every day. The only maintained subsystem that is technically touched by this series is netdev, and David Miller already approved of the relevant patch's inclusion:

http://lkml.org/lkml/2009/8/3/505

So with all due respect, where is the problem? The patches are all professionally developed according to the Linux coding standards, pass checkpatch, are GPL'ed, and work with a freely available platform which you can download today
(http://git.kernel.org/?p=linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git;a=summary)

Quoted 15 lines · Avi Kivity

Quoted 13 lines · Gregory Haskins

Quoted 9 lines

My opinion is that this is a duplication of effort and we'd be better off if everyone contributed to enhancing virtio, which already has widely deployed guest drivers and non-Linux guest support.

It may have merit if it is proven that it is technically superior to virtio (and I don't mean some benchmark in some point in time; I mean design wise). So far I haven't seen any indications that it is.

The design is very different, so hopefully I can start to convince you why it might be interesting.

Quoted 3 lines

We've been through this before I believe. If you can point out specific differences that make venet outperform virtio-net I'll be glad to hear (and steal) them though.

You sure know how to convince someone to collaborate with you, eh?

Unforunately, i've answered that question numerous times, but it apparently falls on deaf ears.

Quoted 6 lines

We've been through this before I believe. If you can point out specific differences that make venet outperform virtio-net I'll be glad to hear (and steal) them though.

You sure know how to convince someone to collaborate with you, eh?

Unforunately, i've answered that question numerous times, but it apparently falls on deaf ears.

Case in point: Take an upstream kernel and you can modprobe the vbus-pcibridge in and virtio devices will work over that transport unmodified.

See http://lkml.org/lkml/2009/8/6/244 for details.

OTOH, Michael's patch is purely targeted at improving virtio-net on kvm, and its likewise constrained by various limitations of that decision (such as its reliance of the PCI model, and the kvm memory scheme).

Quoted 13 lines · Gregory Haskins

Hi Ingo,

1) First off, let me state that I have made every effort to propose this as a solution to integrate with KVM, the most recent of which is April:

   http://lkml.org/lkml/2009/4/21/408

If you read through the various vbus related threads on LKML/KVM posted this year, I think you will see that I made numerous polite offerings to work with people on finding a common solution here, including Michael.

Quoted 4 lines

In the end, Michael decided that go a different route using some of the ideas proposed in vbus + venet-tap to create vhost-net. This is fine, and I respect his decision. But do not try to pin "fracturing" on me, because I tried everything to avoid it. :)

That's good.

So if virtio is fixed to be as fast as vbus, and if there's no other techical advantages of vbus over virtio you'll be glad to drop vbus and stand behind virtio?

Also, are you willing to help virtio to become faster?

Or do you
have arguments why that is impossible to do so and why the only possible solution is vbus? Avi says no such arguments were offered so far.

Quoted 14 lines · Gregory Haskins

Quoted 13 lines · Avi Kivity

Quoted 12 lines · Gregory Haskins

Quoted 9 lines

My opinion is that this is a duplication of effort and we'd be better off if everyone contributed to enhancing virtio, which already has widely deployed guest drivers and non-Linux guest support.

It may have merit if it is proven that it is technically superior to virtio (and I don't mean some benchmark in some point in time; I mean design wise). So far I haven't seen any indications that it is.

The design is very different, so hopefully I can start to convince you why it might be interesting.

Quoted 7 lines

Quoted 3 lines

We've been through this before I believe. If you can point out specific differences that make venet outperform virtio-net I'll be glad to hear (and steal) them though.

You sure know how to convince someone to collaborate with you, eh?

Unforunately, i've answered that question numerous times, but it apparently falls on deaf ears.

I'm trying to find the relevant discussion. The link you gave in the previous mail:

http://lkml.org/lkml/2009/4/21/408

does not offer any design analysis of vbus versus virtio, and why the only fix to virtio is vbus. It offers a comparison and a blanket statement that vbus is superior but no arguments.

(If you've already explained in a past thread then please give me an URL to that reply if possible, or forward me that prior reply. Thanks!)

Quoted 5 lines · Gregory Haskins

Case in point: Take an upstream kernel and you can modprobe the vbus-pcibridge in and virtio devices will work over that transport unmodified.

See http://lkml.org/lkml/2009/8/6/244 for details.

The modprobe you are talking about would need
to be done in guest kernel, correct?

Quoted 4 lines

OTOH, Michael's patch is purely targeted at improving virtio-net on kvm, and its likewise constrained by various limitations of that decision (such as its reliance of the PCI model, and the kvm memory scheme).

vhost is actually not related to PCI in any way. It simply leaves all setup for userspace to do. And the memory scheme was intentionally separated from kvm so that it can easily support e.g. lguest.

Note: No one has ever proposed to change the virtio-ABI.

Quoted 4 lines

I think the reason vbus gets better performance for networking today is that vbus' backends are in the kernel while virtio's backends are currently in userspace.

Well, with all due respect, you also said initially when I announced vbus that in-kernel doesn't matter, and tried to make virtio-net run as fast as venet from userspace ;) Given that we never saw those userspace patches from you that in fact equaled my performance, I assume you were wrong about that statement. Perhaps you were wrong about other things too?

Conversely, I am not afraid of requiring a new driver to optimize the general PV interface. In the long term, this will reduce the amount of reimplementing the same code over and over, reduce system overhead, and it adds new features not previously available (for instance, coalescing and prioritizing interrupts).

Quoted 4 lines

If that's the case, then I don't see any
reason to adopt vbus unless Greg things there are other compelling features over virtio.

Aside from the fact that this is another confusion of the vbus/virtio relationship...yes, of course there are compelling features (IMHO) or I wouldn't be expending effort ;) They are at least compelling enough to put in AlacrityVM.

If upstream KVM doesn't want them, that's KVMs
decision and I am fine with that. Simply never apply my qemu patches to qemu-kvm.git, and KVM will be blissfully unaware if vbus is present.

I
do hope that I can convince the KVM community otherwise, however. :)

I believe strongly that we should avoid putting things in the kernel unless they absolutely have to be. I'm definitely interested in playing with vhost to see if there are ways to put even less in the kernel. In particular, I think it would be a big win to avoid knowledge of slots in the kernel by doing ring translation in userspace. This implies a userspace transition in the fast path. This may or may not be acceptable. I think this is going to be a very interesting experiment and will ultimately determine whether my intuition about the cost of dropping to userspace is right or wrong.

There is a secondary question of venet (a vbus native device) verses virtio-net (a virtio native device that works with PCI or VBUS). If this contention is really around venet vs virtio-net, I may possibly conceed and retract its submission to mainline. I've been pushing it to date because people are using it and I don't see any reason that the driver couldn't be upstream.

Quoted 2 lines

Also, are you willing to help virtio to become faster?

Yes, that is not a problem. Note that virtio in general, and virtio-net/venet in particular are not the primary goal here, however. Improved 802.x and block IO are just positive side-effects of the effort. I started with 802.x networking just to demonstrate the IO layer capabilities, and to test it. It ended up being so good on contrast to existing facilities, that developers in the vbus community started using it for production development.

Ultimately, I created vbus to address areas of performance that have not yet been addressed in things like KVM. Areas such as real-time guests, or RDMA (host bypass) interfaces.

I also designed it in such a way that
we could, in theory, write one set of (linux-based) backends, and have them work across a variety of environments (such as containers/VMs like KVM, lguest, openvz, but also physical systems like blade enclosures and clusters, or even applications running on the host).

Quoted 5 lines

Or do you
have arguments why that is impossible to do so and why the only possible solution is vbus? Avi says no such arguments were offered so far.

Not for lack of trying. I think my points have just been missed everytime I try to describe them. ;) Basically I write a message very similar to this one, and the next conversation starts back from square one. But I digress, let me try again..

Noting that this discussion is really about the layer *below* virtio, not virtio itself (e.g. PCI vs vbus). Lets start with a little background:

-- Background --

So on one level, we have the resource-container technology called "vbus". It lets you create a container on the host, fill it with virtual devices, and assign that container to some context (such as a KVM guest). These "devices" are LKMs, and each device has a very simple verb namespace consisting of a synchronous "call()" method, and a "shm()" method for establishing async channels.

The async channels are just shared-memory with a signal path (e.g. interrupts and hypercalls), which the device+driver can use to overlay things like rings (virtqueues, IOQs), or other shared-memory based constructs of their choosing (such as a shared table). The signal path is designed to minimize enter/exits and reduce spurious signals in a unified way (see shm-signal patch).

call() can be used both for config-space like details, as well as fast-path messaging that require synchronous behavior (such as guest scheduler updates).

All of this is managed via sysfs/configfs.

On the guest, we have a "vbus-proxy" which is how the guest gets access to devices assigned to its container. (as an aside, "virtio" devices can be populated in the container, and then surfaced up to the virtio-bus via that virtio-vbus patch I mentioned).

There is a thing called a "vbus-connector" which is the guest specific part. Its job is to connect the vbus-proxy in the guest, to the vbus container on the host. How it does its job is specific to the connector implementation, but its role is to transport messages between the guest and the host (such as for call() and shm() invocations) and to handle things like discovery and hotswap.

Out of all this, I think the biggest contention point is the design of the vbus-connector that I use in AlacrityVM (Avi, correct me if I am wrong and you object to other aspects as well). I suspect that if I had designed the vbus-connector to surface vbus devices as PCI devices via QEMU, the patches would potentially have been pulled in a while ago.

There are, of course, reasons why vbus does *not* render as PCI, so this is the meat of of your question, I believe.

At a high level, PCI was designed for software-to-hardware interaction, so it makes assumptions about that relationship that do not necessarily apply to virtualization.

For instance:

A) hardware can only generate byte/word sized requests at a time because that is all the pcb-etch and silicon support. So hardware is usually expressed in terms of some number of "registers".

C) the target end-point has no visibility into the CPU machine state other than the parameters passed in the bus-cycle (usually an address and data tuple).

D) device-ids are in a fixed width register and centrally assigned from an authority (e.g. PCI-SIG).

E) Interrupt/MSI routing is per-device oriented

F) Interrupts/MSI are assumed cheap to inject

G) Interrupts/MSI are non-priortizable.

H) Interrupts/MSI are statically established

These assumptions and constraints may be completely different or simply invalid in a virtualized guest. For instance, the hypervisor is just software, and therefore it's not restricted to "etch" constraints. IO requests can be arbitrarily large, just as if you are invoking a library function-call or OS system-call. Likewise, each one of those requests is a branch and a context switch, so it has often has greater performance implications than a simple register bus-cycle in hardware. If you use an MMIO variant, it has to run through the page-fault code to be decoded.

The result is typically decreased performance if you try to do the same thing real hardware does. This is why you usually see hypervisor specific drivers (e.g. virtio-net, vmnet, etc) a common feature.

_Some_ performance oriented items can technically be accomplished in PCI, albeit in a much more awkward way. For instance, you can set up a really fast, low-latency "call()" mechanism using a PIO port on a PCI-model and ioeventfd. As a matter of fact, this is exactly what the vbus pci-bridge does:

The problem here is that this is incredibly awkward to setup. You have all that per-cpu goo and the registration of the memory on the guest. And on the host side, you have all the vmapping of the registered memory, and the file-descriptor to manage. In short, its really painful.

I would much prefer to do this *once*, and then let all my devices simple re-use that infrastructure. This is, in fact, what I do. Here is the device model that a guest sees:

Moving on: _Other_ items cannot be replicated (at least, not without hacking it into something that is no longer PCI.

Things like the pci-id namespace are just silly for software. I would rather have a namespace that does not require central management so people are free to create vbus-backends at will. This is akin to registering a device MAJOR/MINOR, verses using the various dynamic assignment mechanisms. vbus uses a string identifier in place of a pci-id. This is superior IMHO, and not compatible with PCI.

As another example, the connector design coalesces *all* shm-signals into a single interrupt (by prio) that uses the same context-switch mitigation techniques that help boost things like networking. This effectively means we can detect and optimize out ack/eoi cycles from the APIC as the IO load increases (which is when you need it most). PCI has no such concept.

In addition, the signals and interrupts are priority aware, which is useful for things like 802.1p networking where you may establish 8-tx and 8-rx queues for your virtio-net device. x86 APIC really has no usable equivalent, so PCI is stuck here.

Also, the signals can be allocated on-demand for implementing things like IPC channels in response to guest requests since there is no assumption about device-to-interrupt mappings. This is more flexible.

And through all of this, this design would work in any guest even if it doesn't have PCI (e.g. lguest, UML, physical systems, etc).

-- Bottom Line --

The idea here is to generalize all the interesting parts that are common (fast sync+async io, context-switch mitigation, back-end models, memory abstractions, signal-path routing, etc) that a variety of linux based technologies can use (kvm, lguest, openvz, uml, physical systems) and only require the thin "connector" code to port the system around. The idea is to try to get this aspect of PV right once, and at some point in the future, perhaps vbus will be as ubiquitous as PCI. Well, perhaps not *that* ubiquitous, but you get the idea ;)

Then device models like virtio can ride happily on top and we end up with a really robust and high-performance Linux-based stack. I don't buy the argument that we already have PCI so lets use it. I don't think its the best design and I am not afraid to make an investment in a change here because I think it will pay off in the long run.

Quoted 7 lines · Michael S. Tsirkin

Quoted 5 lines · Gregory Haskins

Case in point: Take an upstream kernel and you can modprobe the vbus-pcibridge in and virtio devices will work over that transport unmodified.

See http://lkml.org/lkml/2009/8/6/244 for details.

Quoted 2 lines

The modprobe you are talking about would need
to be done in guest kernel, correct?

Yes, and your point is? "unmodified" (pardon the psuedo pun) modifies "virtio", not "guest".
It means you can take an off-the-shelf kernel
with off-the-shelf virtio (ala distro-kernel) and modprobe vbus-pcibridge and get alacrityvm acceleration.

It is not a design goal of mine to forbid the loading of a new driver, so I am ok with that requirement.

Quoted 4 lines

Quoted 3 lines

OTOH, Michael's patch is purely targeted at improving virtio-net on kvm, and its likewise constrained by various limitations of that decision (such as its reliance of the PCI model, and the kvm memory scheme).

Quoted 4 lines

vhost is actually not related to PCI in any way. It simply leaves all setup for userspace to do. And the memory scheme was intentionally separated from kvm so that it can easily support e.g. lguest.

I think you have missed my point. I mean that vhost requires a separate bus-model (ala qemu-pci).

And no, your memory scheme is not separated,
at least, not very well. It still assumes memory-regions and copy_to_user(), which is very kvm-esque.

Vbus has people using things
like userspace containers (no regions),

and physical hardware (dma
controllers, so no regions or copy_to_user) so your scheme quickly falls apart once you get away from KVM.

Don't get me wrong: That design may have its place. Perhaps you only care about fixing KVM, which is a perfectly acceptable strategy. Its just not a strategy that I think is the best approach. Essentially you are promoting the proliferation of competing backends, and I am trying to unify them (which is ironic that this thread started with concerns I was fragmenting things ;).

The bottom line is, you have a simpler solution that is more finely targeted at KVM and virtio-networking. It fixes probably a lot of problems with the existing implementation, but it still has limitations.

OTOH, what I am promoting is more complex, but more flexible. That is the tradeoff. You can't have both ;)

So do not for one second think
that what you implemented is equivalent, because they are not.

In fact, I believe I warned you about this potential problem when you decided to implement your own version. I think I said something to the effect of "you will either have a subset of functionality, or you will ultimately reinvent what I did". Right now you are in the subset phase.

Perhaps someday you will be in the complete-reinvent phase. Why you wanted to go that route when I had already worked though the issues is something perhaps only you will ever know, but I'm sure you had your reasons. But do note you could have saved yourself grief by reusing my already implemented and tested variant, as I politely offered to work with you on making it meet your needs.
Kind Regards
-Greg

In particular, I think it would be a big win to avoid knowledge of slots in the kernel by doing ring translation in userspace.

There is a secondary question of venet (a vbus native device) verses virtio-net (a virtio native device that works with PCI or VBUS). If this contention is really around venet vs virtio-net, I may possibly conceed and retract its submission to mainline.

I've been pushing it to date because people are using it and I don't see any reason that the driver couldn't be upstream.

-- Issues --

Out of all this, I think the biggest contention point is the design of the vbus-connector that I use in AlacrityVM (Avi, correct me if I am wrong and you object to other aspects as well). I suspect that if I had designed the vbus-connector to surface vbus devices as PCI devices via QEMU, the patches would potentially have been pulled in a while ago.

There are, of course, reasons why vbus does *not* render as PCI, so this is the meat of of your question, I believe.

At a high level, PCI was designed for software-to-hardware interaction, so it makes assumptions about that relationship that do not necessarily apply to virtualization.

As another example, the connector design coalesces *all* shm-signals into a single interrupt (by prio) that uses the same context-switch mitigation techniques that help boost things like networking. This effectively means we can detect and optimize out ack/eoi cycles from the APIC as the IO load increases (which is when you need it most). PCI has no such concept.

In addition, the signals and interrupts are priority aware, which is useful for things like 802.1p networking where you may establish 8-tx and 8-rx queues for your virtio-net device. x86 APIC really has no usable equivalent, so PCI is stuck here.

I'm not hung up on PCI, myself. An idea that might help you get Avi
on-board: do setup in userspace, over PCI. Negotiate hypercall support (e.g. with a PCI capability) and then switch to that for fastpath. Hmm?

Quoted 4 lines · Michael S. Tsirkin

I'm not hung up on PCI, myself. An idea that might help you get Avi
on-board: do setup in userspace, over PCI. Negotiate hypercall support (e.g. with a PCI capability) and then switch to that for fastpath. Hmm?

Hypercalls don't nest well. When a nested guest issues a hypercall, you have to assume it is destined to the enclosing guest, so you can't assign a hypercall-capable device to a nested guest.

mmio and pio don't have this problem since the host can use the address to locate the destination.

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 4 lines

mmio and pio don't have this problem since the host can use the address to locate the destination.

So userspace could map hypercall to address during setup and tell the host kernel?

Quoted 4 lines · Michael S. Tsirkin

Quoted 4 lines

mmio and pio don't have this problem since the host can use the address to locate the destination.

Quoted 3 lines

So userspace could map hypercall to address during setup and tell the host kernel?

Suppose a nested guest has two devices. One a virtual device backed by its host (our guest), and one a virtual device backed by us (the real host), and assigned by the guest to the nested guest. If both devices use hypercalls, there is no way to distinguish between them.

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 6 lines

Suppose a nested guest has two devices. One a virtual device backed by its host (our guest), and one a virtual device backed by us (the real host), and assigned by the guest to the nested guest. If both devices use hypercalls, there is no way to distinguish between them.

Not sure I understand. What I had in mind is that devices would have to either use different hypercalls and map hypercall to address during setup, or pass address with each hypercall. We get the hypercall, translate the address as if it was pio access, and know the destination?

Quoted 11 lines · Michael S. Tsirkin

Quoted 6 lines

Suppose a nested guest has two devices. One a virtual device backed by its host (our guest), and one a virtual device backed by us (the real host), and assigned by the guest to the nested guest. If both devices use hypercalls, there is no way to distinguish between them.

Not sure I understand. What I had in mind is that devices would have to either use different hypercalls and map hypercall to address during setup, or pass address with each hypercall. We get the hypercall, translate the address as if it was pio access, and know the destination?

There are no different hypercalls. There's just one hypercall instruction, and there's no standard on how it's used. If a nested call issues a hypercall instruction, you have no idea if it's calling a Hyper-V hypercall or a vbus/virtio kick.

You could have a protocol where you register the hypercall instruction's address with its recipient, but it quickly becomes a tangled mess.

And for what? pio and hypercalls have the same performance characteristics.

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 15 lines · Avi Kivity

Quoted 13 lines · Michael S. Tsirkin

Quoted 7 lines

Suppose a nested guest has two devices. One a virtual device backed by its host (our guest), and one a virtual device backed by us (the real host), and assigned by the guest to the nested guest. If both devices use hypercalls, there is no way to distinguish between them.

Not sure I understand. What I had in mind is that devices would have to either use different hypercalls and map hypercall to address during setup, or pass address with each hypercall. We get the hypercall, translate the address as if it was pio access, and know the destination?

Quoted 5 lines

There are no different hypercalls. There's just one hypercall instruction, and there's no standard on how it's used. If a nested call issues a hypercall instruction, you have no idea if it's calling a Hyper-V hypercall or a vbus/virtio kick.

userspace will know which it is, because hypercall capability in the device has been activated, and can tell kernel, using something similar to iosignalfd. No?

Quoted 3 lines

You could have a protocol where you register the hypercall instruction's address with its recipient, but it quickly becomes a tangled mess.

I really thought we could pass the io address in register as an input parameter. Is there a way to do this in a secure manner?

Hmm. Doesn't kvm use hypercalls now? How does this work with nesting? For example, in this code in arch/x86/kvm/x86.c:

         switch (nr) {
         case KVM_HC_VAPIC_POLL_IRQ:
                 ret = 0;
                 break;
         case KVM_HC_MMU_OP:
                 r = kvm_pv_mmu_op(vcpu, a0, hc_gpa(vcpu, a1, a2),&ret);
                 break;
         default:
                 ret = -KVM_ENOSYS;
                 break;
         }

how do we know that it's the guest and not the nested guest performing the hypercall?

Quoted 22 lines · Michael S. Tsirkin

Quoted 20 lines · Avi Kivity

Quoted 7 lines · Michael S. Tsirkin

Quoted 7 lines

Suppose a nested guest has two devices. One a virtual device backed by its host (our guest), and one a virtual device backed by us (the real host), and assigned by the guest to the nested guest. If both devices use hypercalls, there is no way to distinguish between them.

Quoted 6 lines

Not sure I understand. What I had in mind is that devices would have to either use different hypercalls and map hypercall to address during setup, or pass address with each hypercall. We get the hypercall, translate the address as if it was pio access, and know the destination?

There are no different hypercalls. There's just one hypercall instruction, and there's no standard on how it's used. If a nested call issues a hypercall instruction, you have no idea if it's calling a Hyper-V hypercall or a vbus/virtio kick.

Quoted 4 lines

userspace will know which it is, because hypercall capability in the device has been activated, and can tell kernel, using something similar to iosignalfd. No?

The host kernel sees a hypercall vmexit. How does it know if it's a nested-guest-to-guest hypercall or a nested-guest-to-host hypercall? The two are equally valid at the same time.

Quoted 21 lines

Quoted 3 lines

You could have a protocol where you register the hypercall instruction's address with its recipient, but it quickly becomes a tangled mess.

I really thought we could pass the io address in register as an input parameter. Is there a way to do this in a secure manner?

Hmm. Doesn't kvm use hypercalls now? How does this work with nesting? For example, in this code in arch/x86/kvm/x86.c:

         switch (nr) {
         case KVM_HC_VAPIC_POLL_IRQ:
                 ret = 0;
                 break;
         case KVM_HC_MMU_OP:
                 r = kvm_pv_mmu_op(vcpu, a0, hc_gpa(vcpu, a1, a2),&ret);
                 break;
         default:
                 ret = -KVM_ENOSYS;
                 break;
         }

Quoted 3 lines

how do we know that it's the guest and not the nested guest performing the hypercall?

The host knows whether the guest or nested guest are running. If the guest is running, it's a guest-to-host hypercall. If the nested guest is running, it's a nested-guest-to-guest hypercall. We don't have nested-guest-to-host hypercalls (and couldn't unless we get agreement on a protocol from all hypervisor vendors).

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 5 lines

The host kernel sees a hypercall vmexit. How does it know if it's a nested-guest-to-guest hypercall or a nested-guest-to-host hypercall? The two are equally valid at the same time.

Here is how this can work - it is similar to MSI if you like:
- by default, the device uses pio kicks
- nested guest driver can enable hypercall capability in the device, probably with pci config cycle
- guest userspace (hypervisor running in guest) will see this request and perform pci config cycle on the "real" device, telling it to which nested guest this device is assigned

- host userspace (hypervisor running in host) will see this. it now knows both which guest the hypercalls will be for, and that the device in question is an emulated one, and can set up kvm appropriately

Quoted 6 lines

The host knows whether the guest or nested guest are running. If the guest is running, it's a guest-to-host hypercall. If the nested guest is running, it's a nested-guest-to-guest hypercall. We don't have nested-guest-to-host hypercalls (and couldn't unless we get agreement on a protocol from all hypervisor vendors).

Not necessarily. What I am saying is we could make this protocol part of guest paravirt driver. the guest that loads the driver and enables the capability, has to agree to the protocol. If it doesn't want to, it does not have to use that driver.

Quoted 1 line · Gregory Haskins

Note: No one has ever proposed to change the virtio-ABI.

virtio-pci is part of the virtio ABI. You are proposing changing that.

You cannot add new kernel modules to guests and expect them to remain supported.

So there is value in reusing existing ABIs

Quoted 6 lines

Quoted 5 lines

I think the reason vbus gets better performance for networking today is that vbus' backends are in the kernel while virtio's backends are currently in userspace.

Quoted 7 lines

Well, with all due respect, you also said initially when I announced vbus that in-kernel doesn't matter, and tried to make virtio-net run as fast as venet from userspace ;) Given that we never saw those userspace patches from you that in fact equaled my performance, I assume you were wrong about that statement. Perhaps you were wrong about other things too?

I'm wrong about a lot of things :-) I haven't yet been convinced that I'm wrong here though.

One of the gray areas here is what constitutes an in-kernel backend. tun/tap is a sort of an in-kernel backend. Userspace is still involved in all of the paths. vhost seems to be an intermediate step between tun/tap and vbus. The fast paths avoid userspace completely. Many of the slow paths involve userspace still (like migration apparently). With vbus, userspace is avoided entirely. In some ways, you could argue that slirp and vbus are opposite ends of the virtual I/O spectrum.

I believe strongly that we should avoid putting things in the kernel unless they absolutely have to be.

I'm definitely interested in playing
with vhost to see if there are ways to put even less in the kernel. In particular, I think it would be a big win to avoid knowledge of slots in the kernel by doing ring translation in userspace.

This implies a
userspace transition in the fast path. This may or may not be acceptable. I think this is going to be a very interesting experiment and will ultimately determine whether my intuition about the cost of dropping to userspace is right or wrong.

Quoted 8 lines

Conversely, I am not afraid of requiring a new driver to optimize the general PV interface. In the long term, this will reduce the amount of reimplementing the same code over and over, reduce system overhead, and it adds new features not previously available (for instance, coalescing and prioritizing interrupts).

I think you have a lot of ideas and I don't know that we've been able to really understand your vision. Do you have any plans on writing a paper about vbus that goes into some of your thoughts in detail?

Quoted 6 lines

Quoted 5 lines

If that's the case, then I don't see any
reason to adopt vbus unless Greg things there are other compelling features over virtio.

Quoted 4 lines

Aside from the fact that this is another confusion of the vbus/virtio relationship...yes, of course there are compelling features (IMHO) or I wouldn't be expending effort ;) They are at least compelling enough to put in AlacrityVM.

This whole AlactricyVM thing is really hitting this nail with a sledgehammer.

pulls in, as long as you're willing to commit to ABI compatibility, we can pull code into QEMU to support vbus. Then you can just offer vbus host and guest drivers instead of forking the kernel.

Quoted 4 lines

If upstream KVM doesn't want them, that's KVMs
decision and I am fine with that. Simply never apply my qemu patches to qemu-kvm.git, and KVM will be blissfully unaware if vbus is present.

As I mentioned before, if you submit patches to upstream QEMU, we'll apply them (after appropriate review). As I said previously, we want to avoid user confusion as much as possible. Maybe this means limiting it to -device or a separate machine type. I'm not sure, but that's something we can discussion on qemu-devel.

The issue here is that vbus is designed to be a generic solution to in-kernel virtual-IO. It will support (via abstraction of key subsystems) a variety of environments that may or may not be similar in facilities to KVM, and therefore it represents the least-common-denominator as far as what external dependencies it requires.

The bottom line is this: despite the tendency for people to jump at "don't put much in the kernel!", the fact is that a "bus" designed for software to software (such as vbus) is almost laughably trivial. Its essentially a list of objects that have an int (dev-id) and char* (dev-type) attribute. All the extra goo that you see me setting up in something like the kvm-connector needs to be done for fast-path _anyway_, so transporting the verbs to query this simple list is not really a big deal.

Note that I didn't really want to go that route. As you know, I tried pushing this straight through kvm first since earlier this year, but I was met with reluctance to even bother truly understanding what I was proposing, comments like "tell me your ideas so I can steal them", and

"sorry, we are going to reinvent our own instead".

This isn't exactly
going to motivate someone to continue pushing these ideas within that community. I was made to feel (purposely?) unwelcome at times. So I can either roll over and die, or start my own project.

Quoted 8 lines · Gregory Haskins

There is a secondary question of venet (a vbus native device) verses virtio-net (a virtio native device that works with PCI or VBUS). If this contention is really around venet vs virtio-net, I may possibly conceed and retract its submission to mainline. I've been pushing it to date because people are using it and I don't see any reason that the driver couldn't be upstream.

That's probably the cause of much confusion. The primary kvm pain point is now networking, so in any vbus discussion we're concentrating on that aspect.

Quoted 10 lines

Quoted 2 lines

Also, are you willing to help virtio to become faster?

Yes, that is not a problem. Note that virtio in general, and virtio-net/venet in particular are not the primary goal here, however. Improved 802.x and block IO are just positive side-effects of the effort. I started with 802.x networking just to demonstrate the IO layer capabilities, and to test it. It ended up being so good on contrast to existing facilities, that developers in the vbus community started using it for production development.

Quoted 3 lines

Ultimately, I created vbus to address areas of performance that have not yet been addressed in things like KVM. Areas such as real-time guests, or RDMA (host bypass) interfaces.

Can you explain how vbus achieves RDMA?

I also don't see the connection to real time guests.

Quoted 7 lines

I also designed it in such a way that
we could, in theory, write one set of (linux-based) backends, and have them work across a variety of environments (such as containers/VMs like KVM, lguest, openvz, but also physical systems like blade enclosures and clusters, or even applications running on the host).

Sorry, I'm still confused. Why would openvz need vbus?

It already has
zero-copy networking since it's a shared kernel. Shared memory should also work seamlessly, you just need to expose the shared memory object on a shared part of the namespace. And of course, anything in the kernel is already shared.

Quoted 34 lines

Quoted 5 lines

Or do you
have arguments why that is impossible to do so and why the only possible solution is vbus? Avi says no such arguments were offered so far.

Not for lack of trying. I think my points have just been missed everytime I try to describe them. ;) Basically I write a message very similar to this one, and the next conversation starts back from square one. But I digress, let me try again..

Noting that this discussion is really about the layer *below* virtio, not virtio itself (e.g. PCI vs vbus). Lets start with a little background:

-- Background --

So on one level, we have the resource-container technology called "vbus". It lets you create a container on the host, fill it with virtual devices, and assign that container to some context (such as a KVM guest). These "devices" are LKMs, and each device has a very simple verb namespace consisting of a synchronous "call()" method, and a "shm()" method for establishing async channels.

The async channels are just shared-memory with a signal path (e.g. interrupts and hypercalls), which the device+driver can use to overlay things like rings (virtqueues, IOQs), or other shared-memory based constructs of their choosing (such as a shared table). The signal path is designed to minimize enter/exits and reduce spurious signals in a unified way (see shm-signal patch).

call() can be used both for config-space like details, as well as fast-path messaging that require synchronous behavior (such as guest scheduler updates).

Quoted 2 lines

All of this is managed via sysfs/configfs.

One point of contention is that this is all managementy stuff and should be kept out of the host kernel. Exposing shared memory, interrupts, and guest hypercalls can all be easily done from userspace (as virtio demonstrates). True, some devices need kernel acceleration, but that's no reason to put everything into the host kernel.

Quoted 6 lines

On the guest, we have a "vbus-proxy" which is how the guest gets access to devices assigned to its container. (as an aside, "virtio" devices can be populated in the container, and then surfaced up to the virtio-bus via that virtio-vbus patch I mentioned).

Quoted 7 lines

There is a thing called a "vbus-connector" which is the guest specific part. Its job is to connect the vbus-proxy in the guest, to the vbus container on the host. How it does its job is specific to the connector implementation, but its role is to transport messages between the guest and the host (such as for call() and shm() invocations) and to handle things like discovery and hotswap.

virtio has an exact parallel here (virtio-pci and friends).

Quoted 6 lines

Out of all this, I think the biggest contention point is the design of the vbus-connector that I use in AlacrityVM (Avi, correct me if I am wrong and you object to other aspects as well). I suspect that if I had designed the vbus-connector to surface vbus devices as PCI devices via QEMU, the patches would potentially have been pulled in a while ago.

Exposing devices as PCI is an important issue for me, as I have to consider non-Linux guests.

Another issue is the host kernel management code which I believe is superfluous.

But the biggest issue is compatibility. virtio exists and has Windows and Linux drivers. Without a fatal flaw in virtio we'll continue to support it.

Given that, why spread to a new model?

Of course, I understand you're interested in non-ethernet, non-block devices. I can't comment on these until I see them. Maybe they can fit the virtio model, and maybe they can't.

Quoted 8 lines

There are, of course, reasons why vbus does *not* render as PCI, so this is the meat of of your question, I believe.

At a high level, PCI was designed for software-to-hardware interaction, so it makes assumptions about that relationship that do not necessarily apply to virtualization.

Quoted 6 lines

For instance:

A) hardware can only generate byte/word sized requests at a time because that is all the pcb-etch and silicon support. So hardware is usually expressed in terms of some number of "registers".

No, hardware happily DMAs to and fro main memory.

Some hardware of
course uses mmio registers extensively, but not virtio hardware. With the recent MSI support no registers are touched in the fast path.

Quoted 5 lines

C) the target end-point has no visibility into the CPU machine state other than the parameters passed in the bus-cycle (usually an address and data tuple).

That's not an issue. Accessing memory is cheap.

Quoted 3 lines

D) device-ids are in a fixed width register and centrally assigned from an authority (e.g. PCI-SIG).

That's not an issue either. Qumranet/Red Hat has donated a range of device IDs for use in virtio.

Device IDs are how devices are associated
with drivers, so you'll need something similar for vbus.

Quoted 3 lines

E) Interrupt/MSI routing is per-device oriented

Please elaborate. What is the issue? How does vbus solve it?

Quoted 3 lines

F) Interrupts/MSI are assumed cheap to inject

Interrupts are not assumed cheap; that's why interrupt mitigation is used (on real and virtual hardware).

Quoted 3 lines

G) Interrupts/MSI are non-priortizable.

They are prioritizable; Linux ignores this though (Windows doesn't). Please elaborate on what the problem is and how vbus solves it.

Quoted 3 lines

H) Interrupts/MSI are statically established

Can you give an example of why this is a problem?

Quoted 14 lines

These assumptions and constraints may be completely different or simply invalid in a virtualized guest. For instance, the hypervisor is just software, and therefore it's not restricted to "etch" constraints. IO requests can be arbitrarily large, just as if you are invoking a library function-call or OS system-call. Likewise, each one of those requests is a branch and a context switch, so it has often has greater performance implications than a simple register bus-cycle in hardware. If you use an MMIO variant, it has to run through the page-fault code to be decoded.

The result is typically decreased performance if you try to do the same thing real hardware does. This is why you usually see hypervisor specific drivers (e.g. virtio-net, vmnet, etc) a common feature.

Quoted 6 lines

_Some_ performance oriented items can technically be accomplished in PCI, albeit in a much more awkward way. For instance, you can set up a really fast, low-latency "call()" mechanism using a PIO port on a PCI-model and ioeventfd. As a matter of fact, this is exactly what the vbus pci-bridge does:

What performance oriented items have been left unaddressed?

virtio and vbus use three communications channels: call from guest to host (implemented as pio and reasonably fast), call from host to guest (implemented as msi and reasonably fast) and shared memory (as fast as it can be). Where does PCI limit you in any way?

Quoted 9 lines

The problem here is that this is incredibly awkward to setup. You have all that per-cpu goo and the registration of the memory on the guest. And on the host side, you have all the vmapping of the registered memory, and the file-descriptor to manage. In short, its really painful.

I would much prefer to do this *once*, and then let all my devices simple re-use that infrastructure. This is, in fact, what I do. Here is the device model that a guest sees:

virtio also reuses the pci code, on both guest and host.

Quoted 10 lines

Moving on: _Other_ items cannot be replicated (at least, not without hacking it into something that is no longer PCI.

Things like the pci-id namespace are just silly for software. I would rather have a namespace that does not require central management so people are free to create vbus-backends at will. This is akin to registering a device MAJOR/MINOR, verses using the various dynamic assignment mechanisms. vbus uses a string identifier in place of a pci-id. This is superior IMHO, and not compatible with PCI.

How do you handle conflicts? Again you need a central authority to hand out names or prefixes.

Quoted 8 lines

As another example, the connector design coalesces *all* shm-signals into a single interrupt (by prio) that uses the same context-switch mitigation techniques that help boost things like networking. This effectively means we can detect and optimize out ack/eoi cycles from the APIC as the IO load increases (which is when you need it most). PCI has no such concept.

That's a bug, not a feature. It means poor scaling as the number of vcpus increases and as the number of devices increases.

Note nothing prevents steering multiple MSIs into a single vector. It's a bad idea though.

Quoted 6 lines

In addition, the signals and interrupts are priority aware, which is useful for things like 802.1p networking where you may establish 8-tx and 8-rx queues for your virtio-net device. x86 APIC really has no usable equivalent, so PCI is stuck here.

x86 APIC is priority aware.

Quoted 5 lines

Also, the signals can be allocated on-demand for implementing things like IPC channels in response to guest requests since there is no assumption about device-to-interrupt mappings. This is more flexible.

Yes. However given that vectors are a scarce resource you're severely limited in that.

And if you're multiplexing everything on one vector,
then you can just as well demultiplex your channels in the virtio driver code.

Quoted 4 lines

And through all of this, this design would work in any guest even if it doesn't have PCI (e.g. lguest, UML, physical systems, etc).

That is true for virtio which works on pci-less lguest and s390.

Quoted 12 lines

-- Bottom Line --

The idea here is to generalize all the interesting parts that are common (fast sync+async io, context-switch mitigation, back-end models, memory abstractions, signal-path routing, etc) that a variety of linux based technologies can use (kvm, lguest, openvz, uml, physical systems) and only require the thin "connector" code to port the system around. The idea is to try to get this aspect of PV right once, and at some point in the future, perhaps vbus will be as ubiquitous as PCI. Well, perhaps not *that* ubiquitous, but you get the idea ;)

That is exactly the design goal of virtio (except it limits itself to virtualization).

Quoted 7 lines

Then device models like virtio can ride happily on top and we end up with a really robust and high-performance Linux-based stack. I don't buy the argument that we already have PCI so lets use it. I don't think its the best design and I am not afraid to make an investment in a change here because I think it will pay off in the long run.

Sorry, I don't think you've shown any quantifiable advantages.

Quoted 9 lines · Gregory Haskins

Quoted 8 lines · Michael S. Tsirkin

Quoted 5 lines · Gregory Haskins

Case in point: Take an upstream kernel and you can modprobe the vbus-pcibridge in and virtio devices will work over that transport unmodified.

See http://lkml.org/lkml/2009/8/6/244 for details.

The modprobe you are talking about would need
to be done in guest kernel, correct?

Quoted 5 lines

Yes, and your point is? "unmodified" (pardon the psuedo pun) modifies "virtio", not "guest".
It means you can take an off-the-shelf kernel
with off-the-shelf virtio (ala distro-kernel) and modprobe vbus-pcibridge and get alacrityvm acceleration.

Heh, by that logic ksplice does not modify running kernel either :)

Quoted 7 lines

It is not a design goal of mine to forbid the loading of a new driver, so I am ok with that requirement.

Quoted 3 lines

Quoted 3 lines

OTOH, Michael's patch is purely targeted at improving virtio-net on kvm, and its likewise constrained by various limitations of that decision (such as its reliance of the PCI model, and the kvm memory scheme).

Quoted 6 lines

Quoted 4 lines

vhost is actually not related to PCI in any way. It simply leaves all setup for userspace to do. And the memory scheme was intentionally separated from kvm so that it can easily support e.g. lguest.

I think you have missed my point. I mean that vhost requires a separate bus-model (ala qemu-pci).

So? That can be in userspace, and can be anything including vbus.

Quoted 4 lines

And no, your memory scheme is not separated,
at least, not very well. It still assumes memory-regions and copy_to_user(), which is very kvm-esque.

I don't think so: works for lguest, kvm, UML and containers

Quoted 3 lines

Vbus has people using things
like userspace containers (no regions),

vhost by default works without regions

Quoted 4 lines

and physical hardware (dma
controllers, so no regions or copy_to_user) so your scheme quickly falls apart once you get away from KVM.

Someone took a driver and is building hardware for it ... so what?

Quoted 7 lines

Don't get me wrong: That design may have its place. Perhaps you only care about fixing KVM, which is a perfectly acceptable strategy. Its just not a strategy that I think is the best approach. Essentially you are promoting the proliferation of competing backends, and I am trying to unify them (which is ironic that this thread started with concerns I was fragmenting things ;).

So, you don't see how venet fragments things? It's pretty obvious ...

Quoted 7 lines

The bottom line is, you have a simpler solution that is more finely targeted at KVM and virtio-networking. It fixes probably a lot of problems with the existing implementation, but it still has limitations.

OTOH, what I am promoting is more complex, but more flexible. That is the tradeoff. You can't have both ;)

We can. connect eventfds to hypercalls, and vhost will work with vbus.

Quoted 8 lines

So do not for one second think
that what you implemented is equivalent, because they are not.

In fact, I believe I warned you about this potential problem when you decided to implement your own version. I think I said something to the effect of "you will either have a subset of functionality, or you will ultimately reinvent what I did". Right now you are in the subset phase.

No. Unlike vbus, vhost supports unmodified guests and live migration.

Quoted 4 lines · Gregory Haskins

There is a secondary question of venet (a vbus native device) verses virtio-net (a virtio native device that works with PCI or VBUS). If this contention is really around venet vs virtio-net, I may possibly conceed and retract its submission to mainline.

For me yes, venet+ioq competing with virtio+virtqueue.

Quoted 2 lines

I've been pushing it to date because people are using it and I don't see any reason that the driver couldn't be upstream.

If virtio is just as fast, they can just use it without knowing it. Clearly, that's better since we support virtio anyway ...

Quoted 12 lines

-- Issues --

Out of all this, I think the biggest contention point is the design of the vbus-connector that I use in AlacrityVM (Avi, correct me if I am wrong and you object to other aspects as well). I suspect that if I had designed the vbus-connector to surface vbus devices as PCI devices via QEMU, the patches would potentially have been pulled in a while ago.

There are, of course, reasons why vbus does *not* render as PCI, so this is the meat of of your question, I believe.

Quoted 3 lines

At a high level, PCI was designed for software-to-hardware interaction, so it makes assumptions about that relationship that do not necessarily apply to virtualization.

I'm not hung up on PCI, myself. An idea that might help you get Avi
on-board: do setup in userspace, over PCI.

Negotiate hypercall support
(e.g. with a PCI capability) and then switch to that for fastpath. Hmm?

Quoted 6 lines

As another example, the connector design coalesces *all* shm-signals into a single interrupt (by prio) that uses the same context-switch mitigation techniques that help boost things like networking. This effectively means we can detect and optimize out ack/eoi cycles from the APIC as the IO load increases (which is when you need it most). PCI has no such concept.

Could you elaborate on this one for me? How does context-switch mitigation work?

Quoted 5 lines

In addition, the signals and interrupts are priority aware, which is useful for things like 802.1p networking where you may establish 8-tx and 8-rx queues for your virtio-net device. x86 APIC really has no usable equivalent, so PCI is stuck here.

By the way, multiqueue support in virtio would be very nice to have,

and seems mostly unrelated to vbus.

Quoted 7 lines · Gregory Haskins

The issue here is that vbus is designed to be a generic solution to in-kernel virtual-IO. It will support (via abstraction of key subsystems) a variety of environments that may or may not be similar in facilities to KVM, and therefore it represents the least-common-denominator as far as what external dependencies it requires.

Maybe it will be easier to evaluate it in the context of these other environments. It's difficult to assess this without an example.

Quoted 10 lines

The bottom line is this: despite the tendency for people to jump at "don't put much in the kernel!", the fact is that a "bus" designed for software to software (such as vbus) is almost laughably trivial. Its essentially a list of objects that have an int (dev-id) and char* (dev-type) attribute. All the extra goo that you see me setting up in something like the kvm-connector needs to be done for fast-path _anyway_, so transporting the verbs to query this simple list is not really a big deal.

It's not laughably trivial when you try to support the full feature set of kvm (for example, live migration will require dirty memory tracking, and exporting all state stored in the kernel to userspace).

Quoted 6 lines

Note that I didn't really want to go that route. As you know, I tried pushing this straight through kvm first since earlier this year, but I was met with reluctance to even bother truly understanding what I was proposing, comments like "tell me your ideas so I can steal them", and

Oh come on, I wrote "steal" as a convenient shorthand for "cross-pollinate your ideas into our code according to the letter and spirit of the GNU General Public License".

Since we're all trying to improve Linux we may as well cooperate.

Quoted 2 lines

"sorry, we are going to reinvent our own instead".

No. Adopting venet/vbus would mean reinventing something that already existed.

Continuing to support virtio/pci is not reinventing anything.

Quoted 6 lines

This isn't exactly
going to motivate someone to continue pushing these ideas within that community. I was made to feel (purposely?) unwelcome at times. So I can either roll over and die, or start my own project.

You haven't convinced me that your ideas are worth the effort of abandoning virtio/pci or maintaining both venet/vbus and virtio/pci.

I'm sorry if that made you feel unwelcome. There's no reason to interpret disagreement as malice though.

Quoted 8 lines · Gregory Haskins

Quoted 7 lines · Michael S. Tsirkin

Quoted 5 lines · Gregory Haskins

Case in point: Take an upstream kernel and you can modprobe the vbus-pcibridge in and virtio devices will work over that transport unmodified.

See http://lkml.org/lkml/2009/8/6/244 for details.

Quoted 8 lines

Quoted 2 lines

The modprobe you are talking about would need
to be done in guest kernel, correct?

Yes, and your point is? "unmodified" (pardon the psuedo pun) modifies "virtio", not "guest".
It means you can take an off-the-shelf kernel
with off-the-shelf virtio (ala distro-kernel) and modprobe vbus-pcibridge and get alacrityvm acceleration.

Heh, by that logic ksplice does not modify running kernel either :)

Quoted 7 lines

It is not a design goal of mine to forbid the loading of a new driver, so I am ok with that requirement.

Quoted 4 lines

Quoted 3 lines

OTOH, Michael's patch is purely targeted at improving virtio-net on kvm, and its likewise constrained by various limitations of that decision (such as its reliance of the PCI model, and the kvm memory scheme).

Quoted 7 lines

Quoted 4 lines

vhost is actually not related to PCI in any way. It simply leaves all setup for userspace to do. And the memory scheme was intentionally separated from kvm so that it can easily support e.g. lguest.

I think you have missed my point. I mean that vhost requires a separate bus-model (ala qemu-pci).

So? That can be in userspace, and can be anything including vbus.

Quoted 3 lines

And no, your memory scheme is not separated,
at least, not very well. It still assumes memory-regions and copy_to_user(), which is very kvm-esque.

I don't think so: works for lguest, kvm, UML and containers

Quoted 2 lines

Vbus has people using things
like userspace containers (no regions),

vhost by default works without regions

Quoted 3 lines

and physical hardware (dma
controllers, so no regions or copy_to_user) so your scheme quickly falls apart once you get away from KVM.

Someone took a driver and is building hardware for it ... so what?

Quoted 3 lines

You haven't convinced me that your ideas are worth the effort of abandoning virtio/pci or maintaining both venet/vbus and virtio/pci.

With all due respect, I didnt ask you do to anything, especially not abandon something you are happy with.

All I did was push guest drivers to LKML. The code in question is independent of KVM, and its proven to improve the experience of using Linux as a platform. There are people interested in using them (by virtue of the number of people that have signed up for the AlacrityVM list, and have mailed me privately about this work).

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

OTOH, Michael's patch is purely targeted at improving virtio-net on kvm, and its likewise constrained by various limitations of that decision (such as its reliance of the PCI model, and the kvm memory scheme).

vhost is actually not related to PCI in any way. It simply leaves all setup for userspace to do. And the memory scheme was intentionally separated from kvm so that it can easily support e.g. lguest.

Quoted 4 lines

Quoted 2 lines

I think you have missed my point. I mean that vhost requires a separate bus-model (ala qemu-pci).

So? That can be in userspace, and can be anything including vbus.

-ENOPARSE

Can you elaborate?

Quoted 6 lines

Quoted 4 lines

And no, your memory scheme is not separated,
at least, not very well. It still assumes memory-regions and copy_to_user(), which is very kvm-esque.

I don't think so: works for lguest, kvm, UML and containers

kvm _esque_ , meaning anything that follows the region+copy_to_user model. Not all things do.

Quoted 4 lines

Quoted 2 lines

Vbus has people using things
like userspace containers (no regions),

vhost by default works without regions

Thats a start, but not good enough if you were trying to achieve the same thing as vbus. As I said before, I've never said you had to achieve the same thing, but do note they are distinctly different with different goals. You are solving a directed problem. I am solving a general problem, and trying to solve it once.

Quoted 5 lines

Quoted 3 lines

and physical hardware (dma
controllers, so no regions or copy_to_user) so your scheme quickly falls apart once you get away from KVM.

Someone took a driver and is building hardware for it ... so what?

What is your point?

Quoted 8 lines

Quoted 6 lines

Don't get me wrong: That design may have its place. Perhaps you only care about fixing KVM, which is a perfectly acceptable strategy. Its just not a strategy that I think is the best approach. Essentially you are promoting the proliferation of competing backends, and I am trying to unify them (which is ironic that this thread started with concerns I was fragmenting things ;).

So, you don't see how venet fragments things? It's pretty obvious ...

I never said it doesn't. venet started as a test harness, but now it is inadvertently fragmenting the virtio-net effort. I admit it. It wasn't intentional, but just worked out that way. Until your vhost idea is vetted and benchmarked, its not even in the running.

Venet is currently
the highest performing 802.x acceleration for KVM that I am aware of, so it will continue to garner interest from users concerned with performance.

But likewise, vhost has the potential to fragment the back-end model. That was my point.

Quoted 8 lines

Quoted 7 lines

The bottom line is, you have a simpler solution that is more finely targeted at KVM and virtio-networking. It fixes probably a lot of problems with the existing implementation, but it still has limitations.

OTOH, what I am promoting is more complex, but more flexible. That is the tradeoff. You can't have both ;)

Quoted 1 line

We can. connect eventfds to hypercalls, and vhost will work with vbus.

-ENOPARSE

vbus doesnt use hypercalls, and I do not see why or how you would connect two backend models together like this. Can you elaborate.

Quoted 10 lines

Quoted 8 lines

So do not for one second think
that what you implemented is equivalent, because they are not.

In fact, I believe I warned you about this potential problem when you decided to implement your own version. I think I said something to the effect of "you will either have a subset of functionality, or you will ultimately reinvent what I did". Right now you are in the subset phase.

No. Unlike vbus, vhost supports unmodified guests and live migration.

By "subset", I am referring to your interfaces and the scope of its applicability. The things you need to do to make vhost work and a vbus device work from a memory and signaling abstration POV are going to be extremely similar.

The difference in how the guest sees them these backends is all contained in the vbus-connector. Therefore, what you *could* have done is simply written a connector that does something like only support "virtio" backends, and surfaced them as regular PCI devices to the guest. Then you could have reused all the abstraction features in vbus, instead of reinventing them (case in point, your region+copy_to_user code). And likewise, anyone using vbus could use your virtio-net backend.

Instead, I am still left with no virtio-net backend implemented, and you were left designing, writing, and testing facilities that I've already completed. So it was duplicative effort.

Kind Regards,
-Greg

Quoted 5 lines

Can you explain how vbus achieves RDMA?

I also don't see the connection to real time guests.

Both of these are still in development. Trying to stay true to the "release early and often" mantra, the core vbus technology is being pushed now so it can be reviewed. Stay tuned for these other developments.

Quoted 9 lines

Quoted 7 lines

I also designed it in such a way that
we could, in theory, write one set of (linux-based) backends, and have them work across a variety of environments (such as containers/VMs like KVM, lguest, openvz, but also physical systems like blade enclosures and clusters, or even applications running on the host).

Sorry, I'm still confused. Why would openvz need vbus?

Its just an example. The point is that I abstracted what I think are the key points of fast-io, memory routing, signal routing, etc, so that it will work in a variety of (ideally, _any_) environments.

There may not be _performance_ motivations for certain classes of VMs because they already have decent support, but they may want a connector anyway to gain some of the new features available in vbus.

And looking forward, the idea is that we have commoditized the backend so we don't need to redo this each time a new container comes along.

Quoted 6 lines

One point of contention is that this is all managementy stuff and should be kept out of the host kernel. Exposing shared memory, interrupts, and guest hypercalls can all be easily done from userspace (as virtio demonstrates). True, some devices need kernel acceleration, but that's no reason to put everything into the host kernel.

See my last reply to Anthony. My two points here are that:

a) having it in-kernel makes it a complete subsystem, which perhaps has diminished value in kvm, but adds value in most other places that we are looking to use vbus.

b) the in-kernel code is being overstated as "complex". We are not talking about your typical virt thing, like an emulated ICH/PCI chipset. Its really a simple list of devices with a handful of attributes. They
are managed using established linux interfaces, like sysfs/configfs.

Quoted 3 lines

Exposing devices as PCI is an important issue for me, as I have to consider non-Linux guests.

Thats your prerogative, but obviously not everyone agrees with you.

Getting non-Linux guests to work is my problem if you chose to not be part of the vbus community.

Quoted 3 lines

Another issue is the host kernel management code which I believe is superfluous.

In your opinion, right?

Quoted 2 lines

Given that, why spread to a new model?

Note: I haven't asked you to (at least, not since April with the vbus-v3 release). Spreading to a new model is currently the role of the AlacrityVM project, since we disagree on the utility of a new model.

Quoted 7 lines

Quoted 5 lines

A) hardware can only generate byte/word sized requests at a time because that is all the pcb-etch and silicon support. So hardware is usually expressed in terms of some number of "registers".

No, hardware happily DMAs to and fro main memory.

Yes, now walk me through how you set up DMA to do something like a call when you do not know addresses apriori. Hint: count the number of MMIO/PIOs you need. If the number is> 1, you've lost.

Quoted 4 lines

Some hardware of
course uses mmio registers extensively, but not virtio hardware. With the recent MSI support no registers are touched in the fast path.

Note we are not talking about virtio here. Just raw PCI and why I advocate vbus over it.

Quoted 4 lines

Quoted 4 lines

D) device-ids are in a fixed width register and centrally assigned from an authority (e.g. PCI-SIG).

Quoted 3 lines

That's not an issue either. Qumranet/Red Hat has donated a range of device IDs for use in virtio.

Yes, and to get one you have to do what? Register it with kvm.git, right? Kind of like registering a MAJOR/MINOR, would you agree? Maybe you do not mind (especially given your relationship to kvm.git), but there are disadvantages to that model for most of the rest of us.

Quoted 3 lines

Device IDs are how devices are associated
with drivers, so you'll need something similar for vbus.

Nope, just like you don't need to do anything ahead of time for using a dynamic misc-device name. You just have both the driver and device know what they are looking for (its part of the ABI).

Quoted 5 lines

Quoted 3 lines

E) Interrupt/MSI routing is per-device oriented

Please elaborate. What is the issue? How does vbus solve it?

There are no "interrupts" in vbus..only shm-signals. You can establish an arbitrary amount of shm regions, each with an optional shm-signal associated with it. To do this, the driver calls dev->shm(), and you get back a shm_signal object.

Underneath the hood, the vbus-connector (e.g. vbus-pcibridge) decides how it maps real interrupts to shm-signals (on a system level, not per device). This can be 1:1, or any other scheme. vbus-pcibridge uses one system-wide interrupt per priority level (today this is 8 levels), each with an IOQ based event channel. "signals" come as an event on that channel.

So the "issue" is that you have no real choice with PCI. You just get device oriented interrupts. With vbus, its abstracted. So you can still get per-device standard MSI, or you can do fancier things like do coalescing and prioritization.

Quoted 6 lines

Quoted 3 lines

F) Interrupts/MSI are assumed cheap to inject

Interrupts are not assumed cheap; that's why interrupt mitigation is used (on real and virtual hardware).

Its all relative. IDT dispatch and EOI overhead are "baseline" on real hardware, whereas they are significantly more expensive to do the vmenters and vmexits on virt (and you have new exit causes, like irq-windows, etc, that do not exist in real HW).

Quoted 6 lines

Quoted 3 lines

G) Interrupts/MSI are non-priortizable.

They are prioritizable; Linux ignores this though (Windows doesn't). Please elaborate on what the problem is and how vbus solves it.

It doesn't work right. The x86 sense of interrupt priority is, sorry to say it, half-assed at best. I've worked with embedded systems that have real interrupt priority support in the hardware, end to end, including the PIC. The LAPIC on the other hand is really weak in this dept, and as you said, Linux doesn't even attempt to use whats there.

Quoted 5 lines

Quoted 3 lines

H) Interrupts/MSI are statically established

Can you give an example of why this is a problem?

Some of the things we are building use the model of having a device that hands out shm-signal in response to guest events (say, the creation of an IPC channel). This would generally be handled by a specific device model instance, and it would need to do this without pre-declaring the MSI vectors (to use PCI as an example).

Quoted 2 lines

What performance oriented items have been left unaddressed?

Well, the interrupt model to name one.

Quoted 3 lines

How do you handle conflicts? Again you need a central authority to hand out names or prefixes.

Not really, no. If you really wanted to be formal about it, you could adopt any series of UUID schemes. For instance, perhaps venet should be "com.novell::virtual-ethernet". Heck, I could use uuidgen.

Quoted 8 lines

Quoted 8 lines

As another example, the connector design coalesces *all* shm-signals into a single interrupt (by prio) that uses the same context-switch mitigation techniques that help boost things like networking. This effectively means we can detect and optimize out ack/eoi cycles from the APIC as the IO load increases (which is when you need it most). PCI has no such concept.

Quoted 3 lines

That's a bug, not a feature. It means poor scaling as the number of vcpus increases and as the number of devices increases.

So the "avi-vbus-connector" can use 1:1, if you prefer. Large vcpu counts (which are not typical) and irq-affinity is not a target application for my design, so I prefer the coalescing model in the vbus-pcibridge included in this series. YMMV

Quoted 3 lines

Note nothing prevents steering multiple MSIs into a single vector. It's a bad idea though.

Yes, it is a bad idea...and not the same thing either. This would effectively create a shared-line scenario in the irq code, which is not what happens in vbus.

Quoted 8 lines

Quoted 6 lines

In addition, the signals and interrupts are priority aware, which is useful for things like 802.1p networking where you may establish 8-tx and 8-rx queues for your virtio-net device. x86 APIC really has no usable equivalent, so PCI is stuck here.

x86 APIC is priority aware.

Have you ever tried to use it?

Quoted 8 lines

Quoted 5 lines

Also, the signals can be allocated on-demand for implementing things like IPC channels in response to guest requests since there is no assumption about device-to-interrupt mappings. This is more flexible.

Yes. However given that vectors are a scarce resource you're severely limited in that.

The connector I am pushing out does not have this limitation.

Quoted 5 lines

And if you're multiplexing everything on one vector,
then you can just as well demultiplex your channels in the virtio driver code.

Only per-device, not system wide.

Quoted 6 lines

Quoted 4 lines

And through all of this, this design would work in any guest even if it doesn't have PCI (e.g. lguest, UML, physical systems, etc).

That is true for virtio which works on pci-less lguest and s390.

Yes, and lguest and s390 had to build their own bus-model to do it, right?

Thank you for bringing this up, because it is one of the main points here. What I am trying to do is generalize the bus to prevent the proliferation of more of these isolated models in the future. Build one, fast, in-kernel model so that we wouldn't need virtio-X, and virtio-Y in the future. They can just reuse the (performance optimized) bus and models, and only need to build the connector to bridge them.

Quoted 3 lines

That is exactly the design goal of virtio (except it limits itself to virtualization).

No, virtio is only part of the picture. It not including the backend models, or how to do memory/signal-path abstraction for in-kernel, for instance. But otherwise, virtio as a device model is compatible with vbus as a bus model. They compliment one another.

Quoted 7 lines

Quoted 7 lines

Then device models like virtio can ride happily on top and we end up with a really robust and high-performance Linux-based stack. I don't buy the argument that we already have PCI so lets use it. I don't think its the best design and I am not afraid to make an investment in a change here because I think it will pay off in the long run.

Quoted 2 lines

Sorry, I don't think you've shown any quantifiable advantages.

We can agree to disagree then, eh? There are certainly quantifiable differences. Waving your hand at the differences to say they are not advantages is merely an opinion, one that is not shared universally.

The bottom line is all of these design distinctions are encapsulated within the vbus subsystem and do not affect the kvm code-base. So agreement with kvm upstream is not a requirement, but would be advantageous for collaboration.

Quoted 6 lines · Michael S. Tsirkin

Quoted 4 lines · Gregory Haskins

There is a secondary question of venet (a vbus native device) verses virtio-net (a virtio native device that works with PCI or VBUS). If this contention is really around venet vs virtio-net, I may possibly conceed and retract its submission to mainline.

Quoted 7 lines

For me yes, venet+ioq competing with virtio+virtqueue.

Quoted 2 lines

I've been pushing it to date because people are using it and I don't see any reason that the driver couldn't be upstream.

If virtio is just as fast, they can just use it without knowing it. Clearly, that's better since we support virtio anyway ...

More specifically: kvm can support whatever it wants. I am not asking kvm to support venet.

If we (the alacrityvm community) decide to keep maintaining venet, _we_ will support it, and I have no problem with that.

As of right now, we are doing some interesting things with it in the lab and its certainly more flexible for us as a platform since we maintain the ABI and feature set. So for now, I do not think its a big deal if they both co-exist, and it has no bearing on KVM upstream.

Quoted 5 lines

It's not laughably trivial when you try to support the full feature set of kvm (for example, live migration will require dirty memory tracking, and exporting all state stored in the kernel to userspace).

Doesn't vhost suffer from the same issue? If not, could I also apply the same technique to support live-migration in vbus?

Quoted 4 lines

Oh come on, I wrote "steal" as a convenient shorthand for "cross-pollinate your ideas into our code according to the letter and spirit of the GNU General Public License".

Is that supposed to make me feel better about working with you? I mean, writing, testing, polishing patches for LKML-type submission is time consuming. If all you are going to do is take those ideas and rewrite it yourself, why should I go through that effort?

And its not like that was the first time you have said that to me.

Quoted 2 lines

Since we're all trying to improve Linux we may as well cooperate.

Well, I don't think anyone can say that I haven't been trying.

Quoted 5 lines

Quoted 2 lines

"sorry, we are going to reinvent our own instead".

No. Adopting venet/vbus would mean reinventing something that already existed.

But yet, it doesn't.

Quoted 2 lines

Continuing to support virtio/pci is not reinventing anything.

No one asked you to do otherwise.

Quoted 3 lines

You haven't convinced me that your ideas are worth the effort of abandoning virtio/pci or maintaining both venet/vbus and virtio/pci.

With all due respect, I didnt ask you do to anything, especially not abandon something you are happy with.

All I did was push guest drivers to LKML. The code in question is independent of KVM, and its proven to improve the experience of using Linux as a platform. There are people interested in using them (by virtue of the number of people that have signed up for the AlacrityVM list, and have mailed me privately about this work).

So where is the problem here?

So, in my system, copy_(to|from)_user() is completely wrong. There is no userspace, only a physical system. In fact, because normal x86 computers do not have DMA controllers, the host system doesn't actually handle any data transfer!

I used virtio-net in both the guest and host systems in my example virtio-over-PCI patch, and succeeded in getting them to communicate. However, the lack of any setup interface means that the devices must be hardcoded into both drivers, when the decision could be up to userspace. I think this is a problem that vbus could solve.

Quoted 3 lines

It's not laughably trivial when you try to support the full feature set of kvm (for example, live migration will require dirty memory tracking, and exporting all state stored in the kernel to userspace).

Doesn't vhost suffer from the same issue? If not, could I also apply the same technique to support live-migration in vbus?

With all due respect, I didnt ask you do to anything, especially not abandon something you are happy with.

All I did was push guest drivers to LKML. The code in question is independent of KVM, and its proven to improve the experience of using Linux as a platform. There are people interested in using them (by virtue of the number of people that have signed up for the AlacrityVM list, and have mailed me privately about this work).

So where is the problem here?

Quoted 5 lines · Ira W. Snyder

So, in my system, copy_(to|from)_user() is completely wrong. There is no userspace, only a physical system. In fact, because normal x86 computers do not have DMA controllers, the host system doesn't actually handle any data transfer!

In fact, modern x86s do have dma engines these days (google for Intel I/OAT), and one of our plans for vhost-net is to allow their use for packets above a certain size. So a patch allowing vhost-net to optionally use a dma engine is a good thing.

Quoted 6 lines

I used virtio-net in both the guest and host systems in my example virtio-over-PCI patch, and succeeded in getting them to communicate. However, the lack of any setup interface means that the devices must be hardcoded into both drivers, when the decision could be up to userspace. I think this is a problem that vbus could solve.

Exposing a knob to userspace is not an insurmountable problem; vhost-net already allows changing the memory layout, for example.

Quoted 5 lines

In fact, modern x86s do have dma engines these days (google for Intel I/OAT), and one of our plans for vhost-net is to allow their use for packets above a certain size. So a patch allowing vhost-net to optionally use a dma engine is a good thing.

Yes, I'm aware that very modern x86 PCs have general purpose DMA engines, even though I don't have any capable hardware. However, I think it is better to support using any PC (with or without DMA engine, any architecture) as the PCI master, and just handle the DMA all from the PCI agent, which is known to have DMA?

Quoted 4 lines

Exposing a knob to userspace is not an insurmountable problem; vhost-net already allows changing the memory layout, for example.

Let me explain the most obvious problem I ran into: setting the MAC addresses used in virtio.

On the host (PCI master), I want eth0 (virtio-net) to get a random MAC address.

On the guest (PCI agent), I want eth0 (virtio-net) to get a specific MAC address, aa:bb:cc:dd:ee:ff.

The virtio feature negotiation code handles this, by seeing the VIRTIO_NET_F_MAC feature in it's configuration space. If BOTH drivers do not have VIRTIO_NET_F_MAC set, then NEITHER will use the specified MAC address. This is because the feature negotiation code only accepts a feature if it is offered by both sides of the connection.

In this case, I must have the guest generate a random MAC address and have the host put aa:bb:cc:dd:ee:ff into the guest's configuration space. This basically means hardcoding the MAC addresses in the Linux drivers, which is a big no-no.

What would I expose to userspace to make this situation manageable?

Quoted 7 lines · Avi Kivity

On 08/17/2009 10:33 PM, Gregory Haskins wrote:

One point of contention is that this is all managementy stuff and should be kept out of the host kernel. Exposing shared memory, interrupts, and guest hypercalls can all be easily done from userspace (as virtio demonstrates). True, some devices need kernel acceleration, but that's no reason to put everything into the host kernel.

See my last reply to Anthony. My two points here are that:

a) having it in-kernel makes it a complete subsystem, which perhaps has diminished value in kvm, but adds value in most other places that we are looking to use vbus.

b) the in-kernel code is being overstated as "complex". We are not talking about your typical virt thing, like an emulated ICH/PCI chipset. Its really a simple list of devices with a handful of attributes. They
are managed using established linux interfaces, like sysfs/configfs.

Quoted 1 line

What performance oriented items have been left unaddressed?

Well, the interrupt model to name one.

Quoted 11 lines · Ira W. Snyder

Quoted 5 lines

In fact, modern x86s do have dma engines these days (google for Intel I/OAT), and one of our plans for vhost-net is to allow their use for packets above a certain size. So a patch allowing vhost-net to optionally use a dma engine is a good thing.

Yes, I'm aware that very modern x86 PCs have general purpose DMA engines, even though I don't have any capable hardware. However, I think it is better to support using any PC (with or without DMA engine, any architecture) as the PCI master, and just handle the DMA all from the PCI agent, which is known to have DMA?

Certainly; but if your PCI agent will support the DMA API, then the same vhost code will work with both I/OAT and your specialized hardware.

Quoted 19 lines

Quoted 4 lines

Exposing a knob to userspace is not an insurmountable problem; vhost-net already allows changing the memory layout, for example.

Let me explain the most obvious problem I ran into: setting the MAC addresses used in virtio.

On the host (PCI master), I want eth0 (virtio-net) to get a random MAC address.

On the guest (PCI agent), I want eth0 (virtio-net) to get a specific MAC address, aa:bb:cc:dd:ee:ff.

The virtio feature negotiation code handles this, by seeing the VIRTIO_NET_F_MAC feature in it's configuration space. If BOTH drivers do not have VIRTIO_NET_F_MAC set, then NEITHER will use the specified MAC address. This is because the feature negotiation code only accepts a feature if it is offered by both sides of the connection.

Quoted 8 lines

In this case, I must have the guest generate a random MAC address and have the host put aa:bb:cc:dd:ee:ff into the guest's configuration space. This basically means hardcoding the MAC addresses in the Linux drivers, which is a big no-no.

What would I expose to userspace to make this situation manageable?

I think in this case you want one side to be virtio-net (I'm guessing the x86) and the other side vhost-net (the ppc boards with the dma engine). virtio-net on x86 would communicate with userspace on the ppc board to negotiate features and get a mac address, the fast path would be between virtio-net and vhost-net (which would use the dma engine to push and pull data).

Quoted 8 lines

I think in this case you want one side to be virtio-net (I'm guessing the x86) and the other side vhost-net (the ppc boards with the dma engine). virtio-net on x86 would communicate with userspace on the ppc board to negotiate features and get a mac address, the fast path would be between virtio-net and vhost-net (which would use the dma engine to push and pull data).

Ah, that seems backwards, but it should work after vhost-net learns how to use the DMAEngine API.

I haven't studied vhost-net very carefully yet. As soon as I saw the copy_(to|from)_user() I stopped reading, because it seemed useless for my case. I'll look again and try to find where vhost-net supports setting MAC addresses and other features.

Also, in my case I'd like to boot Linux with my rootfs over NFS. Is vhost-net capable of this?

I've had Arnd, BenH, and Grant Likely (and others, privately) contact me about devices they are working with that would benefit from something like virtio-over-PCI. I'd like to see vhost-net be merged with the capability to support my use case. There are plenty of others that would benefit, not just myself.

I'm not sure vhost-net is being written with this kind of future use in mind. I'd hate to see it get merged, and then have to change the ABI to support physical-device-to-device usage. It would be better to keep future use in mind now, rather than try and hack it in later.

Quoted 2 lines

Well, the interrupt model to name one.

The performance aspects of your interrupt model are independent of the vbus proxy, or at least they should be. Let's assume for now that your event notification mechanism gives significant performance improvements (which we can't measure independently right now). I don't see a reason why we could not get the same performance out of a paravirtual interrupt controller that uses the same method, and it would be straightforward to implement one and use that together with all the existing emulated PCI devices and virtio devices including vhost_net.

I haven't studied vhost-net very carefully yet. As soon as I saw the copy_(to|from)_user() I stopped reading, because it seemed useless for my case. I'll look again and try to find where vhost-net supports setting MAC addresses and other features.

Also, in my case I'd like to boot Linux with my rootfs over NFS. Is vhost-net capable of this?

I've had Arnd, BenH, and Grant Likely (and others, privately) contact me about devices they are working with that would benefit from something like virtio-over-PCI. I'd like to see vhost-net be merged with the capability to support my use case. There are plenty of others that would benefit, not just myself.

I'm not sure vhost-net is being written with this kind of future use in mind. I'd hate to see it get merged, and then have to change the ABI to support physical-device-to-device usage. It would be better to keep future use in mind now, rather than try and hack it in later.

Thanks for the comments.
Ira

Quoted 7 lines · Avi Kivity

Quoted 5 lines · Ira W. Snyder

So, in my system, copy_(to|from)_user() is completely wrong. There is no userspace, only a physical system. In fact, because normal x86 computers do not have DMA controllers, the host system doesn't actually handle any data transfer!

Quoted 5 lines

In fact, modern x86s do have dma engines these days (google for Intel I/OAT), and one of our plans for vhost-net is to allow their use for packets above a certain size. So a patch allowing vhost-net to optionally use a dma engine is a good thing.

Yes, I'm aware that very modern x86 PCs have general purpose DMA engines, even though I don't have any capable hardware. However, I think it is better to support using any PC (with or without DMA engine, any architecture) as the PCI master, and just handle the DMA all from the PCI agent, which is known to have DMA?

Quoted 10 lines

Quoted 6 lines

I used virtio-net in both the guest and host systems in my example virtio-over-PCI patch, and succeeded in getting them to communicate. However, the lack of any setup interface means that the devices must be hardcoded into both drivers, when the decision could be up to userspace. I think this is a problem that vbus could solve.

Exposing a knob to userspace is not an insurmountable problem; vhost-net already allows changing the memory layout, for example.

Let me explain the most obvious problem I ran into: setting the MAC addresses used in virtio.

On the host (PCI master), I want eth0 (virtio-net) to get a random MAC address.

On the guest (PCI agent), I want eth0 (virtio-net) to get a specific MAC address, aa:bb:cc:dd:ee:ff.

The virtio feature negotiation code handles this, by seeing the VIRTIO_NET_F_MAC feature in it's configuration space. If BOTH drivers do not have VIRTIO_NET_F_MAC set, then NEITHER will use the specified MAC address. This is because the feature negotiation code only accepts a feature if it is offered by both sides of the connection.

In this case, I must have the guest generate a random MAC address and have the host put aa:bb:cc:dd:ee:ff into the guest's configuration space. This basically means hardcoding the MAC addresses in the Linux drivers, which is a big no-no.

What would I expose to userspace to make this situation manageable?

Thanks for the response,
Ira

I think Greg is referring to something like my virtio-over-PCI patch. I'm pretty sure that vhost is completely useless for my situation. I'd like to see vhost work for my use, so I'll try to explain what I'm doing.

I've got a system where I have about 20 computers connected via PCI. The PCI master is a normal x86 system, and the PCI agents are PowerPC systems. The PCI agents act just like any other PCI card, except they are running Linux, and have their own RAM and peripherals.

I wrote a custom driver which imitated a network interface and a serial port. I tried to push it towards mainline, and DavidM rejected it, with the argument, "use virtio, don't add another virtualization layer to the kernel." I think he has a decent argument, so I wrote virtio-over-PCI.

Now, there are some things about virtio that don't work over PCI. Mainly, memory is not truly shared. It is extremely slow to access memory that is "far away", meaning "across the PCI bus." This can be worked around by using a DMA controller to transfer all data, along with an intelligent scheme to perform only writes across the bus. If you're careful, reads are never needed.

So, in my system, copy_(to|from)_user() is completely wrong. There is no userspace, only a physical system.

In fact, because normal x86 computers
do not have DMA controllers, the host system doesn't actually handle any data transfer!

I used virtio-net in both the guest and host systems in my example virtio-over-PCI patch, and succeeded in getting them to communicate. However, the lack of any setup interface means that the devices must be hardcoded into both drivers, when the decision could be up to userspace. I think this is a problem that vbus could solve.

For my own selfish reasons (I don't want to maintain an out-of-tree driver) I'd like to see *something* useful in mainline Linux. I'm happy to answer questions about my setup, just ask.

Ira

Quoted 12 lines · Ira W. Snyder

Quoted 8 lines

I think in this case you want one side to be virtio-net (I'm guessing the x86) and the other side vhost-net (the ppc boards with the dma engine). virtio-net on x86 would communicate with userspace on the ppc board to negotiate features and get a mac address, the fast path would be between virtio-net and vhost-net (which would use the dma engine to push and pull data).

Ah, that seems backwards, but it should work after vhost-net learns how to use the DMAEngine API.

Quoted 5 lines

I haven't studied vhost-net very carefully yet. As soon as I saw the copy_(to|from)_user() I stopped reading, because it seemed useless for my case. I'll look again and try to find where vhost-net supports setting MAC addresses and other features.

It doesn't; all it does is pump the rings, leaving everything else to userspace.

Quoted 3 lines

Also, in my case I'd like to boot Linux with my rootfs over NFS. Is vhost-net capable of this?

It's just another network interface. You'd need an initramfs though to contain the needed userspace.

Quoted 6 lines

I've had Arnd, BenH, and Grant Likely (and others, privately) contact me about devices they are working with that would benefit from something like virtio-over-PCI. I'd like to see vhost-net be merged with the capability to support my use case. There are plenty of others that would benefit, not just myself.

Quoted 5 lines

I'm not sure vhost-net is being written with this kind of future use in mind. I'd hate to see it get merged, and then have to change the ABI to support physical-device-to-device usage. It would be better to keep future use in mind now, rather than try and hack it in later.

Please review and comment then. I'm fairly confident there won't be any ABI issues since vhost-net does so little outside pumping the rings.

Note the signalling paths go through eventfd: when vhost-net wants the other side to look at its ring, it tickles an eventfd which is supposed to trigger an interrupt on the other side. Conversely, when another eventfd is signalled, vhost-net will look at the ring and process any data there. You'll need to wire your signalling to those eventfds, either in userspace or in the kernel.

Quoted 8 lines · Ira W. Snyder

Also, in my case I'd like to boot Linux with my rootfs over NFS. Is vhost-net capable of this?

I've had Arnd, BenH, and Grant Likely (and others, privately) contact me about devices they are working with that would benefit from something like virtio-over-PCI. I'd like to see vhost-net be merged with the capability to support my use case. There are plenty of others that would benefit, not just myself.

Quoted 4 lines

I'm not sure vhost-net is being written with this kind of future use in mind. I'd hate to see it get merged, and then have to change the ABI to support physical-device-to-device usage. It would be better to keep future use in mind now, rather than try and hack it in later.

I still need to think your usage over. I am not so sure this fits what vhost is trying to do. If not, possibly it's better to just have a separate driver for your device.

On a non shared-memory system (where the guest's RAM is not just a chunk of userspace RAM in the host system), virtio's management model seems to fall apart. Feature negotiation doesn't work as one would expect.

This does appear to be solved by vbus, though I haven't written a vbus-over-PCI implementation, so I cannot be completely sure.

I'm not at all clear on how to get feature negotiation to work on a system like mine. From my study of lguest and kvm (see below) it looks like userspace will need to be involved, via a miscdevice.

Ok. I thought I should at least express my concerns while we're discussing this, rather than being too late after finding the time to study the driver.

Off the top of my head, I would think that transporting userspace addresses in the ring (for copy_(to|from)_user()) vs. physical addresses (for DMAEngine) might be a problem. Pinning userspace pages into memory for DMA is a bit of a pain, though it is possible.

There is also the problem of different endianness between host and guest in virtio-net. The struct virtio_net_hdr (include/linux/virtio_net.h) defines fields in host byte order. Which totally breaks if the guest has a different endianness. This is a virtio-net problem though, and is not transport specific.

I've browsed over both the kvm and lguest code, and it looks like they each re-invent a mechanism for transporting interrupts between the host and guest, using eventfd. They both do this by implementing a miscdevice, which is basically their management interface.

See drivers/lguest/lguest_user.c (see write() and LHREQ_EVENTFD) and kvm-kmod-devel-88/x86/kvm_main.c (see kvm_vm_ioctl(), called via kvm_dev_ioctl()) for how they hook up eventfd's.

I can now imagine how two userspace programs (host and guest) could work together to implement a management interface, including hotplug of devices, etc. Of course, this would basically reinvent the vbus management interface into a specific driver.

I think this is partly what Greg is trying to abstract out into generic code. I haven't studied the actual data transport mechanisms in vbus, though I have studied virtio's transport mechanism. I think a generic management interface for virtio might be a good thing to consider, because it seems there are at least two implementations already: kvm and lguest.

Quoted 5 lines

Off the top of my head, I would think that transporting userspace addresses in the ring (for copy_(to|from)_user()) vs. physical addresses (for DMAEngine) might be a problem. Pinning userspace pages into memory for DMA is a bit of a pain, though it is possible.

Oh, the ring doesn't transport userspace addresses. It transports guest addresses, and it's up to vhost to do something with them.

Currently vhost supports two translation modes:

1. virtio address == host virtual address (using copy_to_user)
2. virtio address == offsetted host virtual address (using copy_to_user)

The latter mode is used for kvm guests (with multiple offsets, skipping some details).

I think you need to add a third mode, virtio address == host physical address (using dma engine). Once you do that, and wire up the signalling, things should work.

Quoted 22 lines · Ira W. Snyder

I think Greg is referring to something like my virtio-over-PCI patch. I'm pretty sure that vhost is completely useless for my situation. I'd like to see vhost work for my use, so I'll try to explain what I'm doing.

I've got a system where I have about 20 computers connected via PCI. The PCI master is a normal x86 system, and the PCI agents are PowerPC systems. The PCI agents act just like any other PCI card, except they are running Linux, and have their own RAM and peripherals.

I wrote a custom driver which imitated a network interface and a serial port. I tried to push it towards mainline, and DavidM rejected it, with the argument, "use virtio, don't add another virtualization layer to the kernel." I think he has a decent argument, so I wrote virtio-over-PCI.

Now, there are some things about virtio that don't work over PCI. Mainly, memory is not truly shared. It is extremely slow to access memory that is "far away", meaning "across the PCI bus." This can be worked around by using a DMA controller to transfer all data, along with an intelligent scheme to perform only writes across the bus. If you're careful, reads are never needed.

Quoted 2 lines

So, in my system, copy_(to|from)_user() is completely wrong. There is no userspace, only a physical system.

Can guests do DMA to random host memory? Or is there some kind of IOMMU and DMA API involved? If the later, then note that you'll still need some kind of driver for your device. The question we need to ask ourselves then is whether this driver can reuse bits from vhost.

Quoted 3 lines

In fact, because normal x86 computers
do not have DMA controllers, the host system doesn't actually handle any data transfer!

Is it true that PPC has to initiate all DMA then? How do you manage not to do DMA reads then?

Quoted 5 lines

I used virtio-net in both the guest and host systems in my example virtio-over-PCI patch, and succeeded in getting them to communicate. However, the lack of any setup interface means that the devices must be hardcoded into both drivers, when the decision could be up to userspace. I think this is a problem that vbus could solve.

What you describe (passing setup from host to guest) seems like a feature that guest devices need to support. It seems unlikely that vbus, being a transport layer, can address this.

Quoted 6 lines

For my own selfish reasons (I don't want to maintain an out-of-tree driver) I'd like to see *something* useful in mainline Linux. I'm happy to answer questions about my setup, just ask.

Ira

Thanks Ira, I'll think about it.
A couple of questions:
- Could you please describe what kind of communication needs to happen?
- I'm not familiar with DMA engine in question. I'm guessing it's the usual thing: in/out buffers need to be kernel memory, interface is asynchronous, small limited number of outstanding requests? Is there a userspace interface for it and if yes how does it work?

Quoted 4 lines · Ira W. Snyder

On a non shared-memory system (where the guest's RAM is not just a chunk of userspace RAM in the host system), virtio's management model seems to fall apart. Feature negotiation doesn't work as one would expect.

In your case, virtio-net on the main board accesses PCI config space registers to perform the feature negotiation; software on your PCI cards needs to trap these config space accesses and respond to them according to virtio ABI.

(There's no real guest on your setup, right? just a kernel running on and x86 system and other kernels running on the PCI cards?)

Quoted 3 lines

This does appear to be solved by vbus, though I haven't written a vbus-over-PCI implementation, so I cannot be completely sure.

Even if virtio-pci doesn't work out for some reason (though it should), you can write your own virtio transport and implement its config space however you like.

Quoted 4 lines

I'm not at all clear on how to get feature negotiation to work on a system like mine. From my study of lguest and kvm (see below) it looks like userspace will need to be involved, via a miscdevice.

I don't see why. Is the kernel on the PCI cards in full control of all accesses?

Quoted 9 lines

Ok. I thought I should at least express my concerns while we're discussing this, rather than being too late after finding the time to study the driver.

Off the top of my head, I would think that transporting userspace addresses in the ring (for copy_(to|from)_user()) vs. physical addresses (for DMAEngine) might be a problem. Pinning userspace pages into memory for DMA is a bit of a pain, though it is possible.

Oh, the ring doesn't transport userspace addresses. It transports guest addresses, and it's up to vhost to do something with them.

Currently vhost supports two translation modes:

1. virtio address == host virtual address (using copy_to_user)
2. virtio address == offsetted host virtual address (using copy_to_user)

The latter mode is used for kvm guests (with multiple offsets, skipping some details).

I think you need to add a third mode, virtio address == host physical address (using dma engine). Once you do that, and wire up the signalling, things should work.

Quoted 6 lines

There is also the problem of different endianness between host and guest in virtio-net. The struct virtio_net_hdr (include/linux/virtio_net.h) defines fields in host byte order. Which totally breaks if the guest has a different endianness. This is a virtio-net problem though, and is not transport specific.

Yeah. You'll need to add byteswaps.

Quoted 5 lines

I've browsed over both the kvm and lguest code, and it looks like they each re-invent a mechanism for transporting interrupts between the host and guest, using eventfd. They both do this by implementing a miscdevice, which is basically their management interface.

Quoted 9 lines

See drivers/lguest/lguest_user.c (see write() and LHREQ_EVENTFD) and kvm-kmod-devel-88/x86/kvm_main.c (see kvm_vm_ioctl(), called via kvm_dev_ioctl()) for how they hook up eventfd's.

I can now imagine how two userspace programs (host and guest) could work together to implement a management interface, including hotplug of devices, etc. Of course, this would basically reinvent the vbus management interface into a specific driver.

You don't need anything in the guest userspace (virtio-net) side.

Quoted 7 lines

I think this is partly what Greg is trying to abstract out into generic code. I haven't studied the actual data transport mechanisms in vbus, though I have studied virtio's transport mechanism. I think a generic management interface for virtio might be a good thing to consider, because it seems there are at least two implementations already: kvm and lguest.

Management code in the kernel doesn't really help unless you plan to manage things with echo and cat.

Quoted 18 lines · Avi Kivity

Quoted 5 lines

Off the top of my head, I would think that transporting userspace addresses in the ring (for copy_(to|from)_user()) vs. physical addresses (for DMAEngine) might be a problem. Pinning userspace pages into memory for DMA is a bit of a pain, though it is possible.

Oh, the ring doesn't transport userspace addresses. It transports guest addresses, and it's up to vhost to do something with them.

Currently vhost supports two translation modes:

1. virtio address == host virtual address (using copy_to_user)
2. virtio address == offsetted host virtual address (using copy_to_user)

The latter mode is used for kvm guests (with multiple offsets, skipping some details).

Quoted 3 lines

I think you need to add a third mode, virtio address == host physical address (using dma engine). Once you do that, and wire up the signalling, things should work.

You don't need in fact a third mode. You can mmap the x86 address space into your ppc userspace and use the second mode. All you need then is the dma engine glue and byte swapping.

Quoted 6 lines · Gregory Haskins

Quoted 3 lines

You haven't convinced me that your ideas are worth the effort of abandoning virtio/pci or maintaining both venet/vbus and virtio/pci.

With all due respect, I didnt ask you do to anything, especially not abandon something you are happy with.

Quoted 6 lines

All I did was push guest drivers to LKML. The code in question is independent of KVM, and its proven to improve the experience of using Linux as a platform. There are people interested in using them (by virtue of the number of people that have signed up for the AlacrityVM list, and have mailed me privately about this work).

This thread started because i asked you about your technical arguments why we'd want vbus instead of virtio.

Your answer above
now basically boils down to: "because I want it so, why dont you leave me alone".

What you are doing here is to in essence to fork KVM,

regardless of
the technical counter arguments given against such a fork and regardless of the ample opportunity given to you to demostrate the technical advantages of your code. (in which case KVM would happily migrate to your code)

We all love faster code and better management interfaces and tons of your prior patches got accepted by Avi. This time you didnt even _try_ to improve virtio.

It's not like you posted a lot of virtio
patches which were not applied. You didnt even try and you need to try _much_ harder than that before forking a project.

And fragmentation matters quite a bit. To Linux users, developers, administrators, packagers it's a big deal whether two overlapping pieces of functionality for the same thing exist within the same kernel.

The kernel is not an anarchy where everyone can have their own sys_fork() version or their own sys_write() version. Would you want to have two dozen read() variants, sys_read_oracle() and a sys_read_db2()?

I certainly dont want that. Instead we (at great expense and work) try to reach the best technical solution.

That means we throw away
inferior code and adopt the better one. (with a reasonable migration period)

You are ignoring that principle with hand-waving about 'the community wants this'.

I can assure you, users _DONT WANT_ split
interfaces and incompatible drivers for the same thing. They want stuff that works well.

If the community wants this then why cannot you convince one of the most prominent representatives of that community, the KVM developers?

Furthermore, 99% of your work is KVM

why dont you respect that work by not forking it?

Quoted 4 lines

This thread started because i asked you about your technical arguments why we'd want vbus instead of virtio.

(You mean vbus vs pci, right? virtio works fine, is untouched, and is out-of-scope here)

Right, and I do believe I answered your questions. Do you feel as though this was not a satisfactory response?

Quoted 4 lines

Your answer above
now basically boils down to: "because I want it so, why dont you leave me alone".

Well, with all due respect, please do not put words in my mouth. This is not what I am saying at all.

What I *am* saying is:

fact: this thread is about linux guest drivers to support vbus

fact: these drivers do not touch kvm code.

fact: these drivers to not force kvm to alter its operation in any way.

fact: these drivers do not alter ABIs that KVM currently supports.

Therefore, all this talk about "abandoning", "supporting", and "changing" things in KVM is, premature, irrelevant, and/or, FUD. No one proposed such changes, so I am highlighting this fact to bring the thread back on topic. That KVM talk is merely a distraction at this point in time.

Quoted 4 lines

We all love faster code and better management interfaces and tons of your prior patches got accepted by Avi. This time you didnt even _try_ to improve virtio.

Im sorry, but you are mistaken:

http://lkml.indiana.edu/hypermail/linux/kernel/0904.2/02443.html

You are also wrong to say that I didn't try to avoid creating a
downstream effort first. I believe the public record of the mailing
lists will back me up that I tried politely pushing this directly though kvm first. It was only after Avi recently informed me that they would be building their own version of an in-kernel backend in lieu of working with me to adapt vbus to their needs that I decided to put my own project together.

What should I have done otherwise, in your opinion?

Quoted 5 lines

And fragmentation matters quite a bit. To Linux users, developers, administrators, packagers it's a big deal whether two overlapping pieces of functionality for the same thing exist within the same kernel.

So the only thing that could be construed as overlapping here is venet vs virtio-net. If I dropped the contentious venet and focused on making a virtio-net backend that we can all re-use, do you see that as a path of compromise here?

Quoted 3 lines

I certainly dont want that. Instead we (at great expense and work) try to reach the best technical solution.

This is all I want, as well.

Quoted 4 lines

If the community wants this then why cannot you convince one of the most prominent representatives of that community, the KVM developers?

Its a chicken and egg at times. Perhaps the KVM developers do not have the motivation or time to properly consider such a proposal _until_ the community presents its demand.

Quoted 2 lines

Furthermore, 99% of your work is KVM

Actually, no. Almost none of it is. I think there are about 2-3 patches in the series that touch KVM, the rest are all original (and primarily stand-alone code). AlacrityVM is the application of kvm and vbus (and, of course, Linux) together as a complete unit, but I do not try to hide this relationship.

By your argument, KVM is 99% QEMU+Linux. ;)

Quoted 5 lines

You don't need in fact a third mode. You can mmap the x86 address space into your ppc userspace and use the second mode. All you need then is the dma engine glue and byte swapping.

Hmm, I'll have to think about that.

The ppc is a 32-bit processor, so it has 4GB of address space for everything, including PCI, SDRAM, flash memory, and all other peripherals.

This is exactly like 32bit x86, where you cannot have a PCI card that exposes a 4GB PCI BAR. The system would have no address space left for its own SDRAM.

On my x86 computers, I only have 1GB of physical RAM, and so the ppc's have plenty of room in their address spaces to map the entire x86 RAM into their own address space. That is exactly what I do now. Accesses to ppc physical address 0x80000000 "magically" hit x86 physical address 0x0.

Quoted 5 lines · Gregory Haskins

Quoted 3 lines

It's not laughably trivial when you try to support the full feature set of kvm (for example, live migration will require dirty memory tracking, and exporting all state stored in the kernel to userspace).

Doesn't vhost suffer from the same issue? If not, could I also apply the same technique to support live-migration in vbus?

vhost does this by switching to userspace for the duration of live migration. venet could do this I guess, but you'd need to write a userspace implementation. vhost just reuses existing userspace virtio.

Quoted 9 lines

With all due respect, I didnt ask you do to anything, especially not abandon something you are happy with.

All I did was push guest drivers to LKML. The code in question is independent of KVM, and its proven to improve the experience of using Linux as a platform. There are people interested in using them (by virtue of the number of people that have signed up for the AlacrityVM list, and have mailed me privately about this work).

Quoted 1 line

So where is the problem here?

If virtio net in guest could be improved instead, everyone would benefit.

I am doing this, and I wish more people would join. Instead, you change ABI in a incompatible way.

So now, there's no single place to
work on kvm networking performance. Now, it would all be understandable if the reason was e.g. better performance. But you say yourself it isn't.

Quoted 10 lines · Gregory Haskins

Quoted 7 lines · Avi Kivity

On 08/17/2009 10:33 PM, Gregory Haskins wrote:

One point of contention is that this is all managementy stuff and should be kept out of the host kernel. Exposing shared memory, interrupts, and guest hypercalls can all be easily done from userspace (as virtio demonstrates). True, some devices need kernel acceleration, but that's no reason to put everything into the host kernel.

See my last reply to Anthony. My two points here are that:

Quoted 8 lines

a) having it in-kernel makes it a complete subsystem, which perhaps has diminished value in kvm, but adds value in most other places that we are looking to use vbus.

b) the in-kernel code is being overstated as "complex". We are not talking about your typical virt thing, like an emulated ICH/PCI chipset. Its really a simple list of devices with a handful of attributes. They
are managed using established linux interfaces, like sysfs/configfs.

IMHO the complexity of the code is not so much of a problem. What I see as a problem is the complexity a kernel/user space interface that manages a the devices with global state.

One of the greatest features of Michaels vhost driver is that all the state is associated with open file descriptors that either exist already or belong to the vhost_net misc device. When a process dies, all the file descriptors get closed and the whole state is cleaned up implicitly.

AFAICT, you can't do that with the vbus host model.

Quoted 3 lines

Quoted 2 lines

What performance oriented items have been left unaddressed?

Well, the interrupt model to name one.

The performance aspects of your interrupt model are independent of the vbus proxy, or at least they should be. Let's assume for now that your event notification mechanism gives significant performance improvements (which we can't measure independently right now). I don't see a reason why we could not get the same performance out of a paravirtual interrupt controller that uses the same method, and it would be straightforward to implement one and use that together with all the existing emulated PCI devices and virtio devices including vhost_net.

Quoted 7 lines · Avi Kivity

Quoted 5 lines · Ira W. Snyder

On a non shared-memory system (where the guest's RAM is not just a chunk of userspace RAM in the host system), virtio's management model seems to fall apart. Feature negotiation doesn't work as one would expect.

Quoted 6 lines

In your case, virtio-net on the main board accesses PCI config space registers to perform the feature negotiation; software on your PCI cards needs to trap these config space accesses and respond to them according to virtio ABI.

Is this "real PCI" (physical hardware) or "fake PCI" (software PCI emulation) that you are describing?

The host (x86, PCI master) must use "real PCI" to actually configure the boards, enable bus mastering, etc. Just like any other PCI device, such as a network card.

On the guests (ppc, PCI agents) I cannot add/change PCI functions (the last .[0-9] in the PCI address) nor can I change PCI BAR's once the board has started. I'm pretty sure that would violate the PCI spec, since the PCI master would need to re-scan the bus, and re-assign addresses, which is a task for the BIOS.

Quoted 4 lines

(There's no real guest on your setup, right? just a kernel running on and x86 system and other kernels running on the PCI cards?)

Yes, the x86 (PCI master) runs Linux (booted via PXELinux). The ppc's (PCI agents) also run Linux (booted via U-Boot). They are independent Linux systems, with a physical PCI interconnect.

The x86 has CONFIG_PCI=y, however the ppc's have CONFIG_PCI=n. Linux's PCI stack does bad things as a PCI agent. It always assumes it is a PCI master.

It is possible for me to enable CONFIG_PCI=y on the ppc's by removing the PCI bus from their list of devices provided by OpenFirmware. They can not access PCI via normal methods. PCI drivers cannot work on the ppc's, because Linux assumes it is a PCI master.

To the best of my knowledge, I cannot trap configuration space accesses on the PCI agents. I haven't needed that for anything I've done thus far.

Quoted 4 lines

Quoted 4 lines

This does appear to be solved by vbus, though I haven't written a vbus-over-PCI implementation, so I cannot be completely sure.

Quoted 5 lines

Even if virtio-pci doesn't work out for some reason (though it should), you can write your own virtio transport and implement its config space however you like.

This is what I did with virtio-over-PCI. The way virtio-net negotiates features makes this work non-intuitively.

Quoted 5 lines

Quoted 5 lines

I'm not at all clear on how to get feature negotiation to work on a system like mine. From my study of lguest and kvm (see below) it looks like userspace will need to be involved, via a miscdevice.

Quoted 4 lines

I don't see why. Is the kernel on the PCI cards in full control of all accesses?

I'm not sure what you mean by this. Could you be more specific? This is a normal, unmodified vanilla Linux kernel running on the PCI agents.

In my virtio-over-PCI patch, I hooked two virtio-net's together. I wrote an algorithm to pair the tx/rx queues together. Since virtio-net pre-fills its rx queues with buffers, I was able to use the DMA engine to copy from the tx queue into the pre-allocated memory in the rx queue.

Quoted 5 lines

Quoted 2 lines

Yeah. You'll need to add byteswaps.

I wonder if Rusty would accept a new feature:
VIRTIO_F_NET_LITTLE_ENDIAN, which would allow the virtio-net driver to use LE for all of it's multi-byte fields.

I don't think the transport should have to care about the endianness.

True. It's slowpath setup, so I don't care how fast it is. For reasons outside my control, the x86 (PCI master) is running a RHEL5 system. This means glibc-2.5, which doesn't have eventfd support, AFAIK. I could try and push for an upgrade. This obviously makes cat/echo really nice, it doesn't depend on glibc, only the kernel version.

I don't give much weight to the above, because I can use the eventfd syscalls directly, without glibc support. It is just more painful.

Quoted 3 lines

If virtio net in guest could be improved instead, everyone would benefit.

So if I whip up a virtio-net backend for vbus with a PCI compliant connector, you are happy?

Quoted 3 lines

I am doing this, and I wish more people would join. Instead, you change ABI in a incompatible way.

Only by choice of my particular connector. The ABI is a function of the connector design. So one such model is to terminate the connector in qemu, and surface the resulting objects as PCI devices. I choose not to use this particular design for my connector that I am pushing upstream because I am of the opinion that I can do better by terminating it in the guest directly as a PV optimized bus. However, both connectors can theoretically coexist peacefully.

Quoted 10 lines · Gregory Haskins

Quoted 5 lines

Can you explain how vbus achieves RDMA?

I also don't see the connection to real time guests.

Both of these are still in development. Trying to stay true to the "release early and often" mantra, the core vbus technology is being pushed now so it can be reviewed. Stay tuned for these other developments.

Hopefully you can outline how it works. AFAICT, RDMA and kernel bypass will need device assignment. If you're bypassing the call into the host kernel, it doesn't really matter how that call is made, does it?

Quoted 19 lines

Quoted 11 lines

Quoted 9 lines

I also designed it in such a way that
we could, in theory, write one set of (linux-based) backends, and have them work across a variety of environments (such as containers/VMs like KVM, lguest, openvz, but also physical systems like blade enclosures and
clusters, or even applications running on the host).

Sorry, I'm still confused. Why would openvz need vbus?

Its just an example. The point is that I abstracted what I think are the key points of fast-io, memory routing, signal routing, etc, so that it will work in a variety of (ideally, _any_) environments.

There may not be _performance_ motivations for certain classes of VMs because they already have decent support, but they may want a connector anyway to gain some of the new features available in vbus.

Quoted 3 lines

And looking forward, the idea is that we have commoditized the backend so we don't need to redo this each time a new container comes along.

I'll wait until a concrete example shows up as I still don't understand.

Quoted 9 lines

Quoted 7 lines

One point of contention is that this is all managementy stuff and should be kept out of the host kernel. Exposing shared memory, interrupts, and guest hypercalls can all be easily done from userspace (as virtio demonstrates). True, some devices need kernel acceleration, but that's no reason to put everything into the host kernel.

See my last reply to Anthony. My two points here are that:

Quoted 4 lines

a) having it in-kernel makes it a complete subsystem, which perhaps has diminished value in kvm, but adds value in most other places that we are looking to use vbus.

It's not a complete system unless you want users to administer VMs using echo and cat and configfs. Some userspace support will always be necessary.

Quoted 6 lines

b) the in-kernel code is being overstated as "complex". We are not talking about your typical virt thing, like an emulated ICH/PCI chipset. Its really a simple list of devices with a handful of attributes. They
are managed using established linux interfaces, like sysfs/configfs.

They need to be connected to the real world somehow. What about security? can any user create a container and devices and link them to real interfaces? If not, do you need to run the VM as root?

virtio and vhost-net solve these issues. Does vbus?

The code may be simple to you. But the question is whether it's necessary, not whether it's simple or complex.

Quoted 3 lines

Quoted 3 lines

Exposing devices as PCI is an important issue for me, as I have to consider non-Linux guests.

Quoted 2 lines

Thats your prerogative, but obviously not everyone agrees with you.

I hope everyone agrees that it's an important issue for me and that I have to consider non-Linux guests. I also hope that you're considering non-Linux guests since they have considerable market share.

Quoted 4 lines

Getting non-Linux guests to work is my problem if you chose to not be part of the vbus community.

I won't be writing those drivers in any case.

Quoted 6 lines

Quoted 4 lines

Another issue is the host kernel management code which I believe is superfluous.

In your opinion, right?

Yes, this is why I wrote "I believe".

Quoted 8 lines

Quoted 4 lines

Given that, why spread to a new model?

Note: I haven't asked you to (at least, not since April with the vbus-v3 release). Spreading to a new model is currently the role of the AlacrityVM project, since we disagree on the utility of a new model.

Given I'm not the gateway to inclusion of vbus/venet, you don't need to ask me anything. I'm still free to give my opinion.

Quoted 9 lines

Quoted 9 lines

Quoted 7 lines

A) hardware can only generate byte/word sized requests at a time because
that is all the pcb-etch and silicon support. So hardware is usually expressed in terms of some number of "registers".

No, hardware happily DMAs to and fro main memory.

Quoted 4 lines

Yes, now walk me through how you set up DMA to do something like a call when you do not know addresses apriori. Hint: count the number of MMIO/PIOs you need. If the number is> 1, you've lost.

With virtio, the number is 1 (or less if you amortize). Set up the ring entries and kick.

Quoted 8 lines

Quoted 5 lines

Some hardware of
course uses mmio registers extensively, but not virtio hardware. With the recent MSI support no registers are touched in the fast path.

Note we are not talking about virtio here. Just raw PCI and why I advocate vbus over it.

There's no such thing as raw PCI. Every PCI device has a protocol. The protocol virtio chose is optimized for virtualization.

Quoted 9 lines

Quoted 9 lines

Quoted 6 lines

D) device-ids are in a fixed width register and centrally assigned from an authority (e.g. PCI-SIG).

That's not an issue either. Qumranet/Red Hat has donated a range of device IDs for use in virtio.

Quoted 5 lines

Yes, and to get one you have to do what? Register it with kvm.git, right? Kind of like registering a MAJOR/MINOR, would you agree? Maybe you do not mind (especially given your relationship to kvm.git), but there are disadvantages to that model for most of the rest of us.

Send an email, it's not that difficult. There's also an experimental range.

Quoted 8 lines

Quoted 4 lines

Device IDs are how devices are associated
with drivers, so you'll need something similar for vbus.

Nope, just like you don't need to do anything ahead of time for using a dynamic misc-device name. You just have both the driver and device know what they are looking for (its part of the ABI).

If you get a device ID clash, you fail. If you get a device name clash, you fail in the same way.

Quoted 18 lines

Quoted 6 lines

Quoted 4 lines

E) Interrupt/MSI routing is per-device oriented

Please elaborate. What is the issue? How does vbus solve it?

There are no "interrupts" in vbus..only shm-signals. You can establish an arbitrary amount of shm regions, each with an optional shm-signal associated with it. To do this, the driver calls dev->shm(), and you get back a shm_signal object.

Underneath the hood, the vbus-connector (e.g. vbus-pcibridge) decides how it maps real interrupts to shm-signals (on a system level, not per device). This can be 1:1, or any other scheme. vbus-pcibridge uses one system-wide interrupt per priority level (today this is 8 levels), each with an IOQ based event channel. "signals" come as an event on that channel.

Quoted 5 lines

So the "issue" is that you have no real choice with PCI. You just get device oriented interrupts. With vbus, its abstracted. So you can still get per-device standard MSI, or you can do fancier things like do coalescing and prioritization.

As I've mentioned before, prioritization is available on x86

, and coalescing scales badly.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

F) Interrupts/MSI are assumed cheap to inject

Quoted 8 lines

Quoted 3 lines

Interrupts are not assumed cheap; that's why interrupt mitigation is used (on real and virtual hardware).

Its all relative. IDT dispatch and EOI overhead are "baseline" on real hardware, whereas they are significantly more expensive to do the vmenters and vmexits on virt (and you have new exit causes, like irq-windows, etc, that do not exist in real HW).

irq window exits ought to be pretty rare, so we're only left with injection vmexits. At around 1us/vmexit, even 100,000 interrupts/vcpu (which is excessive) will only cost you 10% cpu time.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

G) Interrupts/MSI are non-priortizable.

Quoted 9 lines

Quoted 3 lines

They are prioritizable; Linux ignores this though (Windows doesn't). Please elaborate on what the problem is and how vbus solves it.

It doesn't work right. The x86 sense of interrupt priority is, sorry to say it, half-assed at best. I've worked with embedded systems that have real interrupt priority support in the hardware, end to end, including the PIC. The LAPIC on the other hand is really weak in this dept, and as you said, Linux doesn't even attempt to use whats there.

Maybe prioritization is not that important then. If it is, it needs to be fixed at the lapic level, otherwise you have no real prioritization wrt non-vbus interrupts.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

H) Interrupts/MSI are statically established

Quoted 8 lines

Quoted 2 lines

Can you give an example of why this is a problem?

Some of the things we are building use the model of having a device that hands out shm-signal in response to guest events (say, the creation of an IPC channel). This would generally be handled by a specific device model instance, and it would need to do this without pre-declaring the MSI vectors (to use PCI as an example).

You're free to demultiplex an MSI to however many consumers you want, there's no need for a new bus for that.

Quoted 5 lines

Quoted 3 lines

What performance oriented items have been left unaddressed?

Well, the interrupt model to name one.

Like I mentioned, you can merge MSI interrupts, but that's not necessarily a good idea.

Quoted 3 lines

Quoted 3 lines

How do you handle conflicts? Again you need a central authority to hand out names or prefixes.

Quoted 4 lines

Not really, no. If you really wanted to be formal about it, you could adopt any series of UUID schemes. For instance, perhaps venet should be "com.novell::virtual-ethernet". Heck, I could use uuidgen.

Do you use DNS. We use PCI-SIG. If Novell is a PCI-SIG member you can get a vendor ID and control your own virtio space.

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

As another example, the connector design coalesces *all* shm-signals into a single interrupt (by prio) that uses the same context-switch mitigation techniques that help boost things like networking. This effectively means we can detect and optimize out ack/eoi cycles from the
APIC as the IO load increases (which is when you need it most). PCI has
no such concept.

Quoted 2 lines

Quoted 2 lines

That's a bug, not a feature. It means poor scaling as the number of vcpus increases and as the number of devices increases.

Quoted 6 lines

Quoted 1 line

So the "avi-vbus-connector" can use 1:1, if you prefer. Large vcpu counts (which are not typical) and irq-affinity is not a target application for my design, so I prefer the coalescing model in the vbus-pcibridge included in this series. YMMV

So far you've left out live migration

Windows,

large guests

and multiqueue out of your design.

If you wish to position vbus/venet for
large scale use you'll need to address all of them.

Quoted 7 lines

Quoted 3 lines

Note nothing prevents steering multiple MSIs into a single vector. It's a bad idea though.

Yes, it is a bad idea...and not the same thing either. This would effectively create a shared-line scenario in the irq code, which is not what happens in vbus.

Ok.

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

In addition, the signals and interrupts are priority aware, which is useful for things like 802.1p networking where you may establish 8-tx and 8-rx queues for your virtio-net device. x86 APIC really has no usable equivalent, so PCI is stuck here.

Quoted 4 lines

Quoted 2 lines

x86 APIC is priority aware.

Have you ever tried to use it?

I haven't, but Windows does.

Quoted 11 lines

Quoted 9 lines

Quoted 6 lines

Also, the signals can be allocated on-demand for implementing things like IPC channels in response to guest requests since there is no assumption about device-to-interrupt mappings. This is more flexible.

Yes. However given that vectors are a scarce resource you're severely limited in that.

The connector I am pushing out does not have this limitation.

Okay.

Quoted 5 lines

Quoted 5 lines

And if you're multiplexing everything on one vector,
then you can just as well demultiplex your channels in the virtio driver code.

Quoted 2 lines

Only per-device, not system wide.

Right. I still think multiplexing interrupts is a bad idea in a large system. In a small system... why would you do it at all?

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

And through all of this, this design would work in any guest even if it doesn't have PCI (e.g. lguest, UML, physical systems, etc).

Quoted 5 lines

Quoted 2 lines

That is true for virtio which works on pci-less lguest and s390.

Yes, and lguest and s390 had to build their own bus-model to do it, right?

They had to build connectors just like you propose to do.

Quoted 8 lines

Thank you for bringing this up, because it is one of the main points here. What I am trying to do is generalize the bus to prevent the proliferation of more of these isolated models in the future. Build one, fast, in-kernel model so that we wouldn't need virtio-X, and virtio-Y in the future. They can just reuse the (performance optimized) bus and models, and only need to build the connector to bridge them.

But you still need vbus-connector-lguest and vbus-connector-s390 because they all talk to the host differently. So what's changed? the names?

Quoted 4 lines

Quoted 4 lines

That is exactly the design goal of virtio (except it limits itself to virtualization).

Quoted 5 lines

No, virtio is only part of the picture. It not including the backend models, or how to do memory/signal-path abstraction for in-kernel, for instance. But otherwise, virtio as a device model is compatible with vbus as a bus model. They compliment one another.

Well, venet doesn't complement virtio-net, and virtio-pci doesn't complement vbus-connector.

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Then device models like virtio can ride happily on top and we end up with a really robust and high-performance Linux-based stack. I don't buy the argument that we already have PCI so lets use it. I don't think
its the best design and I am not afraid to make an investment in a change here because I think it will pay off in the long run.

Quoted 6 lines

Quoted 2 lines

Sorry, I don't think you've shown any quantifiable advantages.

We can agree to disagree then, eh? There are certainly quantifiable differences. Waving your hand at the differences to say they are not advantages is merely an opinion, one that is not shared universally.

I've addressed them one by one. We can agree to disagree on interrupt multiplexing, and the importance of compatibility, Windows, large guests, multiqueue, and DNS vs. PCI-SIG.

Quoted 5 lines

The bottom line is all of these design distinctions are encapsulated within the vbus subsystem and do not affect the kvm code-base. So agreement with kvm upstream is not a requirement, but would be advantageous for collaboration.

Certainly.

Quoted 1 line

Quoted 1 line

On 8/19/2009 at 1:48 AM, in message <[email protected]>, Avi Kivity

Quoted 6 lines · Gregory Haskins

Quoted 3 lines

If virtio net in guest could be improved instead, everyone would benefit.

So if I whip up a virtio-net backend for vbus with a PCI compliant connector, you are happy?

This doesn't improve virtio-net in any way.

Quoted 4 lines

Quoted 4 lines

I am doing this, and I wish more people would join. Instead, you change ABI in a incompatible way.

Quoted 8 lines

Only by choice of my particular connector. The ABI is a function of the connector design. So one such model is to terminate the connector in qemu, and surface the resulting objects as PCI devices. I choose not to use this particular design for my connector that I am pushing upstream because I am of the opinion that I can do better by terminating it in the guest directly as a PV optimized bus. However, both connectors can theoretically coexist peacefully.

virtio already supports this model; see lguest and s390. Transporting virtio over vbus and vbus over something else doesn't gain anything over directly transporting virtio over that something else.

Quoted 14 lines · Avi Kivity

Quoted 12 lines · Gregory Haskins

Quoted 6 lines

Can you explain how vbus achieves RDMA?

I also don't see the connection to real time guests.

Both of these are still in development. Trying to stay true to the "release early and often" mantra, the core vbus technology is being pushed now so it can be reviewed. Stay tuned for these other developments.

Quoted 4 lines

Hopefully you can outline how it works. AFAICT, RDMA and kernel bypass will need device assignment. If you're bypassing the call into the host kernel, it doesn't really matter how that call is made, does it?

This is for things like the setup of queue-pairs, and the transport of door-bells, and ib-verbs. I am not on the team doing that work, so I am not an expert in this area. What I do know is having a flexible and low-latency signal-path was deemed a key requirement.

For real-time, a big part of it is relaying the guest scheduler state to the host, but in a smart way. For instance, the cpu priority for each vcpu is in a shared-table. When the priority is raised, we can simply update the table without taking a VMEXIT. When it is lowered, we need to inform the host of the change in case the underlying task needs to reschedule.

This is where the really fast call() type mechanism is important.

Its also about having the priority flow-end to end, and having the vcpu interrupt state affect the task-priority, etc (e.g. pending interrupts affect the vcpu task prio).

etc, etc.

I can go on and on (as you know ;), but will wait till this work is more concrete and proven.

Basically, what it comes down to is both vbus and vhost need configuration/management. Vbus does it with sysfs/configfs, and vhost does it with ioctls. I ultimately decided to go with sysfs/configfs because, at least that the time I looked, it seemed like the "blessed" way to do user->kernel interfaces.

Quoted 4 lines

They need to be connected to the real world somehow. What about security? can any user create a container and devices and link them to real interfaces? If not, do you need to run the VM as root?

Today it has to be root as a result of weak mode support in configfs, so you have me there. I am looking for help patching this limitation, though.

Quoted 4 lines

I hope everyone agrees that it's an important issue for me and that I have to consider non-Linux guests. I also hope that you're considering non-Linux guests since they have considerable market share.

I didn't mean non-Linux guests are not important. I was disagreeing with your assertion that it only works if its PCI. There are numerous examples of IHV/ISV "bridge" implementations deployed in Windows, no?

If vbus is exposed as a PCI-BRIDGE, how is this different?

Quoted 3 lines

Given I'm not the gateway to inclusion of vbus/venet, you don't need to ask me anything. I'm still free to give my opinion.

Agreed, and I didn't mean to suggest otherwise. It not clear if you are wearing the "kvm maintainer" hat, or the "lkml community member" hat at times, so its important to make that distinction. Otherwise, its not clear if this is edict as my superior, or input as my peer. ;)

Quoted 3 lines

With virtio, the number is 1 (or less if you amortize). Set up the ring entries and kick.

Again, I am just talking about basic PCI here, not the things we build on top.

The point is: the things we build on top have costs associated with them, and I aim to minimize it. For instance, to do a "call()" kind of interface, you generally need to pre-setup some per-cpu mappings so that you can just do a single iowrite32() to kick the call off. Those per-cpu mappings have a cost if you want them to be high-performance, so my argument is that you ideally want to limit the number of times you have to do this. My current design reduces this to "once".

Quoted 3 lines

There's no such thing as raw PCI. Every PCI device has a protocol. The protocol virtio chose is optimized for virtualization.

And its a question of how that protocol scales, more than how the protocol works.

Obviously the general idea of the protocol works, as vbus itself is implemented as a PCI-BRIDGE and is therefore limited to the underlying characteristics that I can get out of PCI (like PIO latency).

Quoted 2 lines

As I've mentioned before, prioritization is available on x86

But as Ive mentioned, it doesn't work very well.

Quoted 2 lines

, and coalescing scales badly.

Depends on what is scaling. Scaling vcpus? Yes, you are right. Scaling the number of devices? No, this is where it improves.

Quoted 4 lines

irq window exits ought to be pretty rare, so we're only left with injection vmexits. At around 1us/vmexit, even 100,000 interrupts/vcpu (which is excessive) will only cost you 10% cpu time.

1us is too much for what I am building, IMHO.

Quoted 3 lines

You're free to demultiplex an MSI to however many consumers you want, there's no need for a new bus for that.

Hmmm...can you elaborate?

Quoted 3 lines

Do you use DNS. We use PCI-SIG. If Novell is a PCI-SIG member you can get a vendor ID and control your own virtio space.

Yeah, we have our own id. I am more concerned about making this design make sense outside of PCI oriented environments.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

That's a bug, not a feature. It means poor scaling as the number of vcpus increases and as the number of devices increases.

vcpu increases, I agree (and am ok with, as I expect low vcpu count machines to be typical).

nr of devices, I disagree. can you elaborate?

Quoted 2 lines

Windows,

Work in progress.

Quoted 3 lines

large guests

Can you elaborate? I am not familiar with the term.

Quoted 3 lines

and multiqueue out of your design.

AFAICT, multiqueue should work quite nicely with vbus. Can you elaborate on where you see the problem?

Quoted 6 lines

Quoted 6 lines

Quoted 3 lines

x86 APIC is priority aware.

Have you ever tried to use it?

Quoted 2 lines

I haven't, but Windows does.

Yeah, it doesn't really work well. Its an extremely rigid model that (IIRC) only lets you prioritize in 16 groups spaced by IDT (0-15 are one level, 16-31 are another, etc). Most of the embedded PICs I have worked with supported direct remapping, etc. But in any case, Linux doesn't support it so we are hosed no matter how good it is.

Quoted 4 lines

Quoted 2 lines

They had to build connectors just like you propose to do.

More importantly, they had to build back-end busses too, no?

Quoted 3 lines

But you still need vbus-connector-lguest and vbus-connector-s390 because they all talk to the host differently. So what's changed? the names?

The fact that they don't need to redo most of the in-kernel backend stuff. Just the connector.

Quoted 3 lines

Well, venet doesn't complement virtio-net, and virtio-pci doesn't complement vbus-connector.

Agreed, but virtio complements vbus by virtue of virtio-vbus.

Quoted 8 lines

Quoted 6 lines

So if I whip up a virtio-net backend for vbus with a PCI compliant connector, you are happy?

This doesn't improve virtio-net in any way.

Any why not? (Did you notice I said "PCI compliant", i.e. over virtio-pci)

Quoted 4 lines

virtio already supports this model; see lguest and s390. Transporting virtio over vbus and vbus over something else doesn't gain anything over directly transporting virtio over that something else.

This is not what I am advocating.

Quoted 1 line

Quoted 1 line

On 8/19/2009 at 3:13 AM, in message <[email protected]>, Avi Kivity

Quoted 6 lines · Gregory Haskins

Quoted 6 lines

Quoted 6 lines

So if I whip up a virtio-net backend for vbus with a PCI compliant connector, you are happy?

Quoted 4 lines

Quoted 2 lines

This doesn't improve virtio-net in any way.

Any why not? (Did you notice I said "PCI compliant", i.e. over virtio-pci)

Because virtio-net will have gained nothing that it didn't have before.

Quoted 8 lines

Quoted 8 lines

virtio already supports this model; see lguest and s390. Transporting virtio over vbus and vbus over something else doesn't gain anything over directly transporting virtio over that something else.

Quoted 3 lines

This is not what I am advocating.

What are you advocating? As far as I can tell your virtio-vbus connector plus the vbus-kvm connector is just that.

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 6 lines

So if I whip up a virtio-net backend for vbus with a PCI compliant connector, you are happy?

This doesn't improve virtio-net in any way.

Quoted 5 lines

Quoted 3 lines

Any why not? (Did you notice I said "PCI compliant", i.e. over virtio-pci)

Because virtio-net will have gained nothing that it didn't have before.

??

*) ABI is virtio-pci compatible, as you like

*) fast-path is in-kernel, as we all like

*) model is in vbus so it would work in all environments that vbus supports.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

virtio already supports this model; see lguest and s390. Transporting virtio over vbus and vbus over something else doesn't gain anything over directly transporting virtio over that something else.

Quoted 7 lines

Quoted 4 lines

This is not what I am advocating.

What are you advocating? As far as I can tell your virtio-vbus connector plus the vbus-kvm connector is just that.

I wouldn't classify it anything like that, no. Its just virtio over vbus.

Quoted 13 lines · Gregory Haskins

Quoted 13 lines

Quoted 13 lines

Quoted 9 lines

Quoted 6 lines

So if I whip up a virtio-net backend for vbus with a PCI compliant connector, you are happy?

This doesn't improve virtio-net in any way.

Any why not? (Did you notice I said "PCI compliant", i.e. over virtio-pci)

Quoted 6 lines

Quoted 2 lines

Because virtio-net will have gained nothing that it didn't have before.

??

*) ABI is virtio-pci compatible, as you like

That's not a gain, that's staying in the same place.

Quoted 2 lines

*) fast-path is in-kernel, as we all like

That's not a gain as we have vhost-net (sure, in development, but your proposed backend isn't even there yet).

Quoted 3 lines

*) model is in vbus so it would work in all environments that vbus supports.

The ABI can be virtio-pci compatible or it can be vbus-comaptible. How can it be both? The ABIs are different.

Note that if you had submitted a virtio-net backend I'd have asked you to strip away all the management / bus layers and we'd have ended up with vhost-net.

Quoted 14 lines

Quoted 14 lines

Quoted 11 lines

Quoted 7 lines

virtio already supports this model; see lguest and s390. Transporting virtio over vbus and vbus over something else doesn't gain anything over
directly transporting virtio over that something else.

This is not what I am advocating.

What are you advocating? As far as I can tell your virtio-vbus connector plus the vbus-kvm connector is just that.

Quoted 3 lines

I wouldn't classify it anything like that, no. Its just virtio over vbus.

We're in a loop. Doesn't virtio over vbus need a virtio-vbus connector? and doesn't vbus need a connector to talk to the hypervisor?

Quoted 7 lines · Gregory Haskins

Quoted 4 lines

This thread started because i asked you about your technical arguments why we'd want vbus instead of virtio.

(You mean vbus vs pci, right? virtio works fine, is untouched, and is out-of-scope here)

I guess he meant venet vs virtio-net. Without venet vbus is currently userless.

Quoted 3 lines

Right, and I do believe I answered your questions. Do you feel as though this was not a satisfactory response?

Others and I have shown you its wrong.

There's no inherent performance
problem in pci. The vbus approach has inherent problems (the biggest of which is compatibility

Quoted 15 lines

Quoted 4 lines

Your answer above
now basically boils down to: "because I want it so, why dont you leave me alone".

Well, with all due respect, please do not put words in my mouth. This is not what I am saying at all.

What I *am* saying is:

fact: this thread is about linux guest drivers to support vbus

fact: these drivers do not touch kvm code.

fact: these drivers to not force kvm to alter its operation in any way.

Quoted 8 lines

fact: these drivers do not alter ABIs that KVM currently supports.

Therefore, all this talk about "abandoning", "supporting", and "changing" things in KVM is, premature, irrelevant, and/or, FUD. No one proposed such changes, so I am highlighting this fact to bring the thread back on topic. That KVM talk is merely a distraction at this point in time.

s/kvm/kvm stack/. virtio/pci is part of the kvm stack, even if it is not part of kvm itself. If vbus/venet were to be merged, users and developers would have to choose one or the other. That's the fragmentation I'm worried about. And you can prefix that with "fact:" as well.

Quoted 9 lines

Quoted 5 lines

We all love faster code and better management interfaces and tons of your prior patches got accepted by Avi. This time you didnt even _try_ to improve virtio.

Im sorry, but you are mistaken:

http://lkml.indiana.edu/hypermail/linux/kernel/0904.2/02443.html

That does nothing to improve virtio.

Existing guests (Linux and
Windows) which support virtio will cease to work if the host moves to vbus-virtio.

Existing hosts (running virtio-pci) won't be able to talk to newer guests running virtio-vbus. The patch doesn't improve performance without the entire vbus stack in the host kernel and a vbus-virtio-net-host host kernel driver.

Perhaps if you posted everything needed to make vbus-virtio work and perform we could compare that to vhost-net and you'll see another reason why vhost-net is the better approach.

Quoted 9 lines

You are also wrong to say that I didn't try to avoid creating a
downstream effort first. I believe the public record of the mailing
lists will back me up that I tried politely pushing this directly though kvm first. It was only after Avi recently informed me that they would be building their own version of an in-kernel backend in lieu of working with me to adapt vbus to their needs that I decided to put my own project together.

There's no way we can adapt vbus to our needs.

Don't you think we'd preferred it rather than writing our own?

the current virtio-net issues
are hurting us.

Our needs are compatibility, performance, and managability. vbus fails all three, your impressive venet numbers notwithstanding.

Quoted 2 lines

What should I have done otherwise, in your opinion?

You could come up with uses where vbus truly is superior to virtio/pci/whatever

(not words about etch constraints).

Showing some of those non-virt uses, for example.

The fact that your only user duplicates existing functionality doesn't help.

Quoted 7 lines

Quoted 7 lines

And fragmentation matters quite a bit. To Linux users, developers, administrators, packagers it's a big deal whether two overlapping pieces of functionality for the same thing exist within the same kernel.

Quoted 5 lines

So the only thing that could be construed as overlapping here is venet vs virtio-net. If I dropped the contentious venet and focused on making a virtio-net backend that we can all re-use, do you see that as a path of compromise here?

That's a step in the right direction.

Quoted 6 lines

Quoted 4 lines

I certainly dont want that. Instead we (at great expense and work) try to reach the best technical solution.

This is all I want, as well.

Note whenever I mention migration, large guests, or Windows you say these are not your design requirements.

The best technical solution will have to consider those.

Quoted 8 lines

Quoted 5 lines

If the community wants this then why cannot you convince one of the most prominent representatives of that community, the KVM developers?

Its a chicken and egg at times. Perhaps the KVM developers do not have the motivation or time to properly consider such a proposal _until_ the community presents its demand.

I've spent quite a lot of time arguing with you, no doubt influenced by the fact that you can write a lot faster than I can read.

Quoted 9 lines

Quoted 3 lines

Furthermore, 99% of your work is KVM

Actually, no. Almost none of it is. I think there are about 2-3 patches in the series that touch KVM, the rest are all original (and primarily stand-alone code). AlacrityVM is the application of kvm and vbus (and, of course, Linux) together as a complete unit, but I do not try to hide this relationship.

Quoted 2 lines

By your argument, KVM is 99% QEMU+Linux. ;)

That's one of the kvm strong points...

Please post results when you have numbers, as I had to
give up my 10GE rig in the lab.
I suspect you will have performance
issues until you at least address GSO, but you may already be there by now.

Quoted 4 lines

There's no inherent performance
problem in pci. The vbus approach has inherent problems (the biggest of which is compatibility

Trying to be backwards compatible in all dimensions is not a design goal, as already stated.

, the second managability).

Quoted 1 line

Where are the management problems?

No, you have shown me that you disagree. I'm sorry, but do not assume they are the same.

I'm sorry, but thats just plain false.

Quoted 4 lines

Existing guests (Linux and
Windows) which support virtio will cease to work if the host moves to vbus-virtio.

Sigh...please re-read "fact" section. And even if this work is accepted upstream as it is, how you configure the host and guest is just that: a configuration. If your guest and host both speak vbus, use it. If they don't, don't use it. Simple as that. Saying anything else is just more FUD, and I can say the same thing about a variety of other configuration options currently available.

Quoted 5 lines

Existing hosts (running virtio-pci) won't be able to talk to newer guests running virtio-vbus. The patch doesn't improve performance without the entire vbus stack in the host kernel and a vbus-virtio-net-host host kernel driver.

<rewind years=2>Existing hosts (running realtek emulation) won't be able to talk to newer guests running virtio-net. Virtio-net doesn't do anything to improve realtek emulation without the entire virtio stack in the host.</rewind>

You gotta start somewhere. You're argument buys you nothing other than
backwards compat, which I've already stated is not a specific goal here.
I am not against "modprobe vbus-pcibridge", and I am sure there are
users out that that do not object to this either.

Quoted 4 lines

Perhaps if you posted everything needed to make vbus-virtio work and perform we could compare that to vhost-net and you'll see another reason why vhost-net is the better approach.

Yet, you must recognize that an alternative outcome is that we can look at issues outside of virtio-net on KVM and perhaps you will see vbus is a better approach.

Quoted 9 lines

Quoted 9 lines

You are also wrong to say that I didn't try to avoid creating a
downstream effort first. I believe the public record of the mailing
lists will back me up that I tried politely pushing this directly though kvm first. It was only after Avi recently informed me that they would be building their own version of an in-kernel backend in lieu of working with me to adapt vbus to their needs that I decided to put my own project together.

Quoted 2 lines

There's no way we can adapt vbus to our needs.

Really? Did you ever bother to ask how? I'm pretty sure you can. And if you couldn't, I would have considered changes to make it work.

Quoted 2 lines

Don't you think we'd preferred it rather than writing our own?

Honestly, I am not so sure based on your responses.

I've already listed numerous examples on why I advocate vbus over PCI, and have already stated I am not competing against virtio.

Quoted 2 lines

Showing some of those non-virt uses, for example.

Actually, Ira's chassis discussed earlier is a classic example. Vbus actually fits neatly into his model, I believe (and much better than the vhost proposals, IMO).

Basically, IMO we want to invert Ira's bus (so that the PPC boards see host-based devices, instead of the other way around). You write a connector that transports the vbus verbs over the PCI link. You write a udev rule that responds to the PPC board "arrival" event to create a new vbus container, and assign the board to that context.

Quoted 2 lines

The fact that your only user duplicates existing functionality doesn't help.

Certainly at some level, that is true and is unfortunate, I agree. In retrospect, I wish I started with something non-overlapping with virtio as the demo, just to avoid this aspect of controversy.

At another level, its the highest-performance 802.x interface for KVM at the moment, since we still have not seen benchmarks for vhost. Given that I have spent a lot of time lately optimizing KVM, I can tell you its not trivial to get it to work better than the userspace virtio. Michael is clearly a smart guy, so the odds are in his favor. But do not count your chickens before they hatch, because its not guaranteed success.

Long story short, my patches are not duplicative on all levels (i.e. performance). Its just another ethernet driver, of which there are probably hundreds of alternatives in the kernel already. You could also argue that we already have multiple models in qemu (realtek, e1000, virtio-net, etc) so this is not without precedent. So really all this "fragmentation" talk is FUD. Lets stay on-point, please.

Can we talk more about that at some point? I think you will see its not some "evil, heavy duty" infrastructure that some comments seem to be trying to paint it as. I think its similar in concept to what you need to do for a vhost like design, but (with all due respect to Michael) a little bit more thought into the necessary abstraction points to allow broader application.

Quoted 5 lines

Quoted 2 lines

Note whenever I mention migration, large guests, or Windows you say these are not your design requirements.

Actually, I don't think I've ever said that, per se. I said that those things are not a priority for me, personally. I never made a design decision that I knew would preclude the support for such concepts. In fact, afaict, the design would support them just fine, given resources the develop them.

For the record: I never once said "vbus is done". There is plenty of work left to do. This is natural (kvm I'm sure wasn't 100% when it went in either, nor is it today)

Quoted 9 lines · Ira W. Snyder

Quoted 7 lines · Avi Kivity

Quoted 5 lines · Ira W. Snyder

On a non shared-memory system (where the guest's RAM is not just a chunk of userspace RAM in the host system), virtio's management model seems to fall apart. Feature negotiation doesn't work as one would expect.

Quoted 10 lines

Quoted 6 lines

In your case, virtio-net on the main board accesses PCI config space registers to perform the feature negotiation; software on your PCI cards needs to trap these config space accesses and respond to them according to virtio ABI.

Is this "real PCI" (physical hardware) or "fake PCI" (software PCI emulation) that you are describing?

Real PCI.

Quoted 10 lines

The host (x86, PCI master) must use "real PCI" to actually configure the boards, enable bus mastering, etc. Just like any other PCI device, such as a network card.

On the guests (ppc, PCI agents) I cannot add/change PCI functions (the last .[0-9] in the PCI address) nor can I change PCI BAR's once the board has started. I'm pretty sure that would violate the PCI spec, since the PCI master would need to re-scan the bus, and re-assign addresses, which is a task for the BIOS.

Yes. Can the boards respond to PCI config space cycles coming from the host, or is the config space implemented in silicon and immutable? (reading on, I see the answer is no). virtio-pci uses the PCI config space to configure the hardware.

Quoted 12 lines

Quoted 4 lines

(There's no real guest on your setup, right? just a kernel running on and x86 system and other kernels running on the PCI cards?)

Yes, the x86 (PCI master) runs Linux (booted via PXELinux). The ppc's (PCI agents) also run Linux (booted via U-Boot). They are independent Linux systems, with a physical PCI interconnect.

The x86 has CONFIG_PCI=y, however the ppc's have CONFIG_PCI=n. Linux's PCI stack does bad things as a PCI agent. It always assumes it is a PCI master.

Quoted 10 lines

It is possible for me to enable CONFIG_PCI=y on the ppc's by removing the PCI bus from their list of devices provided by OpenFirmware. They can not access PCI via normal methods. PCI drivers cannot work on the ppc's, because Linux assumes it is a PCI master.

To the best of my knowledge, I cannot trap configuration space accesses on the PCI agents. I haven't needed that for anything I've done thus far.

Well, if you can't do that, you can't use virtio-pci on the host. You'll need another virtio transport (equivalent to "fake pci" you mentioned above).

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

This does appear to be solved by vbus, though I haven't written a vbus-over-PCI implementation, so I cannot be completely sure.

Even if virtio-pci doesn't work out for some reason (though it should), you can write your own virtio transport and implement its config space however you like.

Quoted 3 lines

This is what I did with virtio-over-PCI. The way virtio-net negotiates features makes this work non-intuitively.

I think you tried to take two virtio-nets and make them talk together? That won't work. You need the code from qemu to talk to virtio-net config space, and vhost-net to pump the rings.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

I'm not at all clear on how to get feature negotiation to work on a system like mine. From my study of lguest and kvm (see below) it looks like userspace will need to be involved, via a miscdevice.

Quoted 7 lines

Quoted 4 lines

I don't see why. Is the kernel on the PCI cards in full control of all accesses?

I'm not sure what you mean by this. Could you be more specific? This is a normal, unmodified vanilla Linux kernel running on the PCI agents.

I meant, does board software implement the config space accesses issued from the host, and it seems the answer is no.

Quoted 6 lines

In my virtio-over-PCI patch, I hooked two virtio-net's together. I wrote an algorithm to pair the tx/rx queues together. Since virtio-net pre-fills its rx queues with buffers, I was able to use the DMA engine to copy from the tx queue into the pre-allocated memory in the rx queue.