Backend libraries for VirtIO device emulation
Hi,
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Currently most virtio backends are embedded directly in various VMMs which emulate a number of devices as well as deal with handling devices that are vhost aware and link with the host kernel. However there seems to be a growing interest in having backends implemented in separate processes, potentially even hosted in other guest VMs.
As far as I can tell there is a lot of duplicated effort in handling the low level navigation of virt queues and buffers. QEMU has code in hw/virtio as well as contrib/libvhost-user which is used by the recent virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a similar set of functionality for it's emulation. The Rust-vmm project has libraries for implementing the device traits.
Another aspect to this is the growing interest in carrying virtio over other hypervisors. I'm wondering if there is enough abstraction possible to have a common library that is hypervisor agnostic? Can a device backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Thoughts?
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Hello Alex,
Quoted 18 lines · Alex Bennée
Hi,
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Currently most virtio backends are embedded directly in various VMMs which emulate a number of devices as well as deal with handling devices that are vhost aware and link with the host kernel. However there seems to be a growing interest in having backends implemented in separate processes, potentially even hosted in other guest VMs.
As far as I can tell there is a lot of duplicated effort in handling the low level navigation of virt queues and buffers. QEMU has code in hw/virtio as well as contrib/libvhost-user which is used by the recent virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a similar set of functionality for it's emulation. The Rust-vmm project has libraries for implementing the device traits.
Quoted 8 lines
Another aspect to this is the growing interest in carrying virtio over other hypervisors. I'm wondering if there is enough abstraction possible to have a common library that is hypervisor agnostic? Can a device backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Thoughts?
I am currently investigating the use of metamodeling to capture virtio concepts. The work is still a draft but you can give a look at
https://github.com/MatiasVara/virtioml. One of the ideas is to base on a metamodel to generate the code for different targets, e.g., rust, c, etc. For the moment, I just played by generating the c header. This could be extended to other languages.
Regards, Matias.
Quoted 8 lines
Signature · 6 lines
-- Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 6 lines · Matias Ezequiel Vara Larsen
Hello Alex,
Quoted 4 lines · Alex Bennée
Hi,
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
<snip>
Quoted 9 lines
Quoted 2 lines
Thoughts?
I am currently investigating the use of metamodeling to capture virtio concepts. The work is still a draft but you can give a look at
https://github.com/MatiasVara/virtioml. One of the ideas is to base on a metamodel to generate the code for different targets, e.g., rust, c, etc. For the moment, I just played by generating the c header. This could be extended to other languages.
Thanks for the pointer, I shall have a look.
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 18 lines · * Alex Bennée ([email protected]) wrote:
Hi,
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Currently most virtio backends are embedded directly in various VMMs which emulate a number of devices as well as deal with handling devices that are vhost aware and link with the host kernel. However there seems to be a growing interest in having backends implemented in separate processes, potentially even hosted in other guest VMs.
As far as I can tell there is a lot of duplicated effort in handling the low level navigation of virt queues and buffers. QEMU has code in hw/virtio as well as contrib/libvhost-user which is used by the recent virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a similar set of functionality for it's emulation. The Rust-vmm project has libraries for implementing the device traits.
Quoted 6 lines
Another aspect to this is the growing interest in carrying virtio over other hypervisors. I'm wondering if there is enough abstraction possible to have a common library that is hypervisor agnostic? Can a device backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
It's a little tricky because it has to interface tightly with the way that the memory-mapping works for the hypervisor, so that the external process can access the memory of the queues.
QEMU's vhost-user has a fair amount of code for handling the mappings, dirty logging for migration, iommu's and things like reset (which is pretty hairy, and probably needs more work).
Dave
Quoted 9 lines
Thoughts?
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 26 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Alex Bennée ([email protected]) wrote:
Hi,
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Currently most virtio backends are embedded directly in various VMMs which emulate a number of devices as well as deal with handling devices that are vhost aware and link with the host kernel. However there seems to be a growing interest in having backends implemented in separate processes, potentially even hosted in other guest VMs.
As far as I can tell there is a lot of duplicated effort in handling the low level navigation of virt queues and buffers. QEMU has code in hw/virtio as well as contrib/libvhost-user which is used by the recent virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a similar set of functionality for it's emulation. The Rust-vmm project has libraries for implementing the device traits.
Quoted 6 lines
Another aspect to this is the growing interest in carrying virtio over other hypervisors. I'm wondering if there is enough abstraction possible to have a common library that is hypervisor agnostic? Can a device backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Quoted 3 lines
It's a little tricky because it has to interface tightly with the way that the memory-mapping works for the hypervisor, so that the external process can access the memory of the queues.
I suspect the problem space can at least be reduced to at least a POSIX-like environment - if that makes things simpler. The setting up of memory-mappings should be the problem of the VMM, which would possibly be hypervisor specific. After that it is simply(?) a question of sharing the appropriate bit of memory between the VMM and the device process.
The other model would be the device process runs inside another guest - most likely a Linux VM. Here the guest kernel can be told an area of memory is special in some way and provide a device node that can be mmaped in more or less the same way. In this configuration it can't even be aware of what the underlying hypervisor is - just a block of memory and a way to receive message queue events.
Quoted 3 lines
QEMU's vhost-user has a fair amount of code for handling the mappings, dirty logging for migration, iommu's and things like reset (which is pretty hairy, and probably needs more work).
I suspect all of these multi-process models just hand wave away details like migration because that really does benefit from a single process with total awareness of the state of the system. That said I wonder how robust a guest can be if the device emulation may go away at any time?
I guess in virtio if you never signal the consumption of a virt-queue it will still be there waiting until you restart the emulation process and pick up from where you left off?
Quoted 12 lines
Dave
Quoted 9 lines
Thoughts?
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 8 lines · Alex Bennée
I suspect all of these multi-process models just hand wave away details like migration because that really does benefit from a single process with total awareness of the state of the system. That said I wonder how robust a guest can be if the device emulation may go away at any time?
I guess in virtio if you never signal the consumption of a virt-queue it will still be there waiting until you restart the emulation process and pick up from where you left off?
The virtio spec seems be fine with individual backends dying. Recovery depends on the hypervisor maintaining state for the backends - not sure if that is easily doable, though.
But the challenge starts earlier: virtio-over-ivshmem is designed to support independent backends lifecycles (and has no state except for guest lifeness in the hypervisor). But while playing with that, I immediately ran into Linux frontend code assuming that a virtio device never dies (virtio console, IIRC). I suspect that there are more cases in Linux, not to speak of other guests.
Jan
Signature · 6 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 33 lines · * Alex Bennée ([email protected]) wrote:
Quoted 26 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Alex Bennée ([email protected]) wrote:
Hi,
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Currently most virtio backends are embedded directly in various VMMs which emulate a number of devices as well as deal with handling devices that are vhost aware and link with the host kernel. However there seems to be a growing interest in having backends implemented in separate processes, potentially even hosted in other guest VMs.
As far as I can tell there is a lot of duplicated effort in handling the low level navigation of virt queues and buffers. QEMU has code in hw/virtio as well as contrib/libvhost-user which is used by the recent virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a similar set of functionality for it's emulation. The Rust-vmm project has libraries for implementing the device traits.
Quoted 6 lines
Another aspect to this is the growing interest in carrying virtio over other hypervisors. I'm wondering if there is enough abstraction possible to have a common library that is hypervisor agnostic? Can a device backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Quoted 3 lines
It's a little tricky because it has to interface tightly with the way that the memory-mapping works for the hypervisor, so that the external process can access the memory of the queues.
Quoted 5 lines
I suspect the problem space can at least be reduced to at least a POSIX-like environment - if that makes things simpler. The setting up of memory-mappings should be the problem of the VMM, which would possibly be hypervisor specific. After that it is simply(?) a question of sharing the appropriate bit of memory between the VMM and the device process.
The 'simply(?)' is actually pretty tricky.
You have to share the mapping of all the RAM blocks in whcih the virtio queues or the data to which they point might reside, and you also have to let the other process know where in Guest physical address space they live. That mapping is also not constant, either with hotplug, or with architecture specific things that cause physical address mapping to change.
Quoted 6 lines
The other model would be the device process runs inside another guest - most likely a Linux VM. Here the guest kernel can be told an area of memory is special in some way and provide a device node that can be mmaped in more or less the same way. In this configuration it can't even be aware of what the underlying hypervisor is - just a block of memory and a way to receive message queue events.
Doing it between VMs works in my mind; but again you still need to handle that mapping.
Quoted 7 lines
Quoted 3 lines
QEMU's vhost-user has a fair amount of code for handling the mappings, dirty logging for migration, iommu's and things like reset (which is pretty hairy, and probably needs more work).
I suspect all of these multi-process models just hand wave away details like migration because that really does benefit from a single process with total awareness of the state of the system.
Vhost-user has it pretty well defined; it works - as long as the user process does dirty map update. Postcopy can also be made to work.
Quoted 2 lines
That said I wonder how
robust a guest can be if the device emulation may go away at any time?
That one I've not thought too much about, but the opposite case; making the separate process survive even when the guest behaves badly/resets/etc is quite nasty.
Dave
Quoted 9 lines
I guess in virtio if you never signal the consumption of a virt-queue it will still be there waiting until you restart the emulation process and pick up from where you left off?
Quoted 5 lines
Dave
Quoted 2 lines
Thoughts?
Quoted 12 lines
Quoted 7 lines
Quoted 7 lines
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]Signature · 1 line
Alex Bennée
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 34 lines · Dr. David Alan Gilbert
Quoted 33 lines · * Alex Bennée ([email protected]) wrote:
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Alex Bennée ([email protected]) wrote:
Hi,
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Currently most virtio backends are embedded directly in various VMMs which emulate a number of devices as well as deal with handling devices that are vhost aware and link with the host kernel. However there seems to be a growing interest in having backends implemented in separate processes, potentially even hosted in other guest VMs.
As far as I can tell there is a lot of duplicated effort in handling the low level navigation of virt queues and buffers. QEMU has code in hw/virtio as well as contrib/libvhost-user which is used by the recent virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a similar set of functionality for it's emulation. The Rust-vmm project has libraries for implementing the device traits.
Quoted 6 lines
Another aspect to this is the growing interest in carrying virtio over other hypervisors. I'm wondering if there is enough abstraction possible to have a common library that is hypervisor agnostic? Can a device backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
It's a little tricky because it has to interface tightly with the way that the memory-mapping works for the hypervisor, so that the external process can access the memory of the queues.
Quoted 7 lines
Quoted 5 lines
I suspect the problem space can at least be reduced to at least a POSIX-like environment - if that makes things simpler. The setting up of memory-mappings should be the problem of the VMM, which would possibly be hypervisor specific. After that it is simply(?) a question of sharing the appropriate bit of memory between the VMM and the device process.
The 'simply(?)' is actually pretty tricky.
Well I am at the start of this journey so may be hand waving a bit ;-)
Lets drill down:
Quoted 2 lines
You have to share the mapping of all the RAM blocks in whcih the virtio queues or the data to which they point might reside
Aren't all the queues in one section of memory?
As for where the data is doesn't this depend on the structure of the device. As I understand the behaviour of virtfs there is a direct relationship between the guest page cache and the host page cache to take advantage of DAX. This by definition means the backend needs access to the entire address space of the guest.
Is this also the case for other devices?
Quoted 5 lines
and you also have
to let the other process know where in Guest physical address space they live. That mapping is also not constant, either with hotplug, or with architecture specific things that cause physical address mapping to change.
It sounds like the solution here would be to have bounce buffers as part of the virtio spec which could be part of the virtio memory block? I guess another option is for guests to keep their internal data (as referenced by virtio drivers) in a fixed guest physical address but that gets real complicated quick and I suspect is harder to audit from a security point of view.
Quoted 21 lines
Quoted 6 lines
The other model would be the device process runs inside another guest - most likely a Linux VM. Here the guest kernel can be told an area of memory is special in some way and provide a device node that can be mmaped in more or less the same way. In this configuration it can't even be aware of what the underlying hypervisor is - just a block of memory and a way to receive message queue events.
Doing it between VMs works in my mind; but again you still need to handle that mapping.
Quoted 7 lines
Quoted 3 lines
QEMU's vhost-user has a fair amount of code for handling the mappings, dirty logging for migration, iommu's and things like reset (which is pretty hairy, and probably needs more work).
I suspect all of these multi-process models just hand wave away details like migration because that really does benefit from a single process with total awareness of the state of the system.
Vhost-user has it pretty well defined; it works - as long as the user process does dirty map update. Postcopy can also be made to work.
Quoted 6 lines
Quoted 2 lines
That said I wonder how
robust a guest can be if the device emulation may go away at any time?That one I've not thought too much about, but the opposite case; making the separate process survive even when the guest behaves badly/resets/etc is quite nasty.
I guess whatever orchestrates the start-up of the VMs has to worry about that. Some of the models I've been looking at have very simplistic setups where the guest VMs described in the platform data to be spawned directly by the hypervisor. I guess in those cases you need to restart everything.
Quoted 24 lines
Dave
Quoted 21 lines
I guess in virtio if you never signal the consumption of a virt-queue it will still be there waiting until you restart the emulation process and pick up from where you left off?
Quoted 12 lines
Dave
Quoted 9 lines
Thoughts?
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]Signature · 1 line
Alex Bennée
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 45 lines · * Alex Bennée ([email protected]) wrote:
Quoted 34 lines · Dr. David Alan Gilbert
Quoted 33 lines · * Alex Bennée ([email protected]) wrote:
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Alex Bennée ([email protected]) wrote:
Hi,
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Currently most virtio backends are embedded directly in various VMMs which emulate a number of devices as well as deal with handling devices that are vhost aware and link with the host kernel. However there seems to be a growing interest in having backends implemented in separate processes, potentially even hosted in other guest VMs.
As far as I can tell there is a lot of duplicated effort in handling the low level navigation of virt queues and buffers. QEMU has code in hw/virtio as well as contrib/libvhost-user which is used by the recent virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a similar set of functionality for it's emulation. The Rust-vmm project has libraries for implementing the device traits.
Quoted 6 lines
Another aspect to this is the growing interest in carrying virtio over other hypervisors. I'm wondering if there is enough abstraction possible to have a common library that is hypervisor agnostic? Can a device backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
It's a little tricky because it has to interface tightly with the way that the memory-mapping works for the hypervisor, so that the external process can access the memory of the queues.
Quoted 7 lines
Quoted 5 lines
I suspect the problem space can at least be reduced to at least a POSIX-like environment - if that makes things simpler. The setting up of memory-mappings should be the problem of the VMM, which would possibly be hypervisor specific. After that it is simply(?) a question of sharing the appropriate bit of memory between the VMM and the device process.
The 'simply(?)' is actually pretty tricky.
Quoted 8 lines
Well I am at the start of this journey so may be hand waving a bit ;-)
Lets drill down:
Quoted 2 lines
You have to share the mapping of all the RAM blocks in whcih the virtio queues or the data to which they point might reside
Aren't all the queues in one section of memory?
I don't think so; it's just allocated in guest RAM which can be split into multiple blocks due to NUMA etc.
Quoted 7 lines
As for where the data is doesn't this depend on the structure of the device. As I understand the behaviour of virtfs there is a direct relationship between the guest page cache and the host page cache to take advantage of DAX. This by definition means the backend needs access to the entire address space of the guest.
Is this also the case for other devices?
virtiofs's DAX shared memory is a bit different from normal virtio queues and data. Normal queues and data just live in normal guest RAM and their location is chosen by the guest.
Quoted 6 lines
Quoted 5 lines
and you also have
to let the other process know where in Guest physical address space they live. That mapping is also not constant, either with hotplug, or with architecture specific things that cause physical address mapping to change.
Quoted 6 lines
It sounds like the solution here would be to have bounce buffers as part of the virtio spec which could be part of the virtio memory block? I guess another option is for guests to keep their internal data (as referenced by virtio drivers) in a fixed guest physical address but that gets real complicated quick and I suspect is harder to audit from a security point of view.
Bounce buffers are expensive - they're used in some things (like SEV encrypted memory).
Quoted 28 lines
Quoted 21 lines
Quoted 6 lines
The other model would be the device process runs inside another guest - most likely a Linux VM. Here the guest kernel can be told an area of memory is special in some way and provide a device node that can be mmaped in more or less the same way. In this configuration it can't even be aware of what the underlying hypervisor is - just a block of memory and a way to receive message queue events.
Doing it between VMs works in my mind; but again you still need to handle that mapping.
Quoted 7 lines
Quoted 3 lines
QEMU's vhost-user has a fair amount of code for handling the mappings, dirty logging for migration, iommu's and things like reset (which is pretty hairy, and probably needs more work).
I suspect all of these multi-process models just hand wave away details like migration because that really does benefit from a single process with total awareness of the state of the system.
Vhost-user has it pretty well defined; it works - as long as the user process does dirty map update. Postcopy can also be made to work.
Quoted 6 lines
Quoted 2 lines
That said I wonder how
robust a guest can be if the device emulation may go away at any time?That one I've not thought too much about, but the opposite case; making the separate process survive even when the guest behaves badly/resets/etc is quite nasty.
Quoted 5 lines
I guess whatever orchestrates the start-up of the VMs has to worry about that. Some of the models I've been looking at have very simplistic setups where the guest VMs described in the platform data to be spawned directly by the hypervisor. I guess in those cases you need to restart everything.
That depends; the orchestrator doesn't normally see a guest reset - even a nasty one.
Dave
Quoted 21 lines
Quoted 21 lines
Dave
Quoted 18 lines
I guess in virtio if you never signal the consumption of a virt-queue it will still be there waiting until you restart the emulation process and pick up from where you left off?
Quoted 12 lines
Dave
Quoted 9 lines
Thoughts?
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Signature · 1 line
Alex BennéeSignature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 2 lines · Alex Bennée
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Yes and no.
vhost-user is currently the most widely-used (cross-VMM) interface for out-of-process VIRTIO device emulation.
However, it's not a library, just a protocol specification involving UNIX domain sockets. You mentioned the various implementations below:
Quoted 6 lines
Currently most virtio backends are embedded directly in various VMMs which emulate a number of devices as well as deal with handling devices that are vhost aware and link with the host kernel. However there seems to be a growing interest in having backends implemented in separate processes, potentially even hosted in other guest VMs.
Quoted 6 lines
As far as I can tell there is a lot of duplicated effort in handling the low level navigation of virt queues and buffers. QEMU has code in hw/virtio as well as contrib/libvhost-user which is used by the recent virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a similar set of functionality for it's emulation. The Rust-vmm project has libraries for implementing the device traits.
You missed DPDK/SPDK.
The issue is that no single library would work for all of these users.
DPDK/SPDK does poll-mode device emulation. Therefore the details are slightly different from the eventfd-based approach that event-driven implementations will use.
rust-vmm crates are written in Rust, so direct code sharing is not possible. Bindings for a C library are possible but is that desirable (it defeats the point of using a safe language, it may be harder to add features for developers mainly using Rust).
Unifying everything does not seem realistic for these sorts of reasons. What we have today is that the Linux kernel uapi headers are shared, so at least the struct and constant definitions don't need to be duplicated by C/C++ projects.
Quoted 3 lines
Another aspect to this is the growing interest in carrying virtio over other hypervisors. I'm wondering if there is enough abstraction possible to have a common library that is hypervisor agnostic? Can a device
That already exists today. Pick a vhost-user device implementation from QEMU, DPDK/SPDK, or rust-vmm and run it with your VMM of choice.
Quoted 3 lines
backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Yes, that is what vhost-user does.
Stefan
Quoted 12 lines · Stefan Hajnoczi
Quoted 2 lines · Alex Bennée
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Yes and no.
vhost-user is currently the most widely-used (cross-VMM) interface for out-of-process VIRTIO device emulation.
However, it's not a library, just a protocol specification involving UNIX domain sockets. You mentioned the various implementations below:
<snip>
Quoted 6 lines
Quoted 4 lines
backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Yes, that is what vhost-user does.
I thought so - but does any vhost-user implementation assume it has access to the entire of the guests memory space? I can see why that might be seen as undesirable from a security point of view.
So aside from virtiofsd and crosvm's various pieces for crostini emulation what other out-of-VMM device emulation daemons are you aware of?
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 21 lines · Alex Bennée
Quoted 12 lines · Stefan Hajnoczi
Quoted 2 lines · Alex Bennée
So the context of my question is what sort of common software layer is required to implement a virtio backend entirely in userspace?
Yes and no.
vhost-user is currently the most widely-used (cross-VMM) interface for out-of-process VIRTIO device emulation.
However, it's not a library, just a protocol specification involving UNIX domain sockets. You mentioned the various implementations below:
<snip>
Quoted 5 lines
Quoted 4 lines
backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Quoted 5 lines
Quoted 1 line
Yes, that is what vhost-user does.
I thought so - but does any vhost-user implementation assume it has access to the entire of the guests memory space? I can see why that might be seen as undesirable from a security point of view.
VMMs typically give the vhost-user device program access to the entirety of guest RAM. That's because unmodified VIRTIO guest drivers typically require access to all of guest memory.
If the guest software cooperates then it's possible to reduce the shared memory region. In other words, the vhost-user protocol doesn't demand that guest RAM is shared, it just requires all addresses to be within the shared memory region, whatever that may be.
Quoted 3 lines
So aside from virtiofsd and crosvm's various pieces for crostini emulation what other out-of-VMM device emulation daemons are you aware of?
I haven't checked but my understanding was that crosvm uses its own out-of-process device interface (not vhost-user).
The previously mentioned vhost-user ones were DPDK
(https://www.dpdk.org/) and SPDK (https://spdk.io/).
Sergio Lopez has implemented vhost-user-blk in Rust:
https://github.com/slp/qsd
I think Nutanix is also using vhost-user, but am not sure and I don't know if that is open source.
There are probably more that I'm not aware of but a web search would turn up.
Stefan
Quoted 4 lines · On Wed, 11 Mar 2020 17:24:11 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 3 lines
I thought so - but does any vhost-user implementation assume it has access to the entire of the guests memory space? I can see why that might be seen as undesirable from a security point of view.
On s390 we do assume we have access to the whole guest memory space if VIRTIO_F_ACCESS_PLATFORM is not set.
Quoted 8 lines
VMMs typically give the vhost-user device program access to the entirety of guest RAM. That's because unmodified VIRTIO guest drivers typically require access to all of guest memory.
If the guest software cooperates then it's possible to reduce the shared memory region. In other words, the vhost-user protocol doesn't demand that guest RAM is shared, it just requires all addresses to be within the shared memory region, whatever that may be.
In that case, I guess, we would have to negotiate VIRTIO_F_ACCESS_PLATFORM. vhost-user already has support for VIRTIO_F_ACCESS_PLATFORM, but in context of vhost-user it means: do IOVA translation. Translation is basically boils down to IOMMU/IOTLB support, and the specs (docs/interop/vhost-user.rst) says that the translated stuff needs to be within a region set via VHOST_USER_SET_MEM_TABLE.
I'm wondering how restricting the memory region shared vhost-user program should work, unless we impose restrictions on what physical addresses may be used for virtio buffers (regardless of whether what lands in the virtqueue is an IOVA or a GPA), and make everybody in the stack aware of this.
Of course sharing the whole guest memory space with the vhost-user program does not necessarily mean the guest can actually access the whole guest memory. For example memory encryption or other means of protecting memory (e.g. s390 protected VMs) may ensure that the vhost-user program can not mess with anything that in is not supposed to.
Regards,
Halil
Quoted 9 lines · Stefan Hajnoczi
That already exists today. Pick a vhost-user device implementation from QEMU, DPDK/SPDK, or rust-vmm and run it with your VMM of choice.
Quoted 3 lines
backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Yes, that is what vhost-user does.
Are there vhost-user based backends that emulates block and console devices?
From my quick read, vhost-user seems to depend on having backend complete access to front-end's memory. Is that true? We are looking for a backend solution that does not assume access to whole of front-end memory. Only the memory regions hosting IO buffers need to be ideally shared between the two VMs. Also the mechanism of memory sharing and signalling between VMs could be very hypervisor specific.
Does vhost-user allow for such hyp-specific memory sharing and signalling interfaces?
- vatsa
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 9 lines · Srivatsa Vaddagiri
Quoted 8 lines · Stefan Hajnoczi
That already exists today. Pick a vhost-user device implementation from QEMU, DPDK/SPDK, or rust-vmm and run it with your VMM of choice.
Quoted 3 lines
backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Quoted 8 lines
Quoted 1 line
Yes, that is what vhost-user does.
Are there vhost-user based backends that emulates block and console devices?
From my quick read, vhost-user seems to depend on having backend complete access to front-end's memory. Is that true? We are looking for a backend solution that does not assume access to whole of front-end memory. Only the memory regions hosting IO buffers need to be ideally shared between the two VMs.
I think it currently is true because generally the pointers passed in the virtqueues are directly into the various internal structures of the kernel (e.g. pagecache, skb etc.).
Now for linux the are ways of declaring particular buffers to be in specific locations - ostensibly for things like DMA-ability. But AFAICT there is no mechanism to communicate the particular locations to be used to the VMM and backend daemon.
Quoted 5 lines
Also the
mechanism of memory sharing and signalling between VMs could be very hypervisor specific.
Does vhost-user allow for such hyp-specific memory sharing and signalling interfaces?
The daemons I've looked at so far are passed a shm fd and a control fd from the VMM. From their point of view it's just posix shared memory and a event queue via a socket. Setting that up is the VMMs problem.
Quoted 2 lines
- vatsa
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 21 lines · Alex Bennée
Quoted 9 lines · Srivatsa Vaddagiri
Quoted 8 lines · Stefan Hajnoczi
That already exists today. Pick a vhost-user device implementation from QEMU, DPDK/SPDK, or rust-vmm and run it with your VMM of choice.
Quoted 3 lines
backend be emulated purely with some shared memory and some sockets for passing messages/kicks from/to the VMM which then deals with the hypervisor specifics of the virtio-transport?
Quoted 8 lines
Quoted 1 line
Yes, that is what vhost-user does.
Are there vhost-user based backends that emulates block and console devices?
From my quick read, vhost-user seems to depend on having backend complete access to front-end's memory. Is that true? We are looking for a backend solution that does not assume access to whole of front-end memory. Only the memory regions hosting IO buffers need to be ideally shared between the two VMs.
Quoted 8 lines
I think it currently is true because generally the pointers passed in the virtqueues are directly into the various internal structures of the kernel (e.g. pagecache, skb etc.).
Now for linux the are ways of declaring particular buffers to be in specific locations - ostensibly for things like DMA-ability. But AFAICT there is no mechanism to communicate the particular locations to be used to the VMM and backend daemon.
Yes, the problem is that guest userspace memory buffers could be anywhere and the virtqueue DMA memory managed by the guest drivers is also not allocated from a specific memory range.
When the guest cooperates (e.g. DPDK applications that use a fixed memory range for virtio-net packet buffers), then it's possible to reduce the shared memory region to not expose all of guest RAM.
Quoted 9 lines
Quoted 5 lines
Also the
mechanism of memory sharing and signalling between VMs could be very hypervisor specific.
Does vhost-user allow for such hyp-specific memory sharing and signalling interfaces?The daemons I've looked at so far are passed a shm fd and a control fd from the VMM. From their point of view it's just posix shared memory and a event queue via a socket. Setting that up is the VMMs problem.
Yes, although I don't think there is any assumption that the fd is POSIX shared memory. The fd just needs to support mmap(2).
Stefan