Enabling hypervisor agnosticism for VirtIO backends
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============
In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest. Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space. To activate the mapping will require some sort of hypercall to the hypervisor. I can see two options at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
CCing people working on Xen+VirtIO and IOREQs. Not trimming the original email to let them read the full context.
My comments below are related to a potential Xen implementation, not because it is the only implementation that matters, but because it is the one I know best.
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 63 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to give to the backend if the backend is not running in dom0. So for Xen I think the memory has to be already mapped and the mapping probably done by the toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allow this kind of mapping.
If we run the backend in Dom0 that we have no problems of course.
Quoted 21 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see two options at this point:- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========
Quoted 11 lines
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs the way they are typically envisioned and described in other environments. Instead, Xen has IOREQ servers. Each of them connects independently to Xen via the IOREQ interface. E.g. today multiple QEMUs could be used as emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
The component responsible for starting a daemon and/or setting up shared interfaces is the toolstack: the xl command and the libxl/libxc libraries.
Oleksandr and others I CCed have been working on ways for the toolstack to create virtio backends and setup memory mappings. They might be able to provide more info on the subject. I do think we miss a way to provide the configuration to the backend and anything else that the backend might require to start doing its job.
Quoted 5 lines
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
I don't think that translating IOREQs to eventfd in the kernel is a good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor interface. Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic, perhaps we could design a new interface. One that could be implementable in the Xen hypervisor itself (like IOREQ) and of course any other hypervisor too.
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 76 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest. Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space. To activate the mapping will require some sort of hypercall to the hypervisor. I can see two options at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
Quoted 6 lines
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
VIRTIO typically uses the vring memory layout but doesn't need to. The VIRTIO device model deals with virtqueues. The shared memory vring layout is part of the VIRTIO transport (PCI, MMIO, and CCW use vrings). Alternative transports with other virtqueue representations are possible (e.g. VIRTIO-over-TCP). They don't need to involve a BE mapping shared memory and processing a vring owned by the FE.
For example, there could be BE hypercalls to pop a virtqueue elements, push a virtqueue elements, and to access buffers (basically DMA read/write). The FE could either be a traditional virtio-mmio/pci device with a vring or use FE hypercalls to add available elements to a virtqueue and get used elements.
I don't know the goals of project Stratos or whether this helps, but it might allow other architectures that have different security, complexity, etc properties.
Stefan
Quoted 6 lines · Stefano Stabellini
CCing people working on Xen+VirtIO and IOREQs. Not trimming the original email to let them read the full context.
My comments below are related to a potential Xen implementation, not because it is the only implementation that matters, but because it is the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and particularly EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.
So I hope you like my approach :)
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000546.html
Let me take this opportunity to explain a bit more about my approach below.
Quoted 4 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 63 lines
Quoted 62 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared memory" between VMs. I think that Jailhouse's ivshmem gives us good insights on this matter and that it can even be an alternative for hypervisor-agnostic solution.
(Please note memory regions in ivshmem appear as a PCI device and can be mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 8 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to give to the backend if the backend is not running in dom0. So for Xen I think the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information is expected to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.
In my approach (virtio-proxy), all those Xen (or hypervisor)-specific stuffs are contained in virtio-proxy, yet another VM, to hide all details.
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 6 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allow this kind of mapping.If we run the backend in Dom0 that we have no problems of course.
One of difficulties on Xen that I found in my approach is that calling such hypervisor intefaces (registering IOREQ, mapping memory) is only allowed on BE servers themselvies and so we will have to extend those interfaces.
This, however, will raise some concern on security and privilege distribution as Stefan suggested.
Quoted 42 lines
Quoted 23 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see two options at this point:- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========Quoted 11 lines
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs the way they are typically envisioned and described in other environments. Instead, Xen has IOREQ servers. Each of them connects independently to Xen via the IOREQ interface. E.g. today multiple QEMUs could be used as emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
Quoted 3 lines
The component responsible for starting a daemon and/or setting up shared interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in Startos jargon?) is a subject to debate in parallel.
Otherwise, is there any good assumption to avoid it right now?
Quoted 7 lines
Oleksandr and others I CCed have been working on ways for the toolstack to create virtio backends and setup memory mappings. They might be able to provide more info on the subject. I do think we miss a way to provide the configuration to the backend and anything else that the backend might require to start doing its job.
Quoted 8 lines
Quoted 5 lines
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisor interface via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels), which is converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by interrupts.)
I don't know what "connect directly" means here, but sending interrupts to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-x mechanism.
Quoted 10 lines
Quoted 6 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add.
As I said above, my proposal does the same thing that you mentioned here :) The difference is that I do call hypervisor interfaces via virtio-proxy.
Quoted 8 lines
The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52I don't think that translating IOREQs to eventfd in the kernel is a good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor interface.
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).
Quoted 8 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic, perhaps we could design a new interface. One that could be implementable in the Xen hypervisor itself (like IOREQ) and of course any other hypervisor too.
Quoted 6 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
My proposal document might help here; All the interfaces required for virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 4 lines
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is a different matter.
Thanks,
-Takahiro Akashi
Hi François,
Quoted 1 line · Fran??ois Ozog
I top post as I find it difficult to identify where to make the comments.
Thank you for the posting.
I think that we should first discuss more about the goal/requirements/ practical use cases for the framework.
Quoted 8 lines
1) BE acceleration
Network and storage backends may actually be executed in SmartNICs. As virtio 1.1 is hardware friendly, there may be SmartNICs with virtio 1.1 PCI VFs. Is it a valid use case for the generic BE framework to be used in this context?
DPDK is used in some BE to significantly accelerate switching. DPDK is also used sometimes in guests. In that case, there are no event injection but just high performance memory scheme. Is this considered as a use case?
I'm not quite familiar with DPDK but it seems to be heavily reliant on not only virtqueues but also kvm/linux features/functionality, say, according to [1].
I'm afraid that DPDK is not suitable for primary (at least, initial) target use.
# In my proposal, virtio-proxy, I have in mind the assumption that we would
# create BE VM as a baremetal application on RTOS (and/or unikernel.)
But as far as virtqueue is concerned, I think we can discuss in general technical details as Alex suggested, including:
- sharing or mapping memory regions for data payload
- efficient notification mechanism
[1] https://www.redhat.com/en/blog/journey-vhost-users-realm
Quoted 5 lines
2) Virtio as OS HAL
Panasonic CTO has been calling for a virtio based HAL and based on the teachings of Google GKI, an internal HAL seem inevitable in the long term. Virtio is then a contender to Google promoted Android HAL. Could the framework be used in that context?
In this case, where will the implementation of "HAL" reside? I don't think the portability of "HAL" code (as a set of virtio BEs) is a requirement here.
-Takahiro Akashi
Quoted 298 lines
Quoted 289 lines · AKASHI Takahiro via Stratos-dev
Quoted 6 lines · Stefano Stabellini
CCing people working on Xen+VirtIO and IOREQs. Not trimming the original email to let them read the full context.
My comments below are related to a potential Xen implementation, not because it is the only implementation that matters, but because it is the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and particularly EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.So I hope you like my approach :)
[1]
https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000546.htmlLet me take this opportunity to explain a bit more about my approach below.
Quoted 4 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 15 lines
Quoted 14 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was
a
Quoted 26 lines
Quoted 26 lines
stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
Quoted 6 lines
Quoted 6 lines
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images
however
Quoted 9 lines
Quoted 9 lines
this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 5 lines
Quoted 5 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would
assume
Quoted 2 lines
Quoted 2 lines
in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared memory" between
VMs. I think that Jailhouse's ivshmem gives us good insights on this matter and that it can even be an alternative for hypervisor-agnostic solution.(Please note memory regions in ivshmem appear as a PCI device and can be mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 8 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to give to the backend if the backend is not running in dom0. So for Xen I think the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information is expected to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.In my approach (virtio-proxy), all those Xen (or hypervisor)-specific stuffs are contained in virtio-proxy, yet another VM, to hide all details.
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 6 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allow this kind of mapping.If we run the backend in Dom0 that we have no problems of course.
One of difficulties on Xen that I found in my approach is that calling such hypervisor intefaces (registering IOREQ, mapping memory) is only allowed on BE servers themselvies and so we will have to extend those interfaces.
This, however, will raise some concern on security and privilege distribution
as Stefan suggested.Quoted 4 lines
Quoted 4 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see twooptions
Quoted 38 lines
Quoted 19 lines
at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========Quoted 11 lines
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs the way they are typically envisioned and described in other environments. Instead, Xen has IOREQ servers. Each of them connects independently to Xen via the IOREQ interface. E.g. today multiple QEMUs could be used as emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
Quoted 3 lines
The component responsible for starting a daemon and/or setting up shared interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in Startos jargon?) is a subject to debate in parallel.
Otherwise, is there any good assumption to avoid it right now?Quoted 7 lines
Oleksandr and others I CCed have been working on ways for the toolstack to create virtio backends and setup memory mappings. They might be able to provide more info on the subject. I do think we miss a way to provide the configuration to the backend and anything else that the backend might require to start doing its job.
Quoted 8 lines
Quoted 5 lines
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisor interface via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels), which is converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)I don't know what "connect directly" means here, but sending interrupts to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-x mechanism.Quoted 3 lines
Quoted 3 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other
kernel
Quoted 7 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add.
As I said above, my proposal does the same thing that you mentioned here :) The difference is that I do call hypervisor interfaces via virtio-proxy.
Quoted 4 lines
The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 4 lines
I don't think that translating IOREQs to eventfd in the kernel is a good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor interface.
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 8 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic, perhaps we could design a new interface. One that could be implementable in the Xen hypervisor itself (like IOREQ) and of course any other hypervisor too.
Quoted 6 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.My proposal document might help here; All the interfaces required for virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 4 lines
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is a different matter.Thanks,
-Takahiro Akashi--
Stratos-dev mailing list
[email protected]
https://op-lists.linaro.org/mailman/listinfo/stratos-devSignature · 3 lines
François-Frédéric Ozog | *Director Business Development* T: +33.67221.6485 [email protected] | Skype: ffozog
Hello, all.
Please see some comments below. And sorry for the possible format issues.
Quoted 113 lines · AKASHI Takahiro
Quoted 6 lines · Stefano Stabellini
CCing people working on Xen+VirtIO and IOREQs. Not trimming the original email to let them read the full context.
My comments below are related to a potential Xen implementation, not because it is the only implementation that matters, but because it is the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and particularly EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.So I hope you like my approach :)
[1]
https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000546.htmlLet me take this opportunity to explain a bit more about my approach below.
Quoted 4 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 15 lines
Quoted 14 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was
a
Quoted 26 lines
Quoted 26 lines
stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
Quoted 6 lines
Quoted 6 lines
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images
however
Quoted 9 lines
Quoted 9 lines
this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 5 lines
Quoted 5 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would
assume
Quoted 2 lines
Quoted 2 lines
in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared memory" between
VMs. I think that Jailhouse's ivshmem gives us good insights on this matter and that it can even be an alternative for hypervisor-agnostic solution.(Please note memory regions in ivshmem appear as a PCI device and can be mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 8 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to give to the backend if the backend is not running in dom0. So for Xen I think the memory has to be already mapped
Quoted 10 lines
In Xen's IOREQ solution (virtio-blk), the following information is expected to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.
Yes, Xenstore was chosen as a simple way to pass configuration info to the backend running in a non-toolstack domain.
I remember, there was a wish to avoid using Xenstore in Virtio backend itself if possible, so for non-toolstack domain, this could done with adjusting devd (daemon that listens for devices and launches backends) to read backend configuration from the Xenstore anyway and pass it to the backend via command line arguments.
But, if ...
Quoted 4 lines
In my approach (virtio-proxy), all those Xen (or hypervisor)-specific stuffs are contained in virtio-proxy, yet another VM, to hide all details.
... the solution how to overcome that is already found and proven to work then even better.
Quoted 10 lines
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 6 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allow this kind of mapping.If we run the backend in Dom0 that we have no problems of course.
Quoted 8 lines
One of difficulties on Xen that I found in my approach is that calling such hypervisor intefaces (registering IOREQ, mapping memory) is only allowed on BE servers themselvies and so we will have to extend those interfaces.
This, however, will raise some concern on security and privilege distribution
as Stefan suggested.
We also faced policy related issues with Virtio backend running in other than Dom0 domain in a "dummy" xsm mode. In our target system we run the backend in a driver
domain (we call it DomD) where the underlying H/W resides. We trust it, so we wrote policy rules (to be used in "flask" xsm mode) to provide it with a little bit more privileges than a simple DomU had.
Now it is permitted to issue device-model, resource and memory mappings, etc calls.
Quoted 43 lines
Quoted 4 lines
Quoted 4 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see twooptions
Quoted 38 lines
Quoted 19 lines
at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========Quoted 11 lines
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs the way they are typically envisioned and described in other environments. Instead, Xen has IOREQ servers. Each of them connects independently to Xen via the IOREQ interface. E.g. today multiple QEMUs could be used as emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
Quoted 14 lines
Quoted 3 lines
The component responsible for starting a daemon and/or setting up shared interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in Startos jargon?) is a subject to debate in parallel.
Otherwise, is there any good assumption to avoid it right now?Quoted 5 lines
Oleksandr and others I CCed have been working on ways for the toolstack to create virtio backends and setup memory mappings. They might be able to provide more info on the subject. I do think we miss a way to provide the configuration to the backend and anything else that the backend might require to start doing its job.
Yes, some work has been done for the toolstack to handle Virtio MMIO devices in
general and Virtio block devices in particular. However, it has not been upstreaned yet.
Updated patches on review now:
https://lore.kernel.org/xen-devel/[email protected]/
There is an additional (also important) activity to improve/fix foreign memory mapping on Arm which I am also involved in.
The foreign memory mapping is proposed to be used for Virtio backends (device emulators) if there is a need to run guest OS completely unmodified. Of course, the more secure way would be to use grant memory mapping. Brietly, the main difference between them is that with foreign mapping the backend
can map any guest memory it wants to map, but with grant mapping it is allowed to map only what was previously granted by the frontend.
So, there might be a problem if we want to pre-map some guest memory in advance or to cache mappings in the backend in order to improve performance (because the mapping/unmapping guest pages every request requires a lot of back and forth to Xen + P2M updates). In a nutshell, currently, in order to map a guest page into the backend address space we need to steal a real physical page from the backend domain. So, with the said optimizations we might end up with no free memory in the backend domain (see XSA-300). And what we try to achieve is to not waste a real domain memory at all by providing safe non-allocated-yet (so unused) address space for the foreign (and grant) pages to be mapped into, this enabling work implies Xen and Linux (and likely DTB bindings) changes. However, as it turned out, for this to work in a proper and safe way some prereq work needs to be done. You can find the related Xen discussion at:
https://lore.kernel.org/xen-devel/[email protected]/
Quoted 19 lines
Quoted 10 lines
Quoted 7 lines
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisor interface via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels), which is converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)
Quoted 4 lines
I don't know what "connect directly" means here, but sending interrupts to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-x mechanism.
Agree that MSI would be more efficient than SPI...
At the moment, in order to notify the frontend, the backend issues a specific device-model call to query Xen to inject a corresponding SPI to the guest.
Quoted 65 lines
Quoted 4 lines
Quoted 4 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other
kernel
Quoted 7 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add.
As I said above, my proposal does the same thing that you mentioned here :) The difference is that I do call hypervisor interfaces via virtio-proxy.
Quoted 4 lines
The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 4 lines
I don't think that translating IOREQs to eventfd in the kernel is a good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor interface.
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 8 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic, perhaps we could design a new interface. One that could be implementable in the Xen hypervisor itself (like IOREQ) and of course any other hypervisor too.
Quoted 6 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.My proposal document might help here; All the interfaces required for virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 4 lines
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is a different matter.Thanks,
-Takahiro Akashi
Signature · 3 lines
Regards, Oleksandr Tyshchenko
I top post as I find it difficult to identify where to make the comments.
1) BE acceleration
Network and storage backends may actually be executed in SmartNICs. As virtio 1.1 is hardware friendly, there may be SmartNICs with virtio 1.1 PCI VFs. Is it a valid use case for the generic BE framework to be used in this context?
DPDK is used in some BE to significantly accelerate switching. DPDK is also used sometimes in guests. In that case, there are no event injection but just high performance memory scheme. Is this considered as a use case?
2) Virtio as OS HAL
Panasonic CTO has been calling for a virtio based HAL and based on the teachings of Google GKI, an internal HAL seem inevitable in the long term. Virtio is then a contender to Google promoted Android HAL. Could the framework be used in that context?
Quoted 289 lines · AKASHI Takahiro via Stratos-dev
Quoted 6 lines · Stefano Stabellini
CCing people working on Xen+VirtIO and IOREQs. Not trimming the original email to let them read the full context.
My comments below are related to a potential Xen implementation, not because it is the only implementation that matters, but because it is the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and particularly EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.So I hope you like my approach :)
[1]
https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000546.htmlLet me take this opportunity to explain a bit more about my approach below.
Quoted 4 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 15 lines
Quoted 14 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was
a
Quoted 26 lines
Quoted 26 lines
stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
Quoted 6 lines
Quoted 6 lines
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images
however
Quoted 9 lines
Quoted 9 lines
this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 5 lines
Quoted 5 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would
assume
Quoted 2 lines
Quoted 2 lines
in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared memory" between
VMs. I think that Jailhouse's ivshmem gives us good insights on this matter and that it can even be an alternative for hypervisor-agnostic solution.(Please note memory regions in ivshmem appear as a PCI device and can be mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 8 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to give to the backend if the backend is not running in dom0. So for Xen I think the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information is expected to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.In my approach (virtio-proxy), all those Xen (or hypervisor)-specific stuffs are contained in virtio-proxy, yet another VM, to hide all details.
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 6 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allow this kind of mapping.If we run the backend in Dom0 that we have no problems of course.
One of difficulties on Xen that I found in my approach is that calling such hypervisor intefaces (registering IOREQ, mapping memory) is only allowed on BE servers themselvies and so we will have to extend those interfaces.
This, however, will raise some concern on security and privilege distribution
as Stefan suggested.Quoted 4 lines
Quoted 4 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see twooptions
Quoted 38 lines
Quoted 19 lines
at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========Quoted 11 lines
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs the way they are typically envisioned and described in other environments. Instead, Xen has IOREQ servers. Each of them connects independently to Xen via the IOREQ interface. E.g. today multiple QEMUs could be used as emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
Quoted 3 lines
The component responsible for starting a daemon and/or setting up shared interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in Startos jargon?) is a subject to debate in parallel.
Otherwise, is there any good assumption to avoid it right now?Quoted 7 lines
Oleksandr and others I CCed have been working on ways for the toolstack to create virtio backends and setup memory mappings. They might be able to provide more info on the subject. I do think we miss a way to provide the configuration to the backend and anything else that the backend might require to start doing its job.
Quoted 8 lines
Quoted 5 lines
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisor interface via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels), which is converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)I don't know what "connect directly" means here, but sending interrupts to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-x mechanism.Quoted 3 lines
Quoted 3 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other
kernel
Quoted 7 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add.
As I said above, my proposal does the same thing that you mentioned here :) The difference is that I do call hypervisor interfaces via virtio-proxy.
Quoted 4 lines
The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 4 lines
I don't think that translating IOREQs to eventfd in the kernel is a good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor interface.
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 8 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic, perhaps we could design a new interface. One that could be implementable in the Xen hypervisor itself (like IOREQ) and of course any other hypervisor too.
Quoted 6 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.My proposal document might help here; All the interfaces required for virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 4 lines
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is a different matter.Thanks,
-Takahiro Akashi--
Stratos-dev mailing list
[email protected]
https://op-lists.linaro.org/mailman/listinfo/stratos-dev
Signature · 3 lines
François-Frédéric Ozog | *Director Business Development* T: +33.67221.6485 [email protected] | Skype: ffozog
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device implementations.
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
Quoted 128 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; Stefano Stabellini <[email protected]>
Cc: Alex Benn??e <[email protected]>; Stratos Mailing List <[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean-Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHello, all.
Please see some comments below. And sorry for the possible format issues.
Quoted 108 lines · AKASHI Takahiro
Quoted 6 lines · Stefano Stabellini
CCing people working on Xen+VirtIO and IOREQs. Not trimming the original email to let them read the full context.
My comments below are related to a potential Xen implementation, not because it is the only implementation that matters, but because it is the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and particularly EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.So I hope you like my approach :)
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000546.html
Let me take this opportunity to explain a bit more about my approach below.
Quoted 4 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 63 lines
Quoted 62 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared memory" between VMs. I think that Jailhouse's ivshmem gives us good insights on this matter and that it can even be an alternative for hypervisor-agnostic solution.
(Please note memory regions in ivshmem appear as a PCI device and can be mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 8 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to give to the backend if the backend is not running in dom0. So for Xen I think the memory has to be already mapped
Quoted 9 lines
In Xen's IOREQ solution (virtio-blk), the following information is expected to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.
Quoted 4 lines
Yes, Xenstore was chosen as a simple way to pass configuration info to the backend running in a non-toolstack domain.
I remember, there was a wish to avoid using Xenstore in Virtio backend itself if possible, so for non-toolstack domain, this could done with adjusting devd (daemon that listens for devices and launches backends) to read backend configuration from the Xenstore anyway and pass it to the backend via command line arguments.
Yes, in current PoC code we're using xenstore to pass device configuration. We also designed a static device configuration parse method for Dom0less or other scenarios don't have xentool. yes, it's from device model command line or a config file.
Quoted 191 lines
But, if ...
Quoted 3 lines
In my approach (virtio-proxy), all those Xen (or hypervisor)-specific stuffs are contained in virtio-proxy, yet another VM, to hide all details.
... the solution how to overcome that is already found and proven to work then even better.
Quoted 7 lines
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 4 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allow this kind of mapping.Quoted 8 lines
Quoted 1 line
If we run the backend in Dom0 that we have no problems of course.
One of difficulties on Xen that I found in my approach is that calling such hypervisor intefaces (registering IOREQ, mapping memory) is only allowed on BE servers themselvies and so we will have to extend those interfaces.
This, however, will raise some concern on security and privilege distribution as Stefan suggested.We also faced policy related issues with Virtio backend running in other than Dom0 domain in a "dummy" xsm mode. In our target system we run the backend in a driver domain (we call it DomD) where the underlying H/W resides. We trust it, so we wrote policy rules (to be used in "flask" xsm mode) to provide it with a little bit more privileges than a simple DomU had. Now it is permitted to issue device-model, resource and memory mappings, etc calls.
Quoted 42 lines
Quoted 42 lines
Quoted 23 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see two options at this point:- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========Quoted 11 lines
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs the way they are typically envisioned and described in other environments. Instead, Xen has IOREQ servers. Each of them connects independently to Xen via the IOREQ interface. E.g. today multiple QEMUs could be used as emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
Quoted 13 lines
Quoted 3 lines
The component responsible for starting a daemon and/or setting up shared interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in Startos jargon?) is a subject to debate in parallel.
Otherwise, is there any good assumption to avoid it right now?Quoted 5 lines
Oleksandr and others I CCed have been working on ways for the toolstack to create virtio backends and setup memory mappings. They might be able to provide more info on the subject. I do think we miss a way to provide the configuration to the backend and anything else that the backend might require to start doing its job.
Yes, some work has been done for the toolstack to handle Virtio MMIO devices in general and Virtio block devices in particular. However, it has not been upstreaned yet. Updated patches on review now:
https://lore.kernel.org/xen-devel/[email protected]/There is an additional (also important) activity to improve/fix foreign memory mapping on Arm which I am also involved in.
The foreign memory mapping is proposed to be used for Virtio backends (device emulators) if there is a need to run guest OS completely unmodified. Of course, the more secure way would be to use grant memory mapping. Brietly, the main difference between them is that with foreign mapping the backend can map any guest memory it wants to map, but with grant mapping it is allowed to map only what was previously granted by the frontend.So, there might be a problem if we want to pre-map some guest memory in advance or to cache mappings in the backend in order to improve performance (because the mapping/unmapping guest pages every request requires a lot of back and forth to Xen + P2M updates). In a nutshell, currently, in order to map a guest page into the backend address space we need to steal a real physical page from the backend domain. So, with the said optimizations we might end up with no free memory in the backend domain (see XSA-300). And what we try to achieve is to not waste a real domain memory at all by providing safe non-allocated-yet (so unused) address space for the foreign (and grant) pages to be mapped into, this enabling work implies Xen and Linux (and likely DTB bindings) changes. However, as it turned out, for this to work in a proper and safe way some prereq work needs to be done. You can find the related Xen discussion at:
https://lore.kernel.org/xen-devel/[email protected]/Quoted 18 lines
Quoted 10 lines
Quoted 7 lines
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisor interface via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels), which is converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by interrupts.)Quoted 3 lines
I don't know what "connect directly" means here, but sending interrupts to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-x mechanism.Agree that MSI would be more efficient than SPI...
At the moment, in order to notify the frontend, the backend issues a specific device-model call to query Xen to inject a corresponding SPI to the guest.Quoted 53 lines
Quoted 10 lines
Quoted 6 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add.
As I said above, my proposal does the same thing that you mentioned here :) The difference is that I do call hypervisor interfaces via virtio-proxy.
Quoted 8 lines
The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52I don't think that translating IOREQs to eventfd in the kernel is a good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor interface.
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 8 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic, perhaps we could design a new interface. One that could be implementable in the Xen hypervisor itself (like IOREQ) and of course any other hypervisor too.
Quoted 6 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.My proposal document might help here; All the interfaces required for virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 4 lines
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 7 lines
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is a different matter.Thanks,
-Takahiro Akashi
Quoted 4 lines
--
Regards,Oleksandr Tyshchenko
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Wei, Oleksandr,
Quoted 10 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device implementations.
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always represent # Linaro's view on this subject nor reflect Alex's concerns. Nevertheless,
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first, Xen and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces has a same set of interfaces (for your "struct vmm_impl" and my "RPC interfaces"). This is not co-incident as we both share the same origin as I said above. And so we will also share the same issues. One of them is a way of "sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too.
(See Alex's original email, too).
On the other hand, my approach aims to create a "single-binary" solution in which the same binary of BE vm could run on any hypervisors. Somehow similar to your "proposal-#2" in [2], but in my solution, all the hypervisor-specific code would be put into another entity (VM), named "virtio-proxy" and the abstracted operations are served via RPC. (In this sense, BE is hypervisor-agnostic but might have OS dependency.) But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Specifically speaking about kvm-tool, I have a concern about its license term; Targeting different hypervisors and different OSs (which I assume includes RTOS's), the resultant library should be license permissive and GPL for kvm-tool might be an issue. Any thoughts?
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 335 lines
Quoted 129 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; Stefano Stabellini <[email protected]>
Cc: Alex Benn??e <[email protected]>; Stratos Mailing List <[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean-Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHello, all.
Please see some comments below. And sorry for the possible format issues.
Quoted 108 lines · AKASHI Takahiro
Quoted 6 lines · Stefano Stabellini
CCing people working on Xen+VirtIO and IOREQs. Not trimming the original email to let them read the full context.
My comments below are related to a potential Xen implementation, not because it is the only implementation that matters, but because it is the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and particularly EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.So I hope you like my approach :)
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000546.html
Let me take this opportunity to explain a bit more about my approach below.
Quoted 4 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 63 lines
Quoted 62 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared memory" between VMs. I think that Jailhouse's ivshmem gives us good insights on this matter and that it can even be an alternative for hypervisor-agnostic solution.
(Please note memory regions in ivshmem appear as a PCI device and can be mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 8 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to give to the backend if the backend is not running in dom0. So for Xen I think the memory has to be already mapped
Quoted 9 lines
In Xen's IOREQ solution (virtio-blk), the following information is expected to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.Quoted 4 lines
Yes, Xenstore was chosen as a simple way to pass configuration info to the backend running in a non-toolstack domain.
I remember, there was a wish to avoid using Xenstore in Virtio backend itself if possible, so for non-toolstack domain, this could done with adjusting devd (daemon that listens for devices and launches backends) to read backend configuration from the Xenstore anyway and pass it to the backend via command line arguments.Yes, in current PoC code we're using xenstore to pass device configuration. We also designed a static device configuration parse method for Dom0less or other scenarios don't have xentool. yes, it's from device model command line or a config file.
Quoted 191 lines
But, if ...
Quoted 3 lines
In my approach (virtio-proxy), all those Xen (or hypervisor)-specific stuffs are contained in virtio-proxy, yet another VM, to hide all details.
... the solution how to overcome that is already found and proven to work then even better.
Quoted 7 lines
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 4 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allow this kind of mapping.Quoted 8 lines
Quoted 1 line
If we run the backend in Dom0 that we have no problems of course.
One of difficulties on Xen that I found in my approach is that calling such hypervisor intefaces (registering IOREQ, mapping memory) is only allowed on BE servers themselvies and so we will have to extend those interfaces.
This, however, will raise some concern on security and privilege distribution as Stefan suggested.We also faced policy related issues with Virtio backend running in other than Dom0 domain in a "dummy" xsm mode. In our target system we run the backend in a driver domain (we call it DomD) where the underlying H/W resides. We trust it, so we wrote policy rules (to be used in "flask" xsm mode) to provide it with a little bit more privileges than a simple DomU had. Now it is permitted to issue device-model, resource and memory mappings, etc calls.
Quoted 42 lines
Quoted 42 lines
Quoted 23 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see two options at this point:- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========Quoted 11 lines
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs the way they are typically envisioned and described in other environments. Instead, Xen has IOREQ servers. Each of them connects independently to Xen via the IOREQ interface. E.g. today multiple QEMUs could be used as emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
Quoted 13 lines
Quoted 3 lines
The component responsible for starting a daemon and/or setting up shared interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in Startos jargon?) is a subject to debate in parallel.
Otherwise, is there any good assumption to avoid it right now?Quoted 5 lines
Oleksandr and others I CCed have been working on ways for the toolstack to create virtio backends and setup memory mappings. They might be able to provide more info on the subject. I do think we miss a way to provide the configuration to the backend and anything else that the backend might require to start doing its job.
Yes, some work has been done for the toolstack to handle Virtio MMIO devices in general and Virtio block devices in particular. However, it has not been upstreaned yet. Updated patches on review now:
https://lore.kernel.org/xen-devel/[email protected]/There is an additional (also important) activity to improve/fix foreign memory mapping on Arm which I am also involved in.
The foreign memory mapping is proposed to be used for Virtio backends (device emulators) if there is a need to run guest OS completely unmodified. Of course, the more secure way would be to use grant memory mapping. Brietly, the main difference between them is that with foreign mapping the backend can map any guest memory it wants to map, but with grant mapping it is allowed to map only what was previously granted by the frontend.So, there might be a problem if we want to pre-map some guest memory in advance or to cache mappings in the backend in order to improve performance (because the mapping/unmapping guest pages every request requires a lot of back and forth to Xen + P2M updates). In a nutshell, currently, in order to map a guest page into the backend address space we need to steal a real physical page from the backend domain. So, with the said optimizations we might end up with no free memory in the backend domain (see XSA-300). And what we try to achieve is to not waste a real domain memory at all by providing safe non-allocated-yet (so unused) address space for the foreign (and grant) pages to be mapped into, this enabling work implies Xen and Linux (and likely DTB bindings) changes. However, as it turned out, for this to work in a proper and safe way some prereq work needs to be done. You can find the related Xen discussion at:
https://lore.kernel.org/xen-devel/[email protected]/Quoted 18 lines
Quoted 10 lines
Quoted 7 lines
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisor interface via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels), which is converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by interrupts.)Quoted 3 lines
I don't know what "connect directly" means here, but sending interrupts to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-x mechanism.Agree that MSI would be more efficient than SPI...
At the moment, in order to notify the frontend, the backend issues a specific device-model call to query Xen to inject a corresponding SPI to the guest.Quoted 53 lines
Quoted 10 lines
Quoted 6 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add.
As I said above, my proposal does the same thing that you mentioned here :) The difference is that I do call hypervisor interfaces via virtio-proxy.
Quoted 8 lines
The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52I don't think that translating IOREQs to eventfd in the kernel is a good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor interface.
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 8 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic, perhaps we could design a new interface. One that could be implementable in the Xen hypervisor itself (like IOREQ) and of course any other hypervisor too.
Quoted 6 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.My proposal document might help here; All the interfaces required for virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 4 lines
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 7 lines
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is a different matter.Thanks,
-Takahiro AkashiQuoted 4 lines
--
Regards,Oleksandr Tyshchenko
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Akashi,
Quoted 42 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Stratos Mailing List <[email protected]>; [email protected] open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHi Wei, Oleksandr,
Quoted 10 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device implementations.
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always represent # Linaro's view on this subject nor reflect Alex's concerns. Nevertheless,
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first, Xen and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
Quoted 9 lines
In particular, the abstraction of hypervisor interfaces has a same set of interfaces (for your "struct
vmm_impl" and my "RPC interfaces"). This is not co-incident as we both share the same origin as I said above. And so we will also share the same issues. One of them is a way of "sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too.
(See Alex's original email, too).
Yes, I agree.
Quoted 9 lines
On the other hand, my approach aims to create a "single-binary" solution in which the same binary of BE vm could run on any hypervisors. Somehow similar to your "proposal-#2" in [2], but in my solution, all the hypervisor-specific code would be put into another entity (VM), named "virtio-proxy" and the abstracted operations are served via RPC. (In this sense, BE is hypervisor-agnostic but might have OS dependency.) But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy completely (I will do it ASAP). But from your description, it seems we need a 3rd VM between FE and BE? My concern is that, if my assumption is right, will it increase the latency in data transport path? Even if we're using some lightweight guest like RTOS or Unikernel,
Quoted 6 lines
Specifically speaking about kvm-tool, I have a concern about its license term; Targeting different hypervisors and different OSs (which I assume includes RTOS's), the resultant library should be license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Yes. If user want to implement a FreeBSD device model, but the virtio library is GPL. Then GPL would be a problem. If we have another good candidate, I am open to it.
Quoted 507 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; Stefano StabelliniQuoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos Mailing List<[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Quoted 5 lines
Quoted 5 lines
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hello, all.
Please see some comments below. And sorry for the possible format
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 2 lines · <mailto:[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines · Stefano Stabellini
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not trimming the
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen implementation,
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters, but because it
is
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and
particularly
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.So I hope you like my approach :)
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more about my approach
below.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 3 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code as possible and
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the front end
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required - one per OS,
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant guests are
trivially
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like environment. The
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some mapped memory
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is eventfd). The
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any hypervisor specific
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the actual virtqueue
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion in the sort
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in encapsulating
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI. There interest
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to abstract
differences
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However we are some
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at least
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying requirements. To that
end I
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that make up the
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with delegated
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the backend VM about
key
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick)events
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into your guest images
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious solution seems
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user space but perhaps
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the appropriate
memory
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding clashing. I
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface to userspace
to then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good insights on this
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-agnostic
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a PCI device and
can be
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 3 lines
Quoted 3 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0. So for Xen I
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information is
expected
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.Yes, Xenstore was chosen as a simple way to pass configuration info to
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in Virtio backend
itself if possible, so for non-toolstack domain, this could done with adjusting devd (daemon that listens for devices and launches backends)
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway and pass it to
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass device
configuration.
Quoted 1 line
We also designed a static device configuration parse method for Dom0less
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device model command
line
Quoted 6 lines
or a config file.
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or hypervisor)-
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM, to hide all
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found and proven to
work then even better.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allowthis
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no problems of course.
One of difficulties on Xen that I found in my approach is that
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping memory) is
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have to extend
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security and privilegedistribution
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
as Stefan suggested.
We also faced policy related issues with Virtio backend running in
other than Dom0 domain in a "dummy" xsm mode. In our target system we run the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W resides. We trust it,
so we wrote policy rules (to be used in "flask" xsm mode) to provide it with a little bit more privileges than a simple DomU had.
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and memory
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see twooptions
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for daemon/helper to trigger
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If using a helper
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid the daemon
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that complexity into
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would result in
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the userspace would
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to userspace ptr"
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall interfaces.
This of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
course assumes the majority of BE guests would be Linux
kernels and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
leaves the bare-metal/unikernel approaches to their own
devices.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Operation
=========The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving
update
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
events and parsing the resultant virt queue for data. The vhost-
user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
specification handles a bunch of setup before that point, mostly
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
detail where the virt queues are set up FD's for memory and
event
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communication. This is where the envisioned stub process would
be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then
either
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communicate with the kernel's abstracted ABI or be re-targeted
as a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
way they are typically envisioned and described in other
environments.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead, Xen has IOREQ servers. Each of them connects
independently to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen via the IOREQ interface. E.g. today multiple QEMUs could be
used as
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
The component responsible for starting a daemon and/or setting up
shared
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in
Startos
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
jargon?) is a subject to debate in parallel. Otherwise, is there any good assumption to avoid it right now?
Quoted 1 line
Oleksandr and others I CCed have been working on ways for the
toolstack
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to create virtio backends and setup memory mappings. They might be
able
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to provide more info on the subject. I do think we miss a way to
provide
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the configuration to the backend and anything else that the
backend
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
might require to start doing its job.
Yes, some work has been done for the toolstack to handle Virtio MMIO
devices in
Quoted 1 line
Quoted 1 line
general and Virtio block devices in particular. However, it has not
been upstreaned yet.
Quoted 2 lines
Quoted 2 lines
Updated patches on review now:
https://lore.kernel.org/xen-devel/1621626361-29076-1-git-send-email-Quoted 2 lines
Quoted 2 lines
There is an additional (also important) activity to improve/fix
foreign memory mapping on Arm which I am also involved in.
Quoted 1 line
Quoted 1 line
The foreign memory mapping is proposed to be used for Virtio backends
(device emulators) if there is a need to run guest OS completely unmodified.
Quoted 1 line
Quoted 1 line
Of course, the more secure way would be to use grant memory mapping.
Brietly, the main difference between them is that with foreign mapping the backend
Quoted 1 line
Quoted 1 line
can map any guest memory it wants to map, but with grant mapping it is
allowed to map only what was previously granted by the frontend.
Quoted 2 lines
Quoted 2 lines
So, there might be a problem if we want to pre-map some guest memory
in advance or to cache mappings in the backend in order to improve performance (because the mapping/unmapping guest pages every request requires a lot of back and forth to Xen + P2M updates). In a nutshell, currently, in order to map a guest page into the backend address space we need to steal a real physical page from the backend domain. So, with the said optimizations we might end up with no free memory in the backend domain (see XSA-300). And what we try to achieve is to not waste a real domain memory at all by providing safe non-allocated-yet (so unused) address space for the foreign (and grant) pages to be mapped into, this enabling work implies Xen and Linux (and likely DTB bindings) changes. However, as it turned out, for this to work in a proper and safe way some prereq work needs to be done.
Quoted 2 lines
Quoted 2 lines
You can find the related Xen discussion at:
https://lore.kernel.org/xen-devel/1627489110-25633-1-git-send-email-Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
One question is how to best handle notification and kicks. The
existing
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user framework uses eventfd to signal the daemon (although
QEMU
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is quite capable of simulating them when you use TCG). Xen has
it's own
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
IOREQ mechanism. However latency is an important factor and
having
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisorinterface
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels),which is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I don't know what "connect directly" means here, but sending
interrupts
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-xmechanism.
Quoted 3 lines
Quoted 3 lines
Agree that MSI would be more efficient than SPI...
At the moment, in order to notify the frontend, the backend issues aspecific device-model call to query Xen to inject a corresponding SPI to the guest.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Could we consider the kernel internally converting IOREQ
messages from
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the Xen hypervisor to eventfd events? Would this scale with
other kernel
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to
Xen via
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the IOREQ interface. We could generalize the IOREQ interface and
make it
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor agnostic. The interface is really trivial and easy to
add.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As I said above, my proposal does the same thing that you mentioned
here :)
Quoted 1 line
Quoted 1 line
Quoted 1 line
The difference is that I do call hypervisor interfaces via virtio-
proxy.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The only Xen-specific part is the notification mechanism, which is
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
event channel. If we replaced the event channel with something
else the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I don't think that translating IOREQs to eventfd in the kernel is
a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor
interface.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 3 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic,
perhaps
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we could design a new interface. One that could be implementable
in the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen hypervisor itself (like IOREQ) and of course any other
hypervisor
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
too.
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't wealso need
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an interface for the backend to inject interrupts into the
frontend? And
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if the backend requires dynamic memory mappings of frontend pages,
then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
we would also need an interface to map/unmap domU pages.
My proposal document might help here; All the interfaces required
for
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 1 line
These interfaces are a lot more problematic than IOREQ: IOREQ is
tiny
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because
it
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is adifferent
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
matter.
Thanks,
-Takahiro Akashi--
Regards,
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
Oleksandr Tyshchenko
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
Stefan
Quoted 65 lines · Wei Chen
Hi Akashi,
Quoted 42 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Stratos Mailing List <[email protected]>; [email protected] open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHi Wei, Oleksandr,
Quoted 10 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device implementations.
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always represent # Linaro's view on this subject nor reflect Alex's concerns. Nevertheless,
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first, Xen and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
Quoted 9 lines
In particular, the abstraction of hypervisor interfaces has a same set of interfaces (for your "struct
vmm_impl" and my "RPC interfaces"). This is not co-incident as we both share the same origin as I said above. And so we will also share the same issues. One of them is a way of "sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too.
(See Alex's original email, too).Yes, I agree.
Quoted 9 lines
On the other hand, my approach aims to create a "single-binary" solution in which the same binary of BE vm could run on any hypervisors. Somehow similar to your "proposal-#2" in [2], but in my solution, all the hypervisor-specific code would be put into another entity (VM), named "virtio-proxy" and the abstracted operations are served via RPC. (In this sense, BE is hypervisor-agnostic but might have OS dependency.) But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Quoted 5 lines
Sorry, I haven't had time to finish reading your virtio-proxy completely (I will do it ASAP). But from your description, it seems we need a 3rd VM between FE and BE? My concern is that, if my assumption is right, will it increase the latency in data transport path? Even if we're using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of payload, we will see latency issue (even in your case) and if we have some solution for it, we won't see it neither in my proposal :)
Quoted 10 lines
Quoted 6 lines
Specifically speaking about kvm-tool, I have a concern about its license term; Targeting different hypervisors and different OSs (which I assume includes RTOS's), the resultant library should be license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Yes. If user want to implement a FreeBSD device model, but the virtio library is GPL. Then GPL would be a problem. If we have another good candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD code
-Takahiro Akashi
Quoted 514 lines
Quoted 507 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; Stefano StabelliniQuoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos Mailing List<[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Quoted 5 lines
Quoted 5 lines
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hello, all.
Please see some comments below. And sorry for the possible format
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 2 lines · <mailto:[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines · Stefano Stabellini
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not trimming the
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen implementation,
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters, but because it
is
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and
particularly
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.So I hope you like my approach :)
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more about my approach
below.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 3 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code as possible and
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the front end
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required - one per OS,
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant guests are
trivially
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like environment. The
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some mapped memory
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is eventfd). The
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any hypervisor specific
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the actual virtqueue
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion in the sort
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in encapsulating
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI. There interest
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to abstract
differences
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However we are some
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at least
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying requirements. To that
end I
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that make up the
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with delegated
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the backend VM about
key
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick)events
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into your guest images
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious solution seems
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user space but perhaps
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the appropriate
memory
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding clashing. I
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface to userspace
to then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good insights on this
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-agnostic
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a PCI device and
can be
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 3 lines
Quoted 3 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0. So for Xen I
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information is
expected
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.Yes, Xenstore was chosen as a simple way to pass configuration info to
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in Virtio backend
itself if possible, so for non-toolstack domain, this could done with adjusting devd (daemon that listens for devices and launches backends)
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway and pass it to
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass device
configuration.
Quoted 1 line
We also designed a static device configuration parse method for Dom0less
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device model command
line
Quoted 6 lines
or a config file.
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or hypervisor)-
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM, to hide all
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found and proven to
work then even better.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allowthis
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no problems of course.
One of difficulties on Xen that I found in my approach is that
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping memory) is
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have to extend
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security and privilegedistribution
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
as Stefan suggested.
We also faced policy related issues with Virtio backend running in
other than Dom0 domain in a "dummy" xsm mode. In our target system we run the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W resides. We trust it,
so we wrote policy rules (to be used in "flask" xsm mode) to provide it with a little bit more privileges than a simple DomU had.
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and memory
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see twooptions
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for daemon/helper to trigger
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If using a helper
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid the daemon
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that complexity into
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would result in
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the userspace would
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to userspace ptr"
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall interfaces.
This of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
course assumes the majority of BE guests would be Linux
kernels and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
leaves the bare-metal/unikernel approaches to their own
devices.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Operation
=========The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving
update
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
events and parsing the resultant virt queue for data. The vhost-
user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
specification handles a bunch of setup before that point, mostly
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
detail where the virt queues are set up FD's for memory and
event
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communication. This is where the envisioned stub process would
be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then
either
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communicate with the kernel's abstracted ABI or be re-targeted
as a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
way they are typically envisioned and described in other
environments.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead, Xen has IOREQ servers. Each of them connects
independently to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen via the IOREQ interface. E.g. today multiple QEMUs could be
used as
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
The component responsible for starting a daemon and/or setting up
shared
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in
Startos
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
jargon?) is a subject to debate in parallel. Otherwise, is there any good assumption to avoid it right now?
Quoted 1 line
Oleksandr and others I CCed have been working on ways for the
toolstack
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to create virtio backends and setup memory mappings. They might be
able
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to provide more info on the subject. I do think we miss a way to
provide
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the configuration to the backend and anything else that the
backend
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
might require to start doing its job.
Yes, some work has been done for the toolstack to handle Virtio MMIO
devices in
Quoted 1 line
Quoted 1 line
general and Virtio block devices in particular. However, it has not
been upstreaned yet.
Quoted 2 lines
Quoted 2 lines
Updated patches on review now:
https://lore.kernel.org/xen-devel/1621626361-29076-1-git-send-email-Quoted 2 lines
Quoted 2 lines
There is an additional (also important) activity to improve/fix
foreign memory mapping on Arm which I am also involved in.
Quoted 1 line
Quoted 1 line
The foreign memory mapping is proposed to be used for Virtio backends
(device emulators) if there is a need to run guest OS completely unmodified.
Quoted 1 line
Quoted 1 line
Of course, the more secure way would be to use grant memory mapping.
Brietly, the main difference between them is that with foreign mapping the backend
Quoted 1 line
Quoted 1 line
can map any guest memory it wants to map, but with grant mapping it is
allowed to map only what was previously granted by the frontend.
Quoted 2 lines
Quoted 2 lines
So, there might be a problem if we want to pre-map some guest memory
in advance or to cache mappings in the backend in order to improve performance (because the mapping/unmapping guest pages every request requires a lot of back and forth to Xen + P2M updates). In a nutshell, currently, in order to map a guest page into the backend address space we need to steal a real physical page from the backend domain. So, with the said optimizations we might end up with no free memory in the backend domain (see XSA-300). And what we try to achieve is to not waste a real domain memory at all by providing safe non-allocated-yet (so unused) address space for the foreign (and grant) pages to be mapped into, this enabling work implies Xen and Linux (and likely DTB bindings) changes. However, as it turned out, for this to work in a proper and safe way some prereq work needs to be done.
Quoted 2 lines
Quoted 2 lines
You can find the related Xen discussion at:
https://lore.kernel.org/xen-devel/1627489110-25633-1-git-send-email-Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
One question is how to best handle notification and kicks. The
existing
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user framework uses eventfd to signal the daemon (although
QEMU
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is quite capable of simulating them when you use TCG). Xen has
it's own
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
IOREQ mechanism. However latency is an important factor and
having
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisorinterface
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels),which is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I don't know what "connect directly" means here, but sending
interrupts
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-xmechanism.
Quoted 3 lines
Quoted 3 lines
Agree that MSI would be more efficient than SPI...
At the moment, in order to notify the frontend, the backend issues aspecific device-model call to query Xen to inject a corresponding SPI to the guest.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Could we consider the kernel internally converting IOREQ
messages from
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the Xen hypervisor to eventfd events? Would this scale with
other kernel
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to
Xen via
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the IOREQ interface. We could generalize the IOREQ interface and
make it
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor agnostic. The interface is really trivial and easy to
add.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As I said above, my proposal does the same thing that you mentioned
here :)
Quoted 1 line
Quoted 1 line
Quoted 1 line
The difference is that I do call hypervisor interfaces via virtio-
proxy.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The only Xen-specific part is the notification mechanism, which is
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
event channel. If we replaced the event channel with something
else the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I don't think that translating IOREQs to eventfd in the kernel is
a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor
interface.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 3 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic,
perhaps
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we could design a new interface. One that could be implementable
in the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen hypervisor itself (like IOREQ) and of course any other
hypervisor
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
too.
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't wealso need
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an interface for the backend to inject interrupts into the
frontend? And
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if the backend requires dynamic memory mappings of frontend pages,
then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
we would also need an interface to map/unmap domU pages.
My proposal document might help here; All the interfaces required
for
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 1 line
These interfaces are a lot more problematic than IOREQ: IOREQ is
tiny
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because
it
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is adifferent
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
matter.
Thanks,
-Takahiro Akashi--
Regards,Quoted 6 lines
Quoted 2 lines
Quoted 1 line
Oleksandr Tyshchenko
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Akashi，
Quoted 99 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Stratos Mailing List <[email protected]>; [email protected] open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsQuoted 8 lines · Wei Chen
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>;Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
Quoted 9 lines
Quoted 9 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 14 lines
Quoted 14 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHi Wei, Oleksandr,
Quoted 7 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device implementations.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always represent # Linaro's view on this subject nor reflect Alex's concerns.
Nevertheless,
Quoted 9 lines
Quoted 9 lines
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first, Xen and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces has a same set of interfaces (for your "struct
vmm_impl" and my "RPC interfaces"). This is not co-incident as we both share the same origin as I saidabove.
Quoted 9 lines
Quoted 6 lines
And so we will also share the same issues. One of them is a way of "sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too.
(See Alex's original email, too).Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a "single-binary"
solution
Quoted 5 lines
Quoted 5 lines
in which the same binary of BE vm could run on any hypervisors. Somehow similar to your "proposal-#2" in [2], but in my solution, all the hypervisor-specific code would be put into another entity (VM), named "virtio-proxy" and the abstracted operations are served via RPC. (In this sense, BE is hypervisor-agnostic but might have OS
dependency.)
Quoted 9 lines
Quoted 3 lines
But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy completely (I will do it ASAP). But from your description, it seems we need a 3rd VM between FE and BE? My concern is that, if my assumption is right, will it increase the latency in data transport path? Even if we're using some lightweight guest like RTOS or Unikernel,
Quoted 5 lines
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of payload, we will see latency issue (even in your case) and if we have some solution for it, we won't see it neither in my proposal :)
Oleksandr has sent a proposal to Xen mailing list to reduce this kind of "mapping/unmapping" operations. So the latency caused by this behavior on Xen may eventually be eliminated, and Linux-KVM doesn't have that problem.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Specifically speaking about kvm-tool, I have a concern about its license term; Targeting different hypervisors and different OSs (which I assume includes RTOS's), the resultant library should be license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Quoted 8 lines
Quoted 3 lines
Yes. If user want to implement a FreeBSD device model, but the virtio library is GPL. Then GPL would be a problem. If we have another good candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD code
Interesting, I will look into them : )
Cheers,
Wei Chen
Quoted 627 lines
-Takahiro Akashi
Quoted 11 lines
Quoted 11 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; StefanoStabellini
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos Mailing ListArnd
Quoted 9 lines
Quoted 9 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 12 lines
Quoted 12 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Quoted 5 lines
Quoted 5 lines
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hello, all.
Please see some comments below. And sorry for the possible format
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano Stabellini
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not trimming
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters, but
because it
Quoted 8 lines
Quoted 8 lines
is
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server). It has been, I believe, well generalized but is still a bit
biased
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more about my
approach
Quoted 6 lines
Quoted 6 lines
below.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · Alex Bennée
Quoted 3 lines
Hi,
One of the goals of Project Stratos is to enable hypervisor
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code as possible
and
Quoted 8 lines
Quoted 8 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the front end
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required - one per OS,
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant guests are
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any abstracted paravirt
type
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution I envisioned
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like environment. The
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some mapped
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is eventfd).
The
Quoted 4 lines
Quoted 4 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any hypervisor specific
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the actual
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion in the
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI. There
interest
Quoted 6 lines
Quoted 6 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to abstract
differences
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation"
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed from the
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However we are
some
Quoted 4 lines
Quoted 4 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at least
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying requirements. To
that
Quoted 7 lines
Quoted 7 lines
end I
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because
the
Quoted 6 lines
Quoted 6 lines
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that make up the
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with delegated
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the backend VM
about
Quoted 11 lines
Quoted 11 lines
key
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick)events
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into your guest
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious solution
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user space but
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest and the BE
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address space needs to
be
Quoted 11 lines
Quoted 11 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding clashing. I
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface to
userspace
Quoted 6 lines
Quoted 6 lines
to then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good insights on
this
Quoted 5 lines
Quoted 5 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-agnostic
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a PCI device
and
Quoted 9 lines
Quoted 9 lines
can be
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 3 lines
Quoted 3 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle"
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0. So for
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interfaceto
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
map the configuration space.
Yes, Xenstore was chosen as a simple way to pass configuration
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in Virtio
backend
Quoted 3 lines
Quoted 3 lines
itself if possible, so for non-toolstack domain, this could done with adjusting devd (daemon that listens for devices and launches backends)
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway and pass it
to
Quoted 6 lines
Quoted 6 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass device
configuration.
Quoted 1 line
We also designed a static device configuration parse method for
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device model
command
Quoted 9 lines
Quoted 9 lines
line
Quoted 6 lines
or a config file.
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or hypervisor)-
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM, to hide
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found and proven
to
Quoted 5 lines
Quoted 5 lines
work then even better.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
# My point is that a "handle" is not mandatory for executing
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a newXen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine privileges to
allow
Quoted 4 lines
Quoted 4 lines
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no problems of
course.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my approach is that
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping memory) is
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have to extend
Quoted 8 lines
Quoted 8 lines
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security and privilegedistribution
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
as Stefan suggested.
We also faced policy related issues with Virtio backend running in
other than Dom0 domain in a "dummy" xsm mode. In our target system we
run
Quoted 2 lines
Quoted 2 lines
the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W resides. We
trust it,
Quoted 1 line
Quoted 1 line
so we wrote policy rules (to be used in "flask" xsm mode) to provide
it
Quoted 8 lines
Quoted 8 lines
with a little bit more privileges than a simple DomU had.
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and memory
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can seetwo
Quoted 4 lines
Quoted 4 lines
options
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for daemon/helper to
trigger
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If using a
helper
Quoted 4 lines
Quoted 4 lines
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid the daemon
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that complexity
into
Quoted 10 lines
Quoted 10 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would result in
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the userspace would
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to userspace
ptr"
Quoted 4 lines
Quoted 4 lines
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall interfaces.
This of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
course assumes the majority of BE guests would be Linux
Quoted 8 lines
Quoted 8 lines
kernels and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
leaves the bare-metal/unikernel approaches to their own
devices.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Operation
=========The core of the operation of VirtIO is fairly simple. Once
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user feature negotiation is done it's a case of
receiving
Quoted 2 lines
Quoted 2 lines
update
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
events and parsing the resultant virt queue for data. The
vhost-
Quoted 2 lines
Quoted 2 lines
user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
specification handles a bunch of setup before that point,
mostly
Quoted 4 lines
Quoted 4 lines
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
detail where the virt queues are set up FD's for memory and
event
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communication. This is where the envisioned stub process
would
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
responsible for getting the daemon up and ready to run. This
is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
currently done inside a big VMM like QEMU but I suspect a
modern
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
approach would be to use the rust-vmm vhost crate. It would
then
Quoted 2 lines
Quoted 2 lines
either
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communicate with the kernel's abstracted ABI or be re-
targeted
Quoted 4 lines
Quoted 4 lines
as a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have
VMMs
Quoted 6 lines
Quoted 6 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
way they are typically envisioned and described in other
environments.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead, Xen has IOREQ servers. Each of them connects
independently to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen via the IOREQ interface. E.g. today multiple QEMUs could
be
Quoted 5 lines
Quoted 5 lines
used as
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
The component responsible for starting a daemon and/or setting
up
Quoted 2 lines
Quoted 2 lines
shared
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interfaces is the toolstack: the xl command and the
libxl/libxc
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
libraries.
I think that VM configuration management (or orchestration in
Startos
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
jargon?) is a subject to debate in parallel. Otherwise, is there any good assumption to avoid it right now?
Quoted 1 line
Oleksandr and others I CCed have been working on ways for the
toolstack
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to create virtio backends and setup memory mappings. They
might be
Quoted 2 lines
Quoted 2 lines
able
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to provide more info on the subject. I do think we miss a way
to
Quoted 6 lines
Quoted 6 lines
provide
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the configuration to the backend and anything else that the
backend
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
might require to start doing its job.
Yes, some work has been done for the toolstack to handle Virtio
MMIO
Quoted 2 lines
Quoted 2 lines
devices in
Quoted 1 line
Quoted 1 line
general and Virtio block devices in particular. However, it has
not
Quoted 3 lines
Quoted 3 lines
been upstreaned yet.
Quoted 2 lines
Quoted 2 lines
Updated patches on review now:
https://lore.kernel.org/xen-devel/1621626361-29076-1-git-send-email-
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
There is an additional (also important) activity to improve/fix
foreign memory mapping on Arm which I am also involved in.
Quoted 1 line
Quoted 1 line
The foreign memory mapping is proposed to be used for Virtio
backends
Quoted 3 lines
Quoted 3 lines
(device emulators) if there is a need to run guest OS completely unmodified.
Quoted 1 line
Quoted 1 line
Of course, the more secure way would be to use grant memory
mapping.
Quoted 1 line
Quoted 1 line
Brietly, the main difference between them is that with foreign mapping
the
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
can map any guest memory it wants to map, but with grant mapping
it is
Quoted 3 lines
Quoted 3 lines
allowed to map only what was previously granted by the frontend.
Quoted 2 lines
Quoted 2 lines
So, there might be a problem if we want to pre-map some guest
memory
Quoted 4 lines
Quoted 4 lines
in advance or to cache mappings in the backend in order to improve performance (because the mapping/unmapping guest pages every request requires a lot of back and forth to Xen + P2M updates). In a nutshell, currently, in order to map a guest page into the backend address space
we
Quoted 1 line
Quoted 1 line
need to steal a real physical page from the backend domain. So, with
the
Quoted 2 lines
Quoted 2 lines
said optimizations we might end up with no free memory in the backend domain (see XSA-300). And what we try to achieve is to not waste a
real
Quoted 2 lines
Quoted 2 lines
domain memory at all by providing safe non-allocated-yet (so unused) address space for the foreign (and grant) pages to be mapped into,
this
Quoted 2 lines
Quoted 2 lines
enabling work implies Xen and Linux (and likely DTB bindings) changes. However, as it turned out, for this to work in a proper and safe way
some
Quoted 3 lines
Quoted 3 lines
prereq work needs to be done.
Quoted 2 lines
Quoted 2 lines
You can find the related Xen discussion at:
https://lore.kernel.org/xen-devel/1627489110-25633-1-git-send-email-
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
One question is how to best handle notification and kicks.
The
Quoted 2 lines
Quoted 2 lines
existing
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user framework uses eventfd to signal the daemon
(although
Quoted 2 lines
Quoted 2 lines
QEMU
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is quite capable of simulating them when you use TCG). Xen
has
Quoted 6 lines
Quoted 6 lines
it's own
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
IOREQ mechanism. However latency is an important factor and
having
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the
backends to
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisorinterface
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels),which is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
I don't know what "connect directly" means here, but sending
interrupts
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-xmechanism.
Quoted 3 lines
Quoted 3 lines
Agree that MSI would be more efficient than SPI...
At the moment, in order to notify the frontend, the backend issuesa
Quoted 1 line
Quoted 1 line
specific device-model call to query Xen to inject a corresponding SPI
to
Quoted 5 lines
Quoted 5 lines
the guest.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Could we consider the kernel internally converting IOREQ
messages from
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the Xen hypervisor to eventfd events? Would this scale with
other kernel
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting
with?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One option we should consider is for each backend to connect
to
Quoted 2 lines
Quoted 2 lines
Xen via
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the IOREQ interface. We could generalize the IOREQ interface
and
Quoted 2 lines
Quoted 2 lines
make it
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor agnostic. The interface is really trivial and easy
to
Quoted 3 lines
Quoted 3 lines
add.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As I said above, my proposal does the same thing that you
mentioned
Quoted 2 lines
Quoted 2 lines
here :)
Quoted 1 line
Quoted 1 line
Quoted 1 line
The difference is that I do call hypervisor interfaces via
virtio-
Quoted 3 lines
Quoted 3 lines
proxy.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The only Xen-specific part is the notification mechanism,
which is
Quoted 8 lines
Quoted 8 lines
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
event channel. If we replaced the event channel with something
else the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I don't think that translating IOREQs to eventfd in the kernel
is
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
good idea: if feels like it would be extra complexity and that
the
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
kernel shouldn't be involved as this is a backend-hypervisor
interface.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Given that we may want to implement BE as a bare-metal
application
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
as I did on Zephyr, I don't think that the translation would not
be
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 1 line
Also, eventfd is very Linux-centric and we are trying to
design an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interface that could work well for RTOSes too. If we want to
do
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
something different, both OS-agnostic and hypervisor-agnostic,
perhaps
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we could design a new interface. One that could be
implementable
Quoted 7 lines
Quoted 7 lines
in the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen hypervisor itself (like IOREQ) and of course any other
hypervisor
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
too.
There is also another problem. IOREQ is probably not be the
only
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't wealso need
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an interface for the backend to inject interrupts into the
frontend? And
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if the backend requires dynamic memory mappings of frontend
pages,
Quoted 4 lines
Quoted 4 lines
then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
we would also need an interface to map/unmap domU pages.
My proposal document might help here; All the interfaces
required
Quoted 5 lines
Quoted 5 lines
for
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 1 line
These interfaces are a lot more problematic than IOREQ: IOREQ
is
Quoted 2 lines
Quoted 2 lines
tiny
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and self-contained. It is easy to add anywhere. A new
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
inject interrupts or map pages is more difficult to manage
because
Quoted 4 lines
Quoted 4 lines
it
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also
apply
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is adifferent
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
matter.
Thanks,
-Takahiro AkashiQuoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
--
Regards,Oleksandr Tyshchenko
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the
Quoted 8 lines
Quoted 1 line
Quoted 1 line
contents to any other person, use it for any purpose, or store or copy
the
Quoted 2 lines
Quoted 1 line
information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hello Alex,
I can tell you my experience from working on a PoC (library)
to allow the implementation of virtio-devices that are hypervisor/OS agnostic. I focused on two use cases:
1. type-I hypervisor in which the backend is running as a VM. This is an in-house hypervisor that does not support VMExits.
2. Linux user-space. In this case, the library is just used to communicate threads. The goal of this use case is merely testing.
I have chosen virtio-mmio as the way to exchange information
between the frontend and backend. I found it hard to synchronize the access to the virtio-mmio layout without VMExits. I had to add some extra bits to allow the front-end and back-end to synchronize, which is required
during the device-status initialization. These extra bits would not be needed in case the hypervisor supports VMExits, e.g., KVM.
Each guest has a memory region that is shared with the backend.
This memory region is used by the frontend to allocate the io-buffers. This region also maps the virtio-mmio layout that is initialized by the backend. For the moment, this region is defined when the guest is created. One limitation is that the memory for io-buffers is fixed. At some point, the guest shall be able to balloon this region. Notifications between the frontend and the backend are implemented by using an hypercall. The hypercall mechanism and the memory allocation are abstracted away by a platform layer that exposes an interface that is hypervisor/os agnostic.
I split the backend into a virtio-device driver and a
backend driver. The virtio-device driver is the virtqueues and the backend driver gets packets from the virtqueue for post-processing. For example, in the case of virtio-net, the backend driver would decide if the packet goes to the hardware or to another virtio-net device. The virtio-device drivers may be
implemented in different ways like by using a single thread, multiple threads, or one thread for all the virtio-devices.
In this PoC, I just tackled two very simple use-cases. These use-cases allowed me to extract some requirements for an hypervisor to support virtio.
Matias
Quoted 104 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest. Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space. To activate the mapping will require some sort of hypercall to the hypervisor. I can see two options at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Quoted 13 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 107 lines · Wei Chen
Hi Akashi，
Quoted 99 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Stratos Mailing List <[email protected]>; [email protected] open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsQuoted 8 lines · Wei Chen
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>;Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
Quoted 9 lines
Quoted 9 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 14 lines
Quoted 14 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHi Wei, Oleksandr,
Quoted 7 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device implementations.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always represent # Linaro's view on this subject nor reflect Alex's concerns.
Nevertheless,
Quoted 9 lines
Quoted 9 lines
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first, Xen and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces has a same set of interfaces (for your "struct
vmm_impl" and my "RPC interfaces"). This is not co-incident as we both share the same origin as I saidabove.
Quoted 9 lines
Quoted 6 lines
And so we will also share the same issues. One of them is a way of "sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too.
(See Alex's original email, too).Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a "single-binary"
solution
Quoted 5 lines
Quoted 5 lines
in which the same binary of BE vm could run on any hypervisors. Somehow similar to your "proposal-#2" in [2], but in my solution, all the hypervisor-specific code would be put into another entity (VM), named "virtio-proxy" and the abstracted operations are served via RPC. (In this sense, BE is hypervisor-agnostic but might have OS
dependency.)
Quoted 9 lines
Quoted 3 lines
But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy completely (I will do it ASAP). But from your description, it seems we need a 3rd VM between FE and BE? My concern is that, if my assumption is right, will it increase the latency in data transport path? Even if we're using some lightweight guest like RTOS or Unikernel,
Quoted 5 lines
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of payload, we will see latency issue (even in your case) and if we have some solution for it, we won't see it neither in my proposal :)
Quoted 3 lines
Oleksandr has sent a proposal to Xen mailing list to reduce this kind of "mapping/unmapping" operations. So the latency caused by this behavior on Xen may eventually be eliminated, and Linux-KVM doesn't have that problem.
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
-Takahiro Akashi
Quoted 657 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Specifically speaking about kvm-tool, I have a concern about its license term; Targeting different hypervisors and different OSs (which I assume includes RTOS's), the resultant library should be license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Quoted 8 lines
Quoted 3 lines
Yes. If user want to implement a FreeBSD device model, but the virtio library is GPL. Then GPL would be a problem. If we have another good candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD codeInteresting, I will look into them : )
Cheers,
Wei ChenQuoted 627 lines
-Takahiro Akashi
Quoted 11 lines
Quoted 11 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; StefanoStabellini
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos Mailing ListArnd
Quoted 9 lines
Quoted 9 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 12 lines
Quoted 12 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Quoted 5 lines
Quoted 5 lines
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hello, all.
Please see some comments below. And sorry for the possible format
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano Stabellini
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not trimming
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters, but
because it
Quoted 8 lines
Quoted 8 lines
is
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server). It has been, I believe, well generalized but is still a bit
biased
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more about my
approach
Quoted 6 lines
Quoted 6 lines
below.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · Alex Bennée
Quoted 3 lines
Hi,
One of the goals of Project Stratos is to enable hypervisor
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code as possible
and
Quoted 8 lines
Quoted 8 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the front end
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required - one per OS,
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant guests are
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any abstracted paravirt
type
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution I envisioned
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like environment. The
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some mapped
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is eventfd).
The
Quoted 4 lines
Quoted 4 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any hypervisor specific
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the actual
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion in the
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI. There
interest
Quoted 6 lines
Quoted 6 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to abstract
differences
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation"
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed from the
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However we are
some
Quoted 4 lines
Quoted 4 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at least
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying requirements. To
that
Quoted 7 lines
Quoted 7 lines
end I
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because
the
Quoted 6 lines
Quoted 6 lines
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that make up the
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with delegated
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the backend VM
about
Quoted 11 lines
Quoted 11 lines
key
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick)events
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into your guest
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious solution
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user space but
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest and the BE
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address space needs to
be
Quoted 11 lines
Quoted 11 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding clashing. I
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface to
userspace
Quoted 6 lines
Quoted 6 lines
to then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good insights on
this
Quoted 5 lines
Quoted 5 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-agnostic
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a PCI device
and
Quoted 9 lines
Quoted 9 lines
can be
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 3 lines
Quoted 3 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle"
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0. So for
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interfaceto
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
map the configuration space.
Yes, Xenstore was chosen as a simple way to pass configuration
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in Virtio
backend
Quoted 3 lines
Quoted 3 lines
itself if possible, so for non-toolstack domain, this could done with adjusting devd (daemon that listens for devices and launches backends)
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway and pass it
to
Quoted 6 lines
Quoted 6 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass device
configuration.
Quoted 1 line
We also designed a static device configuration parse method for
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device model
command
Quoted 9 lines
Quoted 9 lines
line
Quoted 6 lines
or a config file.
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or hypervisor)-
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM, to hide
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found and proven
to
Quoted 5 lines
Quoted 5 lines
work then even better.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
# My point is that a "handle" is not mandatory for executing
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a newXen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine privileges to
allow
Quoted 4 lines
Quoted 4 lines
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no problems of
course.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my approach is that
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping memory) is
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have to extend
Quoted 8 lines
Quoted 8 lines
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security and privilegedistribution
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
as Stefan suggested.
We also faced policy related issues with Virtio backend running in
other than Dom0 domain in a "dummy" xsm mode. In our target system we
run
Quoted 2 lines
Quoted 2 lines
the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W resides. We
trust it,
Quoted 1 line
Quoted 1 line
so we wrote policy rules (to be used in "flask" xsm mode) to provide
it
Quoted 8 lines
Quoted 8 lines
with a little bit more privileges than a simple DomU had.
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and memory
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can seetwo
Quoted 4 lines
Quoted 4 lines
options
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for daemon/helper to
trigger
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If using a
helper
Quoted 4 lines
Quoted 4 lines
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid the daemon
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that complexity
into
Quoted 10 lines
Quoted 10 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would result in
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the userspace would
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to userspace
ptr"
Quoted 4 lines
Quoted 4 lines
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall interfaces.
This of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
course assumes the majority of BE guests would be Linux
Quoted 8 lines
Quoted 8 lines
kernels and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
leaves the bare-metal/unikernel approaches to their own
devices.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Operation
=========The core of the operation of VirtIO is fairly simple. Once
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user feature negotiation is done it's a case of
receiving
Quoted 2 lines
Quoted 2 lines
update
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
events and parsing the resultant virt queue for data. The
vhost-
Quoted 2 lines
Quoted 2 lines
user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
specification handles a bunch of setup before that point,
mostly
Quoted 4 lines
Quoted 4 lines
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
detail where the virt queues are set up FD's for memory and
event
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communication. This is where the envisioned stub process
would
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
responsible for getting the daemon up and ready to run. This
is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
currently done inside a big VMM like QEMU but I suspect a
modern
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
approach would be to use the rust-vmm vhost crate. It would
then
Quoted 2 lines
Quoted 2 lines
either
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communicate with the kernel's abstracted ABI or be re-
targeted
Quoted 4 lines
Quoted 4 lines
as a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have
VMMs
Quoted 6 lines
Quoted 6 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
way they are typically envisioned and described in other
environments.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead, Xen has IOREQ servers. Each of them connects
independently to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen via the IOREQ interface. E.g. today multiple QEMUs could
be
Quoted 5 lines
Quoted 5 lines
used as
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
The component responsible for starting a daemon and/or setting
up
Quoted 2 lines
Quoted 2 lines
shared
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interfaces is the toolstack: the xl command and the
libxl/libxc
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
libraries.
I think that VM configuration management (or orchestration in
Startos
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
jargon?) is a subject to debate in parallel. Otherwise, is there any good assumption to avoid it right now?
Quoted 1 line
Oleksandr and others I CCed have been working on ways for the
toolstack
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to create virtio backends and setup memory mappings. They
might be
Quoted 2 lines
Quoted 2 lines
able
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to provide more info on the subject. I do think we miss a way
to
Quoted 6 lines
Quoted 6 lines
provide
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the configuration to the backend and anything else that the
backend
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
might require to start doing its job.
Yes, some work has been done for the toolstack to handle Virtio
MMIO
Quoted 2 lines
Quoted 2 lines
devices in
Quoted 1 line
Quoted 1 line
general and Virtio block devices in particular. However, it has
not
Quoted 3 lines
Quoted 3 lines
been upstreaned yet.
Quoted 2 lines
Quoted 2 lines
Updated patches on review now:
https://lore.kernel.org/xen-devel/1621626361-29076-1-git-send-email-
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
There is an additional (also important) activity to improve/fix
foreign memory mapping on Arm which I am also involved in.
Quoted 1 line
Quoted 1 line
The foreign memory mapping is proposed to be used for Virtio
backends
Quoted 3 lines
Quoted 3 lines
(device emulators) if there is a need to run guest OS completely unmodified.
Quoted 1 line
Quoted 1 line
Of course, the more secure way would be to use grant memory
mapping.
Quoted 1 line
Quoted 1 line
Brietly, the main difference between them is that with foreign mapping
the
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
can map any guest memory it wants to map, but with grant mapping
it is
Quoted 3 lines
Quoted 3 lines
allowed to map only what was previously granted by the frontend.
Quoted 2 lines
Quoted 2 lines
So, there might be a problem if we want to pre-map some guest
memory
Quoted 4 lines
Quoted 4 lines
in advance or to cache mappings in the backend in order to improve performance (because the mapping/unmapping guest pages every request requires a lot of back and forth to Xen + P2M updates). In a nutshell, currently, in order to map a guest page into the backend address space
we
Quoted 1 line
Quoted 1 line
need to steal a real physical page from the backend domain. So, with
the
Quoted 2 lines
Quoted 2 lines
said optimizations we might end up with no free memory in the backend domain (see XSA-300). And what we try to achieve is to not waste a
real
Quoted 2 lines
Quoted 2 lines
domain memory at all by providing safe non-allocated-yet (so unused) address space for the foreign (and grant) pages to be mapped into,
this
Quoted 2 lines
Quoted 2 lines
enabling work implies Xen and Linux (and likely DTB bindings) changes. However, as it turned out, for this to work in a proper and safe way
some
Quoted 3 lines
Quoted 3 lines
prereq work needs to be done.
Quoted 2 lines
Quoted 2 lines
You can find the related Xen discussion at:
https://lore.kernel.org/xen-devel/1627489110-25633-1-git-send-email-
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
One question is how to best handle notification and kicks.
The
Quoted 2 lines
Quoted 2 lines
existing
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user framework uses eventfd to signal the daemon
(although
Quoted 2 lines
Quoted 2 lines
QEMU
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is quite capable of simulating them when you use TCG). Xen
has
Quoted 6 lines
Quoted 6 lines
it's own
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
IOREQ mechanism. However latency is an important factor and
having
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the
backends to
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisorinterface
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels),which is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
I don't know what "connect directly" means here, but sending
interrupts
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-xmechanism.
Quoted 3 lines
Quoted 3 lines
Agree that MSI would be more efficient than SPI...
At the moment, in order to notify the frontend, the backend issuesa
Quoted 1 line
Quoted 1 line
specific device-model call to query Xen to inject a corresponding SPI
to
Quoted 5 lines
Quoted 5 lines
the guest.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Could we consider the kernel internally converting IOREQ
messages from
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the Xen hypervisor to eventfd events? Would this scale with
other kernel
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting
with?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One option we should consider is for each backend to connect
to
Quoted 2 lines
Quoted 2 lines
Xen via
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the IOREQ interface. We could generalize the IOREQ interface
and
Quoted 2 lines
Quoted 2 lines
make it
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor agnostic. The interface is really trivial and easy
to
Quoted 3 lines
Quoted 3 lines
add.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As I said above, my proposal does the same thing that you
mentioned
Quoted 2 lines
Quoted 2 lines
here :)
Quoted 1 line
Quoted 1 line
Quoted 1 line
The difference is that I do call hypervisor interfaces via
virtio-
Quoted 3 lines
Quoted 3 lines
proxy.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The only Xen-specific part is the notification mechanism,
which is
Quoted 8 lines
Quoted 8 lines
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
event channel. If we replaced the event channel with something
else the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I don't think that translating IOREQs to eventfd in the kernel
is
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
good idea: if feels like it would be extra complexity and that
the
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
kernel shouldn't be involved as this is a backend-hypervisor
interface.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Given that we may want to implement BE as a bare-metal
application
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
as I did on Zephyr, I don't think that the translation would not
be
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 1 line
Also, eventfd is very Linux-centric and we are trying to
design an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interface that could work well for RTOSes too. If we want to
do
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
something different, both OS-agnostic and hypervisor-agnostic,
perhaps
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we could design a new interface. One that could be
implementable
Quoted 7 lines
Quoted 7 lines
in the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen hypervisor itself (like IOREQ) and of course any other
hypervisor
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
too.
There is also another problem. IOREQ is probably not be the
only
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't wealso need
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an interface for the backend to inject interrupts into the
frontend? And
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if the backend requires dynamic memory mappings of frontend
pages,
Quoted 4 lines
Quoted 4 lines
then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
we would also need an interface to map/unmap domU pages.
My proposal document might help here; All the interfaces
required
Quoted 5 lines
Quoted 5 lines
for
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 1 line
These interfaces are a lot more problematic than IOREQ: IOREQ
is
Quoted 2 lines
Quoted 2 lines
tiny
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and self-contained. It is easy to add anywhere. A new
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
inject interrupts or map pages is more difficult to manage
because
Quoted 4 lines
Quoted 4 lines
it
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also
apply
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is adifferent
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
matter.
Thanks,
-Takahiro AkashiQuoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
--
Regards,Oleksandr Tyshchenko
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the
Quoted 8 lines
Quoted 1 line
Quoted 1 line
contents to any other person, use it for any purpose, or store or copy
the
Quoted 2 lines
Quoted 1 line
information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hello,
Quoted 10 lines · AKASHI Takahiro
Hi Matias,
Quoted 4 lines · Matias Ezequiel Vara Larsen
Hello Alex,
I can tell you my experience from working on a PoC (library)
to allow the implementation of virtio-devices that are hypervisor/OS agnostic.What hypervisor are you using for your PoC here?
I am using an in-house hypervisor, which is similar to Jailhouse.
Quoted 6 lines
Quoted 6 lines
I focused on two use cases:
1. type-I hypervisor in which the backend is running as a VM. This is an in-house hypervisor that does not support VMExits.
2. Linux user-space. In this case, the library is just used to communicate threads. The goal of this use case is merely testing.
Quoted 7 lines
Quoted 3 lines
I have chosen virtio-mmio as the way to exchange information
between the frontend and backend. I found it hard to synchronize the access to the virtio-mmio layout without VMExits. I had to add some extra bits to allowCan you explain how MMIOs to registers in virito-mmio layout (which I think means a configuration space?) will be propagated to BE?
In this PoC, the BE guest is created with a fixed number of regions
of memory that represents each device. The BE initializes these regions, and then, waits for the FEs to begin the initialization.
Quoted 9 lines
Quoted 8 lines
the front-end and back-end to synchronize, which is required during the device-status initialization. These extra bits would not be needed in case the hypervisor supports VMExits, e.g., KVM.
Each guest has a memory region that is shared with the backend.
This memory region is used by the frontend to allocate the io-buffers. This region also maps the virtio-mmio layout that is initialized by the backend. For the moment, this region is defined when the guest is created. One limitation is that the memory for io-buffers is fixed.
Quoted 7 lines
So in summary, you have a single memory region that is used for virtio-mmio layout and io-buffers (I think they are for payload) and you assume that the region will be (at lease for now) statically shared between FE and BE so that you can eliminate 'mmap' at every time to access the payload.
Correct?
Yes, It is.
Quoted 9 lines
If so, it can be an alternative solution for memory access issue, and a similar technique is used in some implementations:
- (Jailhouse's) ivshmem
- Arnd's fat virtqueueIn either case, however, you will have to allocate payload from the region and so you will see some impact on FE code (at least at some low level). (In ivshmem,
dma_opsin the kernel is defined for this purpose.) Correct?
Yes, It is. The FE implements a sort of malloc() to organize the allocation of io-buffers from that memory region.
Rethinking about the VMExits, I am not sure how this mechanism may be used when both the FE and the BE are VMs. The use of VMExits may require to involve the hypervisor.
Matias
Quoted 141 lines
-Takahiro Akashi
Quoted 138 lines
At some point, the guest shall be able to balloon this region. Notifications between the frontend and the backend are implemented by using an hypercall. The hypercall mechanism and the memory allocation are abstracted away by a platform layer that exposes an interface that is hypervisor/os agnostic.
I split the backend into a virtio-device driver and a
backend driver. The virtio-device driver is the virtqueues and the backend driver gets packets from the virtqueue for post-processing. For example, in the case of virtio-net, the backend driver would decide if the packet goes to the hardware or to another virtio-net device. The virtio-device drivers may be
implemented in different ways like by using a single thread, multiple threads, or one thread for all the virtio-devices.In this PoC, I just tackled two very simple use-cases. These use-cases allowed me to extract some requirements for an hypervisor to support virtio.
Matias
Quoted 117 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect toObviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest. Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space. To activate the mapping will require some sort of hypercall to the hypervisor. I can see two options at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Signature · 5 lines
Alex Bennée --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Hi Stefan,
Quoted 15 lines · Stefan Hajnoczi
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 3 lines
There have been experiments with something kind of similar in KVM recently (see struct
ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/
Do you know the current status of Elena's work? It was last February that she posted her latest patch and it has not been merged upstream yet.
Quoted 12 lines
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 4 lines
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
Can you please elaborate your thoughts a bit more here?
It seems to me that trapping MMIOs to configuration space and forwarding those events to BE (or device emulation) is a quite straight-forward way to emulate device MMIOs.
Or do you think of something of protocols used in vhost-user?
# On the contrary, virtio-ivshmem only requires a driver to explicitly
# forward a "write" request of MMIO accesses to BE. But I don't think # it's your point.
-Takahiro Akashi
Quoted 1 line
Stefan
Quoted 18 lines · AKASHI Takahiro
Hi Stefan,
Quoted 15 lines · Stefan Hajnoczi
Quoted 13 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 7 lines
Quoted 3 lines
There have been experiments with something kind of similar in KVM recently (see struct
ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/Do you know the current status of Elena's work? It was last February that she posted her latest patch and it has not been merged upstream yet.
Elena worked on this during her Outreachy internship. At the moment no one is actively working on the patches.
Quoted 19 lines
Quoted 12 lines
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 4 lines
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
Can you please elaborate your thoughts a bit more here?
Quoted 8 lines
It seems to me that trapping MMIOs to configuration space and forwarding those events to BE (or device emulation) is a quite straight-forward way to emulate device MMIOs.
Or do you think of something of protocols used in vhost-user?# On the contrary, virtio-ivshmem only requires a driver to explicitly
# forward a "write" request of MMIO accesses to BE. But I don't think # it's your point.
See my first reply to this email thread about alternative interfaces for VIRTIO device emulation. The main thing to note was that although the shared memory vring is used by VIRTIO transports today, the device model actually allows transports to implement virtqueues differently (e.g. making it possible to create a VIRTIO over TCP transport without shared memory in the future).
It's possible to define a hypercall interface as a new VIRTIO transport that provides higher-level virtqueue operations. Doing this is more work than using vrings though since existing guest driver and device emulation code already supports vrings.
I don't know the requirements of Stratos so I can't say if creating a new hypervisor-independent interface (VIRTIO transport) that doesn't rely on shared memory vrings makes sense. I just wanted to raise the idea in case you find that VIRTIO's vrings don't meet your requirements.
Stefan
Hi Stefan,
Quoted 23 lines · Stefan Hajnoczi
Quoted 22 lines · AKASHI Takahiro
Hi Stefan,
Quoted 18 lines · Stefan Hajnoczi
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52There have been experiments with something kind of similar in KVM recently (see struct
ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/
Quoted 6 lines
Quoted 3 lines
Do you know the current status of Elena's work? It was last February that she posted her latest patch and it has not been merged upstream yet.
Elena worked on this during her Outreachy internship. At the moment no one is actively working on the patches.
Does RedHat plan to take over or follow up her work hereafter? # I'm simply asking from my curiosity.
Quoted 28 lines
Quoted 19 lines
Quoted 12 lines
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 4 lines
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
Can you please elaborate your thoughts a bit more here?
Quoted 8 lines
It seems to me that trapping MMIOs to configuration space and forwarding those events to BE (or device emulation) is a quite straight-forward way to emulate device MMIOs.
Or do you think of something of protocols used in vhost-user?# On the contrary, virtio-ivshmem only requires a driver to explicitly
# forward a "write" request of MMIO accesses to BE. But I don't think # it's your point.
Quoted 6 lines
See my first reply to this email thread about alternative interfaces for VIRTIO device emulation. The main thing to note was that although the shared memory vring is used by VIRTIO transports today, the device model actually allows transports to implement virtqueues differently (e.g. making it possible to create a VIRTIO over TCP transport without shared memory in the future).
Do you have any example of such use cases or systems?
Quoted 4 lines
It's possible to define a hypercall interface as a new VIRTIO transport that provides higher-level virtqueue operations. Doing this is more work than using vrings though since existing guest driver and device emulation code already supports vrings.
Personally, I'm open to discuss about your point, but
Quoted 4 lines
I don't know the requirements of Stratos so I can't say if creating a new hypervisor-independent interface (VIRTIO transport) that doesn't rely on shared memory vrings makes sense. I just wanted to raise the idea in case you find that VIRTIO's vrings don't meet your requirements.
While I cannot represent the project's view, what the JIRA task that is assigned to me describes:
Deliverables
* Low level library allowing:
* management of virtio rings and buffers
[and so on]
So supporting the shared memory-based vring is one of our assumptions.
In my understanding, the goal of Stratos project is that we would have several VMs congregated into a SoC, yet sharing most of physical IPs, where the shared memory should be, I assume, the most efficient transport for virtio.
One of target applications would be automotive, I guess.
Alex and Mike should have more to say here.
-Takahiro Akashi
Quoted 1 line
Stefan
Quoted 28 lines · AKASHI Takahiro
Quoted 27 lines · Stefan Hajnoczi
Quoted 25 lines · AKASHI Takahiro
Hi Stefan,
Quoted 15 lines · Stefan Hajnoczi
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52Quoted 3 lines
There have been experiments with something kind of similar in KVM recently (see struct
ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/Do you know the current status of Elena's work? It was last February that she posted her latest patch and it has not been merged upstream yet.
Quoted 5 lines
Quoted 2 lines
Elena worked on this during her Outreachy internship. At the moment no one is actively working on the patches.
Does RedHat plan to take over or follow up her work hereafter? # I'm simply asking from my curiosity.
At the moment I'm not aware of anyone from Red Hat working on it. If someone decides they need this KVM API then that could change.
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 12 lines
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 4 lines
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
Can you please elaborate your thoughts a bit more here?
It seems to me that trapping MMIOs to configuration space and forwarding those events to BE (or device emulation) is a quite straight-forward way to emulate device MMIOs.
Or do you think of something of protocols used in vhost-user?# On the contrary, virtio-ivshmem only requires a driver to explicitly
# forward a "write" request of MMIO accesses to BE. But I don't think # it's your point.
Quoted 8 lines
Quoted 6 lines
See my first reply to this email thread about alternative interfaces for VIRTIO device emulation. The main thing to note was that although the shared memory vring is used by VIRTIO transports today, the device model actually allows transports to implement virtqueues differently (e.g. making it possible to create a VIRTIO over TCP transport without shared memory in the future).
Do you have any example of such use cases or systems?
This aspect of VIRTIO isn't being exploited today AFAIK. But the layering to allow other virtqueue implementations is there. For example, Linux's virtqueue API is independent of struct vring, so existing drivers generally aren't tied to vrings.
Quoted 12 lines
Quoted 4 lines
It's possible to define a hypercall interface as a new VIRTIO transport that provides higher-level virtqueue operations. Doing this is more work than using vrings though since existing guest driver and device emulation code already supports vrings.
Personally, I'm open to discuss about your point, but
Quoted 4 lines
I don't know the requirements of Stratos so I can't say if creating a new hypervisor-independent interface (VIRTIO transport) that doesn't rely on shared memory vrings makes sense. I just wanted to raise the idea in case you find that VIRTIO's vrings don't meet your requirements.
Quoted 7 lines
While I cannot represent the project's view, what the JIRA task that is assigned to me describes:
Deliverables
* Low level library allowing:
* management of virtio rings and buffers
[and so on]
So supporting the shared memory-based vring is one of our assumptions.
If shared memory is allowed then vrings are the natural choice. That way existing virtio code will work with minimal modifications.
Stefan
Hi Wei,
Quoted 108 lines · AKASHI Takahiro
Quoted 107 lines · Wei Chen
Hi Akashi，
Quoted 104 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Stratos Mailing List <[email protected]>; [email protected] open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsQuoted 8 lines · Wei Chen
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>;Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
Quoted 9 lines
Quoted 9 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 14 lines
Quoted 14 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHi Wei, Oleksandr,
Quoted 7 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device implementations.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always represent # Linaro's view on this subject nor reflect Alex's concerns.
Nevertheless,
Quoted 9 lines
Quoted 9 lines
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first, Xen and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces has a same set of interfaces (for your "struct
vmm_impl" and my "RPC interfaces"). This is not co-incident as we both share the same origin as I saidabove.
Quoted 9 lines
Quoted 6 lines
And so we will also share the same issues. One of them is a way of "sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too.
(See Alex's original email, too).Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a "single-binary"
solution
Quoted 5 lines
Quoted 5 lines
in which the same binary of BE vm could run on any hypervisors. Somehow similar to your "proposal-#2" in [2], but in my solution, all the hypervisor-specific code would be put into another entity (VM), named "virtio-proxy" and the abstracted operations are served via RPC. (In this sense, BE is hypervisor-agnostic but might have OS
dependency.)
Quoted 9 lines
Quoted 3 lines
But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy completely (I will do it ASAP). But from your description, it seems we need a 3rd VM between FE and BE? My concern is that, if my assumption is right, will it increase the latency in data transport path? Even if we're using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of payload, we will see latency issue (even in your case) and if we have some solution for it, we won't see it neither in my proposal :)
Quoted 6 lines
Quoted 3 lines
Oleksandr has sent a proposal to Xen mailing list to reduce this kind of "mapping/unmapping" operations. So the latency caused by this behavior on Xen may eventually be eliminated, and Linux-KVM doesn't have that problem.
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
Can you give me the link to the discussion or patch, please?
Thanks,
-Takahiro Akashi
Quoted 659 lines
-Takahiro Akashi
Quoted 657 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Specifically speaking about kvm-tool, I have a concern about its license term; Targeting different hypervisors and different OSs (which I assume includes RTOS's), the resultant library should be license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Quoted 8 lines
Quoted 3 lines
Yes. If user want to implement a FreeBSD device model, but the virtio library is GPL. Then GPL would be a problem. If we have another good candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD codeInteresting, I will look into them : )
Cheers,
Wei ChenQuoted 627 lines
-Takahiro Akashi
Quoted 11 lines
Quoted 11 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; StefanoStabellini
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos Mailing ListArnd
Quoted 9 lines
Quoted 9 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 12 lines
Quoted 12 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Quoted 5 lines
Quoted 5 lines
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hello, all.
Please see some comments below. And sorry for the possible format
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano Stabellini
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not trimming
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters, but
because it
Quoted 8 lines
Quoted 8 lines
is
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server). It has been, I believe, well generalized but is still a bit
biased
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more about my
approach
Quoted 6 lines
Quoted 6 lines
below.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · Alex Bennée
Quoted 3 lines
Hi,
One of the goals of Project Stratos is to enable hypervisor
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code as possible
and
Quoted 8 lines
Quoted 8 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the front end
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required - one per OS,
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant guests are
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any abstracted paravirt
type
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution I envisioned
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like environment. The
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some mapped
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is eventfd).
The
Quoted 4 lines
Quoted 4 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any hypervisor specific
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the actual
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion in the
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI. There
interest
Quoted 6 lines
Quoted 6 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to abstract
differences
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation"
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed from the
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However we are
some
Quoted 4 lines
Quoted 4 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at least
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying requirements. To
that
Quoted 7 lines
Quoted 7 lines
end I
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
want to think about what we need for a backend to function.
Configuration
=============In the type-2 setup this is typically fairly simple because
the
Quoted 6 lines
Quoted 6 lines
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that make up the
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with delegated
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the backend VM
about
Quoted 11 lines
Quoted 11 lines
key
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick)events
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into your guest
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious solution
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user space but
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest and the BE
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address space needs to
be
Quoted 11 lines
Quoted 11 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding clashing. I
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface to
userspace
Quoted 6 lines
Quoted 6 lines
to then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good insights on
this
Quoted 5 lines
Quoted 5 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-agnostic
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a PCI device
and
Quoted 9 lines
Quoted 9 lines
can be
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 3 lines
Quoted 3 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle"
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0. So for
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interfaceto
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
map the configuration space.
Yes, Xenstore was chosen as a simple way to pass configuration
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in Virtio
backend
Quoted 3 lines
Quoted 3 lines
itself if possible, so for non-toolstack domain, this could done with adjusting devd (daemon that listens for devices and launches backends)
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway and pass it
to
Quoted 6 lines
Quoted 6 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass device
configuration.
Quoted 1 line
We also designed a static device configuration parse method for
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device model
command
Quoted 9 lines
Quoted 9 lines
line
Quoted 6 lines
or a config file.
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or hypervisor)-
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM, to hide
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found and proven
to
Quoted 5 lines
Quoted 5 lines
work then even better.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
# My point is that a "handle" is not mandatory for executing
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a newXen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine privileges to
allow
Quoted 4 lines
Quoted 4 lines
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no problems of
course.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my approach is that
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping memory) is
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have to extend
Quoted 8 lines
Quoted 8 lines
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security and privilegedistribution
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
as Stefan suggested.
We also faced policy related issues with Virtio backend running in
other than Dom0 domain in a "dummy" xsm mode. In our target system we
run
Quoted 2 lines
Quoted 2 lines
the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W resides. We
trust it,
Quoted 1 line
Quoted 1 line
so we wrote policy rules (to be used in "flask" xsm mode) to provide
it
Quoted 8 lines
Quoted 8 lines
with a little bit more privileges than a simple DomU had.
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and memory
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can seetwo
Quoted 4 lines
Quoted 4 lines
options
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for daemon/helper to
trigger
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If using a
helper
Quoted 4 lines
Quoted 4 lines
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid the daemon
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that complexity
into
Quoted 10 lines
Quoted 10 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would result in
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the userspace would
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to userspace
ptr"
Quoted 4 lines
Quoted 4 lines
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall interfaces.
This of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
course assumes the majority of BE guests would be Linux
Quoted 8 lines
Quoted 8 lines
kernels and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
leaves the bare-metal/unikernel approaches to their own
devices.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Operation
=========The core of the operation of VirtIO is fairly simple. Once
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user feature negotiation is done it's a case of
receiving
Quoted 2 lines
Quoted 2 lines
update
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
events and parsing the resultant virt queue for data. The
vhost-
Quoted 2 lines
Quoted 2 lines
user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
specification handles a bunch of setup before that point,
mostly
Quoted 4 lines
Quoted 4 lines
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
detail where the virt queues are set up FD's for memory and
event
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communication. This is where the envisioned stub process
would
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
responsible for getting the daemon up and ready to run. This
is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
currently done inside a big VMM like QEMU but I suspect a
modern
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
approach would be to use the rust-vmm vhost crate. It would
then
Quoted 2 lines
Quoted 2 lines
either
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communicate with the kernel's abstracted ABI or be re-
targeted
Quoted 4 lines
Quoted 4 lines
as a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have
VMMs
Quoted 6 lines
Quoted 6 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
way they are typically envisioned and described in other
environments.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead, Xen has IOREQ servers. Each of them connects
independently to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen via the IOREQ interface. E.g. today multiple QEMUs could
be
Quoted 5 lines
Quoted 5 lines
used as
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
The component responsible for starting a daemon and/or setting
up
Quoted 2 lines
Quoted 2 lines
shared
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interfaces is the toolstack: the xl command and the
libxl/libxc
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
libraries.
I think that VM configuration management (or orchestration in
Startos
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
jargon?) is a subject to debate in parallel. Otherwise, is there any good assumption to avoid it right now?
Quoted 1 line
Oleksandr and others I CCed have been working on ways for the
toolstack
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to create virtio backends and setup memory mappings. They
might be
Quoted 2 lines
Quoted 2 lines
able
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to provide more info on the subject. I do think we miss a way
to
Quoted 6 lines
Quoted 6 lines
provide
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the configuration to the backend and anything else that the
backend
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
might require to start doing its job.
Yes, some work has been done for the toolstack to handle Virtio
MMIO
Quoted 2 lines
Quoted 2 lines
devices in
Quoted 1 line
Quoted 1 line
general and Virtio block devices in particular. However, it has
not
Quoted 3 lines
Quoted 3 lines
been upstreaned yet.
Quoted 2 lines
Quoted 2 lines
Updated patches on review now:
https://lore.kernel.org/xen-devel/1621626361-29076-1-git-send-email-
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
There is an additional (also important) activity to improve/fix
foreign memory mapping on Arm which I am also involved in.
Quoted 1 line
Quoted 1 line
The foreign memory mapping is proposed to be used for Virtio
backends
Quoted 3 lines
Quoted 3 lines
(device emulators) if there is a need to run guest OS completely unmodified.
Quoted 1 line
Quoted 1 line
Of course, the more secure way would be to use grant memory
mapping.
Quoted 1 line
Quoted 1 line
Brietly, the main difference between them is that with foreign mapping
the
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
can map any guest memory it wants to map, but with grant mapping
it is
Quoted 3 lines
Quoted 3 lines
allowed to map only what was previously granted by the frontend.
Quoted 2 lines
Quoted 2 lines
So, there might be a problem if we want to pre-map some guest
memory
Quoted 4 lines
Quoted 4 lines
in advance or to cache mappings in the backend in order to improve performance (because the mapping/unmapping guest pages every request requires a lot of back and forth to Xen + P2M updates). In a nutshell, currently, in order to map a guest page into the backend address space
we
Quoted 1 line
Quoted 1 line
need to steal a real physical page from the backend domain. So, with
the
Quoted 2 lines
Quoted 2 lines
said optimizations we might end up with no free memory in the backend domain (see XSA-300). And what we try to achieve is to not waste a
real
Quoted 2 lines
Quoted 2 lines
domain memory at all by providing safe non-allocated-yet (so unused) address space for the foreign (and grant) pages to be mapped into,
this
Quoted 2 lines
Quoted 2 lines
enabling work implies Xen and Linux (and likely DTB bindings) changes. However, as it turned out, for this to work in a proper and safe way
some
Quoted 3 lines
Quoted 3 lines
prereq work needs to be done.
Quoted 2 lines
Quoted 2 lines
You can find the related Xen discussion at:
https://lore.kernel.org/xen-devel/1627489110-25633-1-git-send-email-
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
One question is how to best handle notification and kicks.
The
Quoted 2 lines
Quoted 2 lines
existing
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user framework uses eventfd to signal the daemon
(although
Quoted 2 lines
Quoted 2 lines
QEMU
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is quite capable of simulating them when you use TCG). Xen
has
Quoted 6 lines
Quoted 6 lines
it's own
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
IOREQ mechanism. However latency is an important factor and
having
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the
backends to
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisorinterface
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels),which is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
I don't know what "connect directly" means here, but sending
interrupts
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-xmechanism.
Quoted 3 lines
Quoted 3 lines
Agree that MSI would be more efficient than SPI...
At the moment, in order to notify the frontend, the backend issuesa
Quoted 1 line
Quoted 1 line
specific device-model call to query Xen to inject a corresponding SPI
to
Quoted 5 lines
Quoted 5 lines
the guest.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Could we consider the kernel internally converting IOREQ
messages from
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the Xen hypervisor to eventfd events? Would this scale with
other kernel
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting
with?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One option we should consider is for each backend to connect
to
Quoted 2 lines
Quoted 2 lines
Xen via
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the IOREQ interface. We could generalize the IOREQ interface
and
Quoted 2 lines
Quoted 2 lines
make it
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor agnostic. The interface is really trivial and easy
to
Quoted 3 lines
Quoted 3 lines
add.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As I said above, my proposal does the same thing that you
mentioned
Quoted 2 lines
Quoted 2 lines
here :)
Quoted 1 line
Quoted 1 line
Quoted 1 line
The difference is that I do call hypervisor interfaces via
virtio-
Quoted 3 lines
Quoted 3 lines
proxy.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The only Xen-specific part is the notification mechanism,
which is
Quoted 8 lines
Quoted 8 lines
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
event channel. If we replaced the event channel with something
else the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I don't think that translating IOREQs to eventfd in the kernel
is
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
good idea: if feels like it would be extra complexity and that
the
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
kernel shouldn't be involved as this is a backend-hypervisor
interface.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Given that we may want to implement BE as a bare-metal
application
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
as I did on Zephyr, I don't think that the translation would not
be
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).Quoted 1 line
Also, eventfd is very Linux-centric and we are trying to
design an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interface that could work well for RTOSes too. If we want to
do
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
something different, both OS-agnostic and hypervisor-agnostic,
perhaps
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we could design a new interface. One that could be
implementable
Quoted 7 lines
Quoted 7 lines
in the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen hypervisor itself (like IOREQ) and of course any other
hypervisor
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
too.
There is also another problem. IOREQ is probably not be the
only
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't wealso need
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an interface for the backend to inject interrupts into the
frontend? And
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if the backend requires dynamic memory mappings of frontend
pages,
Quoted 4 lines
Quoted 4 lines
then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
we would also need an interface to map/unmap domU pages.
My proposal document might help here; All the interfaces
required
Quoted 5 lines
Quoted 5 lines
for
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 1 line
These interfaces are a lot more problematic than IOREQ: IOREQ
is
Quoted 2 lines
Quoted 2 lines
tiny
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and self-contained. It is easy to add anywhere. A new
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
inject interrupts or map pages is more difficult to manage
because
Quoted 4 lines
Quoted 4 lines
it
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also
apply
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is adifferent
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
matter.
Thanks,
-Takahiro AkashiQuoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
--
Regards,Oleksandr Tyshchenko
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the
Quoted 8 lines
Quoted 1 line
Quoted 1 line
contents to any other person, use it for any purpose, or store or copy
the
Quoted 2 lines
Quoted 1 line
information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Akashi,
Quoted 156 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月26日 17:41
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly Xin <[email protected]>; Stratos Mailing List <[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHi Wei,
Quoted 9 lines · AKASHI Takahiro
Quoted 8 lines · Wei Chen
Hi Akashi，
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>;Stratos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 1 line
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsQuoted 7 lines · Wei Chen
Hi Akashi,
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; StefanoStabellini
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e <[email protected]>;
Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
Quoted 4 lines
Quoted 4 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHi Wei, Oleksandr,
Quoted 7 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device implementations.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always
represent
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
# Linaro's view on this subject nor reflect Alex's concerns.
Nevertheless,
Quoted 3 lines
Quoted 3 lines
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first,
Xen
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces has a
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
set of interfaces (for your "struct
vmm_impl" and my "RPCinterfaces").
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This is not co-incident as we both share the same origin as I
said
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
above.
Quoted 1 line
Quoted 1 line
And so we will also share the same issues. One of them is a way
of
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
"sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too. (See Alex's original email, too).Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a "single-binary"
solution
Quoted 2 lines
Quoted 2 lines
in which the same binary of BE vm could run on any hypervisors. Somehow similar to your "proposal-#2" in [2], but in my solution,
all
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the hypervisor-specific code would be put into another entity
(VM),
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
named "virtio-proxy" and the abstracted operations are served
via RPC.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
(In this sense, BE is hypervisor-agnostic but might have OS
dependency.)
Quoted 5 lines
Quoted 3 lines
But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy
completely
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(I will do it ASAP). But from your description, it seems we need a 3rd VM between FE and BE? My concern is that, if my assumption is
right,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
will it increase the latency in data transport path? Even if we're using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of payload, we will see latency issue (even in your case) and if we have some
solution
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
for it, we won't see it neither in my proposal :)
Oleksandr has sent a proposal to Xen mailing list to reduce this kind of "mapping/unmapping" operations. So the latency caused by this
Quoted 9 lines
behavior
Quoted 1 line
Quoted 1 line
on Xen may eventually be eliminated, and Linux-KVM doesn't have that
problem.
Quoted 3 lines
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
Can you give me the link to the discussion or patch, please?
It's a RFC discussion. We have tested this RFC patch internally.
https://lists.xenproject.org/archives/html/xen-devel/2021-07/msg01532.html
Quoted 832 lines
Thanks,
-Takahiro AkashiQuoted 5 lines
-Takahiro Akashi
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Specifically speaking about kvm-tool, I have a concern about its license term; Targeting different hypervisors and different OSs (which I assume includes RTOS's), the resultant library should
be
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Yes. If user want to implement a FreeBSD device model, but the
virtio
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
library is GPL. Then GPL would be a problem. If we have another
good
Quoted 23 lines
Quoted 23 lines
Quoted 6 lines
Quoted 1 line
candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD codeInteresting, I will look into them : )
Cheers,
Wei ChenQuoted 11 lines
-Takahiro Akashi
Quoted 8 lines
Quoted 8 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; StefanoStabellini
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos MailingList
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
open.org;
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Arnd
Quoted 4 lines
Quoted 4 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>;
Oleksandr
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Quoted 1 line
Quoted 1 line
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hello, all.
Please see some comments below. And sorry for the possible
format
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano
Stabellini
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not
trimming
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters, but
because it
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working
prototype)[1]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
is based on Xen's virtio implementation (i.e. IOREQ) and
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server). It has been, I believe, well generalized but is still a
bit
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
biased
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
dev/2021-
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more about
my
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
approach
Quoted 10 lines
Quoted 10 lines
below.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 3 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code as
possible
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
Quoted 2 lines
Quoted 2 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the
front end
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required - one
per OS,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant guests
are
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any abstracted
paravirt
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
type
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution I
envisioned
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like environment.
The
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some mapped
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is
eventfd).
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any hypervisor
specific
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the actual
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion in
the
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI. There
interest
Quoted 2 lines
Quoted 2 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to
abstract
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
differences
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall
Standardisation"
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed from the
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However we
are
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
some
Quoted 4 lines
Quoted 4 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at least
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying
requirements. To
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
that
Quoted 2 lines
Quoted 2 lines
end I
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
want to think about what we need for a backend to
function.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Configuration
=============In the type-2 setup this is typically fairly simple
because
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that make
up the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with
delegated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the backend
VM
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
about
Quoted 5 lines
Quoted 5 lines
key
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger(kick)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
events
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- what (if any) resources the backend needs to
connect to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Obviously you can elide over configuration issues by
having
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into your
guest
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious
solution
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user space
but
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest and the
BE
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address space
needs to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can
envision 3
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's
Guest
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding
clashing. I
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface to
userspace
Quoted 4 lines
Quoted 4 lines
to then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
make that address space visible to the backend daemon.
Yet another way here is that we would have well known
"shared
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good
insights on
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
this
Quoted 2 lines
Quoted 2 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-
agnostic
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a PCI
device
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
and
Quoted 4 lines
Quoted 4 lines
can be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mapped locally.)
I want to add this shared memory aspect to my virtio-proxy,
but
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the resultant solution would eventually look similar to
ivshmem.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to
where
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is no
"handle"
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0. So
for
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following
information
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisorinterface
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
map the configuration space.
Yes, Xenstore was chosen as a simple way to pass
configuration
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in
Virtio
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
itself if possible, so for non-toolstack domain, this could done
with
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
adjusting devd (daemon that listens for devices and launches
backends)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway and
pass it
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to
Quoted 6 lines
Quoted 6 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass device
configuration.
Quoted 1 line
We also designed a static device configuration parse method
for
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device
model
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
command
Quoted 7 lines
Quoted 7 lines
line
Quoted 6 lines
or a config file.
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or
hypervisor)-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM, to
hide
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found and
proven
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
to
Quoted 5 lines
Quoted 5 lines
work then even better.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
# My point is that a "handle" is not mandatory for
executing
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent anew
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine privileges
to
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
allow
Quoted 4 lines
Quoted 4 lines
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no problems
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
course.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my approach is
that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping
memory) is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have to
extend
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security andprivilege
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
distribution
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
as Stefan suggested.
We also faced policy related issues with Virtio backend
running in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
other than Dom0 domain in a "dummy" xsm mode. In our target
system we
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
run
Quoted 2 lines
Quoted 2 lines
the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W resides.
We
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trust it,
Quoted 1 line
Quoted 1 line
so we wrote policy rules (to be used in "flask" xsm mode) to
provide
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
it
Quoted 2 lines
Quoted 2 lines
with a little bit more privileges than a simple DomU had.
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and
memory
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. Ican see
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
two
Quoted 4 lines
Quoted 4 lines
options
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for daemon/helper
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
trigger
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If
using a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
helper
Quoted 2 lines
Quoted 2 lines
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid the
daemon
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that
complexity
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
into
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would
result in
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the userspace
would
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to
userspace
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
ptr"
Quoted 2 lines
Quoted 2 lines
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall
interfaces.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
course assumes the majority of BE guests would be
Linux
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
kernels and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
leaves the bare-metal/unikernel approaches to
their own
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
devices.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Operation
=========The core of the operation of VirtIO is fairly simple.
Once
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user feature negotiation is done it's a case of
receiving
Quoted 2 lines
Quoted 2 lines
update
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
events and parsing the resultant virt queue for data.
The
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
vhost-
Quoted 2 lines
Quoted 2 lines
user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
specification handles a bunch of setup before that
point,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mostly
Quoted 2 lines
Quoted 2 lines
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
detail where the virt queues are set up FD's for
memory and
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
event
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communication. This is where the envisioned stub
process
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
would
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
responsible for getting the daemon up and ready to run.
This
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
currently done inside a big VMM like QEMU but I
suspect a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
modern
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
approach would be to use the rust-vmm vhost crate. It
would
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
then
Quoted 2 lines
Quoted 2 lines
either
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communicate with the kernel's abstracted ABI or be re-
targeted
Quoted 4 lines
Quoted 4 lines
as a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't
have
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
VMMs
Quoted 6 lines
Quoted 6 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
way they are typically envisioned and described in other
environments.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead, Xen has IOREQ servers. Each of them connects
independently to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen via the IOREQ interface. E.g. today multiple QEMUs
could
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
used as
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
emulators for a single Xen VM, each of them connecting
to Xen
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
independently via the IOREQ interface.
The component responsible for starting a daemon and/or
setting
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
up
Quoted 2 lines
Quoted 2 lines
shared
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interfaces is the toolstack: the xl command and the
libxl/libxc
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
libraries.
I think that VM configuration management (or orchestration
in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Startos
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
jargon?) is a subject to debate in parallel. Otherwise, is there any good assumption to avoid it right
now?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Oleksandr and others I CCed have been working on ways
for the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
toolstack
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to create virtio backends and setup memory mappings.
They
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
might be
Quoted 2 lines
Quoted 2 lines
able
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to provide more info on the subject. I do think we miss
a way
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
provide
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the configuration to the backend and anything else that
the
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
backend
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
might require to start doing its job.
Yes, some work has been done for the toolstack to handle
Virtio
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
MMIO
Quoted 2 lines
Quoted 2 lines
devices in
Quoted 1 line
Quoted 1 line
general and Virtio block devices in particular. However, it
has
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
not
Quoted 3 lines
Quoted 3 lines
been upstreaned yet.
Quoted 2 lines
Quoted 2 lines
Updated patches on review now:
https://lore.kernel.org/xen-devel/1621626361-29076-1-git-send-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
email-
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
There is an additional (also important) activity to
improve/fix
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
foreign memory mapping on Arm which I am also involved in.
Quoted 1 line
Quoted 1 line
The foreign memory mapping is proposed to be used for Virtio
backends
Quoted 3 lines
Quoted 3 lines
(device emulators) if there is a need to run guest OS completely unmodified.
Quoted 1 line
Quoted 1 line
Of course, the more secure way would be to use grant memory
2 lines not shown · raw message
252 lines not shown · raw message
Quoted 8 lines
8 lines not shown · raw message
1 line not shown · raw message
Part of this message is not shown here. View the raw message.
Apologies for being late to this thread, but I hope to be able to contribute to
this discussion in a meaningful way. I am grateful for the level of interest in
this topic. I would like to draw your attention to Argo as a suitable technology for development of VirtIO's hypervisor-agnostic interfaces.
* Argo is an interdomain communication mechanism in Xen (on x86 and Arm) that
can send and receive hypervisor-mediated notifications and messages
between
domains (VMs). [1] The hypervisor can enforce Mandatory Access Control
over
all communication between domains. It is derived from the earlier v4v,
which
has been deployed on millions of machines with the HP/Bromium uXen
hypervisor
and with OpenXT.
* Argo has a simple interface with a small number of operations that was designed for ease of integration into OS primitives on both Linux
(sockets)
and Windows (ReadFile/WriteFile) [2].
- A unikernel example of using it has also been developed for XTF. [3]
* There has been recent discussion and support in the Xen community for making
revisions to the Argo interface to make it hypervisor-agnostic, and
support
implementations of Argo on other hypervisors. This will enable a single interface for an OS kernel binary to use for inter-VM communication that
will
work on multiple hypervisors -- this applies equally to both backends and frontend implementations. [4]
* Here are the design documents for building VirtIO-over-Argo, to support a hypervisor-agnostic frontend VirtIO transport driver using Argo.
The Development Plan to build VirtIO virtual device support over Argo transport:
https://openxt.atlassian.net/wiki/spaces/DC/pages/1696169985/VirtIO-Argo+Development+Phase+1
A design for using VirtIO over Argo, describing how VirtIO data structures and communication is handled over the Argo transport:
https://openxt.atlassian.net/wiki/spaces/DC/pages/1348763698/VirtIO+Argo
Diagram (from the above document) showing how VirtIO rings are synchronized between domains without using shared memory:
https://openxt.atlassian.net/46e1c93b-2b87-4cb2-951e-abd4377a1194#media-blob-url=true&id=01f7d0e1-7686-4f0b-88e1-457c1d30df40&collection=contentId-1348763698&contextId=1348763698&mimeType=image%2Fpng&name=device-buffer-access-virtio-argo.png&size=243175&width=1106&height=1241
Please note that the above design documents show that the existing VirtIO device drivers, and both vring and virtqueue data structures can be preserved
while interdomain communication can be performed with no shared memory required
for most drivers; (the exceptions where further design is required are those such as virtual framebuffer devices where shared memory regions are intentionally
added to the communication structure beyond the vrings and virtqueues).
An analysis of VirtIO and Argo, informing the design:
https://openxt.atlassian.net/wiki/spaces/DC/pages/1333428225/Analysis+of+Argo+as+a+transport+medium+for+VirtIO
* Argo can be used for a communication path for configuration between the backend
and the toolstack, avoiding the need for a dependency on XenStore, which
is an
advantage for any hypervisor-agnostic design. It is also amenable to a
notification
mechanism that is not based on Xen event channels.
* Argo does not use or require shared memory between VMs and provides an alternative
to the use of foreign shared memory mappings. It avoids some of the
complexities
involved with using grants (eg. XSA-300).
* Argo supports Mandatory Access Control by the hypervisor, satisfying a common
certification requirement.
* The Argo headers are BSD-licensed and the Xen hypervisor implementation is GPLv2 but
accessible via the hypercall interface. The licensing should not present
an obstacle
to adoption of Argo in guest software or implementation by other
hypervisors.
* Since the interface that Argo presents to a guest VM is similar to DMA, a VirtIO-Argo
frontend transport driver should be able to operate with a physical
VirtIO-enabled
smart-NIC if the toolstack and an Argo-aware backend provide support.
The next Xen Community Call is next week and I would be happy to answer questions
about Argo and on this topic. I will also be following this thread.
Christopher
(Argo maintainer, Xen Community)
--------------------------------------------------------------------------------
[1]
An introduction to Argo:
https://static.sched.com/hosted_files/xensummit19/92/Argo%20and%20HMX%20-%20OpenXT%20-%20Christopher%20Clark%20-%20Xen%20Summit%202019.pdf
https://www.youtube.com/watch?v=cnC0Tg3jqJQ
Xen Wiki page for Argo:
https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen
[2]
OpenXT Linux Argo driver and userspace library:
https://github.com/openxt/linux-xen-argo
Windows V4V at OpenXT wiki:
https://openxt.atlassian.net/wiki/spaces/DC/pages/14844007/V4V
Windows v4v driver source:
https://github.com/OpenXT/xc-windows/tree/master/xenv4v
HP/Bromium uXen V4V driver:
https://github.com/uxen-virt/uxen/tree/ascara/windows/uxenv4vlib
[3]
v2 of the Argo test unikernel for XTF:
https://lists.xenproject.org/archives/html/xen-devel/2021-01/msg02234.html
[4]
Argo HMX Transport for VirtIO meeting minutes:
https://lists.xenproject.org/archives/html/xen-devel/2021-02/msg01422.html
VirtIO-Argo Development wiki page:
https://openxt.atlassian.net/wiki/spaces/DC/pages/1696169985/VirtIO-Argo+Development+Phase+1
Quoted 1,067 lines · Wei Chen
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月26日 17:41
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; KalyXin
Quoted 1 line
<[email protected]>; Stratos Mailing List <
Quoted 10 lines
[email protected]; Arnd Bergmann <[email protected] ;
Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;Julien
Quoted 40 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsHi Wei,
Quoted 9 lines · AKASHI Takahiro
Quoted 8 lines · Wei Chen
Hi Akashi，
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>;Stratos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]
;
JulienQuoted 1 line
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsQuoted 2 lines · Wei Chen
Hi Akashi,
Quoted 12 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; StefanoStabellini
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e <[email protected]
;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
Quoted 1 line
Quoted 1 line
open.org; Arnd Bergmann <[email protected]>; Viresh
Kumar
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]
;
Jean-Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIObackends
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Hi Wei, Oleksandr,
Quoted 6 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device
implementations.
Quoted 17 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always
represent
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
# Linaro's view on this subject nor reflect Alex's concerns.
Nevertheless,
Quoted 3 lines
Quoted 3 lines
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first,
Xen
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces has a
Quoted 19 lines
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
set of interfaces (for your "struct
vmm_impl" and my "RPCinterfaces").
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This is not co-incident as we both share the same origin as I
said
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
above.
Quoted 1 line
Quoted 1 line
And so we will also share the same issues. One of them is a way
of
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
"sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too. (See Alex's original email, too).Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a "single-binary"
solution
Quoted 2 lines
Quoted 2 lines
in which the same binary of BE vm could run on any hypervisors. Somehow similar to your "proposal-#2" in [2], but in my
solution,
Quoted 5 lines
all
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the hypervisor-specific code would be put into another entity
(VM),
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
named "virtio-proxy" and the abstracted operations are served
via RPC.
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
(In this sense, BE is hypervisor-agnostic but might have OS
dependency.)
Quoted 5 lines
Quoted 3 lines
But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy
completely
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(I will do it ASAP). But from your description, it seems we need
a
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
3rd VM between FE and BE? My concern is that, if my assumption is
right,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
will it increase the latency in data transport path? Even if
we're
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of
payload,
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
we will see latency issue (even in your case) and if we have some
solution
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
for it, we won't see it neither in my proposal :)
Oleksandr has sent a proposal to Xen mailing list to reduce this kind of "mapping/unmapping" operations. So the latency caused by this
Quoted 9 lines
behavior
Quoted 1 line
Quoted 1 line
on Xen may eventually be eliminated, and Linux-KVM doesn't have that
problem.
Quoted 3 lines
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
Can you give me the link to the discussion or patch, please?
It's a RFC discussion. We have tested this RFC patch internally.
https://lists.xenproject.org/archives/html/xen-devel/2021-07/msg01532.htmlQuoted 6 lines
Thanks,
-Takahiro AkashiQuoted 3 lines
-Takahiro Akashi
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Specifically speaking about kvm-tool, I have a concern about
its
Quoted 37 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
license term; Targeting different hypervisors and different OSs (which I assume includes RTOS's), the resultant library should
be
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Yes. If user want to implement a FreeBSD device model, but the
virtio
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
library is GPL. Then GPL would be a problem. If we have another
good
Quoted 26 lines
Quoted 26 lines
Quoted 6 lines
Quoted 1 line
candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD codeInteresting, I will look into them : )
Cheers,
Wei ChenQuoted 14 lines
-Takahiro Akashi
Quoted 11 lines
Quoted 11 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; StefanoQuoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Stabellini
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos MailingList
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
open.org;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Arnd
Quoted 3 lines
Quoted 3 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 75 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]
;
Jean-Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>;
Oleksandr
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Quoted 1 line
Quoted 1 line
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hello, all.
Please see some comments below. And sorry for the possible
format
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano
Stabellini
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not
trimming
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters, but
because it
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working
prototype)[1]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
is based on Xen's virtio implementation (i.e. IOREQ) and
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server). It has been, I believe, well generalized but is still a
bit
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
biased
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
dev/2021-
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more about
my
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
approach
Quoted 10 lines
Quoted 10 lines
below.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 3 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code as
Quoted 10 lines
possible
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
Quoted 2 lines
Quoted 2 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the
front end
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required - one
per OS,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant
guests
Quoted 4 lines
are
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any abstracted
paravirt
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
type
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution I
envisioned
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like environment.
The
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some
mapped
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is
eventfd).
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any hypervisor
specific
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the actual
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion in
the
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI.
There
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interest
Quoted 2 lines
Quoted 2 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to
abstract
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
differences
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
away as well as the idea of bare-metal Rust backends.
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
We have a card (STR-12) called "Hypercall
Standardisation"
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed from the
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However we
are
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
some
Quoted 2 lines
Quoted 2 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at
least
Quoted 66 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying
requirements. To
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
that
Quoted 2 lines
Quoted 2 lines
end I
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
want to think about what we need for a backend to
function.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Configuration
=============In the type-2 setup this is typically fairly simple
because
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that make
up the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with
delegated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the backend
VM
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
about
Quoted 5 lines
Quoted 5 lines
key
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger(kick)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
events
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- what (if any) resources the backend needs to
connect to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Obviously you can elide over configuration issues by
having
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into your
guest
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious
solution
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user space
but
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest and the
BE
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address space
needs to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can
envision 3
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
approaches:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's
Guest
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding
clashing. I
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface to
userspace
Quoted 2 lines
Quoted 2 lines
to then
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make that address space visible to the backend
daemon.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Yet another way here is that we would have well known
"shared
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good
insights on
Quoted 13 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
this
Quoted 2 lines
Quoted 2 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-
agnostic
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a PCI
device
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
and
Quoted 4 lines
Quoted 4 lines
can be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mapped locally.)
I want to add this shared memory aspect to my
virtio-proxy,
Quoted 6 lines
but
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the resultant solution would eventually look similar to
ivshmem.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
* BE guests boots with a hypervisor handle to memory
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The BE guest is then free to map the FE's memory to
where
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is no
"handle"
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0.
So
Quoted 18 lines
for
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following
information
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisorinterface
Quoted 1 line
Quoted 1 line
Quoted 1 line
to
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
map the configuration space.
Yes, Xenstore was chosen as a simple way to pass
configuration
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in
Virtio
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
itself if possible, so for non-toolstack domain, this could
done
Quoted 33 lines
with
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
adjusting devd (daemon that listens for devices and launches
backends)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway and
pass it
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to
Quoted 6 lines
Quoted 6 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass device
configuration.
Quoted 1 line
We also designed a static device configuration parse method
for
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device
model
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
command
Quoted 7 lines
Quoted 7 lines
line
Quoted 6 lines
or a config file.
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or
hypervisor)-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM, to
hide
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found and
Quoted 12 lines
proven
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
to
Quoted 5 lines
Quoted 5 lines
work then even better.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
# My point is that a "handle" is not mandatory for
executing
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the toolstack (also see below.) Or we would have to invent
a
Quoted 106 lines
new
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine privileges
to
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
allow
Quoted 4 lines
Quoted 4 lines
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no problems
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
course.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my approach is
that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping
memory) is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have to
extend
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security andprivilege
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
distribution
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
as Stefan suggested.
We also faced policy related issues with Virtio backend
running in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
other than Dom0 domain in a "dummy" xsm mode. In our target
system we
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
run
Quoted 2 lines
Quoted 2 lines
the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W resides.
We
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trust it,
Quoted 1 line
Quoted 1 line
so we wrote policy rules (to be used in "flask" xsm mode) to
provide
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
it
Quoted 2 lines
Quoted 2 lines
with a little bit more privileges than a simple DomU had.
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and
memory
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. Ican see
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
two
Quoted 4 lines
Quoted 4 lines
options
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for daemon/helper
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
trigger
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If
using a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
helper
Quoted 2 lines
Quoted 2 lines
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid the
daemon
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that
complexity
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
into
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would
result in
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the userspace
would
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to
userspace
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
ptr"
Quoted 2 lines
Quoted 2 lines
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall
23 lines not shown · raw message
Quoted 11 lines
11 lines not shown · raw message
373 lines not shown · raw message
Part of this message is not shown here. View the raw message.
[ resending message to ensure delivery to the CCd mailing lists post-subscription ]
Apologies for being late to this thread, but I hope to be able to contribute to
this discussion in a meaningful way. I am grateful for the level of interest in
this topic. I would like to draw your attention to Argo as a suitable technology for development of VirtIO's hypervisor-agnostic interfaces.
* Argo is an interdomain communication mechanism in Xen (on x86 and Arm) that
can send and receive hypervisor-mediated notifications and messages
between
domains (VMs). [1] The hypervisor can enforce Mandatory Access Control
over
all communication between domains. It is derived from the earlier v4v,
which
has been deployed on millions of machines with the HP/Bromium uXen
hypervisor
and with OpenXT.
* Argo has a simple interface with a small number of operations that was designed for ease of integration into OS primitives on both Linux
(sockets)
and Windows (ReadFile/WriteFile) [2].
- A unikernel example of using it has also been developed for XTF. [3]
* There has been recent discussion and support in the Xen community for making
revisions to the Argo interface to make it hypervisor-agnostic, and
support
implementations of Argo on other hypervisors. This will enable a single interface for an OS kernel binary to use for inter-VM communication that
will
work on multiple hypervisors -- this applies equally to both backends and frontend implementations. [4]
* Here are the design documents for building VirtIO-over-Argo, to support a hypervisor-agnostic frontend VirtIO transport driver using Argo.
The Development Plan to build VirtIO virtual device support over Argo transport:
https://openxt.atlassian.net/wiki/spaces/DC/pages/1696169985/VirtIO-Argo+Development+Phase+1
A design for using VirtIO over Argo, describing how VirtIO data structures and communication is handled over the Argo transport:
https://openxt.atlassian.net/wiki/spaces/DC/pages/1348763698/VirtIO+Argo
Diagram (from the above document) showing how VirtIO rings are synchronized between domains without using shared memory:
https://openxt.atlassian.net/46e1c93b-2b87-4cb2-951e-abd4377a1194#media-blob-url=true&id=01f7d0e1-7686-4f0b-88e1-457c1d30df40&collection=contentId-1348763698&contextId=1348763698&mimeType=image%2Fpng&name=device-buffer-access-virtio-argo.png&size=243175&width=1106&height=1241
Please note that the above design documents show that the existing VirtIO device drivers, and both vring and virtqueue data structures can be preserved
while interdomain communication can be performed with no shared memory required
for most drivers; (the exceptions where further design is required are those such as virtual framebuffer devices where shared memory regions are intentionally
added to the communication structure beyond the vrings and virtqueues).
An analysis of VirtIO and Argo, informing the design:
https://openxt.atlassian.net/wiki/spaces/DC/pages/1333428225/Analysis+of+Argo+as+a+transport+medium+for+VirtIO
* Argo can be used for a communication path for configuration between the backend
and the toolstack, avoiding the need for a dependency on XenStore, which
is an
advantage for any hypervisor-agnostic design. It is also amenable to a
notification
mechanism that is not based on Xen event channels.
* Argo does not use or require shared memory between VMs and provides an alternative
to the use of foreign shared memory mappings. It avoids some of the
complexities
involved with using grants (eg. XSA-300).
* Argo supports Mandatory Access Control by the hypervisor, satisfying a common
certification requirement.
* The Argo headers are BSD-licensed and the Xen hypervisor implementation is GPLv2 but
accessible via the hypercall interface. The licensing should not present
an obstacle
to adoption of Argo in guest software or implementation by other
hypervisors.
* Since the interface that Argo presents to a guest VM is similar to DMA, a VirtIO-Argo
frontend transport driver should be able to operate with a physical
VirtIO-enabled
smart-NIC if the toolstack and an Argo-aware backend provide support.
The next Xen Community Call is next week and I would be happy to answer questions
about Argo and on this topic. I will also be following this thread.
Christopher
(Argo maintainer, Xen Community)
--------------------------------------------------------------------------------
[1]
An introduction to Argo:
https://static.sched.com/hosted_files/xensummit19/92/Argo%20and%20HMX%20-%20OpenXT%20-%20Christopher%20Clark%20-%20Xen%20Summit%202019.pdf
https://www.youtube.com/watch?v=cnC0Tg3jqJQ
Xen Wiki page for Argo:
https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen
[2]
OpenXT Linux Argo driver and userspace library:
https://github.com/openxt/linux-xen-argo
Windows V4V at OpenXT wiki:
https://openxt.atlassian.net/wiki/spaces/DC/pages/14844007/V4V
Windows v4v driver source:
https://github.com/OpenXT/xc-windows/tree/master/xenv4v
HP/Bromium uXen V4V driver:
https://github.com/uxen-virt/uxen/tree/ascara/windows/uxenv4vlib
[3]
v2 of the Argo test unikernel for XTF:
https://lists.xenproject.org/archives/html/xen-devel/2021-01/msg02234.html
[4]
Argo HMX Transport for VirtIO meeting minutes:
https://lists.xenproject.org/archives/html/xen-devel/2021-02/msg01422.html
VirtIO-Argo Development wiki page:
https://openxt.atlassian.net/wiki/spaces/DC/pages/1696169985/VirtIO-Argo+Development+Phase+1
Quoted 1,110 lines · Wei Chen
Quoted 1,108 lines
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月26日 17:41
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; KalyXin
Quoted 1 line
<[email protected]>; Stratos Mailing List <
Quoted 1 line
[email protected]; Arnd Bergmann <
Quoted 8 lines
Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 4 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsQuoted 11 lines
Hi Wei,
Quoted 8 lines · AKASHI Takahiro
Quoted 7 lines · Wei Chen
Hi Akashi，
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; StefanoStabellini
Quoted 23 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e <[email protected]>;
Stratos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]
;
JulienQuoted 1 line
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backendsOn Tue, Aug 17, 2021 at 08:39:09AM +0000, Wei Chen wrote:
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Hi Akashi,
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; StefanoStabellini
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e <
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
Quoted 1 line
Quoted 1 line
open.org; Arnd Bergmann <[email protected]>; Viresh
Kumar
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]
;
Jean-Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIObackends
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Hi Wei, Oleksandr,
Quoted 4 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM
communities.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device
implementations.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In this case, we need to introduce an intermediate
hypervisor
Quoted 14 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always
represent
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
# Linaro's view on this subject nor reflect Alex's concerns.
Nevertheless,
Quoted 3 lines
Quoted 3 lines
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first,
Xen
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
In particular, the abstraction of hypervisor interfaces has a
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
set of interfaces (for your "struct
vmm_impl" and my "RPCinterfaces").
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This is not co-incident as we both share the same origin as I
said
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
above.
Quoted 1 line
Quoted 1 line
And so we will also share the same issues. One of them is a
way
Quoted 9 lines
of
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
"sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too. (See Alex's original email, too).Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a
"single-binary"
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
solution
Quoted 1 line
Quoted 1 line
in which the same binary of BE vm could run on any
hypervisors.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Somehow similar to your "proposal-#2" in [2], but in my
solution,
Quoted 5 lines
all
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the hypervisor-specific code would be put into another entity
(VM),
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
named "virtio-proxy" and the abstracted operations are served
via RPC.
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
(In this sense, BE is hypervisor-agnostic but might have OS
dependency.)
Quoted 5 lines
Quoted 3 lines
But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy
completely
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(I will do it ASAP). But from your description, it seems we
need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
3rd VM between FE and BE? My concern is that, if my assumption
is
Quoted 2 lines
right,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
will it increase the latency in data transport path? Even if
we're
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of
payload,
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
we will see latency issue (even in your case) and if we have some
solution
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
for it, we won't see it neither in my proposal :)
Oleksandr has sent a proposal to Xen mailing list to reduce this
kind
Quoted 10 lines
Quoted 1 line
Quoted 1 line
of "mapping/unmapping" operations. So the latency caused by this
behavior
Quoted 1 line
Quoted 1 line
on Xen may eventually be eliminated, and Linux-KVM doesn't have that
problem.
Quoted 3 lines
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
Can you give me the link to the discussion or patch, please?
It's a RFC discussion. We have tested this RFC patch internally.
https://lists.xenproject.org/archives/html/xen-devel/2021-07/msg01532.htmlQuoted 6 lines
Thanks,
-Takahiro AkashiQuoted 3 lines
-Takahiro Akashi
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Specifically speaking about kvm-tool, I have a concern about
its
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
license term; Targeting different hypervisors and different
OSs
Quoted 39 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(which I assume includes RTOS's), the resultant library should
be
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Yes. If user want to implement a FreeBSD device model, but the
virtio
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
library is GPL. Then GPL would be a problem. If we have another
good
Quoted 29 lines
Quoted 29 lines
Quoted 6 lines
Quoted 1 line
candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD codeInteresting, I will look into them : )
Cheers,
Wei ChenQuoted 17 lines
-Takahiro Akashi
Quoted 11 lines
Quoted 11 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; StefanoStabellini
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; StratosMailing
Quoted 7 lines
List
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
open.org;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Arnd
Quoted 3 lines
Quoted 3 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 24 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]
;
Jean-Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>;
Oleksandr
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Quoted 1 line
Quoted 1 line
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hello, all.
Please see some comments below. And sorry for the possible
format
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
issues.
Quoted 1 line
Quoted 1 line
Quoted 15 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano
Stabellini
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not
trimming
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters,
but
Quoted 33 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
because it
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working
prototype)[1]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
is based on Xen's virtio implementation (i.e. IOREQ) and
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server). It has been, I believe, well generalized but is still a
bit
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
biased
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
dev/2021-
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more about
my
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
approach
Quoted 10 lines
Quoted 10 lines
below.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2Quoted 3 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable
Quoted 10 lines
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code as
possible
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
Quoted 2 lines
Quoted 2 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the
front end
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required -
one
Quoted 3 lines
per OS,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant
guests
Quoted 4 lines
are
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any abstracted
paravirt
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
type
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution I
envisioned
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like environment.
The
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some
mapped
Quoted 15 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is
eventfd).
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any hypervisor
specific
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the actual
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion
in
Quoted 7 lines
the
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI.
There
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interest
Quoted 2 lines
Quoted 2 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to
abstract
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
differences
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
away as well as the idea of bare-metal Rust
backends.
Quoted 12 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
We have a card (STR-12) called "Hypercall
Standardisation"
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed from the
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However we
are
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
some
Quoted 2 lines
Quoted 2 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at
least
Quoted 15 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying
requirements. To
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
that
Quoted 2 lines
Quoted 2 lines
end I
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
want to think about what we need for a backend to
function.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Configuration
=============In the type-2 setup this is typically fairly simple
because
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
host
Quoted 14 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that make
up the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with
delegated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the backend
VM
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
about
Quoted 5 lines
Quoted 5 lines
key
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) andtrigger
Quoted 10 lines
(kick)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
events
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- what (if any) resources the backend needs to
connect to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Obviously you can elide over configuration issues by
having
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into your
guest
Quoted 16 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious
solution
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user space
but
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest and
the
Quoted 14 lines
BE
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address space
needs to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can
envision 3
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in
it's
Quoted 9 lines
Guest
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding
clashing. I
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface to
userspace
Quoted 2 lines
Quoted 2 lines
to then
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make that address space visible to the backend
daemon.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Yet another way here is that we would have well known
"shared
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good
insights on
Quoted 13 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
this
Quoted 2 lines
Quoted 2 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-
agnostic
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a PCI
device
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
and
Quoted 4 lines
Quoted 4 lines
can be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mapped locally.)
I want to add this shared memory aspect to my
virtio-proxy,
Quoted 5 lines
but
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the resultant solution would eventually look similar to
ivshmem.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
* BE guests boots with a hypervisor handle to
memory
Quoted 12 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The BE guest is then free to map the FE's memory to
where
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is no
"handle"
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0.
So
Quoted 18 lines
for
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following
information
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisorinterface
Quoted 1 line
Quoted 1 line
Quoted 1 line
to
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
map the configuration space.
Yes, Xenstore was chosen as a simple way to pass
configuration
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in
Virtio
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
itself if possible, so for non-toolstack domain, this could
done
Quoted 33 lines
with
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
adjusting devd (daemon that listens for devices and launches
backends)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway and
pass it
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to
Quoted 6 lines
Quoted 6 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass device
configuration.
Quoted 1 line
We also designed a static device configuration parse method
for
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device
model
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
command
Quoted 7 lines
Quoted 7 lines
line
Quoted 6 lines
or a config file.
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or
hypervisor)-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM, to
hide
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found and
Quoted 12 lines
proven
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
to
Quoted 5 lines
Quoted 5 lines
work then even better.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
# My point is that a "handle" is not mandatory for
executing
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have toinvent a
Quoted 3 lines
new
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine
privileges
Quoted 10 lines
to
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
allow
Quoted 4 lines
Quoted 4 lines
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no problems
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
course.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my approach
is
Quoted 38 lines
that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping
memory) is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have to
extend
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security andprivilege
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
distribution
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
as Stefan suggested.
We also faced policy related issues with Virtio backend
running in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
other than Dom0 domain in a "dummy" xsm mode. In our target
system we
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
run
Quoted 2 lines
Quoted 2 lines
the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W resides.
We
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trust it,
Quoted 1 line
Quoted 1 line
so we wrote policy rules (to be used in "flask" xsm mode) to
provide
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
it
Quoted 2 lines
Quoted 2 lines
with a little bit more privileges than a simple DomU had.
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and
memory
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. Ican see
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
two
Quoted 1 line
Quoted 1 line
options
Quoted 11 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for daemon/helper
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
trigger
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If
4 lines not shown · raw message
448 lines not shown · raw message
Part of this message is not shown here. View the raw message.