Enabling hypervisor agnosticism for VirtIO backends

virtio-dev Alex Bennée 11 participants 53 messages Aug 4, 2021 – Oct 4, 2021 3 messages not in the archive Page 2 of 2

lore

Share & tools

lore

AKASHI Takahiro

2021-08-31 06:18 UTC #19 lore raw

In reply to Wei Chen

Wei,

Quoted 162 lines · Wei Chen

Hi Akashi,
Quoted 160 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月26日 17:41
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly Xin <[email protected]>; Stratos Mailing List <[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hi Wei,
Quoted 9 lines · AKASHI Takahiro
Quoted 8 lines · Wei Chen
Hi Akashi，
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>;
Stratos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
[email protected]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 1 line
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Quoted 7 lines · Wei Chen
Hi Akashi,
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano
Stabellini
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e <[email protected]>;
Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
[email protected]
Quoted 4 lines
Quoted 4 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hi Wei, Oleksandr,
Quoted 7 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal here. This proposal is still discussing in Xen and KVM communities. The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device implementations.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
In this case, we need to introduce an intermediate hypervisor layer for VMM abstraction, Which is, I think it's very close to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always
represent
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
# Linaro's view on this subject nor reflect Alex's concerns.
Nevertheless,
Quoted 3 lines
Quoted 3 lines
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at first,
Xen
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
and are based on kvm-tool. (Actually, my work is derived from EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces has a
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
set of interfaces (for your "struct vmm_impl" and my "RPC
interfaces").
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This is not co-incident as we both share the same origin as I
said
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
above.
Quoted 1 line
Quoted 1 line
And so we will also share the same issues. One of them is a way
of
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
"sharing/mapping FE's memory". There is some trade-off between the portability and the performance impact.
So we can discuss the topic here in this ML, too. (See Alex's original email, too).
Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a "single-binary"
solution
Quoted 2 lines
Quoted 2 lines
in which the same binary of BE vm could run on any hypervisors. Somehow similar to your "proposal-#2" in [2], but in my solution,
all
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the hypervisor-specific code would be put into another entity
(VM),
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
named "virtio-proxy" and the abstracted operations are served
via RPC.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
(In this sense, BE is hypervisor-agnostic but might have OS
dependency.)
Quoted 5 lines
Quoted 3 lines
But I know that we need discuss if this is a requirement even in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy
completely
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(I will do it ASAP). But from your description, it seems we need a 3rd VM between FE and BE? My concern is that, if my assumption is
right,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
will it increase the latency in data transport path? Even if we're using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of payload, we will see latency issue (even in your case) and if we have some
solution
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
for it, we won't see it neither in my proposal :)
Oleksandr has sent a proposal to Xen mailing list to reduce this kind of "mapping/unmapping" operations. So the latency caused by this
behavior
Quoted 1 line
Quoted 1 line
on Xen may eventually be eliminated, and Linux-KVM doesn't have that
problem.
Quoted 1 line

Quoted 8 lines

Quoted 5 lines
Quoted 2 lines
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
Can you give me the link to the discussion or patch, please?
It's a RFC discussion. We have tested this RFC patch internally.
https://lists.xenproject.org/archives/html/xen-devel/2021-07/msg01532.html

I'm afraid that I miss something here, but I don't know why this proposed API will lead to eliminating 'mmap' in accessing the queued payload at every request?

-Takahiro Akashi

Quoted 841 lines

Quoted 832 lines
Thanks,
-Takahiro Akashi
Quoted 5 lines
-Takahiro Akashi
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Specifically speaking about kvm-tool, I have a concern about its license term; Targeting different hypervisors and different OSs (which I assume includes RTOS's), the resultant library should
be
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Yes. If user want to implement a FreeBSD device model, but the
virtio
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
library is GPL. Then GPL would be a problem. If we have another
good
Quoted 23 lines
Quoted 23 lines
Quoted 6 lines
Quoted 1 line
candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD code
Interesting, I will look into them : )
Cheers,
Wei Chen
Quoted 11 lines
-Takahiro Akashi
Quoted 8 lines
Quoted 8 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>; Stefano
Stabellini
Quoted 2 lines
Quoted 2 lines
<[email protected]>
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos Mailing
List
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; [email protected]
open.org;
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Arnd
Quoted 4 lines
Quoted 4 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>;
Oleksandr
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Quoted 1 line
Quoted 1 line
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hello, all.
Please see some comments below. And sorry for the possible
format
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano
Stabellini
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not
trimming
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters, but
because it
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working
prototype)[1]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
is based on Xen's virtio implementation (i.e. IOREQ) and
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server). It has been, I believe, well generalized but is still a
bit
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
biased
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
[1] https://op-lists.linaro.org/pipermail/stratos-
dev/2021-
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more about
my
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
approach
Quoted 10 lines
Quoted 10 lines
below.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 3 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code as
possible
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
Quoted 2 lines
Quoted 2 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the
front end
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required - one
per OS,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant guests
are
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any abstracted
paravirt
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
type
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution I
envisioned
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like environment.
The
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some mapped
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is
eventfd).
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any hypervisor
specific
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the actual
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion in
the
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI. There
interest
Quoted 2 lines
Quoted 2 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to
abstract
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
differences
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall
Standardisation"
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed from the
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However we
are
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
some
Quoted 4 lines
Quoted 4 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at least
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying
requirements. To
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
that
Quoted 2 lines
Quoted 2 lines
end I
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
want to think about what we need for a backend to
function.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Configuration
=============
In the type-2 setup this is typically fairly simple
because
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that make
up the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with
delegated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the backend
VM
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
about
Quoted 5 lines
Quoted 5 lines
key
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger
(kick)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
events
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- what (if any) resources the backend needs to
connect to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Obviously you can elide over configuration issues by
having
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into your
guest
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious
solution
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user space
but
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest and the
BE
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address space
needs to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can
envision 3
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's
Guest
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding
clashing. I
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface to
userspace
Quoted 4 lines
Quoted 4 lines
to then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
make that address space visible to the backend daemon.
Yet another way here is that we would have well known
"shared
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good
insights on
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
this
Quoted 2 lines
Quoted 2 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-
agnostic
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a PCI
device
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
and
Quoted 4 lines
Quoted 4 lines
can be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mapped locally.)
I want to add this shared memory aspect to my virtio-proxy,
but
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the resultant solution would eventually look similar to
ivshmem.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to
where
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is no
"handle"
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0. So
for
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following
information
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor
interface
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
map the configuration space.
Yes, Xenstore was chosen as a simple way to pass
configuration
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in
Virtio
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
itself if possible, so for non-toolstack domain, this could done
with
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
adjusting devd (daemon that listens for devices and launches
backends)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway and
pass it
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to
Quoted 6 lines
Quoted 6 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass device
configuration.
Quoted 1 line
We also designed a static device configuration parse method
for
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device
model
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
command
Quoted 7 lines
Quoted 7 lines
line
Quoted 6 lines
or a config file.
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or
hypervisor)-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM, to
hide
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found and
proven
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
to
Quoted 5 lines
Quoted 5 lines
work then even better.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
# My point is that a "handle" is not mandatory for
executing
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a
new
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine privileges
to
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
allow
Quoted 4 lines
Quoted 4 lines
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no problems
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
course.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my approach is
that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping
memory) is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have to
extend
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security and
privilege
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
distribution
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
as Stefan suggested.
We also faced policy related issues with Virtio backend
running in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
other than Dom0 domain in a "dummy" xsm mode. In our target
system we
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
run
Quoted 2 lines
Quoted 2 lines
the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W resides.
We
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trust it,
Quoted 1 line
Quoted 1 line
so we wrote policy rules (to be used in "flask" xsm mode) to
provide
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
it
Quoted 2 lines
Quoted 2 lines
with a little bit more privileges than a simple DomU had.
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and
memory
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I
can see
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
two
Quoted 4 lines
Quoted 4 lines
options
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for daemon/helper
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
trigger
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If
using a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
helper
Quoted 2 lines
Quoted 2 lines
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid the
daemon
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that
complexity
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
into
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would
result in
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the userspace
would
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to
userspace
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
ptr"
Quoted 2 lines
Quoted 2 lines
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall
interfaces.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
course assumes the majority of BE guests would be
Linux
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
kernels and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
leaves the bare-metal/unikernel approaches to
their own
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
devices.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Operation
=========
The core of the operation of VirtIO is fairly simple.
Once
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user feature negotiation is done it's a case of
receiving
Quoted 2 lines
Quoted 2 lines
update
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
events and parsing the resultant virt queue for data.
The
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
vhost-
Quoted 2 lines
Quoted 2 lines
user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
specification handles a bunch of setup before that
point,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mostly
Quoted 2 lines
Quoted 2 lines
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
detail where the virt queues are set up FD's for
memory and
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
event
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communication. This is where the envisioned stub
process
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
would
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
responsible for getting the daemon up and ready to run.
This
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
currently done inside a big VMM like QEMU but I
suspect a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
modern
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
approach would be to use the rust-vmm vhost crate. It
would
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
then
Quoted 2 lines
Quoted 2 lines
either
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communicate with the kernel's abstracted ABI or be re-
targeted
Quoted 4 lines
Quoted 4 lines
as a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't
have
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
VMMs
Quoted 6 lines
Quoted 6 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
way they are typically envisioned and described in other
environments.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead, Xen has IOREQ servers. Each of them connects
independently to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen via the IOREQ interface. E.g. today multiple QEMUs
could
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
used as
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
emulators for a single Xen VM, each of them connecting
to Xen
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
independently via the IOREQ interface.
The component responsible for starting a daemon and/or
setting
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
up
Quoted 2 lines
Quoted 2 lines
shared
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interfaces is the toolstack: the xl command and the
libxl/libxc
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
libraries.
I think that VM configuration management (or orchestration
in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Startos
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
jargon?) is a subject to debate in parallel. Otherwise, is there any good assumption to avoid it right
now?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Oleksandr and others I CCed have been working on ways
for the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
toolstack
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to create virtio backends and setup memory mappings.
They
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
might be
Quoted 2 lines
Quoted 2 lines
able
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to provide more info on the subject. I do think we miss
a way
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
provide
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the configuration to the backend and anything else that
the
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
backend
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
might require to start doing its job.
Yes, some work has been done for the toolstack to handle
Virtio
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
MMIO
Quoted 2 lines
Quoted 2 lines
devices in
Quoted 1 line
Quoted 1 line
general and Virtio block devices in particular. However, it
has
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
not
Quoted 3 lines
Quoted 3 lines
been upstreaned yet.
Quoted 2 lines
Quoted 2 lines
Updated patches on review now:
https://lore.kernel.org/xen-devel/1621626361-29076-1-git-
send-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
email-
Quoted 3 lines
Quoted 3 lines
[email protected]/
Quoted 2 lines
Quoted 2 lines
There is an additional (also important) activity to
improve/fix
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
foreign memory mapping on Arm which I am also involved in.
Quoted 1 line
Quoted 1 line
The foreign memory mapping is proposed to be used for Virtio
backends
Quoted 3 lines
Quoted 3 lines
(device emulators) if there is a need to run guest OS completely unmodified.
Quoted 1 line
Quoted 1 line
Of course, the more secure way would be to use grant memory
mapping.
Quoted 1 line
Quoted 1 line
Brietly, the main difference between them is that with foreign
mapping
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
can map any guest memory it wants to map, but with grant
mapping
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
it is
Quoted 3 lines
Quoted 3 lines
allowed to map only what was previously granted by the frontend.
Quoted 2 lines
Quoted 2 lines
So, there might be a problem if we want to pre-map some
guest
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
in advance or to cache mappings in the backend in order to
improve
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
performance (because the mapping/unmapping guest pages every
request
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
requires a lot of back and forth to Xen + P2M updates). In a
nutshell,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
currently, in order to map a guest page into the backend address
space
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
we
Quoted 1 line
Quoted 1 line
need to steal a real physical page from the backend domain. So,
with
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
said optimizations we might end up with no free memory in the
backend
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
domain (see XSA-300). And what we try to achieve is to not waste
a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
real
Quoted 1 line
Quoted 1 line
domain memory at all by providing safe non-allocated-yet (so
unused)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
address space for the foreign (and grant) pages to be mapped
into,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
this
Quoted 1 line
Quoted 1 line
enabling work implies Xen and Linux (and likely DTB bindings)
changes.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
However, as it turned out, for this to work in a proper and safe
way
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
some
Quoted 3 lines
Quoted 3 lines
prereq work needs to be done.
Quoted 2 lines
Quoted 2 lines
You can find the related Xen discussion at:
https://lore.kernel.org/xen-devel/1627489110-25633-1-git-
send-
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
email-
Quoted 6 lines
Quoted 6 lines
[email protected]/
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
One question is how to best handle notification and
kicks.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The
Quoted 2 lines
Quoted 2 lines
existing
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user framework uses eventfd to signal the daemon
(although
Quoted 2 lines
Quoted 2 lines
QEMU
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is quite capable of simulating them when you use TCG).
Xen
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
has
Quoted 2 lines
Quoted 2 lines
it's own
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
IOREQ mechanism. However latency is an important
factor and
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
having
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the
backends to
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a
hypervisor
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interface
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event
channels),
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
which is
Quoted 1 line
Quoted 1 line
Quoted 1 line
converted to a callback to BE via virtio-
proxy
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(Xen's event channel is internnally
implemented by
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interrupts.)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I don't know what "connect directly" means here, but
sending
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interrupts
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to the opposite side would be best efficient. Ivshmem, I suppose, takes this approach by utilizing PCI's
msi-x
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mechanism.
Quoted 3 lines
Quoted 3 lines
Agree that MSI would be more efficient than SPI... At the moment, in order to notify the frontend, the backend
issues
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
specific device-model call to query Xen to inject a
corresponding SPI
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to
Quoted 6 lines
Quoted 6 lines
the guest.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Could we consider the kernel internally converting
IOREQ
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
messages from
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the Xen hypervisor to eventfd events? Would this scale
with
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
other kernel
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth
experimenting
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
with?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One option we should consider is for each backend to
connect
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
Xen via
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the IOREQ interface. We could generalize the IOREQ
interface
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
Quoted 2 lines
Quoted 2 lines
make it
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor agnostic. The interface is really trivial and
easy
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
to
Quoted 3 lines
Quoted 3 lines
add.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
As I said above, my proposal does the same thing that you
mentioned
Quoted 1 line
Quoted 1 line
here :)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The difference is that I do call hypervisor interfaces via
virtio-
Quoted 3 lines
Quoted 3 lines
proxy.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The only Xen-specific part is the notification mechanism,
which is
Quoted 2 lines
Quoted 2 lines
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
event channel. If we replaced the event channel with
something
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
else the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface would be generic. See:
https://gitlab.com/xen-project/xen/-
/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I don't think that translating IOREQs to eventfd in the
kernel
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
is
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
good idea: if feels like it would be extra complexity
and that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
kernel shouldn't be involved as this is a backend-
hypervisor
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
interface.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Given that we may want to implement BE as a bare-metal
application
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
as I did on Zephyr, I don't think that the translation
would not
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt
handling
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
(or nothing but a callback mechanism).
Quoted 1 line
Also, eventfd is very Linux-centric and we are trying to
design an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interface that could work well for RTOSes too. If we
want to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
do
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
something different, both OS-agnostic and hypervisor-
agnostic,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
perhaps
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we could design a new interface. One that could be
implementable
Quoted 2 lines
Quoted 2 lines
in the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen hypervisor itself (like IOREQ) and of course any
other
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
hypervisor
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
too.
There is also another problem. IOREQ is probably not be
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
only
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2.
Don't we
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
also need
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an interface for the backend to inject interrupts into
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
frontend? And
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if the backend requires dynamic memory mappings of
frontend
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
pages,
Quoted 4 lines
Quoted 4 lines
then
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
we would also need an interface to map/unmap domU pages.
My proposal document might help here; All the interfaces
required
Quoted 2 lines
Quoted 2 lines
for
Quoted 1 line
Quoted 1 line
Quoted 1 line
virtio-proxy (or hypervisor-related interfaces) are listed
as
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
RPC protocols :)
Quoted 1 line
These interfaces are a lot more problematic than IOREQ:
IOREQ
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
is
Quoted 2 lines
Quoted 2 lines
tiny
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and self-contained. It is easy to add anywhere. A new
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
inject interrupts or map pages is more difficult to
manage
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
because
Quoted 2 lines
Quoted 2 lines
it
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would require changes scattered across the various
emulators.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Exactly. I have no confident yet that my approach will
also
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
apply
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not
is a
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
different
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
matter.
Thanks,
-Takahiro Akashi
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
--
Regards,
Oleksandr Tyshchenko
IMPORTANT NOTICE: The contents of this email and any
attachments are
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
confidential and may also be privileged. If you are not the
intended
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
recipient, please notify the sender immediately and do not
disclose
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
contents to any other person, use it for any purpose, or store
or copy
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 1 line
information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments
are
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
contents to any other person, use it for any purpose, or store or
copy the
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Wei Chen

2021-09-01 11:13 UTC #20 lore raw

Hi Akashi,

Quoted 227 lines

-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月31日 14:18
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly Xin <[email protected]>; Stratos Mailing List <[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Wei,
Quoted 8 lines · Wei Chen
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月26日 17:41
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly
Xin
Quoted 1 line
Quoted 1 line
<[email protected]>; Stratos Mailing List <stratos-dev@op-
lists.linaro.org>;
Quoted 1 line
Quoted 1 line
[email protected]; Arnd Bergmann
<[email protected]>;
Quoted 8 lines
Quoted 8 lines
Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 7 lines
Quoted 7 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hi Wei,
On Fri, Aug 20, 2021 at 03:41:50PM +0900, AKASHI Takahiro wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Wei Chen
Quoted 7 lines
Hi Akashi，
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano
Stabellini
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e <[email protected]>;
Stratos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
[email protected]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
Quoted 7 lines
Quoted 7 lines
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 9 lines
Quoted 9 lines
Julien
Quoted 1 line
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Quoted 2 lines · Wei Chen
Hi Akashi,
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano
Stabellini
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e
<[email protected]>;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
[email protected]
Quoted 1 line
Quoted 1 line
open.org; Arnd Bergmann <[email protected]>; Viresh
Kumar
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 8 lines
Quoted 8 lines
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>;
Paul
Quoted 3 lines
Quoted 3 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Hi Wei, Oleksandr,
Quoted 3 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal
here.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This proposal is still discussing in Xen and KVM
communities.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device
implementations.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In this case, we need to introduce an intermediate
hypervisor
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
layer for VMM abstraction, Which is, I think it's very
close
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always
represent
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
# Linaro's view on this subject nor reflect Alex's concerns.
Nevertheless,
Quoted 3 lines
Quoted 3 lines
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at
first,
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and are based on kvm-tool. (Actually, my work is derived
from
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces has
a
Quoted 4 lines
Quoted 4 lines
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
set of interfaces (for your "struct vmm_impl" and my "RPC
interfaces").
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This is not co-incident as we both share the same origin as
I
Quoted 3 lines
Quoted 3 lines
said
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
above.
Quoted 1 line
Quoted 1 line
And so we will also share the same issues. One of them is a
way
Quoted 2 lines
Quoted 2 lines
of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"sharing/mapping FE's memory". There is some trade-off
between
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
the portability and the performance impact.
So we can discuss the topic here in this ML, too.
(See Alex's original email, too).
Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a "single-
binary"
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
solution
Quoted 1 line
Quoted 1 line
in which the same binary of BE vm could run on any
hypervisors.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Somehow similar to your "proposal-#2" in [2], but in my
solution,
Quoted 2 lines
Quoted 2 lines
all
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the hypervisor-specific code would be put into another
entity
Quoted 2 lines
Quoted 2 lines
(VM),
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
named "virtio-proxy" and the abstracted operations are
served
Quoted 4 lines
Quoted 4 lines
via RPC.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
(In this sense, BE is hypervisor-agnostic but might have OS
dependency.)
Quoted 1 line
Quoted 1 line
But I know that we need discuss if this is a requirement
even
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy
completely
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(I will do it ASAP). But from your description, it seems we
need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
3rd VM between FE and BE? My concern is that, if my assumption
is
Quoted 2 lines
Quoted 2 lines
right,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
will it increase the latency in data transport path? Even if
we're
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of
payload,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we will see latency issue (even in your case) and if we have
some
Quoted 5 lines
Quoted 5 lines
solution
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
for it, we won't see it neither in my proposal :)
Oleksandr has sent a proposal to Xen mailing list to reduce this
kind
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
of "mapping/unmapping" operations. So the latency caused by this
behavior
Quoted 1 line
Quoted 1 line
on Xen may eventually be eliminated, and Linux-KVM doesn't have
that
Quoted 8 lines
Quoted 7 lines
problem.
Quoted 3 lines
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
Can you give me the link to the discussion or patch, please?

Quoted 8 lines

Quoted 2 lines
It's a RFC discussion. We have tested this RFC patch internally.
https://lists.xenproject.org/archives/html/xen-devel/2021-
07/msg01532.html
I'm afraid that I miss something here, but I don't know why this proposed API will lead to eliminating 'mmap' in accessing the queued payload at every request?

This API give Xen device model (QEMU or kvmtool) the ability to map whole guest RAM in device model's address space. In this case, device model doesn't need dynamic hypercall to map/unmap payload memory. It can use a flat offset to access payload memory in its address space directly. Just Like KVM device model does now.

Before this API, When device model to map whole guest memory, will severely consume the physical pages of Dom-0/Dom-D.

Quoted 973 lines

-Takahiro Akashi
Quoted 6 lines
Quoted 6 lines
Thanks,
-Takahiro Akashi
Quoted 3 lines
-Takahiro Akashi
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Specifically speaking about kvm-tool, I have a concern about
its
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
license term; Targeting different hypervisors and different
OSs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(which I assume includes RTOS's), the resultant library
should
Quoted 8 lines
Quoted 8 lines
be
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Yes. If user want to implement a FreeBSD device model, but the
virtio
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
library is GPL. Then GPL would be a problem. If we have
another
Quoted 19 lines
Quoted 19 lines
good
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
Quoted 1 line
candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD code
Interesting, I will look into them : )
Cheers,
Wei Chen
Quoted 6 lines
-Takahiro Akashi
Quoted 3 lines
Quoted 3 lines
-Takahiro Akashi
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>;
Stefano
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Stabellini
Quoted 2 lines
Quoted 2 lines
<[email protected]>
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos
Mailing
Quoted 7 lines
Quoted 7 lines
List
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; [email protected]
open.org;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Arnd
Quoted 3 lines
Quoted 3 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 9 lines
Quoted 9 lines
Jean-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>;
Oleksandr
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>;
Paul
Quoted 8 lines
Quoted 8 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Quoted 1 line
Quoted 1 line
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hello, all.
Please see some comments below. And sorry for the
possible
Quoted 10 lines
Quoted 10 lines
format
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano
Stabellini
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not
trimming
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters,
but
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
because it
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working
prototype)[1]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is based on Xen's virtio implementation (i.e. IOREQ)
and
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server). It has been, I believe, well generalized but is still
a
Quoted 11 lines
Quoted 11 lines
bit
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
biased
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
[1] https://op-lists.linaro.org/pipermail/stratos-
dev/2021-
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more
about
Quoted 8 lines
Quoted 8 lines
my
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
approach
Quoted 6 lines
Quoted 6 lines
below.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · Alex Bennée
Quoted 3 lines
Hi,
One of the goals of Project Stratos is to enable
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code
as
Quoted 7 lines
Quoted 7 lines
possible
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
Quoted 2 lines
Quoted 2 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the
front end
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required -
one
Quoted 3 lines
Quoted 3 lines
per OS,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant
guests
Quoted 11 lines
Quoted 11 lines
are
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any abstracted
paravirt
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
type
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution I
envisioned
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like
environment.
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some
mapped
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is
eventfd).
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any
hypervisor
Quoted 3 lines
Quoted 3 lines
specific
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the
actual
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion
in
Quoted 7 lines
Quoted 7 lines
the
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI.
There
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interest
Quoted 2 lines
Quoted 2 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to
abstract
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
differences
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
away as well as the idea of bare-metal Rust
backends.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
We have a card (STR-12) called "Hypercall
Standardisation"
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed from
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However
we
Quoted 4 lines
Quoted 4 lines
are
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
some
Quoted 2 lines
Quoted 2 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at
least
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying
requirements. To
Quoted 1 line
Quoted 1 line
Quoted 1 line
that
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
end I
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
want to think about what we need for a backend to
function.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Configuration
=============
In the type-2 setup this is typically fairly
simple
Quoted 4 lines
Quoted 4 lines
because
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that
make
Quoted 6 lines
Quoted 6 lines
up the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with
delegated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the
backend
Quoted 6 lines
Quoted 6 lines
VM
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
about
Quoted 4 lines
Quoted 4 lines
key
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
details about the system:
- where virt queue memory is in it's address
space
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- how it's going to receive (interrupt) and
trigger
Quoted 6 lines
Quoted 6 lines
(kick)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
events
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- what (if any) resources the backend needs to
connect to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Obviously you can elide over configuration issues
by
Quoted 3 lines
Quoted 3 lines
having
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into
your
Quoted 8 lines
Quoted 8 lines
guest
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious
solution
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user
space
Quoted 9 lines
Quoted 9 lines
but
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest and
the
Quoted 6 lines
Quoted 6 lines
BE
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address space
needs to
Quoted 1 line
Quoted 1 line
Quoted 1 line
be
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can
envision 3
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in
it's
Quoted 6 lines
Quoted 6 lines
Guest
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding
clashing. I
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
userspace
Quoted 2 lines
Quoted 2 lines
to then
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make that address space visible to the backend
daemon.
Quoted 12 lines
Quoted 12 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Yet another way here is that we would have well known
"shared
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good
insights on
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
this
Quoted 2 lines
Quoted 2 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-
agnostic
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
solution.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(Please note memory regions in ivshmem appear as a PCI
device
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
and
Quoted 4 lines
Quoted 4 lines
can be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mapped locally.)
I want to add this shared memory aspect to my virtio-
proxy,
Quoted 2 lines
Quoted 2 lines
but
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the resultant solution would eventually look similar
to
Quoted 3 lines
Quoted 3 lines
ivshmem.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
* BE guests boots with a hypervisor handle to
memory
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The BE guest is then free to map the FE's memory
to
Quoted 6 lines
Quoted 6 lines
where
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the BE's guest physical address space.
I cannot see how this could work for Xen. There is
no
Quoted 4 lines
Quoted 4 lines
"handle"
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0.
So
Quoted 20 lines
Quoted 20 lines
for
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following
information
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor
interface
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
map the configuration space.
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Yes, Xenstore was chosen as a simple way to pass
configuration
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in
Virtio
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
itself if possible, so for non-toolstack domain, this could
done
Quoted 4 lines
Quoted 4 lines
with
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
adjusting devd (daemon that listens for devices and launches
backends)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway
and
Quoted 6 lines
Quoted 6 lines
pass it
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
to
Quoted 4 lines
Quoted 4 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass
device
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
configuration.
Quoted 1 line
We also designed a static device configuration parse
method
Quoted 9 lines
Quoted 9 lines
for
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device
model
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
command
Quoted 3 lines
Quoted 3 lines
line
Quoted 2 lines
or a config file.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or
hypervisor)-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM,
to
Quoted 5 lines
Quoted 5 lines
hide
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found
and
Quoted 6 lines
Quoted 6 lines
proven
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
to
Quoted 4 lines
Quoted 4 lines
work then even better.
Quoted 3 lines
Quoted 3 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
# My point is that a "handle" is not mandatory for
executing
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to
invent a
Quoted 3 lines
Quoted 3 lines
new
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine
privileges
Quoted 6 lines
Quoted 6 lines
to
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
allow
Quoted 4 lines
Quoted 4 lines
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no
problems
Quoted 4 lines
Quoted 4 lines
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
course.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my approach
is
Quoted 6 lines
Quoted 6 lines
that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping
memory) is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have
to
Quoted 8 lines
Quoted 8 lines
extend
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security and
privilege
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
distribution
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
as Stefan suggested.
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
We also faced policy related issues with Virtio backend
running in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
other than Dom0 domain in a "dummy" xsm mode. In our target
system we
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
run
Quoted 2 lines
Quoted 2 lines
the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W
resides.
Quoted 6 lines
Quoted 6 lines
We
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trust it,
Quoted 1 line
Quoted 1 line
so we wrote policy rules (to be used in "flask" xsm mode) to
provide
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
it
Quoted 1 line
Quoted 1 line
with a little bit more privileges than a simple DomU had.
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and
memory
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor.
I
Quoted 6 lines
Quoted 6 lines
can see
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
two
Quoted 4 lines
Quoted 4 lines
options
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for
daemon/helper
Quoted 8 lines
Quoted 8 lines
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
trigger
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If
using a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
helper
Quoted 2 lines
Quoted 2 lines
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid
the
Quoted 4 lines
Quoted 4 lines
daemon
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that
complexity
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
into
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would
result in
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the
hypercall
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the
userspace
Quoted 10 lines
Quoted 10 lines
would
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to
userspace
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
ptr"
Quoted 2 lines
Quoted 2 lines
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall
interfaces.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
course assumes the majority of BE guests would
be
Quoted 10 lines
Quoted 10 lines
Linux
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
kernels and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
leaves the bare-metal/unikernel approaches to
their own
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
devices.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Operation
=========
The core of the operation of VirtIO is fairly
simple.
Quoted 3 lines
Quoted 3 lines
Once
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user feature negotiation is done it's a case
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
receiving
Quoted 2 lines
Quoted 2 lines
update
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
events and parsing the resultant virt queue for
data.
Quoted 7 lines
Quoted 7 lines
The
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
vhost-
Quoted 2 lines
Quoted 2 lines
user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
specification handles a bunch of setup before that
point,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
mostly
Quoted 1 line
Quoted 1 line
to
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
detail where the virt queues are set up FD's for
memory and
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
event
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communication. This is where the envisioned stub
process
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
would
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
responsible for getting the daemon up and ready to
run.
Quoted 6 lines
Quoted 6 lines
This
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
currently done inside a big VMM like QEMU but I
suspect a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
modern
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
approach would be to use the rust-vmm vhost crate.
It
Quoted 4 lines
Quoted 4 lines
would
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
then
Quoted 2 lines
Quoted 2 lines
either
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communicate with the kernel's abstracted ABI or be
re-
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
targeted
Quoted 4 lines
Quoted 4 lines
as a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen
doesn't
Quoted 4 lines
Quoted 4 lines
have
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
VMMs
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
way they are typically envisioned and described in
other
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
environments.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead, Xen has IOREQ servers. Each of them
connects
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
independently to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen via the IOREQ interface. E.g. today multiple
380 lines not shown · raw message

Part of this message is not shown here. View the raw message.

AKASHI Takahiro

2021-09-01 12:29 UTC #21 lore raw

Hi Wei,

Quoted 238 lines · Wei Chen

Hi Akashi,
Quoted 227 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月31日 14:18
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly Xin <[email protected]>; Stratos Mailing List <[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Wei,
Quoted 8 lines · Wei Chen
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月26日 17:41
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly
Xin
Quoted 1 line
Quoted 1 line
<[email protected]>; Stratos Mailing List <stratos-dev@op-
lists.linaro.org>;
Quoted 1 line
Quoted 1 line
[email protected]; Arnd Bergmann
<[email protected]>;
Quoted 8 lines
Quoted 8 lines
Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 7 lines
Quoted 7 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hi Wei,
On Fri, Aug 20, 2021 at 03:41:50PM +0900, AKASHI Takahiro wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Wei Chen
Quoted 7 lines
Hi Akashi，
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano
Stabellini
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e <[email protected]>;
Stratos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
[email protected]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
open.org; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
Quoted 7 lines
Quoted 7 lines
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 9 lines
Quoted 9 lines
Julien
Quoted 1 line
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Quoted 2 lines · Wei Chen
Hi Akashi,
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano
Stabellini
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e
<[email protected]>;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
[email protected]
Quoted 1 line
Quoted 1 line
open.org; Arnd Bergmann <[email protected]>; Viresh
Kumar
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 8 lines
Quoted 8 lines
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>;
Paul
Quoted 3 lines
Quoted 3 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Hi Wei, Oleksandr,
Quoted 3 lines · Wei Chen
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal
here.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This proposal is still discussing in Xen and KVM
communities.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The main work is to decouple the kvmtool from KVM and make other hypervisors can reuse the virtual device
implementations.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In this case, we need to introduce an intermediate
hypervisor
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
layer for VMM abstraction, Which is, I think it's very
close
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't always
represent
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
# Linaro's view on this subject nor reflect Alex's concerns.
Nevertheless,
Quoted 3 lines
Quoted 3 lines
Your idea and my proposal seem to share the same background. Both have the similar goal and currently start with, at
first,
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and are based on kvm-tool. (Actually, my work is derived
from
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces has
a
Quoted 4 lines
Quoted 4 lines
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
set of interfaces (for your "struct vmm_impl" and my "RPC
interfaces").
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This is not co-incident as we both share the same origin as
I
Quoted 3 lines
Quoted 3 lines
said
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
above.
Quoted 1 line
Quoted 1 line
And so we will also share the same issues. One of them is a
way
Quoted 2 lines
Quoted 2 lines
of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"sharing/mapping FE's memory". There is some trade-off
between
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
the portability and the performance impact.
So we can discuss the topic here in this ML, too.
(See Alex's original email, too).
Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a "single-
binary"
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
solution
Quoted 1 line
Quoted 1 line
in which the same binary of BE vm could run on any
hypervisors.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Somehow similar to your "proposal-#2" in [2], but in my
solution,
Quoted 2 lines
Quoted 2 lines
all
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the hypervisor-specific code would be put into another
entity
Quoted 2 lines
Quoted 2 lines
(VM),
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
named "virtio-proxy" and the abstracted operations are
served
Quoted 4 lines
Quoted 4 lines
via RPC.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
(In this sense, BE is hypervisor-agnostic but might have OS
dependency.)
Quoted 1 line
Quoted 1 line
But I know that we need discuss if this is a requirement
even
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-proxy
completely
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(I will do it ASAP). But from your description, it seems we
need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
3rd VM between FE and BE? My concern is that, if my assumption
is
Quoted 2 lines
Quoted 2 lines
right,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
will it increase the latency in data transport path? Even if
we're
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of degree. As far as we execute 'mapping' operations at every fetch of
payload,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we will see latency issue (even in your case) and if we have
some
Quoted 5 lines
Quoted 5 lines
solution
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
for it, we won't see it neither in my proposal :)
Oleksandr has sent a proposal to Xen mailing list to reduce this
kind
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
of "mapping/unmapping" operations. So the latency caused by this
behavior
Quoted 1 line
Quoted 1 line
on Xen may eventually be eliminated, and Linux-KVM doesn't have
that
Quoted 8 lines
Quoted 7 lines
problem.
Quoted 3 lines
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
Can you give me the link to the discussion or patch, please?
Quoted 8 lines
Quoted 2 lines
It's a RFC discussion. We have tested this RFC patch internally.
https://lists.xenproject.org/archives/html/xen-devel/2021-
07/msg01532.html
I'm afraid that I miss something here, but I don't know why this proposed API will lead to eliminating 'mmap' in accessing the queued payload at every request?

Quoted 5 lines

This API give Xen device model (QEMU or kvmtool) the ability to map whole guest RAM in device model's address space. In this case, device model doesn't need dynamic hypercall to map/unmap payload memory. It can use a flat offset to access payload memory in its address space directly. Just Like KVM device model does now.

Thank you. Quickly, let me make sure one thing:
This API itself doesn't do any mapping operations, right? So I suppose that virtio BE guest is responsible to
1) fetch the information about all the memory regions in FE,
2) call this API to allocate a big chunk of unused space in BE,
3) create grant/foreign mappings for FE onto this region(S) in the initialization/configuration of emulated virtio devices.

Is this the way this API is expected to be used?
Does Xen already has an interface for (1)?

-Takahiro Akashi

Quoted 976 lines

Before this API, When device model to map whole guest memory, will severely consume the physical pages of Dom-0/Dom-D.
Quoted 973 lines
-Takahiro Akashi
Quoted 6 lines
Quoted 6 lines
Thanks,
-Takahiro Akashi
Quoted 3 lines
-Takahiro Akashi
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Specifically speaking about kvm-tool, I have a concern about
its
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
license term; Targeting different hypervisors and different
OSs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(which I assume includes RTOS's), the resultant library
should
Quoted 8 lines
Quoted 8 lines
be
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
license permissive and GPL for kvm-tool might be an issue. Any thoughts?
Yes. If user want to implement a FreeBSD device model, but the
virtio
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
library is GPL. Then GPL would be a problem. If we have
another
Quoted 19 lines
Quoted 19 lines
good
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
Quoted 1 line
candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my mind:
* Open-AMP, or
* corresponding Free-BSD code
Interesting, I will look into them : )
Cheers,
Wei Chen
Quoted 6 lines
-Takahiro Akashi
Quoted 3 lines
Quoted 3 lines
-Takahiro Akashi
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>;
Stefano
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Stabellini
Quoted 2 lines
Quoted 2 lines
<[email protected]>
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos
Mailing
Quoted 7 lines
Quoted 7 lines
List
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; [email protected]
open.org;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Arnd
Quoted 3 lines
Quoted 3 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 9 lines
Quoted 9 lines
Jean-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Wei Chen <[email protected]>;
Oleksandr
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>;
Paul
Quoted 8 lines
Quoted 8 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Quoted 1 line
Quoted 1 line
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hello, all.
Please see some comments below. And sorry for the
possible
Quoted 10 lines
Quoted 10 lines
format
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano
Stabellini
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs. Not
trimming
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that matters,
but
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
because it
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the one I know best.
Please note that my proposal (and hence the working
prototype)[1]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is based on Xen's virtio implementation (i.e. IOREQ)
and
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server). It has been, I believe, well generalized but is still
a
Quoted 11 lines
Quoted 11 lines
bit
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
biased
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
[1] https://op-lists.linaro.org/pipermail/stratos-
dev/2021-
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
August/000546.html
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit more
about
Quoted 8 lines
Quoted 8 lines
my
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
approach
Quoted 6 lines
Quoted 6 lines
below.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of code
as
Quoted 7 lines
Quoted 7 lines
possible
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
Quoted 2 lines
Quoted 2 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of the
front end
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are required -
one
Quoted 3 lines
Quoted 3 lines
per OS,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The resultant
guests
Quoted 11 lines
Quoted 11 lines
are
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any abstracted
paravirt
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
type
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution I
envisioned
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like
environment.
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only some
mapped
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this is
eventfd).
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any
hypervisor
Quoted 3 lines
Quoted 3 lines
specific
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the
actual
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an expansion
in
Quoted 7 lines
Quoted 7 lines
the
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest in
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like SCMI.
There
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interest
Quoted 2 lines
Quoted 2 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait interface to
abstract
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
differences
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
away as well as the idea of bare-metal Rust
backends.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
We have a card (STR-12) called "Hypercall
Standardisation"
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed from
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends. However
we
Quoted 4 lines
Quoted 4 lines
are
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
some
Quoted 2 lines
Quoted 2 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to at
least
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying
requirements. To
Quoted 1 line
Quoted 1 line
Quoted 1 line
that
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
end I
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
want to think about what we need for a backend to
function.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Configuration
=============
In the type-2 setup this is typically fairly
simple
Quoted 4 lines
Quoted 4 lines
because
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules that
make
Quoted 6 lines
Quoted 6 lines
up the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2 with
delegated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the
backend
Quoted 6 lines
Quoted 6 lines
VM
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
about
Quoted 4 lines
Quoted 4 lines
key
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
details about the system:
- where virt queue memory is in it's address
space
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- how it's going to receive (interrupt) and
trigger
Quoted 6 lines
Quoted 6 lines
(kick)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
events
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- what (if any) resources the backend needs to
connect to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Obviously you can elide over configuration issues
by
Quoted 3 lines
Quoted 3 lines
having
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into
your
Quoted 8 lines
Quoted 8 lines
guest
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The obvious
solution
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user
space
Quoted 9 lines
Quoted 9 lines
but
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place the
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest and
the
Quoted 6 lines
Quoted 6 lines
BE
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address space
needs to
Quoted 1 line
Quoted 1 line
Quoted 1 line
be
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I can
envision 3
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in
it's
Quoted 6 lines
Quoted 6 lines
Guest
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and avoiding
clashing. I
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard interface
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
userspace
Quoted 2 lines
Quoted 2 lines
to then
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make that address space visible to the backend
daemon.
Quoted 12 lines
Quoted 12 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Yet another way here is that we would have well known
"shared
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us good
insights on
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
this
Quoted 2 lines
Quoted 2 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for hypervisor-
agnostic
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
solution.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(Please note memory regions in ivshmem appear as a PCI
device
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
and
Quoted 4 lines
Quoted 4 lines
can be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mapped locally.)
I want to add this shared memory aspect to my virtio-
proxy,
Quoted 2 lines
Quoted 2 lines
but
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the resultant solution would eventually look similar
to
Quoted 3 lines
Quoted 3 lines
ivshmem.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
* BE guests boots with a hypervisor handle to
memory
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The BE guest is then free to map the FE's memory
to
Quoted 6 lines
Quoted 6 lines
where
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the BE's guest physical address space.
I cannot see how this could work for Xen. There is
no
Quoted 4 lines
Quoted 4 lines
"handle"
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in dom0.
So
Quoted 20 lines
Quoted 20 lines
for
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following
information
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor
interface
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
map the configuration space.
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Yes, Xenstore was chosen as a simple way to pass
configuration
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore in
Virtio
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
itself if possible, so for non-toolstack domain, this could
done
Quoted 4 lines
Quoted 4 lines
with
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
adjusting devd (daemon that listens for devices and launches
backends)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore anyway
and
Quoted 6 lines
Quoted 6 lines
pass it
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
to
Quoted 4 lines
Quoted 4 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass
device
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
configuration.
Quoted 1 line
We also designed a static device configuration parse
method
Quoted 9 lines
Quoted 9 lines
for
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from device
model
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
command
Quoted 3 lines
Quoted 3 lines
line
Quoted 2 lines
or a config file.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or
hypervisor)-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another VM,
to
Quoted 5 lines
Quoted 5 lines
hide
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already found
and
Quoted 6 lines
Quoted 6 lines
proven
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
to
Quoted 4 lines
Quoted 4 lines
work then even better.
Quoted 3 lines
Quoted 3 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
# My point is that a "handle" is not mandatory for
executing
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to
invent a
Quoted 3 lines
Quoted 3 lines
new
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine
privileges
Quoted 6 lines
Quoted 6 lines
to
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
allow
Quoted 4 lines
Quoted 4 lines
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no
problems
Quoted 4 lines
Quoted 4 lines
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
course.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my approach
is
Quoted 6 lines
Quoted 6 lines
that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ, mapping
memory) is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will have
to
Quoted 8 lines
Quoted 8 lines
extend
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
those
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces.
This, however, will raise some concern on security and
privilege
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
distribution
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
as Stefan suggested.
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
We also faced policy related issues with Virtio backend
running in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
other than Dom0 domain in a "dummy" xsm mode. In our target
system we
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
run
Quoted 2 lines
Quoted 2 lines
the backend in a driver
Quoted 1 line
Quoted 1 line
domain (we call it DomD) where the underlying H/W
resides.
Quoted 6 lines
Quoted 6 lines
We
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trust it,
Quoted 1 line
Quoted 1 line
so we wrote policy rules (to be used in "flask" xsm mode) to
provide
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
it
Quoted 1 line
Quoted 1 line
with a little bit more privileges than a simple DomU had.
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Now it is permitted to issue device-model, resource and
memory
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
mappings, etc calls.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
To activate the mapping will
require some sort of hypercall to the hypervisor.
I
Quoted 6 lines
Quoted 6 lines
can see
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
two
Quoted 4 lines
Quoted 4 lines
options
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
at this point:
- expose the handle to userspace for
daemon/helper
Quoted 8 lines
Quoted 8 lines
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
trigger
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mapping via existing hypercall interfaces. If
using a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
helper
Quoted 2 lines
Quoted 2 lines
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
would have a hypervisor specific one to avoid
the
Quoted 4 lines
Quoted 4 lines
daemon
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
having to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
care too much about the details or push that
complexity
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
into
Quoted 2 lines
Quoted 2 lines
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
compile time option for the daemon which would
result in
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
different
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
binaries although a common source base.
- expose a new kernel ABI to abstract the
hypercall
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
differences away
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the guest kernel. In this case the
userspace
Quoted 10 lines
Quoted 10 lines
would
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
essentially
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ask for an abstract "map guest N memory to
userspace
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
ptr"
Quoted 2 lines
Quoted 2 lines
and let
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel deal with the different hypercall
interfaces.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
course assumes the majority of BE guests would
be
Quoted 10 lines
Quoted 10 lines
Linux
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
kernels and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
leaves the bare-metal/unikernel approaches to
their own
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
devices.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Operation
=========
The core of the operation of VirtIO is fairly
simple.
Quoted 3 lines
Quoted 3 lines
Once
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
vhost-user feature negotiation is done it's a case
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
receiving
Quoted 2 lines
Quoted 2 lines
update
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
events and parsing the resultant virt queue for
data.
Quoted 7 lines
Quoted 7 lines
The
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
vhost-
Quoted 2 lines
Quoted 2 lines
user
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
specification handles a bunch of setup before that
point,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
mostly
Quoted 1 line
Quoted 1 line
to
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
detail where the virt queues are set up FD's for
memory and
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
event
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communication. This is where the envisioned stub
process
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
would
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
responsible for getting the daemon up and ready to
run.
Quoted 6 lines
Quoted 6 lines
This
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
currently done inside a big VMM like QEMU but I
suspect a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
modern
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
approach would be to use the rust-vmm vhost crate.
It
Quoted 4 lines
Quoted 4 lines
would
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
then
Quoted 2 lines
Quoted 2 lines
either
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
communicate with the kernel's abstracted ABI or be
re-
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
targeted
Quoted 4 lines
Quoted 4 lines
as a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen
doesn't
Quoted 4 lines
Quoted 4 lines
have
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
VMMs
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
way they are typically envisioned and described in
other
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
environments.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead, Xen has IOREQ servers. Each of them
connects
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
independently to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Xen via the IOREQ interface. E.g. today multiple
QEMUs
Quoted 4 lines
Quoted 4 lines
could
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
used as
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
emulators for a single Xen VM, each of them
connecting
Quoted 4 lines
Quoted 4 lines
to Xen
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
independently via the IOREQ interface.
The component responsible for starting a daemon
and/or
Quoted 8 lines
Quoted 8 lines
setting
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
up
Quoted 2 lines
Quoted 2 lines
shared
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
interfaces is the toolstack: the xl command and the
libxl/libxc
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
libraries.
1 line not shown · raw message
361 lines not shown · raw message

Part of this message is not shown here. View the raw message.

Oleksandr Tyshchenko

2021-09-01 16:26 UTC #22 lore raw

Hi Akashi,

I am sorry for the possible format issues.

Quoted 11 lines

Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
It's a RFC discussion. We have tested this RFC patch internally.
https://lists.xenproject.org/archives/html/xen-devel/2021-
07/msg01532.html
I'm afraid that I miss something here, but I don't know why this proposed API will lead to eliminating 'mmap' in accessing the queued payload at every request?

Quoted 6 lines

Quoted 5 lines
This API give Xen device model (QEMU or kvmtool) the ability to map whole guest RAM in device model's address space. In this case, device model doesn't need dynamic hypercall to map/unmap payload memory. It can use a flat offset to access payload memory in its address space directly. Just Like KVM device model does now.

Yes!

Quoted 3 lines

Thank you. Quickly, let me make sure one thing:
This API itself doesn't do any mapping operations, right?

Right. The only purpose of that "API" is to guery hypervisor to find unallocated address space ranges to map the foreign pages into (instead of stealing real RAM pages),
In a nutshell, if you try to map the whole guest memory in the backend address space on Arm (or even cache some mappings) you might end up with memory exhaustion in the backend domain (XSA-300),
and the possibility to hit XSA-300 is higher if your backend needs to serve several Guests. Of course, this depends on the memory assigned to the backend domain and to the Guest(s) it serves...
We believe that with the proposed solution the backend will be able to handle Guest(s) without wasting it's real RAM. However, please note that proposed Xen + Linux changes which are on review now [1] are far from the final solution and require rework and some prereq work to operate in a proper and safe way.

Quoted 9 lines

So I suppose that virtio BE guest is responsible to
1) fetch the information about all the memory regions in FE,
2) call this API to allocate a big chunk of unused space in BE,
3) create grant/foreign mappings for FE onto this region(S) in the initialization/configuration of emulated virtio devices.
Is this the way this API is expected to be used?

No really, the userspace backend doesn't need to call this API at all, all what backend calls is still xenforeignmemory_map()/xenforeignmemory_unmap(), so let's say "magic" is done by Linux and Xen internally.
You can take a look at the virtio-disk PoC [2] (last 4 patches) to better understand what Wei and I are talking about. There we map the Guest memory at the beginning and just calculate a pointer at runtime. Again, the code is not in good shape, but it is enough to demonstrate the feasibility of the improvement.

Quoted 2 lines

Does Xen already has an interface for (1)?

I am not aware of anything existing. For the PoC I guessed the Guest memory layout in a really hackish way (I got total Guest memory size, so having GUEST_RAMX_BASE/GUEST_RAMX_SIZE in hand just performed calculation). Definitely, it is a no-go, so 1) deserves additional discussion/design.

[1]
https://lore.kernel.org/xen-devel/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
[2]
https://github.com/otyshchenko1/virtio-disk/commits/map_opt_next

Signature · 3 lines

Regards,

Oleksandr Tyshchenko

Wei Chen

2021-09-02 01:31 UTC #23 lore raw

In reply to AKASHI Takahiro

Hi Akashi,

Quoted 309 lines

-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年9月1日 20:29
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly Xin <[email protected]>; Stratos Mailing List <[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; nd <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hi Wei,
Quoted 8 lines · Wei Chen
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月31日 14:18
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly
Xin
Quoted 1 line
Quoted 1 line
<[email protected]>; Stratos Mailing List <stratos-dev@op-
lists.linaro.org>;
Quoted 1 line
Quoted 1 line
[email protected]; Arnd Bergmann
<[email protected]>;
Quoted 8 lines
Quoted 8 lines
Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 7 lines
Quoted 7 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Wei,
On Thu, Aug 26, 2021 at 12:10:19PM +0000, Wei Chen wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月26日 17:41
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>;
Kaly
Quoted 8 lines
Quoted 8 lines
Xin
Quoted 1 line
Quoted 1 line
<[email protected]>; Stratos Mailing List <stratos-dev@op-
lists.linaro.org>;
Quoted 1 line
Quoted 1 line
[email protected]; Arnd Bergmann
<[email protected]>;
Quoted 3 lines
Quoted 3 lines
Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hi Wei,
Quoted 3 lines · AKASHI Takahiro
Quoted 2 lines · Wei Chen
Hi Akashi，
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano
Stabellini
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e
<[email protected]>;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Stratos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
[email protected]
Quoted 1 line
Quoted 1 line
Quoted 1 line
open.org; Arnd Bergmann <[email protected]>; Viresh
Kumar
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>;
Paul
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines · Wei Chen
Quoted 7 lines
Hi Akashi,
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Stabellini
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e
<[email protected]>;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
[email protected]
Quoted 1 line
Quoted 1 line
open.org; Arnd Bergmann <[email protected]>;
Viresh
Quoted 3 lines
Quoted 3 lines
Kumar
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Stefano Stabellini <[email protected]>; [email protected];
Jan
Quoted 7 lines
Quoted 7 lines
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Jean-
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Philippe Brucker <[email protected]>; Mathieu
Poirier
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
<[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>;
Paul
Quoted 3 lines
Quoted 3 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hi Wei, Oleksandr,
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · Wei Chen
Quoted 3 lines
Hi All,
Thanks for Stefano to link my kvmtool for Xen proposal
here.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This proposal is still discussing in Xen and KVM
communities.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The main work is to decouple the kvmtool from KVM and
make
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
other hypervisors can reuse the virtual device
implementations.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In this case, we need to introduce an intermediate
hypervisor
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
layer for VMM abstraction, Which is, I think it's very
close
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't
always
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
represent
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
# Linaro's view on this subject nor reflect Alex's
concerns.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Nevertheless,
Quoted 2 lines
Quoted 2 lines
Your idea and my proposal seem to share the same
background.
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Both have the similar goal and currently start with, at
first,
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and are based on kvm-tool. (Actually, my work is derived
from
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor interfaces
has
Quoted 3 lines
Quoted 3 lines
a
Quoted 2 lines
Quoted 2 lines
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
set of interfaces (for your "struct vmm_impl" and my
"RPC
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces").
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This is not co-incident as we both share the same origin
as
Quoted 4 lines
Quoted 4 lines
I
Quoted 3 lines
Quoted 3 lines
said
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
above.
Quoted 1 line
Quoted 1 line
And so we will also share the same issues. One of them
is a
Quoted 17 lines
Quoted 17 lines
way
Quoted 2 lines
Quoted 2 lines
of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"sharing/mapping FE's memory". There is some trade-off
between
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
the portability and the performance impact. So we can discuss the topic here in this ML, too. (See Alex's original email, too).
Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a "single-
binary"
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
solution
Quoted 1 line
Quoted 1 line
in which the same binary of BE vm could run on any
hypervisors.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Somehow similar to your "proposal-#2" in [2], but in my
solution,
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
all
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the hypervisor-specific code would be put into another
entity
Quoted 2 lines
Quoted 2 lines
(VM),
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
named "virtio-proxy" and the abstracted operations are
served
Quoted 2 lines
Quoted 2 lines
via RPC.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(In this sense, BE is hypervisor-agnostic but might have
OS
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
dependency.)
Quoted 1 line
Quoted 1 line
But I know that we need discuss if this is a requirement
even
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
in Stratos project or not. (Maybe not)
Sorry, I haven't had time to finish reading your virtio-
proxy
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
completely
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(I will do it ASAP). But from your description, it seems
we
Quoted 2 lines
Quoted 2 lines
need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
3rd VM between FE and BE? My concern is that, if my
assumption
Quoted 3 lines
Quoted 3 lines
is
Quoted 2 lines
Quoted 2 lines
right,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
will it increase the latency in data transport path? Even
if
Quoted 4 lines
Quoted 4 lines
we're
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of
degree.
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
As far as we execute 'mapping' operations at every fetch of
payload,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we will see latency issue (even in your case) and if we have
some
Quoted 5 lines
Quoted 5 lines
solution
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
for it, we won't see it neither in my proposal :)
Oleksandr has sent a proposal to Xen mailing list to reduce
this
Quoted 2 lines
Quoted 2 lines
kind
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
of "mapping/unmapping" operations. So the latency caused by
this
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
behavior
Quoted 1 line
Quoted 1 line
on Xen may eventually be eliminated, and Linux-KVM doesn't
have
Quoted 18 lines
Quoted 9 lines
that
Quoted 8 lines
Quoted 7 lines
problem.
Quoted 3 lines
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
Can you give me the link to the discussion or patch, please?
Quoted 8 lines
Quoted 2 lines
It's a RFC discussion. We have tested this RFC patch internally.
https://lists.xenproject.org/archives/html/xen-devel/2021-
07/msg01532.html
I'm afraid that I miss something here, but I don't know why this proposed API will lead to eliminating 'mmap' in accessing the queued payload at every request?
Quoted 5 lines
This API give Xen device model (QEMU or kvmtool) the ability to map whole guest RAM in device model's address space. In this case, device model doesn't need dynamic hypercall to map/unmap payload memory. It can use a flat offset to access payload memory in its address space directly. Just Like KVM device model does now.
Thank you. Quickly, let me make sure one thing:
This API itself doesn't do any mapping operations, right? So I suppose that virtio BE guest is responsible to
1) fetch the information about all the memory regions in FE,
2) call this API to allocate a big chunk of unused space in BE,
3) create grant/foreign mappings for FE onto this region(S) in the initialization/configuration of emulated virtio devices.

Quoted 3 lines

Is this the way this API is expected to be used?
Does Xen already has an interface for (1)?

They are discussing in that thread to find a proper way to do it. Because this API is common, both x86 and Arm should be considered.

Quoted 1,092 lines

-Takahiro Akashi
Quoted 12 lines
Before this API, When device model to map whole guest memory, will severely consume the physical pages of Dom-0/Dom-D.
Quoted 9 lines
-Takahiro Akashi
Quoted 6 lines
Quoted 6 lines
Thanks,
-Takahiro Akashi
Quoted 3 lines
-Takahiro Akashi
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Specifically speaking about kvm-tool, I have a concern
about
Quoted 2 lines
Quoted 2 lines
its
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
license term; Targeting different hypervisors and
different
Quoted 5 lines
Quoted 5 lines
OSs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(which I assume includes RTOS's), the resultant library
should
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
license permissive and GPL for kvm-tool might be an
issue.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Any thoughts?
Yes. If user want to implement a FreeBSD device model, but
the
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
virtio
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
library is GPL. Then GPL would be a problem. If we have
another
Quoted 4 lines
Quoted 4 lines
good
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my
mind:
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
* Open-AMP, or
* corresponding Free-BSD code
Interesting, I will look into them : )
Cheers,
Wei Chen
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-Takahiro Akashi
Quoted 4 lines
Quoted 4 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-
dev/2021-
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
August/000548.html
[2] https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 4 lines
Quoted 4 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>;
Quoted 7 lines
Quoted 7 lines
Stefano
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Stabellini
Quoted 2 lines
Quoted 2 lines
<[email protected]>
Quoted 1 line
Quoted 1 line
Cc Alex Benn??e <[email protected]>; Stratos
Mailing
Quoted 2 lines
Quoted 2 lines
List
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; virtio-
[email protected]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
open.org;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Arnd
Quoted 3 lines
Quoted 3 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected];
Jan
Quoted 7 lines
Quoted 7 lines
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Jean-
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Philippe Brucker <[email protected]>; Mathieu
Poirier
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Wei Chen
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Oleksandr
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Tyshchenko <[email protected]>; Bertrand
Marquis
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>;
Paul
Quoted 3 lines
Quoted 3 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Quoted 1 line
Quoted 1 line
Subject: Re: Enabling hypervisor agnosticism for
VirtIO
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
backends
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hello, all.
Please see some comments below. And sorry for the
possible
Quoted 1 line
Quoted 1 line
format
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
issues.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano
Stabellini
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs.
Not
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
trimming
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential Xen
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
because it is the only implementation that
matters,
Quoted 6 lines
Quoted 6 lines
but
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
because it
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the one I know best.
Please note that my proposal (and hence the
working
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
prototype)[1]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is based on Xen's virtio implementation (i.e.
IOREQ)
Quoted 4 lines
Quoted 4 lines
and
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
particularly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is
still
Quoted 11 lines
Quoted 11 lines
a
Quoted 10 lines
Quoted 10 lines
bit
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
biased
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
[1] https://op-lists.linaro.org/pipermail/stratos-
dev/2021-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
August/000546.html
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Let me take this opportunity to explain a bit more
about
Quoted 6 lines
Quoted 6 lines
my
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
approach
Quoted 4 lines
Quoted 4 lines
below.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-
devel&m=162373754705233&w=2
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
On Wed, 4 Aug 2021, Alex Bennée wrote:
Hi,
One of the goals of Project Stratos is to
enable
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
agnostic
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of
code
Quoted 5 lines
Quoted 5 lines
as
Quoted 4 lines
Quoted 4 lines
possible
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
Quoted 2 lines
Quoted 2 lines
avoid
Quoted 1 line
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side of
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
front end
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
where
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple front-end implementations are
required -
Quoted 4 lines
Quoted 4 lines
one
Quoted 3 lines
Quoted 3 lines
per OS,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
assuming
Quoted 1 line
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The
resultant
Quoted 4 lines
Quoted 4 lines
guests
Quoted 3 lines
Quoted 3 lines
are
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trivially
Quoted 1 line
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any
abstracted
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
paravirt
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
type
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a solution
I
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
envisioned
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
vhost-user
Quoted 1 line
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like
environment.
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface to
Quoted 1 line
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only
some
Quoted 4 lines
Quoted 4 lines
mapped
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this
is
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
eventfd).
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
idea was a
Quoted 1 line
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any
hypervisor
Quoted 1 line
Quoted 1 line
specific
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
setup and
Quoted 1 line
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the
actual
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an
expansion
Quoted 5 lines
Quoted 5 lines
in
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
Quoted 1 line
backends could be created. There is interest
in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like
SCMI.
Quoted 4 lines
Quoted 4 lines
There
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interest
Quoted 2 lines
Quoted 2 lines
in Rust
Quoted 1 line
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait
interface to
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
abstract
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
differences
Quoted 1 line
Quoted 1 line
Quoted 1 line
away as well as the idea of bare-metal Rust
backends.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
We have a card (STR-12) called "Hypercall
Standardisation"
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed
from
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends.
However
Quoted 5 lines
Quoted 5 lines
we
Quoted 4 lines
Quoted 4 lines
are
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
some
Quoted 2 lines
Quoted 2 lines
way off
Quoted 1 line
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need to
at
Quoted 7 lines
Quoted 7 lines
least
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
demonstrate one
Quoted 1 line
Quoted 1 line
Quoted 1 line
portable backend before we start codifying
requirements. To
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
that
Quoted 2 lines
Quoted 2 lines
end I
Quoted 1 line
Quoted 1 line
Quoted 1 line
want to think about what we need for a backend
to
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 6 lines
function.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Configuration
=============
In the type-2 setup this is typically fairly
simple
Quoted 4 lines
Quoted 4 lines
because
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
host
Quoted 1 line
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules
that
Quoted 4 lines
Quoted 4 lines
make
Quoted 3 lines
Quoted 3 lines
up the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
complete
Quoted 1 line
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2
with
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
delegated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
service VMs)
Quoted 1 line
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the
backend
Quoted 5 lines
Quoted 5 lines
VM
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
about
Quoted 3 lines
Quoted 3 lines
key
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
details about the system:
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- where virt queue memory is in it's address
space
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- how it's going to receive (interrupt) and
trigger
Quoted 3 lines
Quoted 3 lines
(kick)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
events
Quoted 1 line
Quoted 1 line
Quoted 1 line
- what (if any) resources the backend needs
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
connect to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Obviously you can elide over configuration
issues
Quoted 9 lines
Quoted 9 lines
by
Quoted 3 lines
Quoted 3 lines
having
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
static
Quoted 1 line
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions into
your
Quoted 4 lines
Quoted 4 lines
guest
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The
obvious
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
Quoted 4 lines
solution
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to user
space
Quoted 6 lines
Quoted 6 lines
but
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take place
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE guest
and
Quoted 5 lines
Quoted 5 lines
the
Quoted 4 lines
Quoted 4 lines
BE
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address
space
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
needs to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
visible
Quoted 1 line
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests. I
can
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
envision 3
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
approaches:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where
in
Quoted 4 lines
Quoted 4 lines
it's
Quoted 3 lines
Quoted 3 lines
Guest
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Physical
Quoted 1 line
Quoted 1 line
Quoted 1 line
Address space is already taken up and
avoiding
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
clashing. I
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
would assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
in this case you would want a standard
interface
Quoted 4 lines
Quoted 4 lines
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
userspace
Quoted 2 lines
Quoted 2 lines
to then
Quoted 1 line
Quoted 1 line
Quoted 1 line
make that address space visible to the
backend
Quoted 3 lines
Quoted 3 lines
daemon.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Yet another way here is that we would have well
known
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
"shared
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
memory" between
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us
good
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
insights on
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
this
Quoted 2 lines
Quoted 2 lines
matter
Quoted 1 line
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for
hypervisor-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
agnostic
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
solution.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as a
PCI
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
device
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
and
Quoted 4 lines
Quoted 4 lines
can be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mapped locally.)
I want to add this shared memory aspect to my
virtio-
Quoted 3 lines
Quoted 3 lines
proxy,
Quoted 2 lines
Quoted 2 lines
but
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the resultant solution would eventually look
similar
Quoted 7 lines
Quoted 7 lines
to
Quoted 3 lines
Quoted 3 lines
ivshmem.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
* BE guests boots with a hypervisor handle to
memory
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The BE guest is then free to map the FE's
memory
Quoted 7 lines
Quoted 7 lines
to
Quoted 6 lines
Quoted 6 lines
where
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the BE's guest physical address space.
I cannot see how this could work for Xen. There
is
Quoted 5 lines
Quoted 5 lines
no
Quoted 4 lines
Quoted 4 lines
"handle"
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running in
dom0.
Quoted 7 lines
Quoted 7 lines
So
Quoted 6 lines
Quoted 6 lines
for
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the
following
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
information
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
is
Quoted 8 lines
Quoted 8 lines
expected
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular
hypervisor
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
interface
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
map the configuration space.
Yes, Xenstore was chosen as a simple way to pass
configuration
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using Xenstore
in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Virtio
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
itself if possible, so for non-toolstack domain, this
could
Quoted 3 lines
Quoted 3 lines
done
Quoted 2 lines
Quoted 2 lines
with
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
adjusting devd (daemon that listens for devices and
launches
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backends)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore
anyway
Quoted 7 lines
Quoted 7 lines
and
Quoted 6 lines
Quoted 6 lines
pass it
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
to
Quoted 4 lines
Quoted 4 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to pass
Quoted 8 lines
Quoted 8 lines
device
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
configuration.
Quoted 1 line
We also designed a static device configuration parse
method
Quoted 4 lines
Quoted 4 lines
for
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from
device
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
model
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
command
Quoted 3 lines
Quoted 3 lines
line
Quoted 2 lines
or a config file.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
But, if ...
Quoted 2 lines
In my approach (virtio-proxy), all those Xen (or
hypervisor)-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
specific
Quoted 1 line
Quoted 1 line
Quoted 1 line
stuffs are contained in virtio-proxy, yet another
VM,
Quoted 6 lines
Quoted 6 lines
to
Quoted 5 lines
Quoted 5 lines
hide
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
all
Quoted 3 lines
Quoted 3 lines
details.
Quoted 2 lines
Quoted 2 lines
... the solution how to overcome that is already
found
Quoted 21 lines
Quoted 21 lines
and
Quoted 6 lines
Quoted 6 lines
proven
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
to
Quoted 4 lines
Quoted 4 lines
work then even better.
Quoted 3 lines
Quoted 3 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
# My point is that a "handle" is not mandatory for
executing
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mapping.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to
invent a
Quoted 3 lines
Quoted 3 lines
new
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor interface and Xen virtual machine
privileges
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
allow
Quoted 1 line
Quoted 1 line
this
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
kind of mapping.
If we run the backend in Dom0 that we have no
problems
Quoted 4 lines
Quoted 4 lines
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
course.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
One of difficulties on Xen that I found in my
approach
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
calling
Quoted 1 line
Quoted 1 line
Quoted 1 line
such hypervisor intefaces (registering IOREQ,
mapping
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
memory) is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
only
Quoted 1 line
Quoted 1 line
Quoted 1 line
allowed on BE servers themselvies and so we will
have
Quoted 5 lines
Quoted 5 lines
to
Quoted 4 lines
Quoted 4 lines
extend
3 lines not shown · raw message
558 lines not shown · raw message

Part of this message is not shown here. View the raw message.

Wei Chen

2021-09-02 01:51 UTC #24 lore raw

Hi Akashi, Oleksandr,

Quoted 362 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年9月2日 9:31
To: AKASHI Takahiro <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly Xin <[email protected]>; Stratos Mailing List <[email protected]>; [email protected]; Arnd Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>; Julien Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; nd <[email protected]>; Xen Devel <[email protected]>
Subject: RE: Enabling hypervisor agnosticism for VirtIO backends
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年9月1日 20:29
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>; Kaly
Xin
Quoted 1 line
<[email protected]>; Stratos Mailing List <stratos-dev@op-
lists.linaro.org>;
Quoted 1 line
[email protected]; Arnd Bergmann
<[email protected]>;
Quoted 8 lines
Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik <[email protected]>; [email protected]; Srivatsa Vaddagiri <[email protected]>; Jean- Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 4 lines
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul Durrant <[email protected]>; nd <[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Quoted 11 lines
Hi Wei,
Quoted 8 lines · Wei Chen
Hi Akashi,
Quoted 6 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月31日 14:18
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano Stabellini <[email protected]>; Alex Benn??e <[email protected]>;
Kaly
Quoted 8 lines
Xin
Quoted 1 line
Quoted 1 line
<[email protected]>; Stratos Mailing List <stratos-dev@op-
lists.linaro.org>;
Quoted 1 line
Quoted 1 line
[email protected]; Arnd Bergmann
<[email protected]>;
Quoted 3 lines
Quoted 3 lines
Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev <[email protected]>;
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Wei,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Wei Chen
Quoted 7 lines
Hi Akashi,
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月26日 17:41
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano
Stabellini
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e <[email protected]>;
Kaly
Quoted 3 lines
Quoted 3 lines
Xin
Quoted 1 line
Quoted 1 line
<[email protected]>; Stratos Mailing List <stratos-dev@op-
lists.linaro.org>;
Quoted 12 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
[email protected]; Arnd Bergmann
<[email protected]>;
Quoted 3 lines
Quoted 3 lines
Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan Kiszka <[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri <[email protected]>;
Jean-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Philippe Brucker <[email protected]>; Mathieu Poirier <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 24 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>; Paul
Durrant
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
Hi Wei,
Quoted 3 lines · AKASHI Takahiro
Quoted 2 lines · Wei Chen
Hi Akashi，
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月18日 13:39
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>; Stefano
Stabellini
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e
<[email protected]>;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Stratos
Quoted 1 line
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>; virtio-
[email protected]
Quoted 1 line
Quoted 1 line
Quoted 1 line
open.org; Arnd Bergmann <[email protected]>; Viresh
Quoted 10 lines
Kumar
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Jan
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Jean-
Quoted 1 line
Quoted 1 line
Quoted 1 line
Philippe Brucker <[email protected]>; Mathieu
Poirier
Quoted 20 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
<[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross <[email protected]>;
Paul
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for VirtIO
backends
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines · Wei Chen
Quoted 7 lines
Hi Akashi,
Quoted 5 lines
-----Original Message-----
From: AKASHI Takahiro <[email protected]>
Sent: 2021年8月17日 16:08
To: Wei Chen <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>;
Stefano
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Stabellini
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Alex Benn??e
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Stratos
Quoted 1 line
Quoted 1 line
Mailing List <[email protected]>;
virtio-
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[email protected]
Quoted 1 line
Quoted 1 line
open.org; Arnd Bergmann <[email protected]>;
Viresh
Quoted 3 lines
Quoted 3 lines
Kumar
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Stefano Stabellini <[email protected]>; [email protected];
Jan
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Jean-
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Philippe Brucker <[email protected]>; Mathieu
Poirier
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
<[email protected]>; Oleksandr Tyshchenko <[email protected]>; Bertrand Marquis <[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross
<[email protected]>;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Paul
Quoted 3 lines
Quoted 3 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Subject: Re: Enabling hypervisor agnosticism for
VirtIO
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
backends
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hi Wei, Oleksandr,
On Mon, Aug 16, 2021 at 10:04:03AM +0000, Wei Chen
Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hi All,
Thanks for Stefano to link my kvmtool for Xen
proposal
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
here.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This proposal is still discussing in Xen and KVM
communities.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The main work is to decouple the kvmtool from KVM
and
Quoted 7 lines
make
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
other hypervisors can reuse the virtual device
implementations.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In this case, we need to introduce an intermediate
hypervisor
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
layer for VMM abstraction, Which is, I think it's
very
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
close
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
to stratos' virtio hypervisor agnosticism work.
# My proposal[1] comes from my own idea and doesn't
Quoted 9 lines
always
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
represent
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
# Linaro's view on this subject nor reflect Alex's
concerns.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Nevertheless,
Quoted 2 lines
Quoted 2 lines
Your idea and my proposal seem to share the same
background.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Both have the similar goal and currently start with,
at
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
first,
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and are based on kvm-tool. (Actually, my work is
derived
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
from
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
EPAM's virtio-disk, which is also based on kvm-tool.)
In particular, the abstraction of hypervisor
interfaces
Quoted 7 lines
has
Quoted 3 lines
Quoted 3 lines
a
Quoted 2 lines
Quoted 2 lines
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
set of interfaces (for your "struct vmm_impl" and my
"RPC
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interfaces").
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This is not co-incident as we both share the same
origin
Quoted 17 lines
as
Quoted 4 lines
Quoted 4 lines
I
Quoted 3 lines
Quoted 3 lines
said
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
above.
Quoted 1 line
Quoted 1 line
And so we will also share the same issues. One of them
is a
Quoted 11 lines
Quoted 11 lines
way
Quoted 2 lines
Quoted 2 lines
of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"sharing/mapping FE's memory". There is some trade-off
between
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
the portability and the performance impact. So we can discuss the topic here in this ML, too. (See Alex's original email, too).
Yes, I agree.
Quoted 1 line
On the other hand, my approach aims to create a
"single-
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
binary"
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
solution
Quoted 1 line
Quoted 1 line
in which the same binary of BE vm could run on any
hypervisors.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Somehow similar to your "proposal-#2" in [2], but in
my
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
solution,
Quoted 2 lines
Quoted 2 lines
all
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the hypervisor-specific code would be put into another
entity
Quoted 2 lines
Quoted 2 lines
(VM),
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
named "virtio-proxy" and the abstracted operations are
served
Quoted 2 lines
Quoted 2 lines
via RPC.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(In this sense, BE is hypervisor-agnostic but might
have
Quoted 3 lines
OS
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
dependency.)
Quoted 1 line
Quoted 1 line
But I know that we need discuss if this is a
requirement
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
even
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
in Stratos project or not. (Maybe not)
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Sorry, I haven't had time to finish reading your virtio-
proxy
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
completely
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(I will do it ASAP). But from your description, it seems
we
Quoted 2 lines
Quoted 2 lines
need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
3rd VM between FE and BE? My concern is that, if my
assumption
Quoted 3 lines
Quoted 3 lines
is
Quoted 2 lines
Quoted 2 lines
right,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
will it increase the latency in data transport path?
Even
Quoted 7 lines
if
Quoted 4 lines
Quoted 4 lines
we're
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
using some lightweight guest like RTOS or Unikernel,
Yes, you're right. But I'm afraid that it is a matter of
degree.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
As far as we execute 'mapping' operations at every fetch
of
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
payload,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we will see latency issue (even in your case) and if we
have
Quoted 45 lines
Quoted 6 lines
Quoted 6 lines
some
Quoted 5 lines
Quoted 5 lines
solution
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
for it, we won't see it neither in my proposal :)
Oleksandr has sent a proposal to Xen mailing list to reduce
this
Quoted 2 lines
Quoted 2 lines
kind
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
of "mapping/unmapping" operations. So the latency caused by
this
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
behavior
Quoted 1 line
Quoted 1 line
on Xen may eventually be eliminated, and Linux-KVM doesn't
have
Quoted 18 lines
Quoted 9 lines
that
Quoted 8 lines
Quoted 7 lines
problem.
Quoted 3 lines
Obviously, I have not yet caught up there in the discussion. Which patch specifically?
Can you give me the link to the discussion or patch, please?
Quoted 8 lines
Quoted 2 lines
It's a RFC discussion. We have tested this RFC patch internally.
https://lists.xenproject.org/archives/html/xen-devel/2021-
07/msg01532.html
I'm afraid that I miss something here, but I don't know why this proposed API will lead to eliminating 'mmap' in accessing the queued payload at every request?
Quoted 5 lines
This API give Xen device model (QEMU or kvmtool) the ability to map whole guest RAM in device model's address space. In this case, device model doesn't need dynamic hypercall to map/unmap payload memory. It can use a flat offset to access payload memory in its address space directly. Just Like KVM device model does now.
Thank you. Quickly, let me make sure one thing:
This API itself doesn't do any mapping operations, right? So I suppose that virtio BE guest is responsible to
1) fetch the information about all the memory regions in FE,
2) call this API to allocate a big chunk of unused space in BE,
3) create grant/foreign mappings for FE onto this region(S) in the initialization/configuration of emulated virtio devices.

Quoted 7 lines

Quoted 3 lines
Is this the way this API is expected to be used?
Does Xen already has an interface for (1)?
They are discussing in that thread to find a proper way to do it. Because this API is common, both x86 and Arm should be considered.

Please ignore my above reply. I hadn't seen Oleksandr had replied this question. Sorry about it!

Quoted 1,163 lines

Quoted 11 lines
-Takahiro Akashi
Quoted 9 lines
Before this API, When device model to map whole guest memory, will severely consume the physical pages of Dom-0/Dom-D.
Quoted 6 lines
-Takahiro Akashi
Quoted 3 lines
Quoted 3 lines
Thanks,
-Takahiro Akashi
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
-Takahiro Akashi
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Specifically speaking about kvm-tool, I have a concern
about
Quoted 2 lines
Quoted 2 lines
its
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
license term; Targeting different hypervisors and
different
Quoted 2 lines
Quoted 2 lines
OSs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(which I assume includes RTOS's), the resultant
library
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
should
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
license permissive and GPL for kvm-tool might be an
issue.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Any thoughts?
Yes. If user want to implement a FreeBSD device model,
but
Quoted 18 lines
the
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
virtio
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
library is GPL. Then GPL would be a problem. If we have
another
Quoted 4 lines
Quoted 4 lines
good
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
candidate, I am open to it.
I have some candidates, particularly for vq/vring, in my
mind:
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
* Open-AMP, or
* corresponding Free-BSD code
Interesting, I will look into them : )
Cheers,
Wei Chen
Quoted 10 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-Takahiro Akashi
Quoted 4 lines
Quoted 4 lines
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-
dev/2021-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
August/000548.html
[2] https://marc.info/?l=xen-
devel&m=162373754705233&w=2
Quoted 24 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
From: Oleksandr Tyshchenko <[email protected]>
Sent: 2021年8月14日 23:38
To: AKASHI Takahiro <[email protected]>;
Quoted 7 lines
Quoted 7 lines
Stefano
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Stabellini
Quoted 2 lines
Quoted 2 lines
<[email protected]>
Quoted 1 line
Quoted 1 line
Cc: Alex Benn??e <[email protected]>; Stratos
Mailing
Quoted 2 lines
Quoted 2 lines
List
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; virtio-
[email protected]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
open.org;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Arnd
Quoted 3 lines
Quoted 3 lines
Bergmann <[email protected]>; Viresh Kumar <[email protected]>; Stefano Stabellini <[email protected]>; [email protected];
Jan
Quoted 5 lines
Quoted 5 lines
Kiszka
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Carl van Schaik
<[email protected]>;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; Srivatsa Vaddagiri
<[email protected]>;
Quoted 12 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Jean-
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Philippe Brucker <[email protected]>; Mathieu
Poirier
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Wei Chen
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Oleksandr
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Tyshchenko <[email protected]>; Bertrand
Marquis
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Artem Mygaiev
<[email protected]>;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Julien
Quoted 1 line
Quoted 1 line
Grall <[email protected]>; Juergen Gross
<[email protected]>;
Quoted 17 lines
Quoted 4 lines
Quoted 4 lines
Paul
Quoted 3 lines
Quoted 3 lines
Durrant
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; Xen Devel <[email protected]>
Quoted 1 line
Quoted 1 line
Subject: Re: Enabling hypervisor agnosticism for
VirtIO
Quoted 12 lines
Quoted 12 lines
Quoted 5 lines
Quoted 5 lines
backends
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hello, all.
Please see some comments below. And sorry for the
possible
Quoted 6 lines
Quoted 6 lines
format
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
issues.
Quoted 2 lines
Quoted 2 lines
On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
Quoted 1 line · <mailto:[email protected]> wrote:
Quoted 1 line
On Wed, Aug 04, 2021 at 12:20:01PM -0700,
Stefano
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Stabellini
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
CCing people working on Xen+VirtIO and IOREQs.
Not
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
trimming
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
the
Quoted 4 lines
Quoted 4 lines
original
Quoted 3 lines
Quoted 3 lines
email to let them read the full context.
My comments below are related to a potential
Xen
Quoted 11 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
implementation,
Quoted 2 lines
Quoted 2 lines
not
Quoted 1 line
Quoted 1 line
because it is the only implementation that
matters,
Quoted 6 lines
Quoted 6 lines
but
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
because it
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
the one I know best.
Please note that my proposal (and hence the
working
Quoted 16 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
prototype)[1]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is based on Xen's virtio implementation (i.e.
IOREQ)
Quoted 4 lines
Quoted 4 lines
and
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
particularly
Quoted 2 lines
Quoted 2 lines
EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is
still
Quoted 8 lines
Quoted 8 lines
a
Quoted 7 lines
Quoted 7 lines
bit
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
biased
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
toward this original design.
So I hope you like my approach :)
[1] https://op-
lists.linaro.org/pipermail/stratos-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
dev/2021-
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
August/000546.html
Quoted 2 lines
Quoted 2 lines
Let me take this opportunity to explain a bit
more
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
about
Quoted 6 lines
Quoted 6 lines
my
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
approach
Quoted 4 lines
Quoted 4 lines
below.
Quoted 3 lines
Quoted 3 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-
devel&m=162373754705233&w=2
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
On Wed, 4 Aug 2021, Alex Bennée wrote:
Quoted 13 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hi,
One of the goals of Project Stratos is to
enable
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
agnostic
Quoted 1 line
Quoted 1 line
backends so we can enable as much re-use of
code
Quoted 5 lines
Quoted 5 lines
as
Quoted 4 lines
Quoted 4 lines
possible
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
Quoted 2 lines
Quoted 2 lines
avoid
Quoted 1 line
Quoted 1 line
repeating ourselves. This is the flip side
of
Quoted 7 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
front end
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
where
Quoted 1 line
Quoted 1 line
multiple front-end implementations are
required -
Quoted 2 lines
Quoted 2 lines
one
Quoted 1 line
Quoted 1 line
per OS,
Quoted 13 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
assuming
Quoted 1 line
Quoted 1 line
you don't just want Linux guests. The
resultant
Quoted 4 lines
Quoted 4 lines
guests
Quoted 3 lines
Quoted 3 lines
are
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
trivially
Quoted 1 line
Quoted 1 line
movable between hypervisors modulo any
abstracted
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
paravirt
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
type
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interfaces.
In my original thumb nail sketch of a
solution
Quoted 51 lines
I
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
envisioned
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
vhost-user
Quoted 1 line
Quoted 1 line
daemons running in a broadly POSIX like
environment.
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface to
Quoted 1 line
Quoted 1 line
the daemon is fairly simple requiring only
some
Quoted 4 lines
Quoted 4 lines
mapped
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
memory
Quoted 2 lines
Quoted 2 lines
and some
Quoted 1 line
Quoted 1 line
sort of signalling for events (on Linux this
is
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
eventfd).
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The
Quoted 2 lines
Quoted 2 lines
idea was a
Quoted 1 line
Quoted 1 line
stub binary would be responsible for any
hypervisor
Quoted 1 line
Quoted 1 line
specific
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
setup and
Quoted 1 line
Quoted 1 line
then launch a common binary to deal with the
actual
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
virtqueue
Quoted 4 lines
Quoted 4 lines
requests
Quoted 3 lines
Quoted 3 lines
themselves.
Since that original sketch we've seen an
expansion
Quoted 5 lines
Quoted 5 lines
in
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
sort
Quoted 2 lines
Quoted 2 lines
of ways
Quoted 1 line
Quoted 1 line
backends could be created. There is interest
in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
encapsulating
Quoted 2 lines
Quoted 2 lines
backends
Quoted 1 line
Quoted 1 line
in RTOSes or unikernels for solutions like
SCMI.
Quoted 4 lines
Quoted 4 lines
There
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interest
Quoted 2 lines
Quoted 2 lines
in Rust
Quoted 1 line
Quoted 1 line
has prompted ideas of using the trait
interface to
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
abstract
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
differences
Quoted 1 line
Quoted 1 line
away as well as the idea of bare-metal Rust
backends.
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
We have a card (STR-12) called "Hypercall
Standardisation"
Quoted 13 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
calls for a description of the APIs needed
from
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hypervisor
Quoted 2 lines
Quoted 2 lines
side to
Quoted 1 line
Quoted 1 line
support VirtIO guests and their backends.
However
Quoted 5 lines
Quoted 5 lines
we
Quoted 4 lines
Quoted 4 lines
are
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
some
Quoted 2 lines
Quoted 2 lines
way off
Quoted 1 line
Quoted 1 line
from that at the moment as I think we need
to
Quoted 8 lines
at
Quoted 7 lines
Quoted 7 lines
least
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
demonstrate one
Quoted 1 line
Quoted 1 line
portable backend before we start codifying
requirements. To
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
that
Quoted 2 lines
Quoted 2 lines
end I
Quoted 1 line
Quoted 1 line
want to think about what we need for a
backend
Quoted 10 lines
to
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
function.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Configuration
=============
In the type-2 setup this is typically fairly
simple
Quoted 2 lines
Quoted 2 lines
because
Quoted 1 line
Quoted 1 line
Quoted 1 line
the
Quoted 18 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
host
Quoted 1 line
Quoted 1 line
system can orchestrate the various modules
that
Quoted 4 lines
Quoted 4 lines
make
Quoted 3 lines
Quoted 3 lines
up the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
complete
Quoted 1 line
Quoted 1 line
system. In the type-1 case (or even type-2
with
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
delegated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
service VMs)
Quoted 1 line
Quoted 1 line
we need some sort of mechanism to inform the
backend
Quoted 6 lines
Quoted 6 lines
VM
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
about
Quoted 4 lines
Quoted 4 lines
key
Quoted 3 lines
Quoted 3 lines
details about the system:
- where virt queue memory is in it's
address
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
space
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- how it's going to receive (interrupt)
and
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
trigger
Quoted 3 lines
Quoted 3 lines
(kick)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
events
Quoted 1 line
Quoted 1 line
- what (if any) resources the backend
needs
Quoted 9 lines
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
connect to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Obviously you can elide over configuration
issues
Quoted 4 lines
Quoted 4 lines
by
Quoted 3 lines
Quoted 3 lines
having
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
static
Quoted 1 line
Quoted 1 line
configurations and baking the assumptions
into
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
your
Quoted 4 lines
Quoted 4 lines
guest
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
images
Quoted 2 lines
Quoted 2 lines
however
Quoted 1 line
Quoted 1 line
this isn't scalable in the long term. The
obvious
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
solution
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
seems
Quoted 2 lines
Quoted 2 lines
to be
Quoted 1 line
Quoted 1 line
extending a subset of Device Tree data to
user
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
space
Quoted 6 lines
Quoted 6 lines
but
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
perhaps
Quoted 4 lines
Quoted 4 lines
there
Quoted 3 lines
Quoted 3 lines
are other approaches?
Before any virtio transactions can take
place
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
appropriate
Quoted 2 lines
Quoted 2 lines
memory
Quoted 1 line
Quoted 1 line
mappings need to be made between the FE
guest
Quoted 11 lines
and
Quoted 5 lines
Quoted 5 lines
the
Quoted 4 lines
Quoted 4 lines
BE
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
guest.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Currently the whole of the FE guests address
space
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
needs to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
be
Quoted 2 lines
Quoted 2 lines
visible
Quoted 1 line
Quoted 1 line
to whatever is serving the virtio requests.
I
Quoted 7 lines
can
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
envision 3
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
approaches:
Quoted 4 lines
Quoted 4 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing
where
Quoted 20 lines
in
Quoted 4 lines
Quoted 4 lines
it's
Quoted 3 lines
Quoted 3 lines
Guest
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Physical
Quoted 1 line
Quoted 1 line
Address space is already taken up and
avoiding
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
clashing. I
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
would assume
Quoted 1 line
Quoted 1 line
in this case you would want a standard
interface
Quoted 4 lines
Quoted 4 lines
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
userspace
Quoted 2 lines
Quoted 2 lines
to then
Quoted 1 line
Quoted 1 line
make that address space visible to the
backend
Quoted 3 lines
Quoted 3 lines
daemon.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Yet another way here is that we would have well
known
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"shared
Quoted 12 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
memory" between
Quoted 1 line
Quoted 1 line
VMs. I think that Jailhouse's ivshmem gives us
good
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
insights on
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
this
Quoted 2 lines
Quoted 2 lines
matter
Quoted 1 line
Quoted 1 line
and that it can even be an alternative for
hypervisor-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
agnostic
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
solution.
Quoted 2 lines
Quoted 2 lines
(Please note memory regions in ivshmem appear as
a
Quoted 4 lines
PCI
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
device
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
and
Quoted 1 line
Quoted 1 line
can be
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mapped locally.)
I want to add this shared memory aspect to my
virtio-
Quoted 3 lines
Quoted 3 lines
proxy,
Quoted 2 lines
Quoted 2 lines
but
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the resultant solution would eventually look
similar
Quoted 4 lines
Quoted 4 lines
to
Quoted 3 lines
Quoted 3 lines
ivshmem.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
* BE guests boots with a hypervisor handle
to
Quoted 11 lines
Quoted 3 lines
Quoted 3 lines
memory
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The BE guest is then free to map the FE's
memory
Quoted 7 lines
Quoted 7 lines
to
Quoted 6 lines
Quoted 6 lines
where
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
it
Quoted 4 lines
Quoted 4 lines
wants in
Quoted 3 lines
Quoted 3 lines
the BE's guest physical address space.
I cannot see how this could work for Xen.
There
Quoted 6 lines
is
Quoted 5 lines
Quoted 5 lines
no
Quoted 4 lines
Quoted 4 lines
"handle"
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
Quoted 2 lines
give
Quoted 1 line
Quoted 1 line
to the backend if the backend is not running
in
Quoted 14 lines
dom0.
Quoted 7 lines
Quoted 7 lines
So
Quoted 6 lines
Quoted 6 lines
for
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Xen I
Quoted 4 lines
Quoted 4 lines
think
Quoted 3 lines
Quoted 3 lines
the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the
following
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
information
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
expected
Quoted 2 lines
Quoted 2 lines
to be exposed to BE via Xenstore:
(I know that this is a tentative approach
though.)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular
Quoted 10 lines
hypervisor
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
interface
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
map the configuration space.
Yes, Xenstore was chosen as a simple way to pass
configuration
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
info to
Quoted 2 lines
Quoted 2 lines
the backend running in a non-toolstack domain.
Quoted 1 line
Quoted 1 line
I remember, there was a wish to avoid using
Xenstore
Quoted 4 lines
in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Virtio
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backend
Quoted 1 line
Quoted 1 line
itself if possible, so for non-toolstack domain, this
Quoted 15 lines
could
Quoted 3 lines
Quoted 3 lines
done
Quoted 2 lines
Quoted 2 lines
with
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
adjusting devd (daemon that listens for devices and
launches
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
backends)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to read backend configuration from the Xenstore
anyway
Quoted 7 lines
Quoted 7 lines
and
Quoted 6 lines
Quoted 6 lines
pass it
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
to
Quoted 4 lines
Quoted 4 lines
the backend via command line arguments.
Quoted 3 lines
Quoted 1 line
Yes, in current PoC code we're using xenstore to
pass
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
device
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
configuration.
Quoted 1 line
We also designed a static device configuration parse
method
Quoted 4 lines
Quoted 4 lines
for
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Dom0less
Quoted 2 lines
Quoted 2 lines
or
Quoted 1 line
other scenarios don't have xentool. yes, it's from
device
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
model
11 lines not shown · raw message
653 lines not shown · raw message

Part of this message is not shown here. View the raw message.

A message that is not in the archive (sent off-list or not yet received): 0100017b33e585a5-06d4248e-b1a7-485e-800c-7ead89e5f916-000000@email.amazonses.com

François Ozog

2021-08-16 08:02 UTC #26 lore raw

I top post as I find it difficult to identify where to make the comments.

1) BE acceleration
Network and storage backends may actually be executed in SmartNICs. As virtio 1.1 is hardware friendly, there may be SmartNICs with virtio 1.1 PCI VFs. Is it a valid use case for the generic BE framework to be used in this context?
DPDK is used in some BE to significantly accelerate switching. DPDK is also used sometimes in guests. In that case, there are no event injection but just high performance memory scheme. Is this considered as a use case?

2) Virtio as OS HAL
Panasonic CTO has been calling for a virtio based HAL and based on the teachings of Google GKI, an internal HAL seem inevitable in the long term. Virtio is then a contender to Google promoted Android HAL. Could the framework be used in that context?

Quoted 289 lines · AKASHI Takahiro via Stratos-dev

Quoted 6 lines · Stefano Stabellini
CCing people working on Xen+VirtIO and IOREQs. Not trimming the original email to let them read the full context.
My comments below are related to a potential Xen implementation, not because it is the only implementation that matters, but because it is the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and particularly EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.
So I hope you like my approach :)
[1]
https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000546.html
Let me take this opportunity to explain a bit more about my approach below.
Quoted 4 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 15 lines
Quoted 14 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was
a
Quoted 26 lines
Quoted 26 lines
stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============
In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
Quoted 6 lines
Quoted 6 lines
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images
however
Quoted 9 lines
Quoted 9 lines
this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 5 lines
Quoted 5 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would
assume
Quoted 2 lines
Quoted 2 lines
in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared memory" between
VMs. I think that Jailhouse's ivshmem gives us good insights on this matter and that it can even be an alternative for hypervisor-agnostic solution.
(Please note memory regions in ivshmem appear as a PCI device and can be mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 8 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to give to the backend if the backend is not running in dom0. So for Xen I think the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information is expected to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.
In my approach (virtio-proxy), all those Xen (or hypervisor)-specific stuffs are contained in virtio-proxy, yet another VM, to hide all details.
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 6 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allow this kind of mapping.
If we run the backend in Dom0 that we have no problems of course.
One of difficulties on Xen that I found in my approach is that calling such hypervisor intefaces (registering IOREQ, mapping memory) is only allowed on BE servers themselvies and so we will have to extend those interfaces.
This, however, will raise some concern on security and privilege distribution
as Stefan suggested.
Quoted 4 lines
Quoted 4 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see two
options
Quoted 38 lines
Quoted 19 lines
at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========
Quoted 11 lines
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs the way they are typically envisioned and described in other environments. Instead, Xen has IOREQ servers. Each of them connects independently to Xen via the IOREQ interface. E.g. today multiple QEMUs could be used as emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
Quoted 3 lines
The component responsible for starting a daemon and/or setting up shared interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in Startos jargon?) is a subject to debate in parallel.
Otherwise, is there any good assumption to avoid it right now?
Quoted 7 lines
Oleksandr and others I CCed have been working on ways for the toolstack to create virtio backends and setup memory mappings. They might be able to provide more info on the subject. I do think we miss a way to provide the configuration to the backend and anything else that the backend might require to start doing its job.
Quoted 8 lines
Quoted 5 lines
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisor interface via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels), which is converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)
I don't know what "connect directly" means here, but sending interrupts to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-x mechanism.
Quoted 3 lines
Quoted 3 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other
kernel
Quoted 7 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add.
As I said above, my proposal does the same thing that you mentioned here :) The difference is that I do call hypervisor interfaces via virtio-proxy.
Quoted 4 lines
The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 4 lines
I don't think that translating IOREQs to eventfd in the kernel is a good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor interface.
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).
Quoted 8 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic, perhaps we could design a new interface. One that could be implementable in the Xen hypervisor itself (like IOREQ) and of course any other hypervisor too.
Quoted 6 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
My proposal document might help here; All the interfaces required for virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 4 lines
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is a different matter.
Thanks,
-Takahiro Akashi
--
Stratos-dev mailing list
[email protected]
https://op-lists.linaro.org/mailman/listinfo/stratos-dev

Signature · 3 lines

François-Frédéric Ozog | *Director Business Development*
T: +33.67221.6485
[email protected] | Skype: ffozog

AKASHI Takahiro

2021-08-13 05:11 UTC #27 lore raw

Hi François,

Quoted 1 line · Fran??ois Ozog

I top post as I find it difficult to identify where to make the comments.

Thank you for the posting.
I think that we should first discuss more about the goal/requirements/ practical use cases for the framework.

Quoted 8 lines

1) BE acceleration
Network and storage backends may actually be executed in SmartNICs. As virtio 1.1 is hardware friendly, there may be SmartNICs with virtio 1.1 PCI VFs. Is it a valid use case for the generic BE framework to be used in this context?
DPDK is used in some BE to significantly accelerate switching. DPDK is also used sometimes in guests. In that case, there are no event injection but just high performance memory scheme. Is this considered as a use case?

I'm not quite familiar with DPDK but it seems to be heavily reliant on not only virtqueues but also kvm/linux features/functionality, say, according to [1].
I'm afraid that DPDK is not suitable for primary (at least, initial) target use.
# In my proposal, virtio-proxy, I have in mind the assumption that we would

# create BE VM as a baremetal application on RTOS (and/or unikernel.)

But as far as virtqueue is concerned, I think we can discuss in general technical details as Alex suggested, including:
- sharing or mapping memory regions for data payload
- efficient notification mechanism

[1] https://www.redhat.com/en/blog/journey-vhost-users-realm

Quoted 5 lines

2) Virtio as OS HAL
Panasonic CTO has been calling for a virtio based HAL and based on the teachings of Google GKI, an internal HAL seem inevitable in the long term. Virtio is then a contender to Google promoted Android HAL. Could the framework be used in that context?

In this case, where will the implementation of "HAL" reside? I don't think the portability of "HAL" code (as a set of virtio BEs) is a requirement here.

-Takahiro Akashi

Quoted 298 lines

Quoted 289 lines · AKASHI Takahiro via Stratos-dev
Quoted 6 lines · Stefano Stabellini
CCing people working on Xen+VirtIO and IOREQs. Not trimming the original email to let them read the full context.
My comments below are related to a potential Xen implementation, not because it is the only implementation that matters, but because it is the one I know best.
Please note that my proposal (and hence the working prototype)[1] is based on Xen's virtio implementation (i.e. IOREQ) and particularly EPAM's virtio-disk application (backend server).
It has been, I believe, well generalized but is still a bit biased toward this original design.
So I hope you like my approach :)
[1]
https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000546.html
Let me take this opportunity to explain a bit more about my approach below.
Quoted 4 lines
Also, please see this relevant email thread:
https://marc.info/?l=xen-devel&m=162373754705233&w=2
Quoted 15 lines
Quoted 14 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was
a
Quoted 26 lines
Quoted 26 lines
stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============
In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
Quoted 6 lines
Quoted 6 lines
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images
however
Quoted 9 lines
Quoted 9 lines
this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest.
Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3
approaches:
Quoted 5 lines
Quoted 5 lines
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would
assume
Quoted 2 lines
Quoted 2 lines
in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
Yet another way here is that we would have well known "shared memory" between
VMs. I think that Jailhouse's ivshmem gives us good insights on this matter and that it can even be an alternative for hypervisor-agnostic solution.
(Please note memory regions in ivshmem appear as a PCI device and can be mapped locally.)
I want to add this shared memory aspect to my virtio-proxy, but the resultant solution would eventually look similar to ivshmem.
Quoted 8 lines
Quoted 4 lines
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space.
I cannot see how this could work for Xen. There is no "handle" to give to the backend if the backend is not running in dom0. So for Xen I think the memory has to be already mapped
In Xen's IOREQ solution (virtio-blk), the following information is expected to be exposed to BE via Xenstore:
(I know that this is a tentative approach though.)
- the start address of configuration space
- interrupt number
- file path for backing storage
- read-only flag
And the BE server have to call a particular hypervisor interface to map the configuration space.
In my approach (virtio-proxy), all those Xen (or hypervisor)-specific stuffs are contained in virtio-proxy, yet another VM, to hide all details.
# My point is that a "handle" is not mandatory for executing mapping.
Quoted 6 lines
and the mapping probably done by the
toolstack (also see below.) Or we would have to invent a new Xen hypervisor interface and Xen virtual machine privileges to allow this kind of mapping.
If we run the backend in Dom0 that we have no problems of course.
One of difficulties on Xen that I found in my approach is that calling such hypervisor intefaces (registering IOREQ, mapping memory) is only allowed on BE servers themselvies and so we will have to extend those interfaces.
This, however, will raise some concern on security and privilege distribution
as Stefan suggested.
Quoted 4 lines
Quoted 4 lines
To activate the mapping will
require some sort of hypercall to the hypervisor. I can see two
options
Quoted 38 lines
Quoted 19 lines
at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========
Quoted 11 lines
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One thing I mentioned before to Alex is that Xen doesn't have VMMs the way they are typically envisioned and described in other environments. Instead, Xen has IOREQ servers. Each of them connects independently to Xen via the IOREQ interface. E.g. today multiple QEMUs could be used as emulators for a single Xen VM, each of them connecting to Xen independently via the IOREQ interface.
Quoted 3 lines
The component responsible for starting a daemon and/or setting up shared interfaces is the toolstack: the xl command and the libxl/libxc libraries.
I think that VM configuration management (or orchestration in Startos jargon?) is a subject to debate in parallel.
Otherwise, is there any good assumption to avoid it right now?
Quoted 7 lines
Oleksandr and others I CCed have been working on ways for the toolstack to create virtio backends and setup memory mappings. They might be able to provide more info on the subject. I do think we miss a way to provide the configuration to the backend and anything else that the backend might require to start doing its job.
Quoted 8 lines
Quoted 5 lines
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Yeah I think, regardless of anything else, we want the backends to connect directly to the Xen hypervisor.
In my approach,
a) BE -> FE: interrupts triggered by BE calling a hypervisor interface via virtio-proxy
b) FE -> BE: MMIO to config raises events (in event channels), which is converted to a callback to BE via virtio-proxy (Xen's event channel is internnally implemented by
interrupts.)
I don't know what "connect directly" means here, but sending interrupts to the opposite side would be best efficient.
Ivshmem, I suppose, takes this approach by utilizing PCI's msi-x mechanism.
Quoted 3 lines
Quoted 3 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other
kernel
Quoted 7 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add.
As I said above, my proposal does the same thing that you mentioned here :) The difference is that I do call hypervisor interfaces via virtio-proxy.
Quoted 4 lines
The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 4 lines
I don't think that translating IOREQs to eventfd in the kernel is a good idea: if feels like it would be extra complexity and that the kernel shouldn't be involved as this is a backend-hypervisor interface.
Given that we may want to implement BE as a bare-metal application as I did on Zephyr, I don't think that the translation would not be a big issue, especially on RTOS's.
It will be some kind of abstraction layer of interrupt handling (or nothing but a callback mechanism).
Quoted 8 lines
Also, eventfd is very Linux-centric and we are trying to design an interface that could work well for RTOSes too. If we want to do something different, both OS-agnostic and hypervisor-agnostic, perhaps we could design a new interface. One that could be implementable in the Xen hypervisor itself (like IOREQ) and of course any other hypervisor too.
Quoted 6 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
My proposal document might help here; All the interfaces required for virtio-proxy (or hypervisor-related interfaces) are listed as RPC protocols :)
Quoted 4 lines
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Exactly. I have no confident yet that my approach will also apply to other hypervisors than Xen.
Technically, yes, but whether people can accept it or not is a different matter.
Thanks,
-Takahiro Akashi
--
Stratos-dev mailing list
[email protected]
https://op-lists.linaro.org/mailman/listinfo/stratos-dev
Signature · 3 lines
François-Frédéric Ozog | *Director Business Development*
T: +33.67221.6485
[email protected] | Skype: ffozog

Alex Bennée

author

2021-09-01 08:59 UTC #28 lore raw

Quoted 39 lines · AKASHI Takahiro

Hi François,
Quoted 1 line · Fran??ois Ozog
I top post as I find it difficult to identify where to make the comments.
Thank you for the posting.
I think that we should first discuss more about the goal/requirements/ practical use cases for the framework.
Quoted 8 lines
1) BE acceleration
Network and storage backends may actually be executed in SmartNICs. As virtio 1.1 is hardware friendly, there may be SmartNICs with virtio 1.1 PCI VFs. Is it a valid use case for the generic BE framework to be used in this context?
DPDK is used in some BE to significantly accelerate switching. DPDK is also used sometimes in guests. In that case, there are no event injection but just high performance memory scheme. Is this considered as a use case?
I'm not quite familiar with DPDK but it seems to be heavily reliant on not only virtqueues but also kvm/linux features/functionality, say, according to [1].
I'm afraid that DPDK is not suitable for primary (at least, initial) target use.
# In my proposal, virtio-proxy, I have in mind the assumption that we would
# create BE VM as a baremetal application on RTOS (and/or unikernel.)
But as far as virtqueue is concerned, I think we can discuss in general technical details as Alex suggested, including:
- sharing or mapping memory regions for data payload
- efficient notification mechanism
[1] https://www.redhat.com/en/blog/journey-vhost-users-realm
Quoted 5 lines
2) Virtio as OS HAL
Panasonic CTO has been calling for a virtio based HAL and based on the teachings of Google GKI, an internal HAL seem inevitable in the long term. Virtio is then a contender to Google promoted Android HAL. Could the framework be used in that context?

Quoted 3 lines

In this case, where will the implementation of "HAL" reside? I don't think the portability of "HAL" code (as a set of virtio BEs) is a requirement here.

When I hear people referring to VirtIO HALs I'm thinking mainly of VirtIO FE's living in a Linux kernel. There are certainly more devices that can get added but the commonality on the guest side I think is pretty much a solved problem (modulo Linux-ism's creeping into the virtio spec).

Signature · 5 lines

Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Stefan Hajnoczi

2021-08-17 12:58 UTC #29 lore raw

In reply to Stefano Stabellini

Quoted 6 lines · Stefano Stabellini

Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?

Quoted 7 lines

One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52

There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/

Quoted 11 lines

There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.

Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.

Stefan

AKASHI Takahiro

2021-08-23 06:25 UTC #30 lore raw

Hi Stefan,

Quoted 15 lines · Stefan Hajnoczi

Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52

Quoted 3 lines

There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/

Do you know the current status of Elena's work? It was last February that she posted her latest patch and it has not been merged upstream yet.

Quoted 12 lines

Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.

Quoted 4 lines

Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.

Can you please elaborate your thoughts a bit more here?

It seems to me that trapping MMIOs to configuration space and forwarding those events to BE (or device emulation) is a quite straight-forward way to emulate device MMIOs.
Or do you think of something of protocols used in vhost-user?

# On the contrary, virtio-ivshmem only requires a driver to explicitly

# forward a "write" request of MMIO accesses to BE. But I don't think
# it's your point.

-Takahiro Akashi

Quoted 1 line

Stefan

Stefan Hajnoczi

2021-08-23 09:59 UTC #31 lore raw

Quoted 18 lines · AKASHI Takahiro

Hi Stefan,
Quoted 15 lines · Stefan Hajnoczi
Quoted 13 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52

Quoted 7 lines

Quoted 3 lines
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/
Do you know the current status of Elena's work? It was last February that she posted her latest patch and it has not been merged upstream yet.

Elena worked on this during her Outreachy internship. At the moment no one is actively working on the patches.

Quoted 19 lines

Quoted 12 lines
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 4 lines
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
Can you please elaborate your thoughts a bit more here?

Quoted 8 lines

It seems to me that trapping MMIOs to configuration space and forwarding those events to BE (or device emulation) is a quite straight-forward way to emulate device MMIOs.
Or do you think of something of protocols used in vhost-user?
# On the contrary, virtio-ivshmem only requires a driver to explicitly
# forward a "write" request of MMIO accesses to BE. But I don't think
# it's your point. 

See my first reply to this email thread about alternative interfaces for VIRTIO device emulation. The main thing to note was that although the shared memory vring is used by VIRTIO transports today, the device model actually allows transports to implement virtqueues differently (e.g. making it possible to create a VIRTIO over TCP transport without shared memory in the future).

It's possible to define a hypercall interface as a new VIRTIO transport that provides higher-level virtqueue operations. Doing this is more work than using vrings though since existing guest driver and device emulation code already supports vrings.

I don't know the requirements of Stratos so I can't say if creating a new hypervisor-independent interface (VIRTIO transport) that doesn't rely on shared memory vrings makes sense. I just wanted to raise the idea in case you find that VIRTIO's vrings don't meet your requirements.

Stefan

AKASHI Takahiro

2021-08-25 10:30 UTC #32 lore raw

Hi Stefan,

Quoted 23 lines · Stefan Hajnoczi

Quoted 22 lines · AKASHI Takahiro
Hi Stefan,
Quoted 18 lines · Stefan Hajnoczi
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/

Quoted 6 lines

Quoted 3 lines
Do you know the current status of Elena's work? It was last February that she posted her latest patch and it has not been merged upstream yet.
Elena worked on this during her Outreachy internship. At the moment no one is actively working on the patches.

Does RedHat plan to take over or follow up her work hereafter? # I'm simply asking from my curiosity.

Quoted 28 lines

Quoted 19 lines
Quoted 12 lines
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 4 lines
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
Can you please elaborate your thoughts a bit more here?
Quoted 8 lines
It seems to me that trapping MMIOs to configuration space and forwarding those events to BE (or device emulation) is a quite straight-forward way to emulate device MMIOs.
Or do you think of something of protocols used in vhost-user?
# On the contrary, virtio-ivshmem only requires a driver to explicitly
# forward a "write" request of MMIO accesses to BE. But I don't think
# it's your point. 

Quoted 6 lines

See my first reply to this email thread about alternative interfaces for VIRTIO device emulation. The main thing to note was that although the shared memory vring is used by VIRTIO transports today, the device model actually allows transports to implement virtqueues differently (e.g. making it possible to create a VIRTIO over TCP transport without shared memory in the future).

Do you have any example of such use cases or systems?

Quoted 4 lines

It's possible to define a hypercall interface as a new VIRTIO transport that provides higher-level virtqueue operations. Doing this is more work than using vrings though since existing guest driver and device emulation code already supports vrings.

Personally, I'm open to discuss about your point, but

Quoted 4 lines

I don't know the requirements of Stratos so I can't say if creating a new hypervisor-independent interface (VIRTIO transport) that doesn't rely on shared memory vrings makes sense. I just wanted to raise the idea in case you find that VIRTIO's vrings don't meet your requirements.

While I cannot represent the project's view, what the JIRA task that is assigned to me describes:
Deliverables
* Low level library allowing:
* management of virtio rings and buffers
[and so on]
So supporting the shared memory-based vring is one of our assumptions.

In my understanding, the goal of Stratos project is that we would have several VMs congregated into a SoC, yet sharing most of physical IPs, where the shared memory should be, I assume, the most efficient transport for virtio.
One of target applications would be automotive, I guess.

Alex and Mike should have more to say here.

-Takahiro Akashi

Quoted 1 line

Stefan

Stefan Hajnoczi

2021-08-25 15:03 UTC #33 lore raw

Quoted 28 lines · AKASHI Takahiro

Quoted 27 lines · Stefan Hajnoczi
Quoted 25 lines · AKASHI Takahiro
Hi Stefan,
Quoted 15 lines · Stefan Hajnoczi
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 3 lines
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/
Do you know the current status of Elena's work? It was last February that she posted her latest patch and it has not been merged upstream yet.

Quoted 5 lines

Quoted 2 lines
Elena worked on this during her Outreachy internship. At the moment no one is actively working on the patches.
Does RedHat plan to take over or follow up her work hereafter? # I'm simply asking from my curiosity.

At the moment I'm not aware of anyone from Red Hat working on it. If someone decides they need this KVM API then that could change.

Quoted 28 lines

Quoted 28 lines
Quoted 27 lines
Quoted 12 lines
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 4 lines
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
Can you please elaborate your thoughts a bit more here?
It seems to me that trapping MMIOs to configuration space and forwarding those events to BE (or device emulation) is a quite straight-forward way to emulate device MMIOs.
Or do you think of something of protocols used in vhost-user?
# On the contrary, virtio-ivshmem only requires a driver to explicitly
# forward a "write" request of MMIO accesses to BE. But I don't think
# it's your point. 

Quoted 8 lines

Quoted 6 lines
See my first reply to this email thread about alternative interfaces for VIRTIO device emulation. The main thing to note was that although the shared memory vring is used by VIRTIO transports today, the device model actually allows transports to implement virtqueues differently (e.g. making it possible to create a VIRTIO over TCP transport without shared memory in the future).
Do you have any example of such use cases or systems?

This aspect of VIRTIO isn't being exploited today AFAIK. But the layering to allow other virtqueue implementations is there. For example, Linux's virtqueue API is independent of struct vring, so existing drivers generally aren't tied to vrings.

Quoted 12 lines

Quoted 4 lines
It's possible to define a hypercall interface as a new VIRTIO transport that provides higher-level virtqueue operations. Doing this is more work than using vrings though since existing guest driver and device emulation code already supports vrings.
Personally, I'm open to discuss about your point, but
Quoted 4 lines
I don't know the requirements of Stratos so I can't say if creating a new hypervisor-independent interface (VIRTIO transport) that doesn't rely on shared memory vrings makes sense. I just wanted to raise the idea in case you find that VIRTIO's vrings don't meet your requirements.

Quoted 7 lines

While I cannot represent the project's view, what the JIRA task that is assigned to me describes:
Deliverables
* Low level library allowing:
* management of virtio rings and buffers
[and so on]
So supporting the shared memory-based vring is one of our assumptions.

If shared memory is allowed then vrings are the natural choice. That way existing virtio code will work with minimal modifications.

Stefan

Alex Bennée

author

2021-09-01 13:20 UTC #34 lore raw

In reply to Stefan Hajnoczi

Quoted 16 lines · Stefan Hajnoczi

[[PGP Signed Part:Undecided]]
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52

Quoted 3 lines

There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/

Reading the cover letter was very useful in showing how this provides a separate channel for signalling IO events to userspace instead of using the normal type-2 vmexit type event. I wonder how deeply tied the userspace facing side of this is to KVM? Could it provide a common FD type interface to IOREQ?

As I understand IOREQ this is currently a direct communication between userspace and the hypervisor using the existing Xen message bus. My worry would be that by adding knowledge of what the underlying hypervisor is we'd end up with excess complexity in the kernel. For one thing we certainly wouldn't want an API version dependency on the kernel to understand which version of the Xen hypervisor it was running on.

Quoted 12 lines

Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.

Quoted 5 lines

Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.

It's true our focus is just VirtIO which does support alternative transport options however most implementations seem to be targeting virtio-mmio for it's relative simplicity and understood semantics (modulo a desire for MSI to reduce round trip latency handling signalling).

Quoted 4 lines

Stefan
[[End of PGP Signed Part]]

Signature · 5 lines

Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Stefan Hajnoczi

2021-09-02 09:12 UTC #35 lore raw

Quoted 23 lines · Alex Bennée

Quoted 16 lines · Stefan Hajnoczi
[[PGP Signed Part:Undecided]]
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 3 lines
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/

Quoted 5 lines

Reading the cover letter was very useful in showing how this provides a separate channel for signalling IO events to userspace instead of using the normal type-2 vmexit type event. I wonder how deeply tied the userspace facing side of this is to KVM? Could it provide a common FD type interface to IOREQ?

I wondered this too after reading Stefano's link to Xen's ioreq. They seem to be quite similar. ioregionfd is closer to have PIO/MMIO vmexits are handled in KVM while I guess ioreq is closer to how Xen handles them, but those are small details.

It may be possible to use the ioreq struct instead of ioregionfd in KVM, but I haven't checked each field.

Quoted 25 lines

As I understand IOREQ this is currently a direct communication between userspace and the hypervisor using the existing Xen message bus. My worry would be that by adding knowledge of what the underlying hypervisor is we'd end up with excess complexity in the kernel. For one thing we certainly wouldn't want an API version dependency on the kernel to understand which version of the Xen hypervisor it was running on.
Quoted 12 lines
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 5 lines
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.

Quoted 5 lines

It's true our focus is just VirtIO which does support alternative transport options however most implementations seem to be targeting virtio-mmio for it's relative simplicity and understood semantics (modulo a desire for MSI to reduce round trip latency handling signalling).

Okay.

Stefan

AKASHI Takahiro

2021-09-03 08:06 UTC #36 lore raw

In reply to Alex Bennée

Alex,

Quoted 23 lines · Alex Benn??e

Quoted 16 lines · Stefan Hajnoczi
[[PGP Signed Part:Undecided]]
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 3 lines
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/

Quoted 5 lines

Reading the cover letter was very useful in showing how this provides a separate channel for signalling IO events to userspace instead of using the normal type-2 vmexit type event. I wonder how deeply tied the userspace facing side of this is to KVM? Could it provide a common FD type interface to IOREQ?

Why do you stick to a "FD" type interface?

Quoted 2 lines

As I understand IOREQ this is currently a direct communication between userspace and the hypervisor using the existing Xen message bus. My

With IOREQ server, IO event occurrences are notified to BE via Xen's event channel, while the actual contexts of IO events (see struct ioreq in ioreq.h) are put in a queue on a single shared memory page which is to be assigned beforehand with xenforeignmemory_map_resource hypervisor call.

Quoted 4 lines

worry would be that by adding knowledge of what the underlying hypervisor is we'd end up with excess complexity in the kernel. For one thing we certainly wouldn't want an API version dependency on the kernel to understand which version of the Xen hypervisor it was running on.

That's exactly what virtio-proxy in my proposal[1] does; All the hypervisor- specific details of IO event handlings are contained in virtio-proxy and virtio BE will communicate with virtio-proxy through a virtqueue (yes, virtio-proxy is seen as yet another virtio device on BE) and will get IO event-related *RPC* callbacks, either MMIO read or write, from virtio-proxy.

See page 8 (protocol flow) and 10 (interfaces) in [1].

If kvm's ioregionfd can fit into this protocol, virtio-proxy for kvm will hopefully be implemented using ioregionfd.

-Takahiro Akashi

[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000548.html

Quoted 32 lines

Quoted 12 lines
Quoted 11 lines
There is also another problem. IOREQ is probably not be the only interface needed. Have a look at
https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also need an interface for the backend to inject interrupts into the frontend? And if the backend requires dynamic memory mappings of frontend pages, then we would also need an interface to map/unmap domU pages.
These interfaces are a lot more problematic than IOREQ: IOREQ is tiny and self-contained. It is easy to add anywhere. A new interface to inject interrupts or map pages is more difficult to manage because it would require changes scattered across the various emulators.
Quoted 5 lines
Something like ioreq is indeed necessary to implement arbitrary devices, but if you are willing to restrict yourself to VIRTIO then other interfaces are possible too because the VIRTIO device model is different from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
It's true our focus is just VirtIO which does support alternative transport options however most implementations seem to be targeting virtio-mmio for it's relative simplicity and understood semantics (modulo a desire for MSI to reduce round trip latency handling signalling).
Quoted 4 lines
Stefan
[[End of PGP Signed Part]]
Signature · 1 line
Alex Bennée

Alex Bennée

author

2021-09-03 09:38 UTC #37 lore raw

Quoted 26 lines · AKASHI Takahiro

Alex,
Quoted 23 lines · Alex Benn??e
Quoted 19 lines · Stefan Hajnoczi
[[PGP Signed Part:Undecided]]
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/

Quoted 7 lines

Quoted 5 lines
Reading the cover letter was very useful in showing how this provides a separate channel for signalling IO events to userspace instead of using the normal type-2 vmexit type event. I wonder how deeply tied the userspace facing side of this is to KVM? Could it provide a common FD type interface to IOREQ?
Why do you stick to a "FD" type interface?

I mean most user space interfaces on POSIX start with a file descriptor and the usual read/write semantics or a series of ioctls.

Quoted 7 lines

Quoted 2 lines
As I understand IOREQ this is currently a direct communication between userspace and the hypervisor using the existing Xen message bus. My
With IOREQ server, IO event occurrences are notified to BE via Xen's event channel, while the actual contexts of IO events (see struct ioreq in ioreq.h) are put in a queue on a single shared memory page which is to be assigned beforehand with xenforeignmemory_map_resource hypervisor call.

If we abstracted the IOREQ via the kernel interface you would probably just want to put the ioreq structure on a queue rather than expose the shared page to userspace.

Quoted 5 lines

Quoted 4 lines
worry would be that by adding knowledge of what the underlying hypervisor is we'd end up with excess complexity in the kernel. For one thing we certainly wouldn't want an API version dependency on the kernel to understand which version of the Xen hypervisor it was running on.

Quoted 8 lines

That's exactly what virtio-proxy in my proposal[1] does; All the hypervisor- specific details of IO event handlings are contained in virtio-proxy and virtio BE will communicate with virtio-proxy through a virtqueue (yes, virtio-proxy is seen as yet another virtio device on BE) and will get IO event-related *RPC* callbacks, either MMIO read or write, from virtio-proxy.
See page 8 (protocol flow) and 10 (interfaces) in [1].

There are two areas of concern with the proxy approach at the moment. The first is how the bootstrap of the virtio-proxy channel happens and the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.

Quoted 7 lines

If kvm's ioregionfd can fit into this protocol, virtio-proxy for kvm will hopefully be implemented using ioregionfd.
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000548.html

Signature · 5 lines

Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

AKASHI Takahiro

2021-09-06 02:24 UTC #38 lore raw

Alex,

Quoted 35 lines · Alex Benn??e

Quoted 32 lines · AKASHI Takahiro
Alex,
Quoted 28 lines · Alex Benn??e
Quoted 16 lines · Stefan Hajnoczi
[[PGP Signed Part:Undecided]]
Quoted 6 lines · Stefano Stabellini
Quoted 5 lines
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Quoted 7 lines
One option we should consider is for each backend to connect to Xen via the IOREQ interface. We could generalize the IOREQ interface and make it hypervisor agnostic. The interface is really trivial and easy to add. The only Xen-specific part is the notification mechanism, which is an event channel. If we replaced the event channel with something else the interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 3 lines
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/
Reading the cover letter was very useful in showing how this provides a separate channel for signalling IO events to userspace instead of using the normal type-2 vmexit type event. I wonder how deeply tied the userspace facing side of this is to KVM? Could it provide a common FD type interface to IOREQ?

Quoted 4 lines

Quoted 1 line
Why do you stick to a "FD" type interface?
I mean most user space interfaces on POSIX start with a file descriptor and the usual read/write semantics or a series of ioctls.

Who do you assume is responsible for implementing this kind of fd semantics, OSs on BE or hypervisor itself?

I think such interfaces can only be easily implemented on type-2 hypervisors.

# In this sense, I don't think rust-vmm, as it is, cannot be

# a general solution.

Quoted 11 lines

Quoted 7 lines
Quoted 2 lines
As I understand IOREQ this is currently a direct communication between userspace and the hypervisor using the existing Xen message bus. My
With IOREQ server, IO event occurrences are notified to BE via Xen's event channel, while the actual contexts of IO events (see struct ioreq in ioreq.h) are put in a queue on a single shared memory page which is to be assigned beforehand with xenforeignmemory_map_resource hypervisor call.
If we abstracted the IOREQ via the kernel interface you would probably just want to put the ioreq structure on a queue rather than expose the shared page to userspace.

Where is that queue?

Quoted 12 lines

Quoted 12 lines
Quoted 4 lines
worry would be that by adding knowledge of what the underlying hypervisor is we'd end up with excess complexity in the kernel. For one thing we certainly wouldn't want an API version dependency on the kernel to understand which version of the Xen hypervisor it was running on.
That's exactly what virtio-proxy in my proposal[1] does; All the hypervisor- specific details of IO event handlings are contained in virtio-proxy and virtio BE will communicate with virtio-proxy through a virtqueue (yes, virtio-proxy is seen as yet another virtio device on BE) and will get IO event-related *RPC* callbacks, either MMIO read or write, from virtio-proxy.

Quoted 4 lines

Quoted 1 line
See page 8 (protocol flow) and 10 (interfaces) in [1].
There are two areas of concern with the proxy approach at the moment. The first is how the bootstrap of the virtio-proxy channel happens and

As I said, from BE point of view, virtio-proxy would be seen as yet another virtio device by which BE could talk to "virtio proxy" vm or whatever else.

This way we guarantee BE's hypervisor-agnosticism instead of having "common" hypervisor interfaces. That is the base of my idea.

Quoted 4 lines

the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.

In configuration phase of virtio device, the latency won't be a big matter. In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue is how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.

Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary" BE.
(I know this is is an arguable requirement, though.)

# Would we better discuss what "hypervisor-agnosticism" means?

-Takahiro Akashi

Quoted 10 lines

Quoted 7 lines
If kvm's ioregionfd can fit into this protocol, virtio-proxy for kvm will hopefully be implemented using ioregionfd.
-Takahiro Akashi
[1] https://op-lists.linaro.org/pipermail/stratos-dev/2021-August/000548.html
Signature · 1 line
Alex Bennée

Christopher Clark

2021-09-07 02:42 UTC #39 lore raw

Quoted 127 lines · AKASHI Takahiro via Stratos-dev

Alex,
Quoted 5 lines · Alex Benn??e
Quoted 2 lines · AKASHI Takahiro
Alex,
Quoted 7 lines
Quoted 7 lines · Alex Benn??e
Quoted 6 lines
Quoted 3 lines · Stefan Hajnoczi
[[PGP Signed Part:Undecided]]
Quoted 1 line · Stefano Stabellini
Quoted 1 line
Could we consider the kernel internally converting IOREQ
messages from
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the Xen hypervisor to eventfd events? Would this scale with
other kernel
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to
Xen via
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the IOREQ interface. We could generalize the IOREQ interface and
make it
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor agnostic. The interface is really trivial and easy to
add.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The only Xen-specific part is the notification mechanism, which is
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
event channel. If we replaced the event channel with something
else the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Reading the cover letter was very useful in showing how this provides
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
separate channel for signalling IO events to userspace instead of
using
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
the normal type-2 vmexit type event. I wonder how deeply tied the userspace facing side of this is to KVM? Could it provide a common FD type interface to IOREQ?
Why do you stick to a "FD" type interface?
I mean most user space interfaces on POSIX start with a file descriptor and the usual read/write semantics or a series of ioctls.
Who do you assume is responsible for implementing this kind of fd semantics, OSs on BE or hypervisor itself?
I think such interfaces can only be easily implemented on type-2 hypervisors.
# In this sense, I don't think rust-vmm, as it is, cannot be
# a general solution.
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
As I understand IOREQ this is currently a direct communication between userspace and the hypervisor using the existing Xen message bus. My
With IOREQ server, IO event occurrences are notified to BE via Xen's
event
Quoted 1 line
Quoted 1 line
channel, while the actual contexts of IO events (see struct ioreq in
ioreq.h)
Quoted 1 line
Quoted 1 line
are put in a queue on a single shared memory page which is to be
assigned
Quoted 5 lines
Quoted 1 line
beforehand with xenforeignmemory_map_resource hypervisor call.
If we abstracted the IOREQ via the kernel interface you would probably just want to put the ioreq structure on a queue rather than expose the shared page to userspace.
Where is that queue?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
worry would be that by adding knowledge of what the underlying hypervisor is we'd end up with excess complexity in the kernel. For
one
Quoted 1 line
Quoted 1 line
Quoted 1 line
thing we certainly wouldn't want an API version dependency on the
kernel
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
to understand which version of the Xen hypervisor it was running on.
That's exactly what virtio-proxy in my proposal[1] does; All the
hypervisor-
Quoted 10 lines
Quoted 7 lines
specific details of IO event handlings are contained in virtio-proxy and virtio BE will communicate with virtio-proxy through a virtqueue (yes, virtio-proxy is seen as yet another virtio device on BE) and will get IO event-related *RPC* callbacks, either MMIO read or write, from virtio-proxy.
See page 8 (protocol flow) and 10 (interfaces) in [1].
There are two areas of concern with the proxy approach at the moment. The first is how the bootstrap of the virtio-proxy channel happens and
As I said, from BE point of view, virtio-proxy would be seen as yet another virtio device by which BE could talk to "virtio proxy" vm or whatever else.
This way we guarantee BE's hypervisor-agnosticism instead of having "common" hypervisor interfaces. That is the base of my idea.
Quoted 4 lines
the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.
In configuration phase of virtio device, the latency won't be a big matter. In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue is
how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.
Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary"
BE.
(I know this is is an arguable requirement, though.)

Quoted 3 lines

# Would we better discuss what "hypervisor-agnosticism" means?
Is there a call that you could recommend that we join to discuss this and

the topics of this thread?
There is definitely interest in pursuing a new interface for Argo that can be implemented in other hypervisors and enable guest binary portability between them, at least on the same hardware architecture, with VirtIO transport as a primary use case.

The notes from the Xen Summit Design Session on VirtIO Cross-Project BoF for Xen and Guest OS, which include context about the several separate approaches to VirtIO on Xen, have now been posted here:
https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00472.html

Christopher

Quoted 4 lines

-Takahiro Akashi

AKASHI Takahiro

2021-09-10 02:51 UTC #40 lore raw

Quoted 130 lines · Christopher Clark

Quoted 127 lines · AKASHI Takahiro via Stratos-dev
Alex,
Quoted 5 lines · Alex Benn??e
Quoted 2 lines · AKASHI Takahiro
Alex,
Quoted 7 lines
Quoted 7 lines · Alex Benn??e
Quoted 6 lines
Quoted 3 lines · Stefan Hajnoczi
[[PGP Signed Part:Undecided]]
Quoted 1 line · Stefano Stabellini
Quoted 1 line
Could we consider the kernel internally converting IOREQ
messages from
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the Xen hypervisor to eventfd events? Would this scale with
other kernel
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
One option we should consider is for each backend to connect to
Xen via
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the IOREQ interface. We could generalize the IOREQ interface and
make it
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
hypervisor agnostic. The interface is really trivial and easy to
add.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The only Xen-specific part is the notification mechanism, which is
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
event channel. If we replaced the event channel with something
else the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
interface would be generic. See:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
There have been experiments with something kind of similar in KVM recently (see struct ioregionfd_cmd):
https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@gmail.com/
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Reading the cover letter was very useful in showing how this provides
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
separate channel for signalling IO events to userspace instead of
using
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
the normal type-2 vmexit type event. I wonder how deeply tied the userspace facing side of this is to KVM? Could it provide a common FD type interface to IOREQ?
Why do you stick to a "FD" type interface?
I mean most user space interfaces on POSIX start with a file descriptor and the usual read/write semantics or a series of ioctls.
Who do you assume is responsible for implementing this kind of fd semantics, OSs on BE or hypervisor itself?
I think such interfaces can only be easily implemented on type-2 hypervisors.
# In this sense, I don't think rust-vmm, as it is, cannot be
# a general solution.
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
As I understand IOREQ this is currently a direct communication between userspace and the hypervisor using the existing Xen message bus. My
With IOREQ server, IO event occurrences are notified to BE via Xen's
event
Quoted 1 line
Quoted 1 line
channel, while the actual contexts of IO events (see struct ioreq in
ioreq.h)
Quoted 1 line
Quoted 1 line
are put in a queue on a single shared memory page which is to be
assigned
Quoted 5 lines
Quoted 1 line
beforehand with xenforeignmemory_map_resource hypervisor call.
If we abstracted the IOREQ via the kernel interface you would probably just want to put the ioreq structure on a queue rather than expose the shared page to userspace.
Where is that queue?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
worry would be that by adding knowledge of what the underlying hypervisor is we'd end up with excess complexity in the kernel. For
one
Quoted 1 line
Quoted 1 line
Quoted 1 line
thing we certainly wouldn't want an API version dependency on the
kernel
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
to understand which version of the Xen hypervisor it was running on.
That's exactly what virtio-proxy in my proposal[1] does; All the
hypervisor-
Quoted 10 lines
Quoted 7 lines
specific details of IO event handlings are contained in virtio-proxy and virtio BE will communicate with virtio-proxy through a virtqueue (yes, virtio-proxy is seen as yet another virtio device on BE) and will get IO event-related *RPC* callbacks, either MMIO read or write, from virtio-proxy.
See page 8 (protocol flow) and 10 (interfaces) in [1].
There are two areas of concern with the proxy approach at the moment. The first is how the bootstrap of the virtio-proxy channel happens and
As I said, from BE point of view, virtio-proxy would be seen as yet another virtio device by which BE could talk to "virtio proxy" vm or whatever else.
This way we guarantee BE's hypervisor-agnosticism instead of having "common" hypervisor interfaces. That is the base of my idea.
Quoted 4 lines
the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.
In configuration phase of virtio device, the latency won't be a big matter. In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue is
how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.
Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary"
BE.
(I know this is is an arguable requirement, though.)

Quoted 4 lines

Quoted 2 lines
# Would we better discuss what "hypervisor-agnosticism" means?
Is there a call that you could recommend that we join to discuss this and the topics of this thread?

Stratos call?
Alex should have more to say.

-Takahiro Akashi

Quoted 18 lines

There is definitely interest in pursuing a new interface for Argo that can be implemented in other hypervisors and enable guest binary portability between them, at least on the same hardware architecture, with VirtIO transport as a primary use case.
The notes from the Xen Summit Design Session on VirtIO Cross-Project BoF for Xen and Guest OS, which include context about the several separate approaches to VirtIO on Xen, have now been posted here:
https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00472.html
Christopher
Quoted 4 lines
-Takahiro Akashi

Alex Bennée

author

2021-09-10 09:45 UTC #41 lore raw

In reply to Christopher Clark

Quoted 5 lines · Christopher Clark

On Sun, Sep 5, 2021 at 7:24 PM AKASHI Takahiro via Stratos-dev <[email protected]> wrote:
Alex,
On Fri, Sep 03, 2021 at 10:28:06AM +0100, Alex Benn??e wrote:

<snip>

Quoted 7 lines

In configuration phase of virtio device, the latency won't be a big matter. In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue is how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.

Quoted 6 lines

Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary" BE.
(I know this is is an arguable requirement, though.)

The proposal for a single binary would always require something to shim between hypervisors. This is still an area of discussion though. Having a compile time selectable approach is practically unavoidable for "bare metal" backends though because there are no other processes/layers that communication with the hypervisor can be delegated to.

Quoted 7 lines

# Would we better discuss what "hypervisor-agnosticism" means?
Is there a call that you could recommend that we join to discuss this and the topics of this thread?
There is definitely interest in pursuing a new interface for Argo that can be implemented in other hypervisors and enable guest binary portability between them, at least on the same hardware architecture,
with VirtIO transport as a primary use case.

There is indeed ;-)

We have a regular open call every two week for the Stratos project which you are welcome to attend. You can find the details on the project overview page:

https://linaro.atlassian.net/wiki/spaces/STR/overview

we regularly have teams from outside the project present their work as well.

Quoted 3 lines

The notes from the Xen Summit Design Session on VirtIO Cross-Project BoF for Xen and Guest OS, which include context about the several separate approaches to VirtIO on Xen, have now been posted here:
https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00472.html

Thanks for the link - looks like a very detailed summary.

Quoted 5 lines

Christopher
-Takahiro Akashi

Signature · 5 lines

Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Stefano Stabellini

2021-09-13 23:51 UTC #42 lore raw

In reply to AKASHI Takahiro

Quoted 11 lines · AKASHI Takahiro

Quoted 4 lines
the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.
In configuration phase of virtio device, the latency won't be a big matter. In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue is how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.

Quoted 6 lines

Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary" BE.
(I know this is is an arguable requirement, though.)

In my experience, latency, performance, and security are far more important than providing a single binary.

In my opinion, we should optimize for the best performance and security, then be practical on the topic of hypervisor agnosticism. For instance, a shared source with a small hypervisor-specific component, with one implementation of the small component for each hypervisor, would provide a good enough hypervisor abstraction. It is good to be hypervisor agnostic, but I wouldn't go extra lengths to have a single binary. I cannot picture a case where a BE binary needs to be moved between different hypervisors and a recompilation is impossible (BE, not FE). Instead, I can definitely imagine detailed requirements on IRQ latency having to be lower than 10us or bandwidth higher than 500 MB/sec.

Instead of virtio-proxy, my suggestion is to work together on a common project and common source with others interested in the same problem.

I would pick something like kvmtool as a basis. It doesn't have to be kvmtools, and kvmtools specifically is GPL-licensed, which is unfortunate because it would help if the license was BSD-style for ease of integration with Zephyr and other RTOSes.

As long as the project is open to working together on multiple hypervisors and deployment models then it is fine. For instance, the shared source could be based on OpenAMP kvmtool [1] (the original kvmtool likely prefers to stay small and narrow-focused on KVM). OpenAMP kvmtool was created to add support for hypervisor-less virtio but they are very open to hypervisors too. It could be a good place to add a Xen implementation, a KVM fatqueue implementation, a Jailhouse implementation, etc. -- work together toward the common goal of a single BE source (not binary) supporting multiple different deployment models.

[1] https://github.com/OpenAMP/kvmtool

François Ozog

2021-09-14 06:09 UTC #43 lore raw

Quoted 44 lines · Stefano Stabellini via Stratos-dev

Quoted 6 lines · AKASHI Takahiro
Quoted 4 lines
the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.
In configuration phase of virtio device, the latency won't be a big
matter.
Quoted 2 lines
In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only
issue is
Quoted 7 lines
how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.
Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single
binary"
Quoted 2 lines
BE.
(I know this is is an arguable requirement, though.)
In my experience, latency, performance, and security are far more important than providing a single binary.
In my opinion, we should optimize for the best performance and security, then be practical on the topic of hypervisor agnosticism. For instance, a shared source with a small hypervisor-specific component, with one implementation of the small component for each hypervisor, would provide a good enough hypervisor abstraction. It is good to be hypervisor agnostic, but I wouldn't go extra lengths to have a single binary. I cannot picture a case where a BE binary needs to be moved between different hypervisors and a recompilation is impossible (BE, not FE). Instead, I can definitely imagine detailed requirements on IRQ latency having to be lower than 10us or bandwidth higher than 500 MB/sec.
Instead of virtio-proxy, my suggestion is to work together on a common project and common source with others interested in the same problem.
I would pick something like kvmtool as a basis. It doesn't have to be kvmtools, and kvmtools specifically is GPL-licensed, which is unfortunate because it would help if the license was BSD-style for ease of integration with Zephyr and other RTOSes.

Quoted 10 lines

As long as the project is open to working together on multiple hypervisors and deployment models then it is fine. For instance, the shared source could be based on OpenAMP kvmtool [1] (the original kvmtool likely prefers to stay small and narrow-focused on KVM). OpenAMP kvmtool was created to add support for hypervisor-less virtio but they are very open to hypervisors too. It could be a good place to add a Xen implementation, a KVM fatqueue implementation, a Jailhouse implementation, etc. -- work together toward the common goal of a single BE source (not binary) supporting multiple different deployment models.

i like the hypervisor-less approach described in the link below. it can also be used to drfine abstract HAL between normal world and TrustZone to implement confidential workloads in the TZ. Virtio-sock is of particular interest.
In addition, this can define a HAL that can be re-used in many contexts :
could we use this to implement something similar to Android Generic Kernel Image stuff ?

Quoted 8 lines

[1] https://github.com/OpenAMP/kvmtool
--
Stratos-dev mailing list
[email protected]
https://op-lists.linaro.org/mailman/listinfo/stratos-dev

Signature · 3 lines

François-Frédéric Ozog | *Director Business Development*
T: +33.67221.6485
[email protected] | Skype: ffozog

Alex Bennée

author

2021-09-14 14:34 UTC #44 lore raw

In reply to Stefano Stabellini

Quoted 22 lines · Stefano Stabellini

Quoted 11 lines · AKASHI Takahiro
Quoted 4 lines
the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.
In configuration phase of virtio device, the latency won't be a big matter. In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue is how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.
Quoted 6 lines
Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary" BE.
(I know this is is an arguable requirement, though.)
In my experience, latency, performance, and security are far more important than providing a single binary.

Quoted 6 lines

In my opinion, we should optimize for the best performance and security, then be practical on the topic of hypervisor agnosticism. For instance, a shared source with a small hypervisor-specific component, with one implementation of the small component for each hypervisor, would provide a good enough hypervisor abstraction. It is good to be hypervisor agnostic, but I wouldn't go extra lengths to have a single binary.

I agree it shouldn't be a primary goal although a single binary working with helpers to bridge the gap would make a cool demo. The real aim of agnosticism is avoid having multiple implementations of the backend itself for no other reason than a change in hypervisor.

Quoted 5 lines

I cannot picture a case where a BE binary needs to be moved between different hypervisors and a recompilation is impossible (BE, not FE). Instead, I can definitely imagine detailed requirements on IRQ latency having to be lower than 10us or bandwidth higher than 500 MB/sec.

Quoted 7 lines

Instead of virtio-proxy, my suggestion is to work together on a common project and common source with others interested in the same problem.
I would pick something like kvmtool as a basis. It doesn't have to be kvmtools, and kvmtools specifically is GPL-licensed, which is unfortunate because it would help if the license was BSD-style for ease of integration with Zephyr and other RTOSes.

This does imply making some choices, especially the implementation language. However I feel that C is really the lowest common denominator here and I get the sense that people would rather avoid it if they could given the potential security implications of a bug prone back end. This is what is prompting interest in Rust.

Quoted 12 lines

As long as the project is open to working together on multiple hypervisors and deployment models then it is fine. For instance, the shared source could be based on OpenAMP kvmtool [1] (the original kvmtool likely prefers to stay small and narrow-focused on KVM). OpenAMP kvmtool was created to add support for hypervisor-less virtio but they are very open to hypervisors too. It could be a good place to add a Xen implementation, a KVM fatqueue implementation, a Jailhouse implementation, etc. -- work together toward the common goal of a single BE source (not binary) supporting multiple different deployment models.
[1] https://github.com/OpenAMP/kvmtool

Signature · 5 lines

Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Trilok Soni

2021-09-15 08:10 UTC #45 lore raw

In reply to Stefano Stabellini

Hello,

Quoted 41 lines · Stefano Stabellini via Stratos-dev

Quoted 11 lines · AKASHI Takahiro
Quoted 4 lines
the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.
In configuration phase of virtio device, the latency won't be a big matter. In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue is how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.
Quoted 6 lines
Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary" BE.
(I know this is is an arguable requirement, though.)
In my experience, latency, performance, and security are far more important than providing a single binary.
In my opinion, we should optimize for the best performance and security, then be practical on the topic of hypervisor agnosticism. For instance, a shared source with a small hypervisor-specific component, with one implementation of the small component for each hypervisor, would provide a good enough hypervisor abstraction. It is good to be hypervisor agnostic, but I wouldn't go extra lengths to have a single binary. I cannot picture a case where a BE binary needs to be moved between different hypervisors and a recompilation is impossible (BE, not FE). Instead, I can definitely imagine detailed requirements on IRQ latency having to be lower than 10us or bandwidth higher than 500 MB/sec.
Instead of virtio-proxy, my suggestion is to work together on a common project and common source with others interested in the same problem.
I would pick something like kvmtool as a basis. It doesn't have to be kvmtools, and kvmtools specifically is GPL-licensed, which is unfortunate because it would help if the license was BSD-style for ease of integration with Zephyr and other RTOSes.

Quoted 9 lines

As long as the project is open to working together on multiple hypervisors and deployment models then it is fine. For instance, the shared source could be based on OpenAMP kvmtool [1] (the original kvmtool likely prefers to stay small and narrow-focused on KVM). OpenAMP kvmtool was created to add support for hypervisor-less virtio but they are very open to hypervisors too. It could be a good place to add a Xen implementation, a KVM fatqueue implementation, a Jailhouse implementation, etc. -- work together toward the common goal of a single BE source (not binary) supporting multiple different deployment models.

I have my reservations on using "kvmtool" to do any development here. "kvmtool" can't be used on the products and it is just a tool for the developers.

The benefit of the solving problem w/ rust-vmm is that some of the crates from this project can be utilized for the real product. Alex has mentioned that "rust-vmm" today has some KVM specific bits but the rust-vmm community is already discussing to remove or re-org them in such a way that other Hypervisors can fit in.

Microsoft has Hyper-V implementation w/ cloud-hypervisor which uses some of the rust-vmm components as well and they had shown interest to add the Hyper-V support in the "rust-vmm" project as well. I don't know the current progress but they had proven it it "cloud-hypervisor" project.

"rust-vmm" project's license will work as well for most of the project developments and I see that "CrosVM" is shipping in the products as well.

---Trilok Soni

Stefano Stabellini

2021-09-15 03:29 UTC #46 lore raw

Quoted 65 lines · Trilok Soni

Quoted 44 lines · Stefano Stabellini via Stratos-dev
Quoted 13 lines · AKASHI Takahiro
Quoted 4 lines
the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.
In configuration phase of virtio device, the latency won't be a big matter.
In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue is
how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.
Quoted 7 lines
Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary"
BE.
(I know this is is an arguable requirement, though.)
In my experience, latency, performance, and security are far more important than providing a single binary.
In my opinion, we should optimize for the best performance and security, then be practical on the topic of hypervisor agnosticism. For instance, a shared source with a small hypervisor-specific component, with one implementation of the small component for each hypervisor, would provide a good enough hypervisor abstraction. It is good to be hypervisor agnostic, but I wouldn't go extra lengths to have a single binary. I cannot picture a case where a BE binary needs to be moved between different hypervisors and a recompilation is impossible (BE, not FE). Instead, I can definitely imagine detailed requirements on IRQ latency having to be lower than 10us or bandwidth higher than 500 MB/sec.
Instead of virtio-proxy, my suggestion is to work together on a common project and common source with others interested in the same problem.
I would pick something like kvmtool as a basis. It doesn't have to be kvmtools, and kvmtools specifically is GPL-licensed, which is unfortunate because it would help if the license was BSD-style for ease of integration with Zephyr and other RTOSes.
Quoted 9 lines
As long as the project is open to working together on multiple hypervisors and deployment models then it is fine. For instance, the shared source could be based on OpenAMP kvmtool [1] (the original kvmtool likely prefers to stay small and narrow-focused on KVM). OpenAMP kvmtool was created to add support for hypervisor-less virtio but they are very open to hypervisors too. It could be a good place to add a Xen implementation, a KVM fatqueue implementation, a Jailhouse implementation, etc. -- work together toward the common goal of a single BE source (not binary) supporting multiple different deployment models.
I have my reservations on using "kvmtool" to do any development here. "kvmtool" can't be used on the products and it is just a tool for the developers.
The benefit of the solving problem w/ rust-vmm is that some of the crates from this project can be utilized for the real product. Alex has mentioned that "rust-vmm" today has some KVM specific bits but the rust-vmm community is already discussing to remove or re-org them in such a way that other Hypervisors can fit in.

Quoted 7 lines

Microsoft has Hyper-V implementation w/ cloud-hypervisor which uses some of the rust-vmm components as well and they had shown interest to add the Hyper-V support in the "rust-vmm" project as well. I don't know the current progress but they had proven it it "cloud-hypervisor" project.
"rust-vmm" project's license will work as well for most of the project developments and I see that "CrosVM" is shipping in the products as well.

Most things in open source start as a developers tool before they become part of a product :)

I am concerned about how "embeddable" rust-vmm is going to be. Do you think it would be possible to run it against an RTOS together with other apps written in C?

Let me make a realistic example. You can imagine a Zephyr instance with simple toolstack functionalities written in C (starting/stopping VMs). One might want to add a virtio backend to it. I am not familiar enough with Rust and rust-vmm to know if it would be feasible and "easy" to run a rust-vmm backend as a Zephyr app.

A C project of the size of kvmtool, but BSD-licensed, could run on Zephyr with only a little porting effort using the POSIX compatibility layer. I think that would be ideal. Anybody aware of a project fulfilling these requirements?

If we have to give up the ability to integrate with an RTOS, then I think QEMU could be the leading choice because is still the main reference implementation for virtio.

Trilok Soni

2021-09-15 23:51 UTC #47 lore raw

Hi Stefano,

Quoted 71 lines · Stefano Stabellini

Quoted 70 lines · Trilok Soni
Quoted 44 lines · Stefano Stabellini via Stratos-dev
Quoted 13 lines · AKASHI Takahiro
Quoted 4 lines
the second is how many context switches are involved in a transaction. Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which I think would imply hypervisor knowledge in the backend binary.
In configuration phase of virtio device, the latency won't be a big matter.
In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue is
how efficiently we can deliver notification to the opposite side. Right? And this is a very common problem whatever approach we would take.
Quoted 7 lines
Anyhow, if we do care the latency in my approach, most of virtio-proxy- related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary"
BE.
(I know this is is an arguable requirement, though.)
In my experience, latency, performance, and security are far more important than providing a single binary.
In my opinion, we should optimize for the best performance and security, then be practical on the topic of hypervisor agnosticism. For instance, a shared source with a small hypervisor-specific component, with one implementation of the small component for each hypervisor, would provide a good enough hypervisor abstraction. It is good to be hypervisor agnostic, but I wouldn't go extra lengths to have a single binary. I cannot picture a case where a BE binary needs to be moved between different hypervisors and a recompilation is impossible (BE, not FE). Instead, I can definitely imagine detailed requirements on IRQ latency having to be lower than 10us or bandwidth higher than 500 MB/sec.
Instead of virtio-proxy, my suggestion is to work together on a common project and common source with others interested in the same problem.
I would pick something like kvmtool as a basis. It doesn't have to be kvmtools, and kvmtools specifically is GPL-licensed, which is unfortunate because it would help if the license was BSD-style for ease of integration with Zephyr and other RTOSes.
Quoted 9 lines
As long as the project is open to working together on multiple hypervisors and deployment models then it is fine. For instance, the shared source could be based on OpenAMP kvmtool [1] (the original kvmtool likely prefers to stay small and narrow-focused on KVM). OpenAMP kvmtool was created to add support for hypervisor-less virtio but they are very open to hypervisors too. It could be a good place to add a Xen implementation, a KVM fatqueue implementation, a Jailhouse implementation, etc. -- work together toward the common goal of a single BE source (not binary) supporting multiple different deployment models.
I have my reservations on using "kvmtool" to do any development here. "kvmtool" can't be used on the products and it is just a tool for the developers.
The benefit of the solving problem w/ rust-vmm is that some of the crates from this project can be utilized for the real product. Alex has mentioned that "rust-vmm" today has some KVM specific bits but the rust-vmm community is already discussing to remove or re-org them in such a way that other Hypervisors can fit in.
Microsoft has Hyper-V implementation w/ cloud-hypervisor which uses some of the rust-vmm components as well and they had shown interest to add the Hyper-V support in the "rust-vmm" project as well. I don't know the current progress but they had proven it it "cloud-hypervisor" project.

Quoted 5 lines

Quoted 2 lines
"rust-vmm" project's license will work as well for most of the project developments and I see that "CrosVM" is shipping in the products as well.
Most things in open source start as a developers tool before they become part of a product :)

Agree, but I had an offline discussions with one the active developer of kvmtool and the confidence of using it in the product was no where near we expected during our evaluation. Same goes the QEMU and one of the biggest problem was no. of security issues against this huge codebase of QEMU.

Quoted 4 lines

I am concerned about how "embeddable" rust-vmm is going to be. Do you think it would be possible to run it against an RTOS together with other apps written in C?

I don't see any limitations of rust-vmm. For example, I am confident that we can port rust-vmm based backend into the QNX as host OS and same goes w/ Zephyr as well. Some work is needed but nothing fundamentally blocking it. We should be able to run it w/ Fuchsia as well with some effort.

---Trilok Soni

Stefano Stabellini

2021-09-16 02:12 UTC #48 lore raw

Quoted 84 lines · Trilok Soni

Quoted 83 lines · Stefano Stabellini
Quoted 81 lines · Trilok Soni
Quoted 49 lines · Stefano Stabellini via Stratos-dev
Quoted 17 lines · AKASHI Takahiro
Quoted 6 lines
the second is how many context switches are involved in a transaction.
Of course with all things there is a trade off. Things involving the very tightest latency would probably opt for a bare metal backend which
I think would imply hypervisor knowledge in the backend binary.
In configuration phase of virtio device, the latency won't be a big matter.
In device operations (i.e. read/write to block devices), if we can resolve 'mmap' issue, as Oleksandr is proposing right now, the only issue
is
how efficiently we can deliver notification to the opposite side. Right?
And this is a very common problem whatever approach we would take.
Quoted 8 lines
Anyhow, if we do care the latency in my approach, most of virtio-proxy-
related code can be re-implemented just as a stub (or shim?) library since the protocols are defined as RPCs.
In this case, however, we would lose the benefit of providing "single binary"
BE.
(I know this is is an arguable requirement, though.)
In my experience, latency, performance, and security are far more important than providing a single binary.
In my opinion, we should optimize for the best performance and security, then be practical on the topic of hypervisor agnosticism. For instance, a shared source with a small hypervisor-specific component, with one implementation of the small component for each hypervisor, would provide a good enough hypervisor abstraction. It is good to be hypervisor agnostic, but I wouldn't go extra lengths to have a single binary. I cannot picture a case where a BE binary needs to be moved between different hypervisors and a recompilation is impossible (BE, not FE). Instead, I can definitely imagine detailed requirements on IRQ latency having to be lower than 10us or bandwidth higher than 500 MB/sec.
Instead of virtio-proxy, my suggestion is to work together on a common project and common source with others interested in the same problem.
I would pick something like kvmtool as a basis. It doesn't have to be kvmtools, and kvmtools specifically is GPL-licensed, which is unfortunate because it would help if the license was BSD-style for ease of integration with Zephyr and other RTOSes.
Quoted 9 lines
As long as the project is open to working together on multiple hypervisors and deployment models then it is fine. For instance, the shared source could be based on OpenAMP kvmtool [1] (the original kvmtool likely prefers to stay small and narrow-focused on KVM). OpenAMP kvmtool was created to add support for hypervisor-less virtio but they are very open to hypervisors too. It could be a good place to add a Xen implementation, a KVM fatqueue implementation, a Jailhouse implementation, etc. -- work together toward the common goal of a single BE source (not binary) supporting multiple different deployment models.
I have my reservations on using "kvmtool" to do any development here. "kvmtool" can't be used on the products and it is just a tool for the developers.
The benefit of the solving problem w/ rust-vmm is that some of the crates from
this project can be utilized for the real product. Alex has mentioned that "rust-vmm" today has some KVM specific bits but the rust-vmm community is already discussing to remove or re-org them in such a way that other Hypervisors can fit in.
Microsoft has Hyper-V implementation w/ cloud-hypervisor which uses some of
the rust-vmm components as well and they had shown interest to add the Hyper-V
support in the "rust-vmm" project as well. I don't know the current progress
but they had proven it it "cloud-hypervisor" project.
"rust-vmm" project's license will work as well for most of the project developments and I see that "CrosVM" is shipping in the products as well.

Quoted 7 lines

Quoted 2 lines
Most things in open source start as a developers tool before they become part of a product :)
Agree, but I had an offline discussions with one the active developer of kvmtool and the confidence of using it in the product was no where near we expected during our evaluation. Same goes the QEMU and one of the biggest problem was no. of security issues against this huge codebase of QEMU.

That is fair, but it is important to recognize that these are *known* security issues.

Does rust-vmm have a security process and a security response team? I tried googling for it but couldn't find relevant info.

QEMU is a very widely used and very well inspected codebase. It has a mailing list to report security issues and a security process. As a consequence we know of many vulnerabilities affecting the code base. As far as I am aware rust-vmm has not been inspected yet with the same level of attention and the same amount of security researchers.

That said, of course it is undeniable that the larger size of QEMU implies a higher amount of security issues. But for this project, we wouldn't be using the whole of QEMU of course. We would be narrowing it down to a build with only few revelant pieces. I imagine that the total LOC count would still be higher but the number of relevant security vulnerabilities would only be a small fraction of the QEMU total.

Quoted 8 lines

Quoted 3 lines
I am concerned about how "embeddable" rust-vmm is going to be. Do you think it would be possible to run it against an RTOS together with other apps written in C?
I don't see any limitations of rust-vmm. For example, I am confident that we can port rust-vmm based backend into the QNX as host OS and same goes w/ Zephyr as well. Some work is needed but nothing fundamentally blocking it. We should be able to run it w/ Fuchsia as well with some effort.

That's good to hear.

Stefan Hajnoczi

2021-08-05 15:48 UTC #49 lore raw

In reply to Alex Bennée

Quoted 76 lines · Alex Bennée

Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============
In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest. Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space. To activate the mapping will require some sort of hypercall to the hypervisor. I can see two options at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.

Quoted 6 lines

- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.

VIRTIO typically uses the vring memory layout but doesn't need to. The VIRTIO device model deals with virtqueues. The shared memory vring layout is part of the VIRTIO transport (PCI, MMIO, and CCW use vrings). Alternative transports with other virtqueue representations are possible (e.g. VIRTIO-over-TCP). They don't need to involve a BE mapping shared memory and processing a vring owned by the FE.

For example, there could be BE hypercalls to pop a virtqueue elements, push a virtqueue elements, and to access buffers (basically DMA read/write). The FE could either be a traditional virtio-mmio/pci device with a vring or use FE hypercalls to add available elements to a virtqueue and get used elements.

I don't know the goals of project Stratos or whether this helps, but it might allow other architectures that have different security, complexity, etc properties.

Stefan

Matias Ezequiel Vara Larsen

2021-08-19 09:13 UTC #50 lore raw

In reply to Alex Bennée

Hello Alex,

I can tell you my experience from working on a PoC (library)
to allow the implementation of virtio-devices that are hypervisor/OS agnostic. I focused on two use cases:
1. type-I hypervisor in which the backend is running as a VM. This is an in-house hypervisor that does not support VMExits.
2. Linux user-space. In this case, the library is just used to communicate threads. The goal of this use case is merely testing.

I have chosen virtio-mmio as the way to exchange information
between the frontend and backend. I found it hard to synchronize the access to the virtio-mmio layout without VMExits. I had to add some extra bits to allow the front-end and back-end to synchronize, which is required
during the device-status initialization. These extra bits would not be needed in case the hypervisor supports VMExits, e.g., KVM.

Each guest has a memory region that is shared with the backend.
This memory region is used by the frontend to allocate the io-buffers. This region also maps the virtio-mmio layout that is initialized by the backend. For the moment, this region is defined when the guest is created. One limitation is that the memory for io-buffers is fixed. At some point, the guest shall be able to balloon this region. Notifications between the frontend and the backend are implemented by using an hypercall. The hypercall mechanism and the memory allocation are abstracted away by a platform layer that exposes an interface that is hypervisor/os agnostic.

I split the backend into a virtio-device driver and a
backend driver. The virtio-device driver is the virtqueues and the backend driver gets packets from the virtqueue for post-processing. For example, in the case of virtio-net, the backend driver would decide if the packet goes to the hardware or to another virtio-net device. The virtio-device drivers may be
implemented in different ways like by using a single thread, multiple threads, or one thread for all the virtio-devices.

In this PoC, I just tackled two very simple use-cases. These use-cases allowed me to extract some requirements for an hypervisor to support virtio.

Matias

Quoted 104 lines · Alex Bennée

Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============
In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest. Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space. To activate the mapping will require some sort of hypercall to the hypervisor. I can see two options at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.

Quoted 13 lines

Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Signature · 5 lines
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

A message that is not in the archive (sent off-list or not yet received): 20210820060558.GB13452@laputa

Matias Ezequiel Vara Larsen

2021-08-21 14:10 UTC #52 lore raw

Hello,

Quoted 10 lines · AKASHI Takahiro

Hi Matias,
Quoted 4 lines · Matias Ezequiel Vara Larsen
Hello Alex,
I can tell you my experience from working on a PoC (library)
to allow the implementation of virtio-devices that are hypervisor/OS agnostic.
What hypervisor are you using for your PoC here?

I am using an in-house hypervisor, which is similar to Jailhouse.

Quoted 6 lines

Quoted 6 lines
I focused on two use cases:
1. type-I hypervisor in which the backend is running as a VM. This is an in-house hypervisor that does not support VMExits.
2. Linux user-space. In this case, the library is just used to communicate threads. The goal of this use case is merely testing.

Quoted 7 lines

Quoted 3 lines
I have chosen virtio-mmio as the way to exchange information
between the frontend and backend. I found it hard to synchronize the access to the virtio-mmio layout without VMExits. I had to add some extra bits to allow
Can you explain how MMIOs to registers in virito-mmio layout (which I think means a configuration space?) will be propagated to BE?

In this PoC, the BE guest is created with a fixed number of regions
of memory that represents each device. The BE initializes these regions, and then, waits for the FEs to begin the initialization.

Quoted 9 lines

Quoted 8 lines
the front-end and back-end to synchronize, which is required during the device-status initialization. These extra bits would not be needed in case the hypervisor supports VMExits, e.g., KVM.
Each guest has a memory region that is shared with the backend.
This memory region is used by the frontend to allocate the io-buffers. This region also maps the virtio-mmio layout that is initialized by the backend. For the moment, this region is defined when the guest is created. One limitation is that the memory for io-buffers is fixed.

Quoted 7 lines

So in summary, you have a single memory region that is used for virtio-mmio layout and io-buffers (I think they are for payload) and you assume that the region will be (at lease for now) statically shared between FE and BE so that you can eliminate 'mmap' at every time to access the payload.
Correct?

Yes, It is.

Quoted 9 lines

If so, it can be an alternative solution for memory access issue, and a similar technique is used in some implementations:
- (Jailhouse's) ivshmem
- Arnd's fat virtqueue
In either case, however, you will have to allocate payload from the region and so you will see some impact on FE code (at least at some low level). (In ivshmem, dma_ops in the kernel is defined for this purpose.) Correct?

Yes, It is. The FE implements a sort of malloc() to organize the allocation of io-buffers from that memory region.

Rethinking about the VMExits, I am not sure how this mechanism may be used when both the FE and the BE are VMs. The use of VMExits may require to involve the hypervisor.

Matias

Quoted 141 lines

-Takahiro Akashi
Quoted 138 lines
At some point, the guest shall be able to balloon this region. Notifications between the frontend and the backend are implemented by using an hypercall. The hypercall mechanism and the memory allocation are abstracted away by a platform layer that exposes an interface that is hypervisor/os agnostic.
I split the backend into a virtio-device driver and a
backend driver. The virtio-device driver is the virtqueues and the backend driver gets packets from the virtqueue for post-processing. For example, in the case of virtio-net, the backend driver would decide if the packet goes to the hardware or to another virtio-net device. The virtio-device drivers may be
implemented in different ways like by using a single thread, multiple threads, or one thread for all the virtio-devices.
In this PoC, I just tackled two very simple use-cases. These use-cases allowed me to extract some requirements for an hypervisor to support virtio.
Matias
Quoted 117 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============
In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest. Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space. To activate the mapping will require some sort of hypercall to the hypervisor. I can see two options at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Signature · 5 lines
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

A message that is not in the archive (sent off-list or not yet received): 20210823012029.GB40863@laputa

Matias Ezequiel Vara Larsen

2021-10-04 11:35 UTC #54 lore raw

Quoted 33 lines · AKASHI Takahiro

Hi Matias,
Quoted 30 lines · Matias Ezequiel Vara Larsen
Hello,
Quoted 10 lines · AKASHI Takahiro
Hi Matias,
Quoted 4 lines · Matias Ezequiel Vara Larsen
Hello Alex,
I can tell you my experience from working on a PoC (library)
to allow the implementation of virtio-devices that are hypervisor/OS agnostic.
What hypervisor are you using for your PoC here?
I am using an in-house hypervisor, which is similar to Jailhouse.
Quoted 13 lines
Quoted 9 lines
I focused on two use cases:
1. type-I hypervisor in which the backend is running as a VM. This is an in-house hypervisor that does not support VMExits.
2. Linux user-space. In this case, the library is just used to communicate threads. The goal of this use case is merely testing.
I have chosen virtio-mmio as the way to exchange information
between the frontend and backend. I found it hard to synchronize the access to the virtio-mmio layout without VMExits. I had to add some extra bits to allow
Can you explain how MMIOs to registers in virito-mmio layout (which I think means a configuration space?) will be propagated to BE?

Quoted 7 lines

Quoted 3 lines
In this PoC, the BE guest is created with a fixed number of regions
of memory that represents each device. The BE initializes these regions, and then, waits for the FEs to begin the initialization.
Let me ask you in another way; When FE tries to write a register in configuration space, say QueueSel, how is BE notified of this event?

In my PoC, it is never notified when FE writes to a register. For example, the QueueSel is only used in one of the steps of the device status configuration. The BE is only notified when the
FE is in that step. When the FE is setting up the vrings, it sets the address, set the QueueSel, and then blocks until the BE can get the values. The BE gets the values and resumes the FE, which moves to the next step.

Quoted 29 lines

Quoted 29 lines
Quoted 9 lines
Quoted 8 lines
the front-end and back-end to synchronize, which is required during the device-status initialization. These extra bits would not be needed in case the hypervisor supports VMExits, e.g., KVM.
Each guest has a memory region that is shared with the backend.
This memory region is used by the frontend to allocate the io-buffers. This region also maps the virtio-mmio layout that is initialized by the backend. For the moment, this region is defined when the guest is created. One limitation is that the memory for io-buffers is fixed.
Quoted 7 lines
So in summary, you have a single memory region that is used for virtio-mmio layout and io-buffers (I think they are for payload) and you assume that the region will be (at lease for now) statically shared between FE and BE so that you can eliminate 'mmap' at every time to access the payload.
Correct?
Yes, It is.
Quoted 9 lines
If so, it can be an alternative solution for memory access issue, and a similar technique is used in some implementations:
- (Jailhouse's) ivshmem
- Arnd's fat virtqueue
In either case, however, you will have to allocate payload from the region and so you will see some impact on FE code (at least at some low level). (In ivshmem, dma_ops in the kernel is defined for this purpose.) Correct?

Quoted 8 lines

Quoted 5 lines
Yes, It is. The FE implements a sort of malloc() to organize the allocation of io-buffers from that memory region.
Rethinking about the VMExits, I am not sure how this mechanism may be used when both the FE and the BE are VMs. The use of VMExits may require to involve the hypervisor.
Maybe I misunderstand something. Are FE/BE not VMs in your PoC?

Yes, both are VMs. I meant, in case that both are VMs AND a VMExit mechanism is used, such a mechanism would require the hypervisor to forward the traps. In my PoC, both are VMs BUT there is not a VMExit mechanism.

Matias

Quoted 144 lines

-Takahiro Akashi
Quoted 142 lines
Matias
Quoted 141 lines
-Takahiro Akashi
Quoted 138 lines
At some point, the guest shall be able to balloon this region. Notifications between the frontend and the backend are implemented by using an hypercall. The hypercall mechanism and the memory allocation are abstracted away by a platform layer that exposes an interface that is hypervisor/os agnostic.
I split the backend into a virtio-device driver and a
backend driver. The virtio-device driver is the virtqueues and the backend driver gets packets from the virtqueue for post-processing. For example, in the case of virtio-net, the backend driver would decide if the packet goes to the hardware or to another virtio-net device. The virtio-device drivers may be
implemented in different ways like by using a single thread, multiple threads, or one thread for all the virtio-devices.
In this PoC, I just tackled two very simple use-cases. These use-cases allowed me to extract some requirements for an hypervisor to support virtio.
Matias
Quoted 117 lines · Alex Bennée
Hi,
One of the goals of Project Stratos is to enable hypervisor agnostic backends so we can enable as much re-use of code as possible and avoid repeating ourselves. This is the flip side of the front end where multiple front-end implementations are required - one per OS, assuming you don't just want Linux guests. The resultant guests are trivially movable between hypervisors modulo any abstracted paravirt type interfaces.
In my original thumb nail sketch of a solution I envisioned vhost-user daemons running in a broadly POSIX like environment. The interface to the daemon is fairly simple requiring only some mapped memory and some sort of signalling for events (on Linux this is eventfd). The idea was a stub binary would be responsible for any hypervisor specific setup and then launch a common binary to deal with the actual virtqueue requests themselves.
Since that original sketch we've seen an expansion in the sort of ways backends could be created. There is interest in encapsulating backends in RTOSes or unikernels for solutions like SCMI. There interest in Rust has prompted ideas of using the trait interface to abstract differences away as well as the idea of bare-metal Rust backends.
We have a card (STR-12) called "Hypercall Standardisation" which calls for a description of the APIs needed from the hypervisor side to support VirtIO guests and their backends. However we are some way off from that at the moment as I think we need to at least demonstrate one portable backend before we start codifying requirements. To that end I want to think about what we need for a backend to function.
Configuration
=============
In the type-2 setup this is typically fairly simple because the host system can orchestrate the various modules that make up the complete system. In the type-1 case (or even type-2 with delegated service VMs) we need some sort of mechanism to inform the backend VM about key details about the system:
- where virt queue memory is in it's address space
- how it's going to receive (interrupt) and trigger (kick) events
- what (if any) resources the backend needs to connect to
Obviously you can elide over configuration issues by having static configurations and baking the assumptions into your guest images however this isn't scalable in the long term. The obvious solution seems to be extending a subset of Device Tree data to user space but perhaps there are other approaches?
Before any virtio transactions can take place the appropriate memory mappings need to be made between the FE guest and the BE guest. Currently the whole of the FE guests address space needs to be visible to whatever is serving the virtio requests. I can envision 3 approaches:
* BE guest boots with memory already mapped
This would entail the guest OS knowing where in it's Guest Physical Address space is already taken up and avoiding clashing. I would assume in this case you would want a standard interface to userspace to then make that address space visible to the backend daemon.
* BE guests boots with a hypervisor handle to memory
The BE guest is then free to map the FE's memory to where it wants in the BE's guest physical address space. To activate the mapping will require some sort of hypercall to the hypervisor. I can see two options at this point:
- expose the handle to userspace for daemon/helper to trigger the mapping via existing hypercall interfaces. If using a helper you would have a hypervisor specific one to avoid the daemon having to care too much about the details or push that complexity into a compile time option for the daemon which would result in different binaries although a common source base.
- expose a new kernel ABI to abstract the hypercall differences away in the guest kernel. In this case the userspace would essentially ask for an abstract "map guest N memory to userspace ptr" and let the kernel deal with the different hypercall interfaces. This of course assumes the majority of BE guests would be Linux kernels and leaves the bare-metal/unikernel approaches to their own devices.
Operation
=========
The core of the operation of VirtIO is fairly simple. Once the vhost-user feature negotiation is done it's a case of receiving update events and parsing the resultant virt queue for data. The vhost-user specification handles a bunch of setup before that point, mostly to detail where the virt queues are set up FD's for memory and event communication. This is where the envisioned stub process would be responsible for getting the daemon up and ready to run. This is currently done inside a big VMM like QEMU but I suspect a modern approach would be to use the rust-vmm vhost crate. It would then either communicate with the kernel's abstracted ABI or be re-targeted as a build option for the various hypervisors.
One question is how to best handle notification and kicks. The existing vhost-user framework uses eventfd to signal the daemon (although QEMU is quite capable of simulating them when you use TCG). Xen has it's own IOREQ mechanism. However latency is an important factor and having events go through the stub would add quite a lot.
Could we consider the kernel internally converting IOREQ messages from the Xen hypervisor to eventfd events? Would this scale with other kernel hypercall interfaces?
So any thoughts on what directions are worth experimenting with?
Signature · 5 lines
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Alex Bennée

author

2021-09-01 08:56 UTC #55 lore raw

In reply to Matias Ezequiel Vara Larsen

Quoted 10 lines · Matias Ezequiel Vara Larsen

Hello Alex,
I can tell you my experience from working on a PoC (library)
to allow the implementation of virtio-devices that are hypervisor/OS agnostic. I focused on two use cases:
1. type-I hypervisor in which the backend is running as a VM. This is an in-house hypervisor that does not support VMExits.
2. Linux user-space. In this case, the library is just used to communicate threads. The goal of this use case is merely testing.

Quoted 6 lines

I have chosen virtio-mmio as the way to exchange information
between the frontend and backend. I found it hard to synchronize the access to the virtio-mmio layout without VMExits. I had to add some extra bits to allow the front-end and back-end to synchronize, which is required
during the device-status initialization. These extra bits would not be needed in case the hypervisor supports VMExits, e.g., KVM.

The support for a vmexit seems rather fundamental to type-2 hypervisors (like KVM) as the VMM is intrinsically linked to a vCPUs run loop. This makes handling a condition like a bit of MMIO fairly natural to implement. For type-1 cases the line of execution between "guest accesses MMIO" and "something services that request" is a little trickier to pin down. Ultimately at that point you are relying on the hypervisor itself to make the scheduling decision to stop executing the guest and allow the backend to do it's thing. We don't really want to expose the exact details about that as it probably varies a lot between hypervisors. However would a backend API semantic that expresses:

- guest has done some MMIO
- hypervisor has stopped execution of guest
- guest will be restarted when response conditions are set by backend

cover the needs of a virtio backend and could the userspace facing portion of that be agnostic?

Quoted 24 lines

Each guest has a memory region that is shared with the backend.
This memory region is used by the frontend to allocate the io-buffers. This region also maps the virtio-mmio layout that is initialized by the backend. For the moment, this region is defined when the guest is created. One limitation is that the memory for io-buffers is fixed. At some point, the guest shall be able to balloon this region. Notifications between the frontend and the backend are implemented by using an hypercall. The hypercall mechanism and the memory allocation are abstracted away by a platform layer that exposes an interface that is hypervisor/os agnostic.
I split the backend into a virtio-device driver and a
backend driver. The virtio-device driver is the virtqueues and the backend driver gets packets from the virtqueue for post-processing. For example, in the case of virtio-net, the backend driver would decide if the packet goes to the hardware or to another virtio-net device. The virtio-device drivers may be
implemented in different ways like by using a single thread, multiple threads, or one thread for all the virtio-devices.
In this PoC, I just tackled two very simple use-cases. These use-cases allowed me to extract some requirements for an hypervisor to support virtio.
Matias

<snip>

Signature · 5 lines

Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]