Constraining where a guest may allocate virtio accessible resources

Hi,

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of a guests memory space there needs to be some mechanism to limit the where that guest may place things. A simple VirtIO device can be expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

* QEMU KVM

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Option 1 - Kernel Command Line
==============================

This isn't without precedent - the kernel supports options like "memmap" which can with the appropriate amount of crafting be used to carve out sections of bad ram from the physical address space. Other formulations can be used to mark specific areas of the address space as particular types of memory.

However there are cons to this approach as it then becomes a job for whatever builds the VMM command lines to ensure the both the backend and the kernel know where things are. It is also very Linux centric and doesn't solve the problem for other guest OSes. Considering the rest of VirtIO can be made discover-able this seems like it would be a backward step.

Option 2 - Additional Platform Data
===================================

This would be extending using something like device tree or ACPI tables which could define regions of memory that would inform the low level memory allocation routines where they could allocate from. There is already of the concept of "dma-ranges" in device tree which can be a per-device property which defines the region of space that is DMA coherent for a device.

There is the question of how you tie regions declared here with the eventual instantiating of the VirtIO devices?

For a fully distributed set of backends (one backend per device per worker VM) you would need several different regions. Would each region be tied to each device or just a set of areas the guest would allocate from in sequence?

Option 3 - Abusing PCI Regions
==============================

One of the reasons to use the VirtIO PCI backend it to help with automatic probing and setup. Could we define a new PCI region which on backend just maps to RAM but from the front-ends point of view is a region it can allocate it's virtqueues? Could we go one step further and just let the host to define and allocate the virtqueue in the reserved PCI space and pass the base of it somehow?

Options 4 - Extend VirtIO Config
================================

Another approach would be to extend the VirtIO configuration and start-up handshake to supply these limitations to the guest. This could be handled by the addition of a feature bit (VIRTIO_F_HOST_QUEUE?) and additional configuration information.

One problem I can foresee is device initialisation is usually done fairly late in the start-up of a kernel by which time any memory zoning restrictions will likely need to have informed the kernels low level memory management. Does that mean we would have to combine such a feature behaviour with a another method anyway?

Option 5 - Additional Device
============================

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

Closing Thoughts and Open Questions
===================================

Currently all of this is considering just virtqueues themselves but of course only a subset of devices interact purely by virtqueue messages. Network and Block devices often end up filling up additional structures in memory that are usually across the whole of system memory. To achieve better isolation you either need to ensure that specific bits of kernel allocation are done in certain regions (i.e. block cache in "shared" region) or implement some sort of bounce buffer [1] that allows you to bring data from backend to frontend (which is more like the channel concept of Xen's PV).

I suspect the solution will end up being a combination of all of these approaches. There setup of different systems might mean we need a plethora of ways to carve out and define regions in ways a kernel can understand and make decisions about.

I think there will always have to be an element of VirtIO config involved as that is *the* mechanism by which front/back end negotiate if they can get up and running in a way they are both happy with.

One potential approach would be to introduce the concept of a region id at the VirtIO config level which is simply a reasonably unique magic number that virtio driver passes down into the kernel when requesting memory for it's virtqueues. It could then be left to the kernel to associate use that id when identifying the physical address range to allocate from. This seems a bit of a loose binding between the driver level and the kernel level but perhaps that is preferable to allow for flexibility about how such regions are discovered by kernels?

I hope this message hasn't rambled on to much. I feel this is a complex topic and I'm want to be sure I've thought through all the potential options before starting to prototype a solution. For those that have made it this far the final questions are:

- is constraining guest allocation of virtqueues a reasonable requirement?

- could virtqueues ever be directly host/hypervisor assigned?

- should there be a tight or loose coupling between front-end driver and kernel/hypervisor support for allocating memory?

Of course if this is all solvable with existing code I'd be more than happy but please let me know how ;-)

Quoted 127 lines · Alex Bennée

Hi,

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of a guests memory space there needs to be some mechanism to limit the where that guest may place things. A simple VirtIO device can be expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

* QEMU KVM

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Option 1 - Kernel Command Line
==============================

This isn't without precedent - the kernel supports options like "memmap" which can with the appropriate amount of crafting be used to carve out sections of bad ram from the physical address space. Other formulations can be used to mark specific areas of the address space as particular types of memory.

However there are cons to this approach as it then becomes a job for whatever builds the VMM command lines to ensure the both the backend and the kernel know where things are. It is also very Linux centric and doesn't solve the problem for other guest OSes. Considering the rest of VirtIO can be made discover-able this seems like it would be a backward step.

Option 2 - Additional Platform Data
===================================

This would be extending using something like device tree or ACPI tables which could define regions of memory that would inform the low level memory allocation routines where they could allocate from. There is already of the concept of "dma-ranges" in device tree which can be a per-device property which defines the region of space that is DMA coherent for a device.

There is the question of how you tie regions declared here with the eventual instantiating of the VirtIO devices?

For a fully distributed set of backends (one backend per device per worker VM) you would need several different regions. Would each region be tied to each device or just a set of areas the guest would allocate from in sequence?

Option 3 - Abusing PCI Regions
==============================

One of the reasons to use the VirtIO PCI backend it to help with automatic probing and setup. Could we define a new PCI region which on backend just maps to RAM but from the front-ends point of view is a region it can allocate it's virtqueues? Could we go one step further and just let the host to define and allocate the virtqueue in the reserved PCI space and pass the base of it somehow?

Options 4 - Extend VirtIO Config
================================

Another approach would be to extend the VirtIO configuration and start-up handshake to supply these limitations to the guest. This could be handled by the addition of a feature bit (VIRTIO_F_HOST_QUEUE?) and additional configuration information.

One problem I can foresee is device initialisation is usually done fairly late in the start-up of a kernel by which time any memory zoning restrictions will likely need to have informed the kernels low level memory management. Does that mean we would have to combine such a feature behaviour with a another method anyway?

Option 5 - Additional Device
============================

Quoted 8 lines

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Quoted 142 lines · Jan Kiszka

Quoted 127 lines · Alex Bennée

Hi,

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of a guests memory space there needs to be some mechanism to limit the where that guest may place things. A simple VirtIO device can be expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

* QEMU KVM

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Option 1 - Kernel Command Line
==============================

This isn't without precedent - the kernel supports options like "memmap" which can with the appropriate amount of crafting be used to carve out sections of bad ram from the physical address space. Other formulations can be used to mark specific areas of the address space as particular types of memory.

However there are cons to this approach as it then becomes a job for whatever builds the VMM command lines to ensure the both the backend and the kernel know where things are. It is also very Linux centric and doesn't solve the problem for other guest OSes. Considering the rest of VirtIO can be made discover-able this seems like it would be a backward step.

Option 2 - Additional Platform Data
===================================

This would be extending using something like device tree or ACPI tables which could define regions of memory that would inform the low level memory allocation routines where they could allocate from. There is already of the concept of "dma-ranges" in device tree which can be a per-device property which defines the region of space that is DMA coherent for a device.

There is the question of how you tie regions declared here with the eventual instantiating of the VirtIO devices?

For a fully distributed set of backends (one backend per device per worker VM) you would need several different regions. Would each region be tied to each device or just a set of areas the guest would allocate from in sequence?

Option 3 - Abusing PCI Regions
==============================

One of the reasons to use the VirtIO PCI backend it to help with automatic probing and setup. Could we define a new PCI region which on backend just maps to RAM but from the front-ends point of view is a region it can allocate it's virtqueues? Could we go one step further and just let the host to define and allocate the virtqueue in the reserved PCI space and pass the base of it somehow?

Options 4 - Extend VirtIO Config
================================

Another approach would be to extend the VirtIO configuration and start-up handshake to supply these limitations to the guest. This could be handled by the addition of a feature bit (VIRTIO_F_HOST_QUEUE?) and additional configuration information.

One problem I can foresee is device initialisation is usually done fairly late in the start-up of a kernel by which time any memory zoning restrictions will likely need to have informed the kernels low level memory management. Does that mean we would have to combine such a feature behaviour with a another method anyway?

Option 5 - Additional Device
============================

Quoted 8 lines

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

Quoted 4 lines

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Yes, I think IOMMU interfaces are worth investigating more too. IOMMUs are now widely implemented in Linux and virtualization software. That means guest modifications aren't necessary and unmodified guest applications will run.

Applications that need the best performance can use a static mapping while applications that want the strongest isolation can map/unmap DMA buffers dynamically.

Quoted 153 lines · Stefan Hajnoczi

Quoted 142 lines · Jan Kiszka

Quoted 127 lines · Alex Bennée

Hi,

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of a guests memory space there needs to be some mechanism to limit the where that guest may place things. A simple VirtIO device can be expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

* QEMU KVM

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Option 1 - Kernel Command Line
==============================

This isn't without precedent - the kernel supports options like "memmap" which can with the appropriate amount of crafting be used to carve out sections of bad ram from the physical address space. Other formulations can be used to mark specific areas of the address space as particular types of memory.

However there are cons to this approach as it then becomes a job for whatever builds the VMM command lines to ensure the both the backend and the kernel know where things are. It is also very Linux centric and doesn't solve the problem for other guest OSes. Considering the rest of VirtIO can be made discover-able this seems like it would be a backward step.

Option 2 - Additional Platform Data
===================================

This would be extending using something like device tree or ACPI tables which could define regions of memory that would inform the low level memory allocation routines where they could allocate from. There is already of the concept of "dma-ranges" in device tree which can be a per-device property which defines the region of space that is DMA coherent for a device.

There is the question of how you tie regions declared here with the eventual instantiating of the VirtIO devices?

For a fully distributed set of backends (one backend per device per worker VM) you would need several different regions. Would each region be tied to each device or just a set of areas the guest would allocate from in sequence?

Option 3 - Abusing PCI Regions
==============================

One of the reasons to use the VirtIO PCI backend it to help with automatic probing and setup. Could we define a new PCI region which on backend just maps to RAM but from the front-ends point of view is a region it can allocate it's virtqueues? Could we go one step further and just let the host to define and allocate the virtqueue in the reserved PCI space and pass the base of it somehow?

Options 4 - Extend VirtIO Config
================================

Another approach would be to extend the VirtIO configuration and start-up handshake to supply these limitations to the guest. This could be handled by the addition of a feature bit (VIRTIO_F_HOST_QUEUE?) and additional configuration information.

One problem I can foresee is device initialisation is usually done fairly late in the start-up of a kernel by which time any memory zoning restrictions will likely need to have informed the kernels low level memory management. Does that mean we would have to combine such a feature behaviour with a another method anyway?

Option 5 - Additional Device
============================

Quoted 8 lines

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

Quoted 4 lines

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Yes, I think IOMMU interfaces are worth investigating more too. IOMMUs are now widely implemented in Linux and virtualization software. That means guest modifications aren't necessary and unmodified guest applications will run.

Quoted 3 lines

Applications that need the best performance can use a static mapping while applications that want the strongest isolation can map/unmap DMA buffers dynamically.

I do not see yet that you can model with an IOMMU a static, not guest controlled window.

And IOMMU implies guest modifications as well (you need its driver). It just happened to be there now in newer guests. A virtio shared memory transport could be introduced similarly.

But the biggest challenge would be that a static mode would allow for a trivial hypervisor side model. Otherwise, we would only try to achieve a simpler secure model by adding complexity elsewhere.

I'm not arguing against vIOMMU per se. It's there, it is and will be widely used. It's just not solving all issues.

Jan

Signature · 2 lines

Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Quoted 158 lines · Jan Kiszka

Quoted 156 lines · Stefan Hajnoczi

Quoted 142 lines · Jan Kiszka

Quoted 127 lines · Alex Bennée

Hi,

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of a guests memory space there needs to be some mechanism to limit the where that guest may place things. A simple VirtIO device can be expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

* QEMU KVM

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Option 1 - Kernel Command Line
==============================

This isn't without precedent - the kernel supports options like "memmap" which can with the appropriate amount of crafting be used to carve out sections of bad ram from the physical address space. Other formulations can be used to mark specific areas of the address space as particular types of memory.

However there are cons to this approach as it then becomes a job for whatever builds the VMM command lines to ensure the both the backend and the kernel know where things are. It is also very Linux centric and doesn't solve the problem for other guest OSes. Considering the rest of VirtIO can be made discover-able this seems like it would be a backward step.

Option 2 - Additional Platform Data
===================================

This would be extending using something like device tree or ACPI tables which could define regions of memory that would inform the low level memory allocation routines where they could allocate from. There is already of the concept of "dma-ranges" in device tree which can be a per-device property which defines the region of space that is DMA coherent for a device.

There is the question of how you tie regions declared here with the eventual instantiating of the VirtIO devices?

For a fully distributed set of backends (one backend per device per worker VM) you would need several different regions. Would each region be tied to each device or just a set of areas the guest would allocate from in sequence?

Option 3 - Abusing PCI Regions
==============================

One of the reasons to use the VirtIO PCI backend it to help with automatic probing and setup. Could we define a new PCI region which on backend just maps to RAM but from the front-ends point of view is a region it can allocate it's virtqueues? Could we go one step further and just let the host to define and allocate the virtqueue in the reserved PCI space and pass the base of it somehow?

Options 4 - Extend VirtIO Config
================================

Another approach would be to extend the VirtIO configuration and start-up handshake to supply these limitations to the guest. This could be handled by the addition of a feature bit (VIRTIO_F_HOST_QUEUE?) and additional configuration information.

One problem I can foresee is device initialisation is usually done fairly late in the start-up of a kernel by which time any memory zoning restrictions will likely need to have informed the kernels low level memory management. Does that mean we would have to combine such a feature behaviour with a another method anyway?

Option 5 - Additional Device
============================

Quoted 8 lines

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

Quoted 4 lines

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Yes, I think IOMMU interfaces are worth investigating more too. IOMMUs are now widely implemented in Linux and virtualization software. That means guest modifications aren't necessary and unmodified guest applications will run.

Applications that need the best performance can use a static mapping while applications that want the strongest isolation can map/unmap DMA buffers dynamically.

Quoted 2 lines

I do not see yet that you can model with an IOMMU a static, not guest controlled window.

Well basically the IOMMU will have as part of the topology description and range of addresses devices behind it are allowed to access. What's the problem with that?

Quoted 5 lines

And IOMMU implies guest modifications as well (you need its driver). It just happened to be there now in newer guests. A virtio shared memory transport could be introduced similarly.

Quoted 12 lines

But the biggest challenge would be that a static mode would allow for a trivial hypervisor side model. Otherwise, we would only try to achieve a simpler secure model by adding complexity elsewhere.

I'm not arguing against vIOMMU per se. It's there, it is and will be widely used. It's just not solving all issues.

Jan

Signature · 2 lines

Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Quoted 31 lines

Quoted 31 lines

Quoted 30 lines

Quoted 17 lines

Quoted 11 lines

Option 5 - Additional Device
============================

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

Quoted 4 lines

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Yes, I think IOMMU interfaces are worth investigating more too. IOMMUs are now widely implemented in Linux and virtualization software. That means guest modifications aren't necessary and unmodified guest applications will run.

Applications that need the best performance can use a static mapping while applications that want the strongest isolation can map/unmap DMA buffers dynamically.

Quoted 7 lines

Quoted 2 lines

I do not see yet that you can model with an IOMMU a static, not guest controlled window.

Well basically the IOMMU will have as part of the topology description and range of addresses devices behind it are allowed to access. What's the problem with that?

I didn't look at the detail of the vIOMMU from that perspective, but our requirement would be that it would just statically communicate to the guest where DMA windows are, rather than allowing the guest to configure that (which is the normal usage of an IOMMU).

In addition, it would only address the memory transfer topic. We would still be left with the current issue of virtio that the hypervisor's device model needs to understand all supported device types.

Jan

Quoted 39 lines · Jan Kiszka

Quoted 31 lines

Quoted 31 lines

Quoted 30 lines

Quoted 17 lines

Quoted 11 lines

Option 5 - Additional Device
============================

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

Quoted 4 lines

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Yes, I think IOMMU interfaces are worth investigating more too. IOMMUs are now widely implemented in Linux and virtualization software. That means guest modifications aren't necessary and unmodified guest applications will run.

Applications that need the best performance can use a static mapping while applications that want the strongest isolation can map/unmap DMA buffers dynamically.

Quoted 7 lines

Quoted 2 lines

I do not see yet that you can model with an IOMMU a static, not guest controlled window.

Well basically the IOMMU will have as part of the topology description and range of addresses devices behind it are allowed to access. What's the problem with that?

Quoted 4 lines

I didn't look at the detail of the vIOMMU from that perspective, but our requirement would be that it would just statically communicate to the guest where DMA windows are, rather than allowing the guest to configure that (which is the normal usage of an IOMMU).

Right, I got that - IOMMUs aren't necessarily fully configurable though. E.g. some IOMMUs are restricted in the # of bits they can address.

Quoted 4 lines

In addition, it would only address the memory transfer topic. We would still be left with the current issue of virtio that the hypervisor's device model needs to understand all supported device types.

Quoted 1 line

Jan

I'd expect the DMA API would try to paper over that likely using bounce buffering. If you want to avoid copies, that's a harder problem generally.

Quoted 53 lines · Michael S. Tsirkin

Quoted 39 lines · Jan Kiszka

Quoted 31 lines

Quoted 31 lines

Quoted 30 lines

Quoted 17 lines

Quoted 11 lines

Option 5 - Additional Device
============================

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

Quoted 4 lines

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Yes, I think IOMMU interfaces are worth investigating more too. IOMMUs are now widely implemented in Linux and virtualization software. That means guest modifications aren't necessary and unmodified guest applications will run.

Applications that need the best performance can use a static mapping while applications that want the strongest isolation can map/unmap DMA buffers dynamically.

Quoted 7 lines

Quoted 2 lines

I do not see yet that you can model with an IOMMU a static, not guest controlled window.

Well basically the IOMMU will have as part of the topology description and range of addresses devices behind it are allowed to access. What's the problem with that?

Quoted 4 lines

I didn't look at the detail of the vIOMMU from that perspective, but our requirement would be that it would just statically communicate to the guest where DMA windows are, rather than allowing the guest to configure that (which is the normal usage of an IOMMU).

Right, I got that - IOMMUs aren't necessarily fully configurable though. E.g. some IOMMUs are restricted in the # of bits they can address.

Quoted 4 lines

In addition, it would only address the memory transfer topic. We would still be left with the current issue of virtio that the hypervisor's device model needs to understand all supported device types.

Quoted 6 lines

Quoted 1 line

Jan

I'd expect the DMA API would try to paper over that likely using bounce buffering. If you want to avoid copies, that's a harder problem generally.

Here I was referring to the permutations of the control path in a device model when switching from, say, a storage to a network virtio device. With PCI and MMIO (didn't check Channel I/O, but that's not portable anyway), you need to patch the "first-level" hypervisor when you want to add a brand-new virtio-sound device and the hypervisor is not yet aware of it. For minimized setups, I would prefer to only reconfigure it and just add a new backend service app or VM. Naturally, that model also shrinks the logic the core hypervisor needs to provide for virtio.

Jan

Quoted 70 lines · Jan Kiszka

Quoted 53 lines · Michael S. Tsirkin

Quoted 39 lines · Jan Kiszka

Quoted 31 lines

Quoted 31 lines

Quoted 30 lines

Quoted 17 lines

Quoted 11 lines

Option 5 - Additional Device
============================

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

Quoted 4 lines

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Yes, I think IOMMU interfaces are worth investigating more too. IOMMUs are now widely implemented in Linux and virtualization software. That means guest modifications aren't necessary and unmodified guest applications will run.

Applications that need the best performance can use a static mapping while applications that want the strongest isolation can map/unmap DMA buffers dynamically.

Quoted 7 lines

Quoted 2 lines

I do not see yet that you can model with an IOMMU a static, not guest controlled window.

Well basically the IOMMU will have as part of the topology description and range of addresses devices behind it are allowed to access. What's the problem with that?

Quoted 4 lines

I didn't look at the detail of the vIOMMU from that perspective, but our requirement would be that it would just statically communicate to the guest where DMA windows are, rather than allowing the guest to configure that (which is the normal usage of an IOMMU).

Right, I got that - IOMMUs aren't necessarily fully configurable though. E.g. some IOMMUs are restricted in the # of bits they can address.

Quoted 4 lines

In addition, it would only address the memory transfer topic. We would still be left with the current issue of virtio that the hypervisor's device model needs to understand all supported device types.

Quoted 6 lines

Quoted 1 line

Jan

I'd expect the DMA API would try to paper over that likely using bounce buffering. If you want to avoid copies, that's a harder problem generally.

Here I was referring to the permutations of the control path in a device model when switching from, say, a storage to a network virtio device. With PCI and MMIO (didn't check Channel I/O, but that's not portable anyway), you need to patch the "first-level" hypervisor when you want to add a brand-new virtio-sound device and the hypervisor is not yet aware of it. For minimized setups, I would prefer to only reconfigure it and just add a new backend service app or VM. Naturally, that model also shrinks the logic the core hypervisor needs to provide for virtio.

Quoted 1 line

Jan

Hmm that went woosh over my head a bit, sorry.
If it's important for this discussion, a diagram might help.

Quoted 127 lines · Alex Bennée

Hi,

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of a guests memory space there needs to be some mechanism to limit the where that guest may place things. A simple VirtIO device can be expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

* QEMU KVM

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Option 1 - Kernel Command Line
==============================

This isn't without precedent - the kernel supports options like "memmap" which can with the appropriate amount of crafting be used to carve out sections of bad ram from the physical address space. Other formulations can be used to mark specific areas of the address space as particular types of memory.

However there are cons to this approach as it then becomes a job for whatever builds the VMM command lines to ensure the both the backend and the kernel know where things are. It is also very Linux centric and doesn't solve the problem for other guest OSes. Considering the rest of VirtIO can be made discover-able this seems like it would be a backward step.

Option 2 - Additional Platform Data
===================================

This would be extending using something like device tree or ACPI tables which could define regions of memory that would inform the low level memory allocation routines where they could allocate from. There is already of the concept of "dma-ranges" in device tree which can be a per-device property which defines the region of space that is DMA coherent for a device.

There is the question of how you tie regions declared here with the eventual instantiating of the VirtIO devices?

For a fully distributed set of backends (one backend per device per worker VM) you would need several different regions. Would each region be tied to each device or just a set of areas the guest would allocate from in sequence?

Option 3 - Abusing PCI Regions
==============================

One of the reasons to use the VirtIO PCI backend it to help with automatic probing and setup. Could we define a new PCI region which on backend just maps to RAM but from the front-ends point of view is a region it can allocate it's virtqueues? Could we go one step further and just let the host to define and allocate the virtqueue in the reserved PCI space and pass the base of it somehow?

Options 4 - Extend VirtIO Config
================================

Another approach would be to extend the VirtIO configuration and start-up handshake to supply these limitations to the guest. This could be handled by the addition of a feature bit (VIRTIO_F_HOST_QUEUE?) and additional configuration information.

One problem I can foresee is device initialisation is usually done fairly late in the start-up of a kernel by which time any memory zoning restrictions will likely need to have informed the kernels low level memory management. Does that mean we would have to combine such a feature behaviour with a another method anyway?

Option 5 - Additional Device
============================

Quoted 8 lines

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Quoted 5 lines

Closing Thoughts and Open Questions
===================================

Quoted 9 lines

Currently all of this is considering just virtqueues themselves but of course only a subset of devices interact purely by virtqueue messages. Network and Block devices often end up filling up additional structures in memory that are usually across the whole of system memory. To achieve better isolation you either need to ensure that specific bits of kernel allocation are done in certain regions (i.e. block cache in "shared" region) or implement some sort of bounce buffer [1] that allows you to bring data from backend to frontend (which is more like the channel concept of Xen's PV).

For [1], look at https://lkml.org/lkml/2020/3/26/700 or at
http://git.kiszka.org/?p=linux.git;a=blob;f=drivers/virtio/virtio_ivshmem.c;hb=refs/heads/queues/jailhouse
(which should be using swiotlb one day).

Quoted 26 lines

I suspect the solution will end up being a combination of all of these approaches. There setup of different systems might mean we need a plethora of ways to carve out and define regions in ways a kernel can understand and make decisions about.

I think there will always have to be an element of VirtIO config involved as that is *the* mechanism by which front/back end negotiate if they can get up and running in a way they are both happy with.

One potential approach would be to introduce the concept of a region id at the VirtIO config level which is simply a reasonably unique magic number that virtio driver passes down into the kernel when requesting memory for it's virtqueues. It could then be left to the kernel to associate use that id when identifying the physical address range to allocate from. This seems a bit of a loose binding between the driver level and the kernel level but perhaps that is preferable to allow for flexibility about how such regions are discovered by kernels?

I hope this message hasn't rambled on to much. I feel this is a complex topic and I'm want to be sure I've thought through all the potential options before starting to prototype a solution. For those that have made it this far the final questions are:

- is constraining guest allocation of virtqueues a reasonable requirement?

Quoted 8 lines

- could virtqueues ever be directly host/hypervisor assigned?

- should there be a tight or loose coupling between front-end driver and kernel/hypervisor support for allocating memory?

Of course if this is all solvable with existing code I'd be more than happy but please let me know how ;-)

Queues are a central element of virtio, but there is a (maintainability & security) benefit if you can keep them away from the hosting hypervisor, limit their interpretation and negotiation to the backend driver in a host process or in a backend guest VM. So I would be careful with coupling things too tightly.

One of the issues I see in virtio for use in minimalistic hypervisors is the need to be aware of the different virtio devices when using PCI or MMIO transports. That's where a shared memory transport come into play.

Jan

Quoted 18 lines · Alex Bennée

Hi,

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of a guests memory space there needs to be some mechanism to limit the where that guest may place things. A simple VirtIO device can be expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Quoted 5 lines

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

Quoted 4 lines

Option 5 - Additional Device
============================

Quoted 8 lines

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

I think you forgot one approach: virtual IOMMU. That is the advanced form of the grant table approach. The backend still "sees" the full address space of the frontend, but it will not be able to access all of it and there might even be a translation going on. Well, like IOMMUs work.

However, this implies dynamics that are under guest control, namely of the frontend guest. And such dynamics can be counterproductive for certain scenarios. That's where this static windows of shared memory came up.

Quoted 4 lines

Closing Thoughts and Open Questions
===================================

Quoted 9 lines

Currently all of this is considering just virtqueues themselves but of course only a subset of devices interact purely by virtqueue messages. Network and Block devices often end up filling up additional structures in memory that are usually across the whole of system memory. To achieve better isolation you either need to ensure that specific bits of kernel allocation are done in certain regions (i.e. block cache in "shared" region) or implement some sort of bounce buffer [1] that allows you to bring data from backend to frontend (which is more like the channel concept of Xen's PV).

For [1], look at https://lkml.org/lkml/2020/3/26/700 or at
http://git.kiszka.org/?p=linux.git;a=blob;f=drivers/virtio/virtio_ivshmem.c;hb=refs/heads/queues/jailhouse
(which should be using swiotlb one day).

Quoted 25 lines

I suspect the solution will end up being a combination of all of these approaches. There setup of different systems might mean we need a plethora of ways to carve out and define regions in ways a kernel can understand and make decisions about.

I think there will always have to be an element of VirtIO config involved as that is *the* mechanism by which front/back end negotiate if they can get up and running in a way they are both happy with.

One potential approach would be to introduce the concept of a region id at the VirtIO config level which is simply a reasonably unique magic number that virtio driver passes down into the kernel when requesting memory for it's virtqueues. It could then be left to the kernel to associate use that id when identifying the physical address range to allocate from. This seems a bit of a loose binding between the driver level and the kernel level but perhaps that is preferable to allow for flexibility about how such regions are discovered by kernels?

I hope this message hasn't rambled on to much. I feel this is a complex topic and I'm want to be sure I've thought through all the potential options before starting to prototype a solution. For those that have made it this far the final questions are:

- is constraining guest allocation of virtqueues a reasonable requirement?

Quoted 8 lines

- could virtqueues ever be directly host/hypervisor assigned?

- should there be a tight or loose coupling between front-end driver and kernel/hypervisor support for allocating memory?

Of course if this is all solvable with existing code I'd be more than happy but please let me know how ;-)

Queues are a central element of virtio, but there is a (maintainability & security) benefit if you can keep them away from the hosting hypervisor, limit their interpretation and negotiation to the backend driver in a host process or in a backend guest VM. So I would be careful with coupling things too tightly.

One of the issues I see in virtio for use in minimalistic hypervisors is the need to be aware of the different virtio devices when using PCI or MMIO transports. That's where a shared memory transport come into play.

Hi,

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of a guests memory space there needs to be some mechanism to limit the where that guest may place things. A simple VirtIO device can be expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

* QEMU KVM

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Option 1 - Kernel Command Line
==============================

This isn't without precedent - the kernel supports options like "memmap" which can with the appropriate amount of crafting be used to carve out sections of bad ram from the physical address space. Other formulations can be used to mark specific areas of the address space as particular types of memory.

However there are cons to this approach as it then becomes a job for whatever builds the VMM command lines to ensure the both the backend and the kernel know where things are. It is also very Linux centric and doesn't solve the problem for other guest OSes. Considering the rest of VirtIO can be made discover-able this seems like it would be a backward step.

Option 2 - Additional Platform Data
===================================

This would be extending using something like device tree or ACPI tables which could define regions of memory that would inform the low level memory allocation routines where they could allocate from. There is already of the concept of "dma-ranges" in device tree which can be a per-device property which defines the region of space that is DMA coherent for a device.

There is the question of how you tie regions declared here with the eventual instantiating of the VirtIO devices?

For a fully distributed set of backends (one backend per device per worker VM) you would need several different regions. Would each region be tied to each device or just a set of areas the guest would allocate from in sequence?

Option 3 - Abusing PCI Regions
==============================

One of the reasons to use the VirtIO PCI backend it to help with automatic probing and setup. Could we define a new PCI region which on backend just maps to RAM but from the front-ends point of view is a region it can allocate it's virtqueues? Could we go one step further and just let the host to define and allocate the virtqueue in the reserved PCI space and pass the base of it somehow?

Options 4 - Extend VirtIO Config
================================

Another approach would be to extend the VirtIO configuration and start-up handshake to supply these limitations to the guest. This could be handled by the addition of a feature bit (VIRTIO_F_HOST_QUEUE?) and additional configuration information.

One problem I can foresee is device initialisation is usually done fairly late in the start-up of a kernel by which time any memory zoning restrictions will likely need to have informed the kernels low level memory management. Does that mean we would have to combine such a feature behaviour with a another method anyway?

Option 5 - Additional Device
============================

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

Closing Thoughts and Open Questions
===================================

Currently all of this is considering just virtqueues themselves but of course only a subset of devices interact purely by virtqueue messages. Network and Block devices often end up filling up additional structures in memory that are usually across the whole of system memory. To achieve better isolation you either need to ensure that specific bits of kernel allocation are done in certain regions (i.e. block cache in "shared" region) or implement some sort of bounce buffer [1] that allows you to bring data from backend to frontend (which is more like the channel concept of Xen's PV).

I suspect the solution will end up being a combination of all of these approaches. There setup of different systems might mean we need a plethora of ways to carve out and define regions in ways a kernel can understand and make decisions about.

I think there will always have to be an element of VirtIO config involved as that is *the* mechanism by which front/back end negotiate if they can get up and running in a way they are both happy with.

One potential approach would be to introduce the concept of a region id at the VirtIO config level which is simply a reasonably unique magic number that virtio driver passes down into the kernel when requesting memory for it's virtqueues. It could then be left to the kernel to associate use that id when identifying the physical address range to allocate from. This seems a bit of a loose binding between the driver level and the kernel level but perhaps that is preferable to allow for flexibility about how such regions are discovered by kernels?

I hope this message hasn't rambled on to much. I feel this is a complex topic and I'm want to be sure I've thought through all the potential options before starting to prototype a solution. For those that have made it this far the final questions are:

- is constraining guest allocation of virtqueues a reasonable requirement?

- could virtqueues ever be directly host/hypervisor assigned?

- should there be a tight or loose coupling between front-end driver and kernel/hypervisor support for allocating memory?

Of course if this is all solvable with existing code I'd be more than happy but please let me know how ;-)

Regards,

Signature · 6 lines

Alex Bennée

[1] Example bounce buffer approach

Subject: [PATCH 0/5] virtio on Type-1 hypervisor
Message-Id: <[email protected]>

Quoted 57 lines · Alex Bennée

Hi,

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of a guests memory space there needs to be some mechanism to limit the where that guest may place things. A simple VirtIO device can be expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

* QEMU KVM

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

Quoted 8 lines

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Something that's unclear to me is whether you envision each device to have its own dedicated memory it can access,
or broadly to have a couple of groups of devices,
kind of like e.g. there are 32 bit and 64 bit DMA capable pci devices, or like we have devices with VIRTIO_F_ACCESS_PLATFORM and without it?

Quoted 11 lines

Option 5 - Additional Device
============================

The final approach would be to tie the allocation of virtqueues to memory regions as defined by additional devices. For example the proposed IVSHMEMv2 spec offers the ability for the hypervisor to present a fixed non-mappable region of the address space. Other proposals like virtio-mem allow for hot plugging of "physical" memory into the guest (conveniently treatable as separate shareable memory objects for QEMU ;-).

Another approach would be supplying this information through virtio-iommu. That already has topology information, and can be used together with VIRTIO_F_ACCESS_PLATFORM to limit device access to memory.
As virtio iommu is fairly new I kind of like this approach myself - not a lot of legacy to contend with.

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

To support the concept of a VirtIO backend having limited visibility of

a guests memory space there needs to be some mechanism to limit the where that guest may place things.

A simple VirtIO device can be
expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

* QEMU KVM

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Quoted 6 lines · Alex Bennée

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

Quoted 1 line

To support the concept of a VirtIO backend having limited visibility of

It's unclear what we're discussing. Does "VirtIO backend" mean vhost-user devices?

Can you describe what you are trying to do?

Quoted 2 lines

a guests memory space there needs to be some mechanism to limit the where that guest may place things.

Or an enforcing IOMMU? In other words, an IOMMU that only gives access to memory that has been put forth for DMA.

This was discussed recently in the context of the ongoing vfio-over-socket work ("RFC: use VFIO over a UNIX domain socket to implement device offloading" on qemu-devel). The idea is to use the VFIO protocol but over UNIX domain sockets to another host userspace process instead of over ioctls to the kernel VFIO drivers. This would allow arbitary devices to be emulated in a separate process from QEMU. As a first step I suggested DMA_READ/DMA_WRITE protocol messages, even though this will have poor performance.

I think finding a solution for an enforcing IOMMU is preferrable to guest cooperation. The problem with guest cooperation is that you may be able to get new VIRTIO guest drivers to restrict where the virtqueues are placed, but what about applications (e.g. O_DIRECT disk I/O, network packets) with memory buffers at arbitrary addresses?

Modifying guest applications to honor buffer memory restrictions is too disruptive for most use cases.

Quoted 17 lines

A simple VirtIO device can be
expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

Quoted 7 lines

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

This paragraph doesn't reflect my understanding. There can be multiple RAMBlocks. There isn't necessarily just 1 contiguous piece of RAM.

Quoted 3 lines

* QEMU KVM

Quoted 4 lines

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

I don't understand the "although" part. Isn't the situation the same as with TCG, where guest physical memory ranges can cross RAMBlock boundaries?

Quoted 6 lines

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

I'm surprised about the DMA coherency sentence. Dont't VFIO and other userspace I/O APIs provide the DMA APIs allowing applications to deal with caches/coherency?

Quoted 13 lines

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

Quoted 4 lines

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

As mentioned above, I don't think it's possible to do this without modifying applications - which is not possible in many use cases. Instead we could improve IOMMU support so that this works transparently.

Quoted 7 lines · Stefan Hajnoczi

Quoted 6 lines · Alex Bennée

This follows on from the discussion in the last thread I raised:

Subject: Backend libraries for VirtIO device emulation
Date: Fri, 06 Mar 2020 18:33:57 +0000
Message-ID: <[email protected]>

Quoted 7 lines

Quoted 1 line

To support the concept of a VirtIO backend having limited visibility of

It's unclear what we're discussing. Does "VirtIO backend" mean vhost-user devices?

Can you describe what you are trying to do?

Yes - although eventually the vhost-user device might be hosted in a separate VM. See this contrived architecture diagram:

                                                   |                                                     
                Secure World                       |          Non-secure world                 
                                                   |   +--------------------+  +---------------+  
                                                   |   |c1AB                |  |cGRE           |  
                                                   |   |                    |  |               |  
                                                   |   |     Primary OS     |  |   Secondary   |  
                                                   |   |      (android)     |  |      VM       |  
         +--------------+                          |   |                    |  |               |  
         |cYEL          |                          |   |                    |  |   (Backend)   |  
         |              |                          |   |                    |  +---------------+  
         |              |                          |   |                    |                   
         |  Untrusted   |                          |   |                    |                   
         |              |                          |   |                    |  +---------------+  
   EL0   |   Service    |                          |   |                    |  |cGRE           |
    .    |              |                          |   |                    |  |               |
    .    |              |                          :   | +----------------+ |  |   Secondary   |
    .    |              |                          |   | |{io} VirtIO     | |  |      VM       |
   EL1   |              |                          |   | |                | |  |               |
         |  (Backend)   |                          |   | +----------------+ |  |   (Backend)   |
         +--------------+                          |   +----------------+---+  +---------------+
                                                   |                                        
         +-------------------------------------+   |   +---------------------------------------+
         |cPNK                                 |   |   |cGRE                                   |
   EL2   |        Secure Hypervisor            |   |   |          Non-secure Hypervisor        |
         |                                     |   |   |                                       |
         +-------------------------------------+   |   +---------------------------------------+
                                                   +-----------------------------------------------
         +-------------------------------------------------------------------------------------+
         |cRED                                                                                 |
   EL3   |                                  Secure Firmware                                    |
         |                                                                                     |
         +-------------------------------------------------------------------------------------+
  ----=-----------------------------------------------------------------------------------------   
         +------------------------+ +-------------------------+ +------------------------------+
         | c444                   | | {s}                c444 | | {io}                    c444 |
   HW    |        Compute         | |         Storage         | |             I/O              |
         |  (CPUs, GPUs, Accel)   | |  (Flash, Secure Flash)  | |   Network, USB, Peripherals  |
         |                        | |                         | |                              |
         +------------------------+ +-------------------------+ +------------------------------+

Here the primary OS is connected to the work through VirtIO devices (acting as a common HAL). Each individual device might have a secondary VM associated with it. Some devices might be virtual - for example a 3rd party DRM module. It would be un-trusted so doesn't run as part of the secure firmware but it might still need to access secure resources like a key store or a video port.

For all these backends they should only have access to the minimum amount of the primary OS's memory space that they need to fulfil their job. While the non-secure hypervisor could be something like KVM it's likely the secure one will be a much more lightweight type-1 hypervisor.

My interest in including TCG in this mix is for early prototyping and ease of debugging when working with this towering array of layers ;-)

Quoted 6 lines

Quoted 2 lines

a guests memory space there needs to be some mechanism to limit the where that guest may place things.

Or an enforcing IOMMU? In other words, an IOMMU that only gives access to memory that has been put forth for DMA.

Quoted 8 lines

This was discussed recently in the context of the ongoing vfio-over-socket work ("RFC: use VFIO over a UNIX domain socket to implement device offloading" on qemu-devel). The idea is to use the VFIO protocol but over UNIX domain sockets to another host userspace process instead of over ioctls to the kernel VFIO drivers. This would allow arbitary devices to be emulated in a separate process from QEMU. As a first step I suggested DMA_READ/DMA_WRITE protocol messages, even though this will have poor performance.

This is still mediated by a kernel though right?

Quoted 5 lines

I think finding a solution for an enforcing IOMMU is preferrable to guest cooperation. The problem with guest cooperation is that you may be able to get new VIRTIO guest drivers to restrict where the virtqueues are placed, but what about applications (e.g. O_DIRECT disk I/O, network packets) with memory buffers at arbitrary addresses?

The virtqueues are the simple case but yes it gets complex for the rest of the data - the simple case is handled by a bounce buffer which the guest then copies from into it's own secure address space.

Quoted 34 lines

Modifying guest applications to honor buffer memory restrictions is too disruptive for most use cases.

Quoted 17 lines

A simple VirtIO device can be
expressed purely in virt resources, for example:

* status, feature and config fields
* notification/doorbell
* one or more virtqueues

Using a PCI backend the location of everything but the virtqueues it controlled by the mapping of the PCI device so something that is controllable by the host/hypervisor. However the guest is free to allocate the virtqueues anywhere in the virtual address space of system RAM.

In theory this shouldn't matter because sharing virtual pages is just a matter of putting the appropriate translations in place. However there are multiple ways the host and guest may interact:

Quoted 7 lines

* QEMU TCG

QEMU sees a block of system memory in it's virtual address space that has a one to one mapping with the guests physical address space. If QEMU want to share a subset of that address space it can only realistically do it for a contiguous region of it's address space which implies the guest must use a contiguous region of it's physical address space.

This paragraph doesn't reflect my understanding. There can be multiple RAMBlocks. There isn't necessarily just 1 contiguous piece of RAM.

Quoted 3 lines

* QEMU KVM

Quoted 8 lines

Quoted 4 lines

The situation here is broadly the same - although both QEMU and the guest are seeing a their own virtual views of a linear address space which may well actually be a fragmented set of physical pages on the host.

I don't understand the "although" part. Isn't the situation the same as with TCG, where guest physical memory ranges can cross RAMBlock boundaries?

You are correct - I was over simplifying. This is why I was thinking about the virtio-mem device. That would have it's own RAMBlock which could be the only one with an associated shared memory object.

Quoted 7 lines

Quoted 6 lines

KVM based guests have additional constraints if they ever want to access real hardware in the host as you need to ensure any address accessed by the guest can be eventually translated into an address that can physically access the bus which a device in one (for device pass-through). The area also has to be DMA coherent so updates from a bus are reliably visible to software accessing the same address space.

Quoted 3 lines

I'm surprised about the DMA coherency sentence. Dont't VFIO and other userspace I/O APIs provide the DMA APIs allowing applications to deal with caches/coherency?

Yes - but the kernel has to ensure the buffers used by these APIs are allocated in regions that meet the requirements.

Quoted 18 lines

Quoted 17 lines

* Xen (and other type-1's?)

Here the situation is a little different because the guest explicitly makes it's pages visible to other domains by way of grant tables. The guest is still free to use whatever parts of its address space it wishes to. Other domains then request access to those pages via the hypervisor.

In theory the requester is free to map the granted pages anywhere in its own address space. However there are differences between the architectures on how well this is supported.

So I think this makes a case for having a mechanism by which the guest can restrict it's allocation to a specific area of the guest physical address space. The question is then what is the best way to inform the guest kernel of the limitation?

Quoted 3 lines

As mentioned above, I don't think it's possible to do this without modifying applications - which is not possible in many use cases. Instead we could improve IOMMU support so that this works transparently.

Yes. So the IOMMU allows the guest to mark all the pages associated with a particular device and it's transactions but how do we map that to the userspace view which in controlled in software?

Option 2 - Additional Platform Data
===================================

This would be extending using something like device tree or ACPI tables which could define regions of memory that would inform the low level memory allocation routines where they could allocate from. There is already of the concept of "dma-ranges" in device tree which can be a per-device property which defines the region of space that is DMA coherent for a device.

There is the question of how you tie regions declared here with the eventual instantiating of the VirtIO devices?

For a fully distributed set of backends (one backend per device per worker VM) you would need several different regions. Would each region be tied to each device or just a set of areas the guest would allocate from in sequence?