virtio: support devices that own their virtqueue memory
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v2 Latest12 patches 11 repliesIn review
- v1 RFC12 patches 19 repliesSuperseded
Changes in v2 from the cover letter
- Remove the unused map sync API (Michael)
- Drop the sync operations from virtio_map_ops (Michael)
- Drop the stale mask expansion instead of correcting it (Michael)
- Treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
- Read the region's memory type and accept only a coherent one
- Rewrite the region allocator over gen_pool
- Return -ENOMEM instead of -EIO from a failed packed ring mapping
- Drop the patch validating premapped addresses through the map
- Drop the patch reporting a bounded pool's exhaustion as -ENOSPC
- Drop the range withheld for a virtqueue's first descriptor chain
- Drop the .rst and describe the region in the headers instead
Patches in v2 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/12 | virtio_ring: remove the unused map sync APIdrivers/ include/ tools/ | +0 | 1 reply | |
| 2/12 | virtio: drop the sync operations from virtio_map_opsdrivers/ drivers/ drivers/ +1 more | +0 | 1 reply | |
| 3/12 | vdpa: drop the VIRTIO_DEVICE_F_MASK example valuedrivers/ | +3 | 1 reply | |
| 4/12 | virtio_ring: return -ENOMEM when a packed ring mapping failsdrivers/ | +2 | 8 replies | |
| 5/12 | virtio: add the VIRTIO_F_DMB feature bitinclude/ | +23 | 0 replies | |
| 6/12 | virtio_pci: read the device memory buffer registersdrivers/ include/ include/ | +67 | 0 replies | |
| 7/12 | virtio_pci: create virtqueues with the device's mapping tokendrivers/ | +13 | 0 replies | |
| 8/12 | virtio: add a device memory buffer region allocatordrivers/ drivers/ drivers/ +3 more | +856 | 0 replies | |
| 9/12 | virtio: locate the device memory buffer after feature negotiationdrivers/ | +14 | 0 replies | |
| 10/12 | virtio: treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORMdrivers/ include/ | +20 | 0 replies | |
| 11/12 | virtio_pci: support VIRTIO_F_DMBdrivers/ | +58 | 0 replies | |
| 12/12 | virtio: expose device memory buffer occupancy over debugfsdrivers/ | +162 | 0 replies |
Reviews trailers per patch
4 of 12 patches have a Reviewed-by or Acked-by.
| Patch | Eugenio Pérez |
|---|---|
| 1/12 | A-b |
| 2/12 | A-b |
| 3/12 | A-b |
| 4/12 | A-b A-b |
| 5/12 | |
| 6/12 | |
| 7/12 | |
| 8/12 | |
| 9/12 | |
| 10/12 | |
| 11/12 | |
| 12/12 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH v2 00/12] virtio: support devices that own their virtqueue memory
Virtio drivers use guest memory to back virtqueues and their buffers. That means a VMM needs to be able to map guest memory. That is ok in the normal virt case. It gets icky with confidential computing (where we use swiotlb as workaround) and it defeats the purpose of isolated vhost-user backing devices, because they end up with full RAM access to the guest.
So instead, I'm proposing an extension to virtio which allows it to give each virtio device its own dedicated memory region to communicate with the host, called DMB (Device Memory Buffer). A trusted hypervisor can force DMB to be present, which then enables safer, more isolated and resilient communication between guest and host.
With DMB, the device provides a shared memory region that both parties agree is the full memory map both have access to. All memory offsets that previously would have been into guest RAM, are then offsets into this shared memory buffer region. One nice property of this is that it is a generic mechanism in the virtio transport layer, so higher level drivers work unmodified.
I was exploring to use swiotlb instead to create individual pools. But that approach has multiple downsides:
1. Swiotlb is an OS primitive which is not available in all Operating Systems. DMB however lives in the virtio transport layer, which means we can add support for it in any OS independent of generic layers. This helps with Windows support.
2. We munge DMA space together. DMB provides a separate DMA space per virtio device. This means we can for example implement a device in vhost-user and give the implementing process only visibility to the DMB region, not all of guest memory. That reduces the exposure the vhost-user provider has, improving security.
3. Devices can opt-in. A hypervisor can choose to use standard virtio semantics for self-implemented devices (e.g. NSM), while requiring DMB for devices implemented by less trustworthy providers. The non-trustworthy devices do not get any visibility into the trustworthy ones, even with DMB in place for both.
I also looked at virtio-iommu. To restrict DMA visibility, virtio-iommu allows the guest to open specific windows into guest memory to the device, but it comes with its own bag of problems, such as dynamic allocations and complicated device <-> iommu connections that need to be represented reliably.
== Limitations ==
- Only PCI is wired up.
- Feature bit 44 and the two registers at offsets 0x40 and 0x42 of the PCI common configuration are provisional: the OASIS technical committee has the specification and has allocated none of them.
https://lore.kernel.org/virtio-comment/[email protected]/
== Testing ==
I ran this against a device that offers DMB and not VIRTIO_F_ACCESS_PLATFORM, with a 16 MiB region on each device. A kernel without patch 10 finds the region and then hangs with nothing in the log; with it, 512 MiB of O_DIRECT block reads and 137 MiB of loopback network traffic go through the regions with no allocation failure.
Patches 1 to 4 do not need the rest of the series. They remove an API no driver calls and a stale worked example in a vdpa comment. Patch 3 and patch 4 carry Fixes:; patch 4's are older than this series, a failed mapping arriving from a packed ring as -EIO, which fails an I/O that on a split ring is only back-pressure.
I wrote this series with an AI coding assistant, which drafted the code, the changelogs and this cover letter. I reviewed and reworked all of it, and every commit carries an Assisted-by: trailer.
Alex
v1: https://lore.kernel.org/all/[email protected]/
v1 -> v2:
- Remove the unused map sync API (Michael)
- Drop the sync operations from virtio_map_ops (Michael)
- Drop the stale mask expansion instead of correcting it (Michael)
- Treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
- Read the region's memory type and accept only a coherent one
- Rewrite the region allocator over gen_pool
- Return -ENOMEM instead of -EIO from a failed packed ring mapping
- Drop the patch validating premapped addresses through the map
- Drop the patch reporting a bounded pool's exhaustion as -ENOSPC
- Drop the range withheld for a virtqueue's first descriptor chain
- Drop the .rst and describe the region in the headers instead
Alexander Graf (12):
virtio_ring: remove the unused map sync API
virtio: drop the sync operations from virtio_map_ops
vdpa: drop the VIRTIO_DEVICE_F_MASK example value
virtio_ring: return -ENOMEM when a packed ring mapping fails
virtio: add the VIRTIO_F_DMB feature bit
virtio_pci: read the device memory buffer registers
virtio_pci: create virtqueues with the device's mapping token
virtio: add a device memory buffer region allocator
virtio: locate the device memory buffer after feature negotiation
virtio: treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
virtio_pci: support VIRTIO_F_DMB
virtio: expose device memory buffer occupancy over debugfs
drivers/vdpa/vdpa.c | 7 +- drivers/vdpa/vdpa_user/iova_domain.c | 20 - drivers/vdpa/vdpa_user/iova_domain.h | 8 - drivers/vdpa/vdpa_user/vduse_dev.c | 40 -- drivers/virtio/Kconfig | 17 + drivers/virtio/Makefile | 3 +- drivers/virtio/virtio.c | 25 +- drivers/virtio/virtio_dmb.c | 959 +++++++++++++++++++++++++ drivers/virtio/virtio_dmb.h | 28 + drivers/virtio/virtio_pci_modern.c | 75 +- drivers/virtio/virtio_pci_modern_dev.c | 47 +- drivers/virtio/virtio_ring.c | 93 +-- include/linux/virtio.h | 11 +- include/linux/virtio_config.h | 41 +- include/linux/virtio_pci_modern.h | 2 + include/uapi/linux/virtio_config.h | 24 +- include/uapi/linux/virtio_pci.h | 19 + tools/virtio/linux/dma-mapping.h | 7 - 18 files changed, 1217 insertions(+), 209 deletions(-) create mode 100644 drivers/virtio/virtio_dmb.c create mode 100644 drivers/virtio/virtio_dmb.h
base-commit: fc02acf6ac0ccde0c805c2daa9148683cdd01ba8