Skip to contentSkip to conversation

Add VIRTIO_F_DMB (Device Memory Buffer)

PATCH Superseded Alexander Graf 3 participants 4 messages Aug 17 – Aug 18

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 Add VIRTIO_F_DMB (Device Memory Buffer)

Virtio devices assume R/W access of guest RAM to perform DMA into it to access its virt queues and data to operate on. This basic assumption leads to multiple problems:

Confidential Compute - In SEV-SNP or TDX, the host has no direct access
to all of guest RAM. We work around DMA access by having the guest limit DMA access to a common shared memory window between guest and host (swiotlb).

Vhost-user - vhost-user backends require access to all guest memory, not
only the memory they are actually supposed to access. This is fine for trusted backends, but if you want to limit visibility into guest, you need to do weird stunts like bouncing memory in the vmm [1].

To restrict DMA visibility, virtio-iommu allows the guest to open specific windows into guest memory to the device, but it comes with its own bag of problems, such as dynamic allocations and complicated device <-> iommu connections that need to be represented reliably.

Instead, introduce a simple mechanism that lets a device communicate through its own dedicated shared memory region: Device Memory Buffer. A device that offers Device Memory Buffer exposes a shared memory region, owned by the device, that holds the virtqueues and the buffers they reference.

Once negotiated, an imaginary per-device IOMMU maps the device's IOVA space 1:1 onto the region. Every data address is then an address into it, never a physical or bus address, and the device reaches no other memory to process virtqueues.

The device reports the region's shmid in dmb_shm_id on PCI and DMBSHMId on MMIO, so the shmid does not collide with device-specific or future transport-specific shared memory regions. The driver locates the region through the existing Shared Memory Regions facility, VIRTIO_F_DMB needs a transport that supports SHM.

The device also reports the region's memory type. The only available type currently is 0 (cache coherent).

[1] https://lore.kernel.org/qemu-devel/[email protected]/T/#t

---

v1 -> v2:
- Replace the fixed reserved shmid 2 and its registry with a shmid the device reports per transport, as Stefan Hajnoczi suggested on v1
- Spell out per-transport compatibility for dmb_shm_id and DMBSHMId
- Set FAILED when the driver cannot find the region, rather than refuse to negotiate
- List VIRTIO_F_DMB under CCW's features reserved for future use

v2 -> v3:
- Require the region to be memory the driver shares with the device
- Require coherency; couple VIRTIO_F_DMB to VIRTIO_F_ORDER_PLATFORM, replacing v2's sentence on the areas' interaction with it
- Reserve offset 0 in the region
- Except the Device Memory Buffer from the shared-region prohibition
- Allow a device to expose further shared memory regions
- Move bit 44 into the queue and feature negotiation reserved list
- Aim two cross-references at the Reserved Feature Bits chapter

v3 -> v4:
- Decouple VIRTIO_F_DMB from VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_ORDER_PLATFORM
- Describe the mapping as a per-device IOMMU instead of as offsets into the region (Michael)
- Report the region's memory type in the transport (Michael)
- Let a device require either VIRTIO_F_DMB or VIRTIO_F_ACCESS_PLATFORM (Michael)
- Require the region's base address to be aligned (Michael)
- State the property the mapping must have instead of how to map (Michael)
- State the negotiation condition once per normative section (Michael)
- Rewrite description

 conformance.tex    |   2 +
 content.tex        |  20 +++++-
 shared-mem.tex     | 158 ++++++++++++++++++++++++++++++++++++++++++++-
 transport-ccw.tex  |   1 +
 transport-mmio.tex |  27 ++++++++
 transport-pci.tex  |  33 ++++++++++
 6 files changed, 238 insertions(+), 3 deletions(-)
conformance.tex +2 -0
index 9af31e2..cf4259d 100644
@@ -104,6 +104,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Initialization}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Cleanup}
 \item \ref{drivernormative:Reserved Feature Bits}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device and driver capabilities}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts}
@@ -191,6 +192,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Packed Virtqueues / Scatter-Gather Support}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Shared Memory Regions}
+\item \ref{devicenormative:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
 \item \ref{devicenormative:Reserved Feature Bits}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device and driver capabilities}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects}
content.tex +18 -2
index 243ce2d..27c299b 100644
@@ -102,10 +102,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, 41, 42 and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 40, and 43] Feature bits reserved for extensions to the queue and
+\item[24 to 40, 43 and 44] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}
 
-\item[44 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[45 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}
@@ -946,6 +946,14 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
    suspend the device by set the SUSPEND bit to 1.
    See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
 
+  \item[VIRTIO_F_DMB(44)] This feature indicates that the device offers a
+  Device Memory Buffer: a shared memory region, owned by the device,
+  that holds the virtqueues and the buffers they reference.  When this feature
+  is negotiated, all data DMA operations get routed via a static, imaginary
+  per-device transparent IOMMU which creates a 1:1 mapping of the device's
+  IOVA space and the Device Memory Buffer region.
+  See \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}~\nameref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -982,6 +990,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
 A driver SHOULD accept VIRTIO_F_NOTIF_CONFIG_DATA if it is offered.
 
+For the driver requirements that apply once VIRTIO_F_DMB has been
+negotiated, see
+\ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}~\nameref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
@@ -1011,6 +1023,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 and presents a PCI SR-IOV capability structure, otherwise
 it MUST NOT offer VIRTIO_F_SR_IOV.
 
+For the device requirements that apply once VIRTIO_F_DMB has been
+negotiated, see
+\ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}~\nameref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:
shared-mem.tex +157 -1

162 lines not on this page · raw message

transport-ccw.tex +1 -0
index fe6957d..2c277bf 100644
@@ -610,6 +610,7 @@ \subsection{Features reserved for future use}\label{sec:Virtio Transport Options
 \item VIRTIO_F_ADMIN_VQ
 \item VIRTIO_F_RING_RESET
 \item Shared memory regions including VIRTIO_PMEM_F_SHMEM_REGION
+\item VIRTIO_F_DMB
 
 \end{itemize}
 
transport-mmio.tex +27 -0
index cddbcc9..2b0c3fe 100644
@@ -248,6 +248,21 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
     apply to the queue selected by writing to \field{QueueSel}.
   }
   \hline
+  \mmioreg{DMBSHMId}{Device Memory Buffer shared memory ID}{0x0c4}{R}{%
+    If VIRTIO_F_DMB has been negotiated, reading from this register
+    returns the \field{shmid} of the shared memory region that is the
+    Device Memory Buffer (see \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+    This register is valid only if VIRTIO_F_DMB has been negotiated.
+  }
+  \hline
+  \mmioreg{DMBMemType}{Device Memory Buffer memory type}{0x0c8}{R}{%
+    Reading from this register returns the memory type of the Device
+    Memory Buffer region (see \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+    This register is valid whenever the device offers VIRTIO_F_DMB,
+    because the driver reads it in order to decide whether to accept
+    the feature.
+  }
+  \hline
   \mmioreg{ConfigGeneration}{Configuration atomicity value}{0x0fc}{R}{
     Reading from this register returns a value describing a version of the device-specific configuration space (see \field{Config}).
     The driver can then access the configuration space and, when finished, read \field{ConfigGeneration} again.
@@ -313,6 +328,12 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
 when queue reset has completed.
 (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
 
+If the device offers VIRTIO_F_DMB, \field{DMBSHMId} MUST equal the
+\field{shmid} of a shared memory region that the device exposes through
+\field{SHMSel}, \field{SHMLenLow}, \field{SHMLenHigh}, \field{SHMBaseLow}
+and \field{SHMBaseHigh}, and \field{DMBMemType} MUST report a memory type
+that this specification defines.
+
 \drivernormative{\subsubsection}{MMIO Device Register Layout}{Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}
 The driver MUST NOT access memory locations not described in the
 table \ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}
@@ -367,6 +388,12 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
 that were used before the queue reset.
 (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
 
+The driver MUST NOT read \field{DMBSHMId} unless VIRTIO_F_DMB has been
+negotiated.
+
+The driver MUST NOT read \field{DMBMemType} unless the device offers
+VIRTIO_F_DMB.
+
 \subsection{MMIO-specific Initialization And Device Operation}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation}
 
 \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization}
transport-pci.tex +33 -0
index 95b08b8..71c2b64 100644
@@ -325,6 +325,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         /* About the administration virtqueue. */
         le16 admin_queue_index;         /* read-only for driver */
         le16 admin_queue_num;         /* read-only for driver */
+
+        /* About the Device Memory Buffer. */
+        le16 dmb_shm_id;                /* read-only for driver */
+        le16 dmb_mem_type;              /* read-only for driver */
 };
 \end{lstlisting}
 
@@ -428,6 +432,22 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 	The value 0 indicates no supported administration virtqueues.
 	This field is valid only if VIRTIO_F_ADMIN_VQ has been
 	negotiated.
+
+\item[\field{dmb_shm_id}]
+        The device uses this to report the \field{shmid} of the shared
+        memory region that is the Device Memory Buffer (see
+        \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+        This field is read-only for the driver.  This field is valid
+        only if VIRTIO_F_DMB has been negotiated.
+
+\item[\field{dmb_mem_type}]
+        The device uses this to report the memory type of the Device
+        Memory Buffer region (see
+        \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+        This field is read-only for the driver.  Unlike
+        \field{dmb_shm_id}, this field is valid whenever the device
+        offers VIRTIO_F_DMB, because the driver reads it in order to
+        decide whether to accept the feature.
 \end{description}
 
 \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -495,6 +515,13 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 to ensure that indices of valid admin queues fit into
 a 16 bit range beyond all other virtqueues.
 
+If the device offers VIRTIO_F_DMB, the device MUST present the common
+configuration structure with a \field{length} that is large enough to
+include the \field{dmb_mem_type} field, and \field{dmb_shm_id} MUST equal
+the \field{id} of a VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capability that the
+device presents.  Since \field{id} in \field{virtio_pci_cap} is 8 bits,
+the high-order 8 bits of \field{dmb_shm_id} MUST be zero.
+
 \drivernormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
 
 The driver MUST NOT write to \field{device_feature}, \field{num_queues},
@@ -531,6 +558,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 The driver MAY configure fewer administration virtqueues than
 supported by the device.
 
+The driver MUST NOT read \field{dmb_shm_id} unless VIRTIO_F_DMB has
+been negotiated.
+
+The driver MUST NOT read \field{dmb_mem_type} unless the device offers
+VIRTIO_F_DMB.
+
 \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 
 The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
Signature · 1 line
2.47.1
#1 lore raw
Quoted 38 lines · Alexander Graf

Virtio devices assume R/W access of guest RAM to perform DMA into it to access its virt queues and data to operate on. This basic assumption leads to multiple problems:

Confidential Compute - In SEV-SNP or TDX, the host has no direct access
to all of guest RAM. We work around DMA access by having the guest limit DMA access to a common shared memory window between guest and host (swiotlb).

Vhost-user - vhost-user backends require access to all guest memory, not
only the memory they are actually supposed to access. This is fine for trusted backends, but if you want to limit visibility into guest, you need to do weird stunts like bouncing memory in the vmm [1].

To restrict DMA visibility, virtio-iommu allows the guest to open specific windows into guest memory to the device, but it comes with its own bag of problems, such as dynamic allocations and complicated device <-> iommu connections that need to be represented reliably.

Instead, introduce a simple mechanism that lets a device communicate through its own dedicated shared memory region: Device Memory Buffer. A device that offers Device Memory Buffer exposes a shared memory region, owned by the device, that holds the virtqueues and the buffers they reference.

Once negotiated, an imaginary per-device IOMMU maps the device's IOVA space 1:1 onto the region. Every data address is then an address into it, never a physical or bus address, and the device reaches no other memory to process virtqueues.

The device reports the region's shmid in dmb_shm_id on PCI and DMBSHMId on MMIO, so the shmid does not collide with device-specific or future transport-specific shared memory regions. The driver locates the region through the existing Shared Memory Regions facility, VIRTIO_F_DMB needs a transport that supports SHM.

Quoted 8 lines

The device also reports the region's memory type. The only available type currently is 0 (cache coherent).

[1] https://lore.kernel.org/qemu-devel/[email protected]/T/#t

I like the "internal iommu" idea. addresses my concerns nicely!

Quoted 84 lines

---

v1 -> v2:
- Replace the fixed reserved shmid 2 and its registry with a shmid the device reports per transport, as Stefan Hajnoczi suggested on v1
- Spell out per-transport compatibility for dmb_shm_id and DMBSHMId
- Set FAILED when the driver cannot find the region, rather than refuse to negotiate
- List VIRTIO_F_DMB under CCW's features reserved for future use

v2 -> v3:
- Require the region to be memory the driver shares with the device
- Require coherency; couple VIRTIO_F_DMB to VIRTIO_F_ORDER_PLATFORM, replacing v2's sentence on the areas' interaction with it
- Reserve offset 0 in the region
- Except the Device Memory Buffer from the shared-region prohibition
- Allow a device to expose further shared memory regions
- Move bit 44 into the queue and feature negotiation reserved list
- Aim two cross-references at the Reserved Feature Bits chapter

v3 -> v4:
- Decouple VIRTIO_F_DMB from VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_ORDER_PLATFORM
- Describe the mapping as a per-device IOMMU instead of as offsets into the region (Michael)
- Report the region's memory type in the transport (Michael)
- Let a device require either VIRTIO_F_DMB or VIRTIO_F_ACCESS_PLATFORM (Michael)
- Require the region's base address to be aligned (Michael)
- State the property the mapping must have instead of how to map (Michael)
- State the negotiation condition once per normative section (Michael)
- Rewrite description

 conformance.tex    |   2 +
 content.tex        |  20 +++++-
 shared-mem.tex     | 158 ++++++++++++++++++++++++++++++++++++++++++++-
 transport-ccw.tex  |   1 +
 transport-mmio.tex |  27 ++++++++
 transport-pci.tex  |  33 ++++++++++
 6 files changed, 238 insertions(+), 3 deletions(-)
conformance.tex +2 -0
index 9af31e2..cf4259d 100644
@@ -104,6 +104,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Initialization}
 \item \ref{drivernormative:General Initialization And Device Operation / Device Cleanup}
 \item \ref{drivernormative:Reserved Feature Bits}
+\item \ref{drivernormative:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device and driver capabilities}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects}
 \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts}
@@ -191,6 +192,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Packed Virtqueues / Scatter-Gather Support}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Shared Memory Regions}
+\item \ref{devicenormative:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
 \item \ref{devicenormative:Reserved Feature Bits}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device and driver capabilities}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects}
content.tex +2 -2
index 243ce2d..27c299b 100644
@@ -102,10 +102,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, 41, 42 and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 40, and 43] Feature bits reserved for extensions to the queue and
+\item[24 to 40, 43 and 44] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}
 
-\item[44 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[45 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 954

Quoted 9 lines
content.tex +6 -0
@@ -946,6 +946,14 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
    suspend the device by set the SUSPEND bit to 1.
    See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
 
+  \item[VIRTIO_F_DMB(44)] This feature indicates that the device offers a
+  Device Memory Buffer: a shared memory region, owned by the device,
+  that holds the virtqueues and the buffers they reference.  When this feature
+  is negotiated, all data DMA operations get routed via a static, imaginary
+  per-device transparent IOMMU which creates a 1:1 mapping of the device's
+  IOVA space

DMA address space

Quoted 50 lines

and the Device Memory Buffer region.

content.tex +10 -0
+  See \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}~\nameref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -982,6 +990,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
 A driver SHOULD accept VIRTIO_F_NOTIF_CONFIG_DATA if it is offered.
 
+For the driver requirements that apply once VIRTIO_F_DMB has been
+negotiated, see
+\ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}~\nameref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
@@ -1011,6 +1023,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 and presents a PCI SR-IOV capability structure, otherwise
 it MUST NOT offer VIRTIO_F_SR_IOV.
 
+For the device requirements that apply once VIRTIO_F_DMB has been
+negotiated, see
+\ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}~\nameref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:
shared-mem.tex +13 -1
index 6e6f6c4..405745d 100644
@@ -38,5 +38,161 @@ \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De
 Device / Shared Memory Regions}
 Shared memory regions MUST NOT expose shared memory regions which
 are used to control the operation of the device, nor to stream
-data.
+data, other than the Device Memory Buffer when VIRTIO_F_DMB has been
+negotiated (see
+\ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+
+\subsection{Device Memory Buffer}\label{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
+
+When VIRTIO_F_DMB (see \ref{sec:Reserved Feature Bits}) is
+negotiated, the device offers a shared memory region, the
+\emph{Device Memory Buffer}, identified by a \field{shmid} that the
+device reports through a transport-specific mechanism (for example
+\field{dmb_shm_id} in the common configuration structure on the PCI
+transport, see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout},
+and \field{DMBSHMId} on the MMIO transport, see

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 63

Quoted 10 lines
shared-mem.tex +10 -0
+\ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout};
+see the transport chapters).
+The driver reads that \field{shmid} and locates the region using the
+transport-specific shared memory region enumeration mechanism, in the
+same way as any other shared memory region.  Because both the
+\field{shmid} report and that enumeration are transport-specific,
+VIRTIO_F_DMB is available only on transports that define a shared memory
+region discovery mechanism.
+
+When VIRTIO_F_DMB is active, all data DMA operations (not IRQ logic) get

(excluding notifications)?

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 65

Quoted 2 lines
shared-mem.tex +2 -0
+routed via a static, imaginary per-device transparent IOMMU which
+creates a 1:1 mapping of the device's IOVA space

DMA address space

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 5 lines

and the DMB SHM region.

shared-mem.tex +4 -0
+The virtio device then performs all data access against the DMB
+region and no longer against the transport bus's DMA address space.
+
+With VIRTIO_F_DMB active,

negotiated

Quoted 1 line

address 0 is reserved. Drivers

and devices

Quoted 1 line

can

internally

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 79

Quoted 11 lines

use address 0

shared-mem.tex +10 -0
+for internal semantics, such as "not in use" or "failed allocation".
+
+VIRTIO_F_DMB is independent of VIRTIO_F_ACCESS_PLATFORM.  A device MAY
+offer both, and MAY require one of them, but neither implies the other.
+
+VIRTIO_F_DMB does not change the memory ordering requirements for
+the Descriptor, Driver and Device Areas. Those ordering requirements must
+continue to align between the driver and the device observing the Device
+Memory Buffer. Which barriers the driver emits is governed by
+VIRTIO_F_ORDER_PLATFORM as it is for any other device (see \ref{sec:Reserved Feature Bits}).

"as it is for any other device" is confusing and useless.

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 3 lines
shared-mem.tex +3 -0
+
+To allow a driver to understand whether the DMB backing SHM region is cache
+coherent,

in the future we'll presumably have more types?

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 5 lines

the device reports the memory type of the DMB region through a

shared-mem.tex +4 -0
+transport-specific mechanism.  Currently, the following types are defined:
+
+\begin{description}
+\item[0] The region is cache coherent.

between the device and the driver?

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 4 lines

A write that the driver performs to

shared-mem.tex +3 -0
+  the region becomes visible to the device, and a write that the device
+  performs to the region becomes visible to the driver, without any cache
+  maintenance operation

operations.

Quoted 1 line

by the driver.

by either.

But it's still confusing. First, it really depends on how driver mapped the memory. For example i think I can map any memory uncached and viola it works.

Second this also might depend on the ordering flag thinkably? If it does not, say so.

Third there's ordering of this
wrt notifications since you excluded them.

Quoted 1 line

+\end{description}

It's confusing here. It is not clear what this 0 is unless you just read the patch.

Alternatively, if you want to use "Device Buffer memory type" or DMB memory type" or whatever make it
very prominent, like a subsection of its own and use consistently not sometimes like this sometimes " the memory type of the DMB region" etc.

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 95

Quoted 7 lines
shared-mem.tex +4 -0
+All other values are reserved.
 
 
 
+
+The region's length is reported by the shared memory region descriptor and
+bounds the amount of virtqueue data that can be in flight at once.

no idea what this means and what virtqueue data is

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 101

Quoted 6 lines
shared-mem.tex +6 -0
+
+\drivernormative{\subsubsection}{Device Memory Buffer}{Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
+
+The driver MUST NOT accept VIRTIO_F_DMB unless the memory type that the
+device reports through the transport-specific mechanism is a value that this
+specification defines and that the driver supports.

i am not sure this will work well for extensions. what if three are several ways to map for example? I would just drop it I think unless you can show e.g. in the commit log some other possible types and how this will be extensible.

Also generally we give detailed text without MUST etc in non conformance sections and then with a bit of duplication put conformance parts in a separate coformance sections.

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 104

Quoted 3 lines
shared-mem.tex +3 -0
+
+The requirements in the remainder of this section apply once VIRTIO_F_DMB
+has been negotiated.

if

Quoted 23 lines
shared-mem.tex +23 -0
+
+The driver MUST read the \field{shmid} that the device reports through the
+transport-specific mechanism and MUST locate that shared memory region using
+the transport-specific shared memory region enumeration mechanism.  If the
+driver cannot locate the region, it MUST set the FAILED
+\field{device status} bit.
+
+The driver MUST place the Descriptor Area, Driver Area and Device Area of
+every virtqueue within the Device Memory Buffer region, and MUST treat all
+addresses as 1:1 IOMMU mapped into the Device Memory Buffer region.
+The driver MUST choose each area's address such that the address satisfies
+the alignment that the virtqueue layout in use requires of that area.
+
+For every buffer that the driver makes available to the device, and for
+every indirect descriptor table if VIRTIO_F_INDIRECT_DESC has been
+negotiated, the driver MUST place it within the Device Memory Buffer region
+and MUST treat all addresses as 1:1 IOMMU mapped into the Device Memory
+Buffer region.
+
+The driver MUST NOT supply an address that, together with the length of the
+structure it refers to, lies outside the Device Memory Buffer region, and
+MUST NOT make available a buffer that does not fit within the region.
+

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 136

Quoted 10 lines
shared-mem.tex +9 -0
+The driver MUST NOT supply an address of 0 as the address of a Descriptor
+Area, Driver Area or Device Area, as the \field{addr} field of a descriptor,
+or as the address of an indirect descriptor table.
+
+The driver MUST access the Device Memory Buffer region through a mapping by
+which a write the driver performs to the region becomes visible to the
+device, and a write the device performs to the region becomes visible to the
+driver.
 How the driver establishes such a mapping is up to the driver and
+the platform.

what does this mean?

Quoted 19 lines

+\devicenormative{\subsubsection}{Device Memory Buffer}{Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}

shared-mem.tex +16 -0
+
 
 
+A device MUST NOT offer VIRTIO_F_DMB unless the transport provides a
+shared memory region discovery mechanism.
+
+A device MAY require that the driver accept VIRTIO_F_DMB, that the driver
+accept VIRTIO_F_ACCESS_PLATFORM, or that the driver accept at least one of
+the two.  A device that requires at least one of the two MUST fail to set
+the FEATURES_OK \field{device status} bit when the driver writes it having
+accepted neither.
+
+A device MUST NOT offer VIRTIO_F_DMB unless the platform permits the driver to
+access the Device Memory Buffer region as memory that is shared with the
+device.
+
+A device that offers VIRTIO_F_DMB MUST report the memory type of the Device
+Memory Buffer region through the transport-specific mechanism, and MUST

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 164

Quoted 10 lines
shared-mem.tex +10 -0
+report a value that this specification defines.
+
+A device that reports memory type 0 MUST ensure that a write that the driver
+performs to the Device Memory Buffer region becomes visible to the device,
+and a write that the device performs to the region becomes visible to the
+driver, without any cache maintenance operation by the driver.
+
+A device that offers VIRTIO_F_DMB MUST expose exactly one shared memory
+region for the Device Memory Buffer and MUST report that region's
+\field{shmid} through the transport-specific mechanism.

I think this is better in a transport specific section.

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 168

Quoted 4 lines
shared-mem.tex +4 -0
+
+A device that offers VIRTIO_F_DMB MUST report a Device Memory Buffer region
+whose base address is aligned to at least the largest alignment that any
+virtqueue layout requires of an area.

kinda vague. and what is "any virtqueue layout" anyway?

Basically 16 bytes?

shared-mem.tex

Quoted 2 lines
shared-mem.tex +2 -0
+
+A device that offers VIRTIO_F_DMB MAY expose

can expose

Quoted 2 lines

further shared memory regions in
+addition to the Device Memory Buffer;

u call this soemtimes dmb region sometimes just dmb. pls be consistent.

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 180

Quoted 10 lines

the Device Memory Buffer is the one

shared-mem.tex +9 -0
+region whose \field{shmid} the device reports through the transport-specific
+mechanism.
+
+The remaining requirements in this section apply once VIRTIO_F_DMB has been
+negotiated.
+
+The device MUST interpret every data address as an address into the Device
+Memory Buffer. It MUST NOT interpret any such address as a physical or bus
+address.

pls go back to the internal iommu terminology.

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 186

Quoted 5 lines
shared-mem.tex +5 -0
+Before accessing any structure that a driver-supplied address refers to, the
+device MUST verify that the address together with the length of that
+structure lies within the Device Memory Buffer region and does not overflow.
+If this verification fails, the device MUST NOT perform the access and MUST
+set the DEVICE_NEEDS_RESET \field{device status} bit.

and here

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 194

Quoted 8 lines
shared-mem.tex +8 -0
+
+If the driver supplies an address of 0 as the address of a Descriptor Area,
+Driver Area or Device Area, the device MUST NOT access that area, and MUST
+set the DEVICE_NEEDS_RESET \field{device status} bit.  If the driver
+supplies an address of 0 as the \field{addr} field of a descriptor or as the
+address of an indirect descriptor table, the device SHOULD NOT access the
+structure that address refers to, and SHOULD set the DEVICE_NEEDS_RESET
+\field{device status} bit.

so just protect fist 16 bytes in the iommu?

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 197

Quoted 3 lines
shared-mem.tex +3 -0
+
+The device MUST NOT access any memory outside the Device Memory Buffer
+region in order to process virtqueues.

again, iommu

Quoted 21 lines
transport-ccw.tex +1 -0
index fe6957d..2c277bf 100644
@@ -610,6 +610,7 @@ \subsection{Features reserved for future use}\label{sec:Virtio Transport Options
 \item VIRTIO_F_ADMIN_VQ
 \item VIRTIO_F_RING_RESET
 \item Shared memory regions including VIRTIO_PMEM_F_SHMEM_REGION
+\item VIRTIO_F_DMB
 
 \end{itemize}
 
transport-mmio.tex +0 -0
index cddbcc9..2b0c3fe 100644
@@ -248,6 +248,21 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
     apply to the queue selected by writing to \field{QueueSel}.
   }
   \hline

transport-mmio.tex \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi · line 260

Quoted 10 lines
transport-mmio.tex +10 -0
+  \mmioreg{DMBSHMId}{Device Memory Buffer shared memory ID}{0x0c4}{R}{%
+    If VIRTIO_F_DMB has been negotiated, reading from this register
+    returns the \field{shmid} of the shared memory region that is the
+    Device Memory Buffer (see \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+    This register is valid only if VIRTIO_F_DMB has been negotiated.
+  }
+  \hline
+  \mmioreg{DMBMemType}{Device Memory Buffer memory type}{0x0c8}{R}{%
+    Reading from this register returns the memory type of the Device
+    Memory Buffer region (see \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).

again inconsistent naming

transport-mmio.tex \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi · line 263

Quoted 3 lines
transport-mmio.tex +3 -0
+    This register is valid whenever the device offers VIRTIO_F_DMB,
+    because the driver reads it in order to decide whether to accept
+    the feature.

I would add (as opposed to after the feature has been negotiated)

Quoted 98 lines
transport-mmio.tex +14 -0
+  }
+  \hline
   \mmioreg{ConfigGeneration}{Configuration atomicity value}{0x0fc}{R}{
     Reading from this register returns a value describing a version of the device-specific configuration space (see \field{Config}).
     The driver can then access the configuration space and, when finished, read \field{ConfigGeneration} again.
@@ -313,6 +328,12 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
 when queue reset has completed.
 (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
 
+If the device offers VIRTIO_F_DMB, \field{DMBSHMId} MUST equal the
+\field{shmid} of a shared memory region that the device exposes through
+\field{SHMSel}, \field{SHMLenLow}, \field{SHMLenHigh}, \field{SHMBaseLow}
+and \field{SHMBaseHigh}, and \field{DMBMemType} MUST report a memory type
+that this specification defines.
+
 \drivernormative{\subsubsection}{MMIO Device Register Layout}{Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}
 The driver MUST NOT access memory locations not described in the
 table \ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}
@@ -367,6 +388,12 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
 that were used before the queue reset.
 (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}).
 
+The driver MUST NOT read \field{DMBSHMId} unless VIRTIO_F_DMB has been
+negotiated.
+
+The driver MUST NOT read \field{DMBMemType} unless the device offers
+VIRTIO_F_DMB.
+
 \subsection{MMIO-specific Initialization And Device Operation}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation}
 
 \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization}
transport-pci.tex +33 -0
index 95b08b8..71c2b64 100644
@@ -325,6 +325,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         /* About the administration virtqueue. */
         le16 admin_queue_index;         /* read-only for driver */
         le16 admin_queue_num;         /* read-only for driver */
+
+        /* About the Device Memory Buffer. */
+        le16 dmb_shm_id;                /* read-only for driver */
+        le16 dmb_mem_type;              /* read-only for driver */
 };
 \end{lstlisting}
 
@@ -428,6 +432,22 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 	The value 0 indicates no supported administration virtqueues.
 	This field is valid only if VIRTIO_F_ADMIN_VQ has been
 	negotiated.
+
+\item[\field{dmb_shm_id}]
+        The device uses this to report the \field{shmid} of the shared
+        memory region that is the Device Memory Buffer (see
+        \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+        This field is read-only for the driver.  This field is valid
+        only if VIRTIO_F_DMB has been negotiated.
+
+\item[\field{dmb_mem_type}]
+        The device uses this to report the memory type of the Device
+        Memory Buffer region (see
+        \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+        This field is read-only for the driver.  Unlike
+        \field{dmb_shm_id}, this field is valid whenever the device
+        offers VIRTIO_F_DMB, because the driver reads it in order to
+        decide whether to accept the feature.
 \end{description}
 
 \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -495,6 +515,13 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 to ensure that indices of valid admin queues fit into
 a 16 bit range beyond all other virtqueues.
 
+If the device offers VIRTIO_F_DMB, the device MUST present the common
+configuration structure with a \field{length} that is large enough to
+include the \field{dmb_mem_type} field, and \field{dmb_shm_id} MUST equal
+the \field{id} of a VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capability that the
+device presents.  Since \field{id} in \field{virtio_pci_cap} is 8 bits,
+the high-order 8 bits of \field{dmb_shm_id} MUST be zero.
+
 \drivernormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
 
 The driver MUST NOT write to \field{device_feature}, \field{num_queues},
@@ -531,6 +558,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 The driver MAY configure fewer administration virtqueues than
 supported by the device.
 
+The driver MUST NOT read \field{dmb_shm_id} unless VIRTIO_F_DMB has
+been negotiated.
+
+The driver MUST NOT read \field{dmb_mem_type} unless the device offers
+VIRTIO_F_DMB.
+
 \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 
 The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
Signature · 1 line
2.47.1
#2 lore raw
Quoted 42 lines · Michael S. Tsirkin
Quoted 41 lines · Alexander Graf

Virtio devices assume R/W access of guest RAM to perform DMA into it to access its virt queues and data to operate on. This basic assumption leads to multiple problems:

Confidential Compute - In SEV-SNP or TDX, the host has no direct access
to all of guest RAM. We work around DMA access by having the guest limit DMA access to a common shared memory window between guest and host (swiotlb).

Vhost-user - vhost-user backends require access to all guest memory, not
only the memory they are actually supposed to access. This is fine for trusted backends, but if you want to limit visibility into guest, you need to do weird stunts like bouncing memory in the vmm [1].

To restrict DMA visibility, virtio-iommu allows the guest to open specific windows into guest memory to the device, but it comes with its own bag of problems, such as dynamic allocations and complicated device <-> iommu connections that need to be represented reliably.

Instead, introduce a simple mechanism that lets a device communicate through its own dedicated shared memory region: Device Memory Buffer. A device that offers Device Memory Buffer exposes a shared memory region, owned by the device, that holds the virtqueues and the buffers they reference.

Once negotiated, an imaginary per-device IOMMU maps the device's IOVA space 1:1 onto the region. Every data address is then an address into it, never a physical or bus address, and the device reaches no other memory to process virtqueues.

The device reports the region's shmid in dmb_shm_id on PCI and DMBSHMId on MMIO, so the shmid does not collide with device-specific or future transport-specific shared memory regions. The driver locates the region through the existing Shared Memory Regions facility, VIRTIO_F_DMB needs a transport that supports SHM.

The device also reports the region's memory type. The only available type currently is 0 (cache coherent).

Quoted 6 lines
Quoted 5 lines

[1] https://lore.kernel.org/qemu-devel/[email protected]/T/#t

I like the "internal iommu" idea. addresses my concerns nicely!

Woohoo! :)

[...]

Quoted 11 lines

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 954

Quoted 10 lines
content.tex +6 -0
@@ -946,6 +946,14 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
     suspend the device by set the SUSPEND bit to 1.
     See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
 
+  \item[VIRTIO_F_DMB(44)] This feature indicates that the device offers a
+  Device Memory Buffer: a shared memory region, owned by the device,
+  that holds the virtqueues and the buffers they reference.  When this feature
+  is negotiated, all data DMA operations get routed via a static, imaginary
+  per-device transparent IOMMU which creates a 1:1 mapping of the device's
+  IOVA space

DMA address space

The precedence in the doc is "bus address space". I'll use that one instead to stay consistent.

[...]

Quoted 25 lines
Quoted 25 lines
shared-mem.tex +16 -1
index 6e6f6c4..405745d 100644
@@ -38,5 +38,161 @@ \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De
  Device / Shared Memory Regions}
  Shared memory regions MUST NOT expose shared memory regions which
  are used to control the operation of the device, nor to stream
-data.
+data, other than the Device Memory Buffer when VIRTIO_F_DMB has been
+negotiated (see
+\ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+
+\subsection{Device Memory Buffer}\label{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
+
+When VIRTIO_F_DMB (see \ref{sec:Reserved Feature Bits}) is
+negotiated, the device offers a shared memory region, the
+\emph{Device Memory Buffer}, identified by a \field{shmid} that the
+device reports through a transport-specific mechanism (for example
+\field{dmb_shm_id} in the common configuration structure on the PCI
+transport, see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout},
+and \field{DMBSHMId} on the MMIO transport, see
+\ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout};
+see the transport chapters).
+The driver reads that \field{shmid} and locates the region using the
Quoted 8 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 63

Quoted 7 lines
shared-mem.tex +7 -0
+transport-specific shared memory region enumeration mechanism, in the
+same way as any other shared memory region.  Because both the
+\field{shmid} report and that enumeration are transport-specific,
+VIRTIO_F_DMB is available only on transports that define a shared memory
+region discovery mechanism.
+
+When VIRTIO_F_DMB is active, all data DMA operations (not IRQ logic) get

(excluding notifications)?

Good catch. Reworded.

Quoted 18 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 65

Quoted 2 lines
shared-mem.tex +2 -0
+routed via a static, imaginary per-device transparent IOMMU which
+creates a 1:1 mapping of the device's IOVA space

DMA address space

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 5 lines

and the DMB SHM region.

shared-mem.tex +4 -0
+The virtio device then performs all data access against the DMB
+region and no longer against the transport bus's DMA address space.
+
+With VIRTIO_F_DMB active,

negotiated

Quoted 1 line

address 0 is reserved. Drivers

and devices

Quoted 1 line

can

internally

Quoted 14 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 79

Quoted 11 lines

use address 0

shared-mem.tex +10 -0
+for internal semantics, such as "not in use" or "failed allocation".
+
+VIRTIO_F_DMB is independent of VIRTIO_F_ACCESS_PLATFORM.  A device MAY
+offer both, and MAY require one of them, but neither implies the other.
+
+VIRTIO_F_DMB does not change the memory ordering requirements for
+the Descriptor, Driver and Device Areas. Those ordering requirements must
+continue to align between the driver and the device observing the Device
+Memory Buffer. Which barriers the driver emits is governed by
+VIRTIO_F_ORDER_PLATFORM as it is for any other device (see \ref{sec:Reserved Feature Bits}).

"as it is for any other device" is confusing and useless.

Ack on all the above

Quoted 5 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 4 lines
shared-mem.tex +3 -0
+
+To allow a driver to understand whether the DMB backing SHM region is cache
+coherent,

in the future we'll presumably have more types?

We discussed this in earlier versions: If you want to build this as a way to use virtio on systems without DMA (which do exist, there are some funky PCI implementations out there), we can define a WC mode for example which then would allow not-super-fast, but still workable I/O in such environments.

I just don't really want to deal with defining all of the nuances of such a mode right now. IMHO that's best done when we actually cross that bridge and someone can validate that what they put into the spec actually works in real life.

Quoted 13 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 5 lines

the device reports the memory type of the DMB region through a

shared-mem.tex +4 -0
+transport-specific mechanism.  Currently, the following types are defined:
+
+\begin{description}
+\item[0] The region is cache coherent.

between the device and the driver?

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 4 lines

A write that the driver performs to

shared-mem.tex +3 -0
+  the region becomes visible to the device, and a write that the device
+  performs to the region becomes visible to the driver, without any cache
+  maintenance operation

operations.

Quoted 7 lines
Quoted 1 line

by the driver.

by either.

But it's still confusing. First, it really depends on how driver mapped the memory. For example i think I can map any memory uncached and viola it works.

That depends. On older ARM systems, if you map uncached, the host (which has it marked as cached) would not be able to see it. On x86 and modern server class ARM systems, mapping uncached would work, yes.

Quoted 8 lines

Second this also might depend on the ordering flag thinkably? If it does not, say so.

Third there's ordering of this
wrt notifications since you excluded them.

Quoted 1 line

+\end{description}

Quoted 8 lines

It's confusing here. It is not clear what this 0 is unless you just read the patch.

Alternatively, if you want to use "Device Buffer memory type" or DMB memory type" or whatever make it
very prominent, like a subsection of its own and use consistently not sometimes like this sometimes " the memory type of the DMB region" etc.

Great idea!

Quoted 8 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 95

Quoted 7 lines
shared-mem.tex +4 -0
+All other values are reserved.
 
 
+
+The region's length is reported by the shared memory region descriptor and
+bounds the amount of virtqueue data that can be in flight at once.

no idea what this means and what virtqueue data is

Clarified

Quoted 11 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 101

Quoted 7 lines
shared-mem.tex +6 -0
+
+\drivernormative{\subsubsection}{Device Memory Buffer}{Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
+
+The driver MUST NOT accept VIRTIO_F_DMB unless the memory type that the
+device reports through the transport-specific mechanism is a value that this
+specification defines and that the driver supports.

i am not sure this will work well for extensions. what if three are several ways to map for example? I would just drop it I think unless you can show e.g. in the commit log some other possible types and how this will be extensible.

If there are several ways to map, we can make the field become a bitmap with special meaning that 0 means "writeback" :).

I don't think this will realistically happen. The host will want to tell the guest the best caching mode to use at all times. Anyone with sensible hardware will want writeback. But if we want to keep this generic (which is a sensible idea IMHO), it's good to keep at least the field reserved that allows us to indicate different semantics if we need a different caching mode.

Quoted 3 lines

Also generally we give detailed text without MUST etc in non conformance sections and then with a bit of duplication put conformance parts in a separate coformance sections.

This is the conformance section, no? Or do you mean that we should mention the above as well in the normal section?

Let's bounce this spec update once more. I'll also post a new version of the Linux patches.

Alex

#3 lore raw
Quoted 164 lines · Alexander
Quoted 42 lines · Michael S. Tsirkin
Quoted 41 lines · Alexander Graf

Virtio devices assume R/W access of guest RAM to perform DMA into it to access its virt queues and data to operate on. This basic assumption leads to multiple problems:

Confidential Compute - In SEV-SNP or TDX, the host has no direct access
to all of guest RAM. We work around DMA access by having the guest limit DMA access to a common shared memory window between guest and host (swiotlb).

Vhost-user - vhost-user backends require access to all guest memory, not
only the memory they are actually supposed to access. This is fine for trusted backends, but if you want to limit visibility into guest, you need to do weird stunts like bouncing memory in the vmm [1].

To restrict DMA visibility, virtio-iommu allows the guest to open specific windows into guest memory to the device, but it comes with its own bag of problems, such as dynamic allocations and complicated device <-> iommu connections that need to be represented reliably.

Instead, introduce a simple mechanism that lets a device communicate through its own dedicated shared memory region: Device Memory Buffer. A device that offers Device Memory Buffer exposes a shared memory region, owned by the device, that holds the virtqueues and the buffers they reference.

Once negotiated, an imaginary per-device IOMMU maps the device's IOVA space 1:1 onto the region. Every data address is then an address into it, never a physical or bus address, and the device reaches no other memory to process virtqueues.

The device reports the region's shmid in dmb_shm_id on PCI and DMBSHMId on MMIO, so the shmid does not collide with device-specific or future transport-specific shared memory regions. The driver locates the region through the existing Shared Memory Regions facility, VIRTIO_F_DMB needs a transport that supports SHM.

The device also reports the region's memory type. The only available type currently is 0 (cache coherent).

Quoted 6 lines
Quoted 5 lines

[1] https://lore.kernel.org/qemu-devel/[email protected]/T/#t

I like the "internal iommu" idea. addresses my concerns nicely!

Woohoo! :)

[...]

Quoted 11 lines

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 954

Quoted 10 lines
content.tex +6 -0
@@ -946,6 +946,14 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
     suspend the device by set the SUSPEND bit to 1.
     See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
 
+  \item[VIRTIO_F_DMB(44)] This feature indicates that the device offers a
+  Device Memory Buffer: a shared memory region, owned by the device,
+  that holds the virtqueues and the buffers they reference.  When this feature
+  is negotiated, all data DMA operations get routed via a static, imaginary
+  per-device transparent IOMMU which creates a 1:1 mapping of the device's
+  IOVA space

DMA address space

The precedence in the doc is "bus address space". I'll use that one instead to stay consistent.

[...]

Quoted 25 lines
Quoted 25 lines
shared-mem.tex +16 -1
index 6e6f6c4..405745d 100644
@@ -38,5 +38,161 @@ \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De
  Device / Shared Memory Regions}
  Shared memory regions MUST NOT expose shared memory regions which
  are used to control the operation of the device, nor to stream
-data.
+data, other than the Device Memory Buffer when VIRTIO_F_DMB has been
+negotiated (see
+\ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}).
+
+\subsection{Device Memory Buffer}\label{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
+
+When VIRTIO_F_DMB (see \ref{sec:Reserved Feature Bits}) is
+negotiated, the device offers a shared memory region, the
+\emph{Device Memory Buffer}, identified by a \field{shmid} that the
+device reports through a transport-specific mechanism (for example
+\field{dmb_shm_id} in the common configuration structure on the PCI
+transport, see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout},
+and \field{DMBSHMId} on the MMIO transport, see
+\ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout};
+see the transport chapters).
+The driver reads that \field{shmid} and locates the region using the
Quoted 8 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 63

Quoted 7 lines
shared-mem.tex +7 -0
+transport-specific shared memory region enumeration mechanism, in the
+same way as any other shared memory region.  Because both the
+\field{shmid} report and that enumeration are transport-specific,
+VIRTIO_F_DMB is available only on transports that define a shared memory
+region discovery mechanism.
+
+When VIRTIO_F_DMB is active, all data DMA operations (not IRQ logic) get

(excluding notifications)?

Good catch. Reworded.

Quoted 18 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 65

Quoted 2 lines
shared-mem.tex +2 -0
+routed via a static, imaginary per-device transparent IOMMU which
+creates a 1:1 mapping of the device's IOVA space

DMA address space

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 5 lines

and the DMB SHM region.

shared-mem.tex +4 -0
+The virtio device then performs all data access against the DMB
+region and no longer against the transport bus's DMA address space.
+
+With VIRTIO_F_DMB active,

negotiated

Quoted 1 line

address 0 is reserved. Drivers

and devices

Quoted 1 line

can

internally

Quoted 14 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 79

Quoted 11 lines

use address 0

shared-mem.tex +10 -0
+for internal semantics, such as "not in use" or "failed allocation".
+
+VIRTIO_F_DMB is independent of VIRTIO_F_ACCESS_PLATFORM.  A device MAY
+offer both, and MAY require one of them, but neither implies the other.
+
+VIRTIO_F_DMB does not change the memory ordering requirements for
+the Descriptor, Driver and Device Areas. Those ordering requirements must
+continue to align between the driver and the device observing the Device
+Memory Buffer. Which barriers the driver emits is governed by
+VIRTIO_F_ORDER_PLATFORM as it is for any other device (see \ref{sec:Reserved Feature Bits}).

"as it is for any other device" is confusing and useless.

Ack on all the above

Quoted 5 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 4 lines
shared-mem.tex +3 -0
+
+To allow a driver to understand whether the DMB backing SHM region is cache
+coherent,

in the future we'll presumably have more types?

We discussed this in earlier versions: If you want to build this as a way to use virtio on systems without DMA (which do exist, there are some funky PCI implementations out there), we can define a WC mode for example which then would allow not-super-fast, but still workable I/O in such environments.

Quoted 4 lines

I just don't really want to deal with defining all of the nuances of such a mode right now. IMHO that's best done when we actually cross that bridge and someone can validate that what they put into the spec actually works in real life.

Well sure. But taking your example, a system which allows cacheable mappings will for sure also allow WC? So a single type is not good, we need like a bitmap. And once we do, let's just use feature bits?

IOW when/if we define it, we'll just use a separate feature bit. No?

Quoted 18 lines
Quoted 18 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 6 lines

the device reports the memory type of the DMB region through a

shared-mem.tex +4 -0
+transport-specific mechanism.  Currently, the following types are defined:
+
+\begin{description}
+\item[0] The region is cache coherent.

between the device and the driver?

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De

Quoted 4 lines

A write that the driver performs to

shared-mem.tex +3 -0
+  the region becomes visible to the device, and a write that the device
+  performs to the region becomes visible to the driver, without any cache
+  maintenance operation

operations.

Quoted 1 line

by the driver.

by either.

Quoted 7 lines
Quoted 3 lines

But it's still confusing. First, it really depends on how driver mapped the memory. For example i think I can map any memory uncached and viola it works.

That depends. On older ARM systems, if you map uncached, the host (which has it marked as cached) would not be able to see it.

I mean with a physical device. Or have host map it uncached.

Quoted 3 lines

On x86 and modern
server class ARM systems, mapping uncached would work, yes.

So you see already we apparently support 2 mapping types?

Quoted 50 lines
Quoted 8 lines

Second this also might depend on the ordering flag thinkably? If it does not, say so.

Third there's ordering of this
wrt notifications since you excluded them.

Quoted 1 line

+\end{description}

Quoted 8 lines

It's confusing here. It is not clear what this 0 is unless you just read the patch.

Alternatively, if you want to use "Device Buffer memory type" or DMB memory type" or whatever make it
very prominent, like a subsection of its own and use consistently not sometimes like this sometimes " the memory type of the DMB region" etc.

Great idea!

Quoted 8 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 95

Quoted 7 lines
shared-mem.tex +4 -0
+All other values are reserved.
 
 
+
+The region's length is reported by the shared memory region descriptor and
+bounds the amount of virtqueue data that can be in flight at once.

no idea what this means and what virtqueue data is

Clarified

Quoted 11 lines

shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 101

Quoted 7 lines
shared-mem.tex +6 -0
+
+\drivernormative{\subsubsection}{Device Memory Buffer}{Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
+
+The driver MUST NOT accept VIRTIO_F_DMB unless the memory type that the
+device reports through the transport-specific mechanism is a value that this
+specification defines and that the driver supports.

i am not sure this will work well for extensions. what if three are several ways to map for example? I would just drop it I think unless you can show e.g. in the commit log some other possible types and how this will be extensible.

If there are several ways to map, we can make the field become a bitmap with special meaning that 0 means "writeback" :).

Quoted 6 lines

I don't think this will realistically happen. The host will want to tell the guest the best caching mode to use at all times. Anyone with sensible hardware will want writeback. But if we want to keep this generic (which is a sensible idea IMHO), it's good to keep at least the field reserved that allows us to indicate different semantics if we need a different caching mode.

We can simply say guests must check this field is 0 and disable the feature if not 0 if you feel so strongly. I doubt it will be useful and normally we don't do a thing like this -we can always add another feature bit. But hey.

Quoted 7 lines
Quoted 4 lines

Also generally we give detailed text without MUST etc in non conformance sections and then with a bit of duplication put conformance parts in a separate coformance sections.

This is the conformance section, no?

yes

Quoted 2 lines

Or do you mean that we should
mention the above as well in the normal section?

so we have a detailed description without MUST etc in regular sections and conformance sections just repeat the things with MUST/SHOULD etc.

Quoted 6 lines

Let's bounce this spec update once more. I'll also post a new version of the Linux patches.

Alex