virtio-pmem: PMEM device spec

Posting virtio specification for virtio pmem device. Virtio pmem is a paravirtualized device which allows the guest to bypass page cache. Virtio pmem kernel driver is merged in Upstream Kernel 5.3. Also, Qemu device is merged in Qemu 4.1.

Reviewed-by Stefan Hajnoczi <[email protected]>

Signed-off-by Pankaj Gupta <[email protected]>

---
v5 -> v6
Text change in security implication section - Cornelia Added r-o-b from Stefan

 conformance.tex |  16 +++++-
 content.tex     |   1 +
 virtio-pmem.tex | 129 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 144 insertions(+), 2 deletions(-)
 create mode 100644 virtio-pmem.tex

conformance.tex +14 -2

index 94d7a06..7331003 100644
@@ -31,7 +31,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Driver Conformance / Sound Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance} or
-\ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}.
+\ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance},
+\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}

@@ -55,7 +56,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Device Conformance / Sound Device Conformance},
 \ref{sec:Conformance / Device Conformance / Memory Device Conformance},
 \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance} or
-\ref{sec:Conformance / Device Conformance / SCMI Device Conformance}.
+\ref{sec:Conformance / Device Conformance / SCMI Device Conformance},
+\ref{sec:Conformance / Device Conformance / PMEM Driver Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}

@@ -301,6 +303,16 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / SCMI Device / Device Operation / Setting Up eventq Buffers}
 \end{itemize}
 
+\conformance{\subsection}{PMEM Driver Conformance}\label{sec:Conformance / Driver Conformance / PMEM Driver Conformance}
+
+A PMEM driver MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{devicenormative:Device Types / PMEM Device / Device Initialization}
+\item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue flush}
+\item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue return}
+\end{itemize}
+
 \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
 
 A device MUST conform to the following normative statements:

content.tex +1 -0

index 31b02e1..08d4a92 100644
@@ -6583,6 +6583,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 \input{virtio-mem.tex}
 \input{virtio-i2c.tex}
 \input{virtio-scmi.tex}
+\input{virtio-pmem.tex}
 
 \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 

virtio-pmem.tex new file +129 -1

new file mode 100644
index 0000000..93ab3c1
@@ -0,0 +1,129 @@
+\section{PMEM Device}\label{sec:Device Types / PMEM Device}
+
+The virtio pmem device is a persistent memory (NVDIMM) device
+that provides a virtio based asynchronous flush mechanism. This avoids
+the need for a separate page cache in the guest and keeps the page cache
+only in the host. Under memory pressure, the host makes use of
+efficient memory reclaim decisions for page cache pages of all the
+guests. This helps to reduce the memory footprint and fits more guests
+in the host system.
+
+The virtio pmem device provides access to byte-addressable persistent
+memory. The persistent memory is a directly accessible range of system memory.
+Data written to this memory is made persistent by separately sending a
+flush command. Writes that have been flushed are preserved across device
+reset and power failure.
+
+\subsection{Device ID}\label{sec:Device Types / PMEM Device / Device ID}
+  27
+
+\subsection{Virtqueues}\label{sec:Device Types / PMEM Device / Virtqueues}
+\begin{description}
+\item[0] req_vq
+\end{description}
+
+\subsection{Feature bits}\label{sec:Device Types / PMEM Device / Feature bits}
+
+There are currently no feature bits defined for this device.
+
+\subsection{Device configuration layout}\label{sec:Device Types / PMEM Device / Device configuration layout}
+
+\begin{lstlisting}
+struct virtio_pmem_config {
+       le64 start;
+       le64 size;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{start}] contains the physical address of the first byte of the persistent memory region.
+
+\item[\field{size}] contains the length of this address range.
+\end{description}
+
+\begin{enumerate}
+\item Driver vpmem start is read from \field{start}.
+\item Driver vpmem end is read from \field{size}.
+\end{enumerate}
+
+\subsection{Driver Initialization}\label{sec:Device Types / PMEM Driver / Driver Initialization}
+
+The driver determines the start address and size of the persistent memory region in preparation for reading or writing data.
+
+The driver initializes req_vq in preparation for making flush requests.
+
+\subsection{Driver Operations}\label{sec:Device Types / PMEM Driver / Driver Operation / Request Queues}
+
+Requests have the following format:
+
+\begin{lstlisting}
+struct virtio_pmem_req {
+        le32 type;
+};
+\end{lstlisting}
+
+\field{type} is the request command type.
+
+Possible request types are:
+
+\begin{lstlisting}
+#define VIRTIO_PMEM_REQ_TYPE_FLUSH      0
+\end{lstlisting}
+
+\subsection{Device Operations}\label{sec:Device Types / PMEM Driver / Device Operation}
+\devicenormative{\subsubsection}{Device Operation: Virtqueue flush}{Device Types / PMEM Device / Device Operation / Virtqueue flush}
+
+The device MUST ensure that all writes completed before a flush request persist across device reset and power failure before completing the flush request.
+
+\subsubsection{Device Operations}\label{sec:Device Types / PMEM Driver / Device Operation / Virtqueue return}
+\begin{lstlisting}
+struct virtio_pmem_resp {
+        le32 ret;
+};
+\end{lstlisting}
+
+\field{ret} is the value which the device returns after command completion.
+
+\devicenormative{\subsubsection}{Device Operation: Virtqueue return}{Device Types / PMEM Device / Device Operation / Virtqueue return}
+
+The device MUST return "0" for success and "-1" for failure.
+
+\subsection{Possible security implications}\label{sec:Device Types / PMEM Device / Possible Security Implications}
+
+There could be potential security implications depending on how
+memory mapped backing device is used. By default device emulation
+is done with SHARED memory mapping. There is a contract between driver
+and device to access shared memory region for read or write operations.
+
+If a malicious driver or device maps the same memory region, the attacker
+can make use of known side channel attacks to predict the current state of data.
+If both attacker and victim somehow execute same shared code after a flush
+or evict operation, with difference in execution timing attacker could infer
+another device's data.
+
+\subsection{Countermeasures}\label{sec:Device Types / PMEM Device / Possible Security Implications / Countermeasures}
+
+\subsubsection{With SHARED mapping}\label{sec:Device Types / PMEM Device / Possible Security Implications / Countermeasures / SHARED}
+
+If a device's backing region is shared between multiple devices, this may act
+as a metric for side channel attacks. As a counter measure every device
+should have its own (not shared with another driver) SHARED backing memory.
+
+\subsubsection{With PRIVATE mapping}\label{sec:Device Types / PMEM Device / Possible Security Implications / Countermeasures / PRIVATE}
+There maybe be chances of side channels attack with PRIVATE
+memory mapping similar to SHARED with read-only shared mappings.
+PRIVATE is not used for virtio pmem making this usecase
+irrelevant.
+
+\subsubsection{Workload specific mapping}\label{sec:Device Types / PMEM Device / Possible Security Implications / Countermeasures / Workload}
+When using SHARED mappings with a workload that is a single application
+inside the driver where the risk in sharing data is very low or
+nonexisting, the device sharing the same backing region with a SHARED
+mapping can be used as a valid configuration.
+
+\subsubsection{Prevent cache eviction}\label{sec:Device Types / PMEM Device / Possible Security Implications / Countermeasures / Cache eviction}
+Don't allow device shared region eviction from driver filesystem trim or discard
+like commands with virtio pmem. This rules out any possibility of evict-reload
+cache side channel attacks if backing region is shared (SHARED)
+between mutliple devices. Though if we use per device backing file with
+shared mapping this countermeasure is not required.
--

2.25.1

Posting virtio specification for virtio pmem device. Virtio pmem is a paravirtualized device which allows the guest to bypass page cache. Virtio pmem kernel driver is merged in Upstream Kernel 5.3. Also, Qemu device is merged in Qemu 4.1.

Reviewed-by Stefan Hajnoczi <[email protected]>

Signed-off-by Pankaj Gupta <[email protected]>

---
v5 -> v6
Text change in security implication section - Cornelia Added r-o-b from Stefan

 conformance.tex |  16 +++++-
 content.tex     |   1 +
 virtio-pmem.tex | 129 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 144 insertions(+), 2 deletions(-)
 create mode 100644 virtio-pmem.tex

conformance.tex +7 -2

index 94d7a06..7331003 100644
@@ -31,7 +31,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Driver Conformance / Sound Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance} or
-\ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}.
+\ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance},
+\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}

@@ -55,7 +56,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Device Conformance / Sound Device Conformance},
 \ref{sec:Conformance / Device Conformance / Memory Device Conformance},
 \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance} or
-\ref{sec:Conformance / Device Conformance / SCMI Device Conformance}.
+\ref{sec:Conformance / Device Conformance / SCMI Device Conformance},
+\ref{sec:Conformance / Device Conformance / PMEM Driver Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}

@@ -301,6 +303,16 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / SCMI Device / Device Operation / Setting Up eventq Buffers}
 \end{itemize}
 
+\conformance{\subsection}{PMEM Driver Conformance}\label{sec:Conformance / Driver Conformance / PMEM Driver Conformance}
+
+A PMEM driver MUST conform to the following normative statements:

Quoted 58 lines

Posting virtio specification for virtio pmem device. Virtio pmem is a paravirtualized device which allows the guest to bypass page cache. Virtio pmem kernel driver is merged in Upstream Kernel 5.3. Also, Qemu device is merged in Qemu 4.1.

Reviewed-by Stefan Hajnoczi <[email protected]>

Signed-off-by Pankaj Gupta <[email protected]>

---
v5 -> v6
Text change in security implication section - Cornelia Added r-o-b from Stefan

 conformance.tex |  16 +++++-
 content.tex     |   1 +
 virtio-pmem.tex | 129 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 144 insertions(+), 2 deletions(-)
 create mode 100644 virtio-pmem.tex

conformance.tex +14 -2

index 94d7a06..7331003 100644
@@ -31,7 +31,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Driver Conformance / Sound Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance} or
-\ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}.
+\ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance},
+\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}

@@ -55,7 +56,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Device Conformance / Sound Device Conformance},
 \ref{sec:Conformance / Device Conformance / Memory Device Conformance},
 \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance} or
-\ref{sec:Conformance / Device Conformance / SCMI Device Conformance}.
+\ref{sec:Conformance / Device Conformance / SCMI Device Conformance},
+\ref{sec:Conformance / Device Conformance / PMEM Driver Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}

@@ -301,6 +303,16 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / SCMI Device / Device Operation / Setting Up eventq Buffers}
 \end{itemize}
 
+\conformance{\subsection}{PMEM Driver Conformance}\label{sec:Conformance / Driver Conformance / PMEM Driver Conformance}
+
+A PMEM driver MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{devicenormative:Device Types / PMEM Device / Device Initialization}
+\item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue flush}
+\item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue return}
+\end{itemize}
+
 \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
 

conformance.tex \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} · line 318

Quoted 1 line

conformance.tex +0 -0

 A device MUST conform to the following normative statements:

So, I ran into some problems while trying to apply this.

First, this is against an older version of the tree, so it did not apply cleanly.

Second, the conformance section looks off: we don't have a driver conformance section, and the device conformance section ends up being called a driver conformance section. Unfortunately, I had neglected to review that properly :(

I think the following would be correct; I'm inclined to apply this spec with it as an editorial edit. Any comments/objections?

conformance.tex +12 -2

index c52f1a40be2d..c2d79596f515 100644
@@ -56,8 +56,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Device Conformance / Sound Device Conformance},
 \ref{sec:Conformance / Device Conformance / Memory Device Conformance},
 \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance},
-\ref{sec:Conformance / Device Conformance / SCMI Device Conformance} or
-\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}.
+\ref{sec:Conformance / Device Conformance / SCMI Device Conformance},
+\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or
+\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}

@@ -570,6 +571,15 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / GPIO Device / requestq Operation}
 \end{itemize}
 
+\conformance{\subsection}{PMEM Device Conformance}\label{sec:Conformance / Device Conformance / PMEM Device Conformance}
+
+A PMEM device MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue flush}
+\item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue return}
+\end{itemize}
+
 \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
 A conformant implementation MUST be either transitional or
 non-transitional, see \ref{intro:Legacy

Quoted 4 lines

So, I ran into some problems while trying to apply this.

First, this is against an older version of the tree, so it did not apply cleanly.

Sorry! somehow i missed this.

Quoted 6 lines

Second, the conformance section looks off: we don't have a driver conformance section, and the device conformance section ends up being called a driver conformance section. Unfortunately, I had neglected to review that properly :(

Quoted 2 lines

I think the following would be correct; I'm inclined to apply this spec with it as an editorial edit. Any comments/objections?

my bad :(

Thank you for your help!
Hope this makes into master now.

Quoted 6 lines · Pankaj Gupta

Quoted 4 lines

So, I ran into some problems while trying to apply this.

First, this is against an older version of the tree, so it did not apply cleanly.

Sorry! somehow i missed this.

No worries, I fixed that up; but rebasing makes things easier for the person applying it.

Quoted 6 lines

Quoted 6 lines

Second, the conformance section looks off: we don't have a driver conformance section, and the device conformance section ends up being called a driver conformance section. Unfortunately, I had neglected to review that properly :(

Quoted 7 lines

Quoted 2 lines

I think the following would be correct; I'm inclined to apply this spec with it as an editorial edit. Any comments/objections?

my bad :(

Thank you for your help!
Hope this makes into master now.

I pushed this out now. If something is off, we can fix that on top.

Thank you!

@Taylor Stark Since now have virtio pmem spec merged, we can propose spec changes for "virtio-pmem: Support PCI BAR-relative addresses" support.

Best regards,
Pankaj

Quoted 6 lines

@Taylor Stark Since now have virtio pmem spec merged, we can propose spec changes for "virtio-pmem: Support PCI BAR-relative addresses" support.

Best regards,
Pankaj

Thanks Pankaj, will do! I'll post the change late next week (I'm currently on vacation but being bad and checking email).

Taylor