Skip to contentSkip to conversation

Vitqueue State Synchronization

PATCH In review Jason Wang 8 participants 115 messages Jul 6, 2021 – Aug 8, 2021 Page 1 of 2

lore
Share & tools
lore

Jason Wang

author
#0 lore raw

v2 cover Vitqueue State Synchronization

Hi All:

This is an updated version to implement virtqueue state synchronization which is a must for the migration support.

The first patch introduces virtqueue states as a new basic facility of the virtio device. This is used by the driver to save and restore virtqueue state. The states were split into available state and used state to ease the transport specific implementation. It is also allowed for the device to have its own device specific way to save and resotre extra virtqueue states like in flight request.

The second patch introduce a new status bit STOP. This bit is used for the driver to stop the device. The major difference from reset is that STOP must preserve all the virtqueue state plus the device state.

A driver can then:

- Get the virtqueue state if STOP status bit is set
- Set the virtqueue state after FEATURE_OK but before DRIVER_OK

Device specific state synchronization could be built on top.

Please review.

Thanks

Changes from V1:

- introduce used_idx for synchronize used state for split virtqueue, this is needed since the used ring is read only by the driver.
- stick to the uni-directional state machine, this means driver is forbidden to clear STOP, the only way to 'resume' the device is to reset and re-initialization the device. This simplify the implementation and make it works more like a vhost device.
- mandate the virtqueue state setting during device initialization
- clarify the steps that are required for saving the virtqueue state
- allow the device to have its specifc way to save and restore extra virtqueue state
- rename DEVICE_STOPPED to STOP
- various other tweaks

Jason Wang (2):
virtio: introduce virtqueue state as basic facility
virtio: introduce STOP status bit

 content.tex | 180 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 180 insertions(+)
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH v2 1/2 virtio: introduce virtqueue state as basic facility

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

This facility could be used by both migration and device diagnostic.

---
 content.tex | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
content.tex +117 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to get
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last Driver Ring
+Wrap Counter that is observed by the device. This starts at the value
+set by the driver, and is flipped when reaching the end of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This starts
+at the value set by the driver, and increases. It is easy to see this
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} indicates where the device would write the next used
+descriptor head to the descriptor ring. This starts at the value set
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This starts
+at the value set by the driver, and is flipped when reaching the end
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.
+
+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -420,6 +530,9 @@ \section{Device Initialization}\label{sec:General Initialization And Device Oper
    device, optional per-bus setup, reading and possibly writing the
    device's virtio configuration space, and population of virtqueues.
 
+\item\label{itm:General Initialization And Device Operation / Device
+  Initialization / Virtqueue State Setup} When VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state.
+
 \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is
    ``live''.
 \end{enumerate}
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state.
+  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

Quoted 7 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Quoted 7 lines

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

This facility could be used by both migration and device diagnostic.

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Thoughts?

Quoted 84 lines
---
 content.tex | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
content.tex +72 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to get
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last Driver Ring
+Wrap Counter that is observed by the device. This starts at the value
+set by the driver, and is flipped when reaching the end of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This starts
+at the value set by the driver, and increases. It is easy to see this
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 469

Quoted 10 lines
content.tex +10 -0
+
+The \field{used_idx} indicates where the device would write the next used
+descriptor head to the descriptor ring. This starts at the value set
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This starts
+at the value set by the driver, and is flipped when reaching the end
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.

Above only fully describes the vq state if descriptors
are used in order or at least all out of order descriptors are consumed at time of save.

Adding later option to devices such as net will need extra spec work.

Quoted 14 lines
content.tex +14 -0
+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 494

Quoted 10 lines
content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its

may have?
should also go into a normative section

Quoted 29 lines
content.tex +10 -0
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -420,6 +530,9 @@ \section{Device Initialization}\label{sec:General Initialization And Device Oper
    device, optional per-bus setup, reading and possibly writing the
    device's virtio configuration space, and population of virtqueues.
 
+\item\label{itm:General Initialization And Device Operation / Device
+  Initialization / Virtqueue State Setup} When VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state.
+
 \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is
    ``live''.
 \end{enumerate}
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state.
+  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

This facility could be used by both migration and device diagnostic.

Quoted 8 lines
Quoted 1 line

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Hi!

In my opinion we should go the other way around: To make features as orthogonal/independent as possible, and just make them work together if we have to. In this particular case, I think it should be easier to decide how to report status, its needs, etc for a VF, and then open the possibility for the PF to query or set them, reusing format, behavior, etc. as much as possible.

I think that the most controversial point about doing it non-SR IOV way is the exposing of these features/fields to the guest using specific transport facilities, like PCI common config. However I think it should not be hard for the hypervisor to intercept them and even to expose them conditionally. Please correct me if this guessing was not right and you had other concerns.

Quoted 93 lines

Thoughts?

Quoted 91 lines
---
 content.tex | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
content.tex +79 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to get
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last Driver Ring
+Wrap Counter that is observed by the device. This starts at the value
+set by the driver, and is flipped when reaching the end of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This starts
+at the value set by the driver, and increases. It is easy to see this
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} indicates where the device would write the next used
+descriptor head to the descriptor ring. This starts at the value set
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This starts
+at the value set by the driver, and is flipped when reaching the end
Quoted 9 lines

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 469

Quoted 3 lines
content.tex +3 -0
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.

Above only fully describes the vq state if descriptors
are used in order or at least all out of order descriptors are consumed at time of save.

I think that the most straightforward solution would be to add something similar to VHOST_USER_GET_INFLIGHT_FD, but without the _FD part.

Thanks!

Quoted 62 lines

Adding later option to devices such as net will need extra spec work.

Quoted 14 lines
content.tex +14 -0
+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 494

Quoted 10 lines
content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its

may have?
should also go into a normative section

Quoted 29 lines
content.tex +10 -1
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -420,6 +530,9 @@ \section{Device Initialization}\label{sec:General Initialization And Device Oper
    device, optional per-bus setup, reading and possibly writing the
    device's virtio configuration space, and population of virtqueues.
 
+\item\label{itm:General Initialization And Device Operation / Device
+  Initialization / Virtqueue State Setup} When VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state.
+
 \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is
    ``live''.
 \end{enumerate}
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state.
+  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
--

2.25.1

#4 lore raw

Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

Quoted 34 lines · Eugenio Perez Martin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

This facility could be used by both migration and device diagnostic.

Quoted 8 lines
Quoted 1 line

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Hi!

In my opinion we should go the other way around: To make features as orthogonal/independent as possible, and just make them work together if we have to. In this particular case, I think it should be easier to decide how to report status, its needs, etc for a VF, and then open the possibility for the PF to query or set them, reusing format, behavior, etc. as much as possible.

Quoted 6 lines

I think that the most controversial point about doing it non-SR IOV way is the exposing of these features/fields to the guest using specific transport facilities, like PCI common config. However I think it should not be hard for the hypervisor to intercept them and even to expose them conditionally. Please correct me if this guessing was not right and you had other concerns.

Possibly. I'd like to see some guidance on how this all will work in practice then. Maybe make it all part of a non-normative section for now.
I think that the feature itself is not very useful outside of migration so we don't really gain much by adding it as is without all the other missing pieces.
I would say let's see more of the whole picture before we commit.

Quoted 171 lines
Quoted 93 lines

Thoughts?

Quoted 91 lines
---
 content.tex | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
content.tex +79 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to get
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last Driver Ring
+Wrap Counter that is observed by the device. This starts at the value
+set by the driver, and is flipped when reaching the end of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This starts
+at the value set by the driver, and increases. It is easy to see this
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} indicates where the device would write the next used
+descriptor head to the descriptor ring. This starts at the value set
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This starts
+at the value set by the driver, and is flipped when reaching the end
Quoted 9 lines

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 469

Quoted 3 lines
content.tex +3 -0
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.

Above only fully describes the vq state if descriptors
are used in order or at least all out of order descriptors are consumed at time of save.

I think that the most straightforward solution would be to add something similar to VHOST_USER_GET_INFLIGHT_FD, but without the _FD part.

Thanks!

Quoted 62 lines

Adding later option to devices such as net will need extra spec work.

Quoted 14 lines
content.tex +14 -0
+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 494

Quoted 10 lines
content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its

may have?
should also go into a normative section

Quoted 29 lines
content.tex +10 -1
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -420,6 +530,9 @@ \section{Device Initialization}\label{sec:General Initialization And Device Oper
    device, optional per-bus setup, reading and possibly writing the
    device's virtio configuration space, and population of virtqueues.
 
+\item\label{itm:General Initialization And Device Operation / Device
+  Initialization / Virtqueue State Setup} When VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state.
+
 \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is
    ``live''.
 \end{enumerate}
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state.
+  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
--

2.25.1

#5 lore raw

Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

Quoted 39 lines · Michael S. Tsirkin
Quoted 31 lines · Eugenio Perez Martin
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

Quoted 9 lines
Quoted 3 lines

This facility could be used by both migration and device diagnostic.

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Hi!

In my opinion we should go the other way around: To make features as orthogonal/independent as possible, and just make them work together if we have to. In this particular case, I think it should be easier to decide how to report status, its needs, etc for a VF, and then open the possibility for the PF to query or set them, reusing format, behavior, etc. as much as possible.

Quoted 6 lines

I think that the most controversial point about doing it non-SR IOV way is the exposing of these features/fields to the guest using specific transport facilities, like PCI common config. However I think it should not be hard for the hypervisor to intercept them and even to expose them conditionally. Please correct me if this guessing was not right and you had other concerns.

Quoted 7 lines

Possibly. I'd like to see some guidance on how this all will work in practice then. Maybe make it all part of a non-normative section for now.
I think that the feature itself is not very useful outside of migration so we don't really gain much by adding it as is without all the other missing pieces.
I would say let's see more of the whole picture before we commit.

I agree here. I also can't see the whole picture for SRIOV case.

I'll try to combine the admin control queue suggested in previous patch set to my proposal of PF managing the VF migration.

Feature negotiation is part of virtio device-driver communication and not part of the migration software that should manage the migration process.

For me, seems like queue state is something that should be internal and not be exposed to guest drivers that see this as a new feature.

Quoted 170 lines
Quoted 170 lines
Quoted 95 lines

Thoughts?

Quoted 90 lines
---
  content.tex | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 117 insertions(+)
content.tex +78 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
  types. It is RECOMMENDED that devices generate version 4
  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to get
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last Driver Ring
+Wrap Counter that is observed by the device. This starts at the value
+set by the driver, and is flipped when reaching the end of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This starts
+at the value set by the driver, and increases. It is easy to see this
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} indicates where the device would write the next used
+descriptor head to the descriptor ring. This starts at the value set
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This starts
Quoted 9 lines

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 469

Quoted 4 lines
content.tex +4 -0
+at the value set by the driver, and is flipped when reaching the end
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.

Above only fully describes the vq state if descriptors
are used in order or at least all out of order descriptors are consumed at time of save.

I think that the most straightforward solution would be to add something similar to VHOST_USER_GET_INFLIGHT_FD, but without the _FD part.

Thanks!

Quoted 60 lines

Adding later option to devices such as net will need extra spec work.

Quoted 14 lines
content.tex +14 -0
+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 494

Quoted 10 lines
content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its

may have?
should also go into a normative section

Quoted 29 lines
content.tex +10 -1
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+
  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
  We start with an overview of device initialization, then expand on the
@@ -420,6 +530,9 @@ \section{Device Initialization}\label{sec:General Initialization And Device Oper
     device, optional per-bus setup, reading and possibly writing the
     device's virtio configuration space, and population of virtqueues.
 
+\item\label{itm:General Initialization And Device Operation / Device
+  Initialization / Virtqueue State Setup} When VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state.
+
  \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is
     ``live''.
  \end{enumerate}
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
    transport specific.
    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state.
+  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
  \end{description}
 
  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
--

2.25.1

author
#6 lore raw

Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

Quoted 49 lines · Max Gurtovoy
Quoted 39 lines · Michael S. Tsirkin
Quoted 31 lines · Eugenio Perez Martin
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

Quoted 9 lines
Quoted 3 lines

This facility could be used by both migration and device diagnostic.

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Hi!

In my opinion we should go the other way around: To make features as orthogonal/independent as possible, and just make them work together if we have to. In this particular case, I think it should be easier to decide how to report status, its needs, etc for a VF, and then open the possibility for the PF to query or set them, reusing format, behavior, etc. as much as possible.

Quoted 6 lines

I think that the most controversial point about doing it non-SR IOV way is the exposing of these features/fields to the guest using specific transport facilities, like PCI common config. However I think it should not be hard for the hypervisor to intercept them and even to expose them conditionally. Please correct me if this guessing was not right and you had other concerns.

Quoted 7 lines

Possibly. I'd like to see some guidance on how this all will work in practice then. Maybe make it all part of a non-normative section for now.
I think that the feature itself is not very useful outside of migration so we don't really gain much by adding it as is without all the other missing pieces.
I would say let's see more of the whole picture before we commit.

Quoted 1 line

I agree here. I also can't see the whole picture for SRIOV case.

Again, it's not related to SR-IOV at all. It tries to introduce basic facility in the virtio level which can work for all types of virtio device.

Transport such as PCI need to implement its own way to access those state. It's not hard to implement them simply via capability.

It works like other basic facility like device status, features etc.

For SR-IOV, it doesn't prevent you from implementing that via the admin virtqueue.

Quoted 2 lines

I'll try to combine the admin control queue suggested in previous patch set to my proposal of PF managing the VF migration.

Note that, the admin virtqueue should be transport indepedent when trying to introduce them.

Quoted 4 lines

Feature negotiation is part of virtio device-driver communication and not part of the migration software that should manage the migration process.

For me, seems like queue state is something that should be internal and not be exposed to guest drivers that see this as a new feature.

This is not true, we have the case of nested virtualization. As mentioned in another thread, it's the hypervisor that need to choose between hiding or shadowing the internal virtqueue state.

Thanks

Quoted 196 lines
Quoted 195 lines
Quoted 195 lines
Quoted 113 lines

Thoughts?

Quoted 107 lines

---
  content.tex | 117
++++++++++++++++++++++++++++++++++++++++++++++++++++

  1 file changed, 117 insertions(+)
content.tex +0 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic 

Facilities of a Virtio Device / Expo
  types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.

+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}

content.tex +20 -0
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to get
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would 

read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.

content.tex +11 -0
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would 

read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the

content.tex +3 -0
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last 

Driver Ring
+Wrap Counter that is observed by the device. This starts at the value

content.tex +3 -0
+set by the driver, and is flipped when reaching the end of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap 

Counters}.

content.tex +7 -0
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state 

contains:

content.tex +9 -0
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This 

starts
+at the value set by the driver, and increases. It is easy to see this

content.tex +3 -0
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / 

Virtqueues / The Virtqueue Used Ring}

content.tex +11 -0
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} indicates where the device would write the 

next used

content.tex +4 -0
+descriptor head to the descriptor ring. This starts at the value set
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This 

starts
+at the value set by the driver, and is flipped when reaching the end

Quoted 9 lines
Quoted 4 lines
content.tex +3 -0
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap 

Counters}.

Above only fully describes the vq state if descriptors
are used in order or at least all out of order descriptors are consumed at time of save.

I think that the most straightforward solution would be to add something similar to VHOST_USER_GET_INFLIGHT_FD, but without the _FD part.

Thanks!

Quoted 67 lines

Adding later option to devices such as net will need extra spec work.

Quoted 16 lines

+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}

content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities 

of a Virtio Device / Virtqueue State}

content.tex +3 -0
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 494

Quoted 10 lines
content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its

may have?
should also go into a normative section

Quoted 34 lines
content.tex +3 -0
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+

  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}

  We start with an overview of device initialization, then expand on the

content.tex +0 -0
@@ -420,6 +530,9 @@ \section{Device 

Initialization}\label{sec:General Initialization And Device Oper device, optional per-bus setup, reading and possibly writing the device's virtio configuration space, and population of virtqueues.

+\item\label{itm:General Initialization And Device Operation / Device
+  Initialization / Virtqueue State Setup} When
VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state. +
  \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is ``live''.
  \end{enumerate}

content.tex +0 -0
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature 
 Bits}\label{sec:Reserved Feature Bits}

    transport specific.
    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.

+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state. +  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
  \end{description}

  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

Signature · 1 line
2.25.1
#7 lore raw

Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

Quoted 71 lines · Jason Wang
Quoted 49 lines · Max Gurtovoy
Quoted 39 lines · Michael S. Tsirkin
Quoted 31 lines · Eugenio Perez Martin
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

Quoted 9 lines
Quoted 3 lines

This facility could be used by both migration and device diagnostic.

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Hi!

In my opinion we should go the other way around: To make features as orthogonal/independent as possible, and just make them work together if we have to. In this particular case, I think it should be easier to decide how to report status, its needs, etc for a VF, and then open the possibility for the PF to query or set them, reusing format, behavior, etc. as much as possible.

Quoted 6 lines

I think that the most controversial point about doing it non-SR IOV way is the exposing of these features/fields to the guest using specific transport facilities, like PCI common config. However I think it should not be hard for the hypervisor to intercept them and even to expose them conditionally. Please correct me if this guessing was not right and you had other concerns.

Quoted 7 lines

Possibly. I'd like to see some guidance on how this all will work in practice then. Maybe make it all part of a non-normative section for now.
I think that the feature itself is not very useful outside of migration so we don't really gain much by adding it as is without all the other missing pieces.
I would say let's see more of the whole picture before we commit.

Quoted 1 line

I agree here. I also can't see the whole picture for SRIOV case.

Again, it's not related to SR-IOV at all. It tries to introduce basic facility in the virtio level which can work for all types of virtio device.

Transport such as PCI need to implement its own way to access those state. It's not hard to implement them simply via capability.

It works like other basic facility like device status, features etc.

For SR-IOV, it doesn't prevent you from implementing that via the admin virtqueue.

Quoted 2 lines

I'll try to combine the admin control queue suggested in previous patch set to my proposal of PF managing the VF migration.

Note that, the admin virtqueue should be transport indepedent when trying to introduce them.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Feature negotiation is part of virtio device-driver communication and not part of the migration software that should manage the migration process.

For me, seems like queue state is something that should be internal and not be exposed to guest drivers that see this as a new feature.

This is not true, we have the case of nested virtualization. As mentioned in another thread, it's the hypervisor that need to choose between hiding or shadowing the internal virtqueue state.

Thanks

In the nested environment, do you mean the Level 1 is Real PF with X VFs and in Level 2 the X VF seen as PFs in the guests and expose another Y VFs ?

If so, the guest PF will manage the migration for it's Y VFs.

Quoted 207 lines
Quoted 206 lines
Quoted 205 lines
Quoted 205 lines
Quoted 119 lines

Thoughts?

Quoted 111 lines

---
  content.tex | 117
++++++++++++++++++++++++++++++++++++++++++++++++++++

  1 file changed, 117 insertions(+)
content.tex +0 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic 

Facilities of a Virtio Device / Expo
  types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.

+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}

content.tex +8 -0
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to 

get

content.tex +12 -0
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device 

would read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.

content.tex +11 -0
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device 

would read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the

content.tex +3 -0
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last 

Driver Ring
+Wrap Counter that is observed by the device. This starts at the value
+set by the driver, and is flipped when reaching the end of the ring.

content.tex +2 -0
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring 

Wrap Counters}.

content.tex +7 -0
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state 

contains:

content.tex +9 -0
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This 

starts
+at the value set by the driver, and increases. It is easy to see this

content.tex +3 -0
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / 

Virtqueues / The Virtqueue Used Ring}

content.tex +11 -0
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} indicates where the device would write the 

next used
+descriptor head to the descriptor ring. This starts at the value set

content.tex +3 -0
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This 

starts
+at the value set by the driver, and is flipped when reaching the end

Quoted 9 lines
Quoted 4 lines
content.tex +3 -0
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring 

Wrap Counters}.

Above only fully describes the vq state if descriptors
are used in order or at least all out of order descriptors are consumed at time of save.

I think that the most straightforward solution would be to add something similar to VHOST_USER_GET_INFLIGHT_FD, but without the _FD part.

Thanks!

Quoted 71 lines

Adding later option to devices such as net will need extra spec work.

Quoted 25 lines

+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}

content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities 

of a Virtio Device / Virtqueue State}

content.tex +2 -0
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST 

ingore

content.tex +5 -0
+the read and write to the virtqueue state.
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if 

the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 494

Quoted 4 lines
content.tex +4 -0
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its

may have?
should also go into a normative section

Quoted 35 lines
content.tex +3 -0
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+

  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}

  We start with an overview of device initialization, then expand on the

content.tex +0 -0
@@ -420,6 +530,9 @@ \section{Device 

Initialization}\label{sec:General Initialization And Device Oper device, optional per-bus setup, reading and possibly writing the device's virtio configuration space, and population of virtqueues.

+\item\label{itm:General Initialization And Device Operation /
Device
+  Initialization / Virtqueue State Setup} When
VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state. +
  \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is ``live''.
  \end{enumerate}

content.tex +0 -0
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature 
 Bits}\label{sec:Reserved Feature Bits}

    transport specific.
    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.

+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state. +  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
  \end{description}

  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

Signature · 1 line
2.25.1
author
#8 lore raw

Re: [virtio-comment] Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

Quoted 75 lines · Max Gurtovoy
Quoted 73 lines · Jason Wang
Quoted 49 lines · Max Gurtovoy
Quoted 39 lines · Michael S. Tsirkin
Quoted 31 lines · Eugenio Perez Martin
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

Quoted 9 lines
Quoted 3 lines

This facility could be used by both migration and device diagnostic.

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Hi!

In my opinion we should go the other way around: To make features as orthogonal/independent as possible, and just make them work together if we have to. In this particular case, I think it should be easier to decide how to report status, its needs, etc for a VF, and then open the possibility for the PF to query or set them, reusing format, behavior, etc. as much as possible.

Quoted 6 lines

I think that the most controversial point about doing it non-SR IOV way is the exposing of these features/fields to the guest using specific transport facilities, like PCI common config. However I think it should not be hard for the hypervisor to intercept them and even to expose them conditionally. Please correct me if this guessing was not right and you had other concerns.

Quoted 7 lines

Possibly. I'd like to see some guidance on how this all will work in practice then. Maybe make it all part of a non-normative section for now.
I think that the feature itself is not very useful outside of migration so we don't really gain much by adding it as is without all the other missing pieces.
I would say let's see more of the whole picture before we commit.

Quoted 1 line

I agree here. I also can't see the whole picture for SRIOV case.

Again, it's not related to SR-IOV at all. It tries to introduce basic facility in the virtio level which can work for all types of virtio device.

Transport such as PCI need to implement its own way to access those state. It's not hard to implement them simply via capability.

It works like other basic facility like device status, features etc.

For SR-IOV, it doesn't prevent you from implementing that via the admin virtqueue.

Quoted 2 lines

I'll try to combine the admin control queue suggested in previous patch set to my proposal of PF managing the VF migration.

Note that, the admin virtqueue should be transport indepedent when trying to introduce them.

Quoted 3 lines

Feature negotiation is part of virtio device-driver communication and not part of the migration software that should manage the migration process.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

For me, seems like queue state is something that should be internal and not be exposed to guest drivers that see this as a new feature.

This is not true, we have the case of nested virtualization. As mentioned in another thread, it's the hypervisor that need to choose between hiding or shadowing the internal virtqueue state.

Thanks

In the nested environment, do you mean the Level 1 is Real PF with X VFs and in Level 2 the X VF seen as PFs in the guests and expose another Y VFs ?

I meant PF is managed in L0. And the VF is assigned to L2 guest. In this case, we can expose the virtqueue state feature to L1 guest for migration L2 guest.

Quoted 2 lines

If so, the guest PF will manage the migration for it's Y VFs.

Does this mean you want to pass PF to L1 guest?

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#9 lore raw

Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

In reply to Michael S. Tsirkin
Quoted 39 lines · Michael S. Tsirkin
Quoted 31 lines · Eugenio Perez Martin
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

Quoted 9 lines
Quoted 3 lines

This facility could be used by both migration and device diagnostic.

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Hi!

In my opinion we should go the other way around: To make features as orthogonal/independent as possible, and just make them work together if we have to. In this particular case, I think it should be easier to decide how to report status, its needs, etc for a VF, and then open the possibility for the PF to query or set them, reusing format, behavior, etc. as much as possible.

Quoted 6 lines

I think that the most controversial point about doing it non-SR IOV way is the exposing of these features/fields to the guest using specific transport facilities, like PCI common config. However I think it should not be hard for the hypervisor to intercept them and even to expose them conditionally. Please correct me if this guessing was not right and you had other concerns.

Quoted 6 lines

Possibly. I'd like to see some guidance on how this all will work in practice then. Maybe make it all part of a non-normative section for now.
I think that the feature itself is not very useful outside of migration so we don't really gain much by adding it as is without all the other missing pieces.

For networking device, the only missing part is the transport implementation of the virtqueue state.

Quoted 1 line

I would say let's see more of the whole picture before we commit.

I will include an implementation of PCI as an example.

Thanks

Quoted 170 lines
Quoted 170 lines
Quoted 95 lines

Thoughts?

Quoted 90 lines
---
  content.tex | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 117 insertions(+)
content.tex +78 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
  types. It is RECOMMENDED that devices generate version 4
  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to get
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last Driver Ring
+Wrap Counter that is observed by the device. This starts at the value
+set by the driver, and is flipped when reaching the end of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This starts
+at the value set by the driver, and increases. It is easy to see this
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} indicates where the device would write the next used
+descriptor head to the descriptor ring. This starts at the value set
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This starts
Quoted 9 lines

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 469

Quoted 4 lines
content.tex +4 -0
+at the value set by the driver, and is flipped when reaching the end
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.

Above only fully describes the vq state if descriptors
are used in order or at least all out of order descriptors are consumed at time of save.

I think that the most straightforward solution would be to add something similar to VHOST_USER_GET_INFLIGHT_FD, but without the _FD part.

Thanks!

Quoted 60 lines

Adding later option to devices such as net will need extra spec work.

Quoted 14 lines
content.tex +14 -0
+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 494

Quoted 10 lines
content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its

may have?
should also go into a normative section

Quoted 29 lines
content.tex +10 -1
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+
  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
  We start with an overview of device initialization, then expand on the
@@ -420,6 +530,9 @@ \section{Device Initialization}\label{sec:General Initialization And Device Oper
     device, optional per-bus setup, reading and possibly writing the
     device's virtio configuration space, and population of virtqueues.
 
+\item\label{itm:General Initialization And Device Operation / Device
+  Initialization / Virtqueue State Setup} When VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state.
+
  \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is
     ``live''.
  \end{enumerate}
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
    transport specific.
    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state.
+  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
  \end{description}
 
  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
--

2.25.1

author
#10 lore raw

Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

Quoted 49 lines · Jason Wang
Quoted 39 lines · Michael S. Tsirkin
Quoted 31 lines · Eugenio Perez Martin
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

Quoted 9 lines
Quoted 3 lines

This facility could be used by both migration and device diagnostic.

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Hi!

In my opinion we should go the other way around: To make features as orthogonal/independent as possible, and just make them work together if we have to. In this particular case, I think it should be easier to decide how to report status, its needs, etc for a VF, and then open the possibility for the PF to query or set them, reusing format, behavior, etc. as much as possible.

Quoted 6 lines

I think that the most controversial point about doing it non-SR IOV way is the exposing of these features/fields to the guest using specific transport facilities, like PCI common config. However I think it should not be hard for the hypervisor to intercept them and even to expose them conditionally. Please correct me if this guessing was not right and you had other concerns.

Quoted 6 lines

Possibly. I'd like to see some guidance on how this all will work in practice then. Maybe make it all part of a non-normative section for now.
I think that the feature itself is not very useful outside of migration so we don't really gain much by adding it as is without all the other missing pieces.

Quoted 1 line

For networking device, the only missing part is the transport implementation of the virtqueue state.

So I've posted a patch to implement the virtqueue state for PCI. This should be sufficient for a virtio-PCI device to be migrated.

Thanks

Quoted 204 lines
Quoted 3 lines

I would say let's see more of the whole picture before we commit.

I will include an implementation of PCI as an example.

Thanks

Quoted 194 lines
Quoted 194 lines
Quoted 112 lines

Thoughts?

Quoted 107 lines

---
  content.tex | 117
++++++++++++++++++++++++++++++++++++++++++++++++++++

  1 file changed, 117 insertions(+)
content.tex +0 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic 

Facilities of a Virtio Device / Expo
  types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.

+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}

content.tex +20 -0
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to get
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would 

read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.

content.tex +11 -0
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would 

read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the

content.tex +3 -0
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last 

Driver Ring
+Wrap Counter that is observed by the device. This starts at the value

content.tex +3 -0
+set by the driver, and is flipped when reaching the end of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap 

Counters}.

content.tex +7 -0
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state 

contains:

content.tex +9 -0
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This 

starts
+at the value set by the driver, and increases. It is easy to see this

content.tex +3 -0
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / 

Virtqueues / The Virtqueue Used Ring}

content.tex +11 -0
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} indicates where the device would write the 

next used

content.tex +4 -0
+descriptor head to the descriptor ring. This starts at the value set
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This 

starts
+at the value set by the driver, and is flipped when reaching the end

Quoted 9 lines
Quoted 4 lines
content.tex +3 -0
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap 

Counters}.

Above only fully describes the vq state if descriptors
are used in order or at least all out of order descriptors are consumed at time of save.

I think that the most straightforward solution would be to add something similar to VHOST_USER_GET_INFLIGHT_FD, but without the _FD part.

Thanks!

Quoted 67 lines

Adding later option to devices such as net will need extra spec work.

Quoted 16 lines

+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}

content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities 

of a Virtio Device / Virtqueue State}

content.tex +3 -0
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 494

Quoted 10 lines
content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its

may have?
should also go into a normative section

Quoted 34 lines
content.tex +3 -0
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+

  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}

  We start with an overview of device initialization, then expand on the

content.tex +0 -0
@@ -420,6 +530,9 @@ \section{Device 

Initialization}\label{sec:General Initialization And Device Oper device, optional per-bus setup, reading and possibly writing the device's virtio configuration space, and population of virtqueues.

+\item\label{itm:General Initialization And Device Operation / Device
+  Initialization / Virtqueue State Setup} When
VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state. +
  \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is ``live''.
  \end{enumerate}

content.tex +0 -0
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature 
 Bits}\label{sec:Reserved Feature Bits}

    transport specific.
    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.

+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state. +  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
  \end{description}

  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

Signature · 1 line
2.25.1
author
#11 lore raw

Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

In reply to Eugenio Perez Martin
Quoted 22 lines · Eugenio Perez Martin
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

Quoted 9 lines
Quoted 3 lines

This facility could be used by both migration and device diagnostic.

Hi Jason!
I feel that for use-cases such as SRIOV,
the facility to save/restore vq should be part of a PF that is there needs to be a way for one virtio device to address the state of another one.

Quoted 5 lines

Hi!

In my opinion we should go the other way around: To make features as orthogonal/independent as possible, and just make them work together if we have to.

I agree.

Quoted 4 lines

In this particular case, I think it should be easier to
decide how to report status, its needs, etc for a VF, and then open the possibility for the PF to query or set them, reusing format, behavior, etc. as much as possible.

Yes, that's why I introduce virtqueue state as basic facility and reuse the device status bit for stop.

Quoted 6 lines

I think that the most controversial point about doing it non-SR IOV way is the exposing of these features/fields to the guest using specific transport facilities, like PCI common config. However I think it should not be hard for the hypervisor to intercept them and even to expose them conditionally.

This could be done via admin virtqueue. I'm working on the device slicing at virtio level via admin virtqueue, this means, all the basic facility of a device slice could be carried via admin virtqueue: device status, configuration, probing, features negotiation etc.

Thanks

Quoted 170 lines

Please correct me if this guessing was not
right and you had other concerns.

Quoted 92 lines

Thoughts?

Quoted 90 lines
---
  content.tex | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 117 insertions(+)
content.tex +78 -0
index 620c0e2..8877b6f 100644
@@ -385,6 +385,116 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
  types. It is RECOMMENDED that devices generate version 4
  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Virtqueue State}\label{sec:Virtqueues / Virtqueue State}
+
+When VIRTIO_F_RING_STATE is negotiated, the driver can set and
+get the device internal virtqueue state through the following
+fields. The way to access those fields is transport specific.
+
+\subsection{\field{Available State} Field}
+
+The \field{Available State} field is two bytes for the driver to get
+or set the state that is used by the virtqueue to read for the next
+available buffer.
+
+When VIRTIO_F_RING_PACKED is not negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 16;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read
+for the next index from the virtqueue available ring(modulo the queue
+ size). This starts at the value set by the driver, and increases.
+
+When VIRTIO_F_RING_PACKED is negotiated, it contains:
+
+\begin{lstlisting}
+le16 {
+        last_avail_idx : 15;
+        last_avail_wrap_counter : 1;
+} avail_state;
+\end{lstlisting}
+
+The \field{last_avail_idx} field indicates where the device would read for
+the next descriptor head from the descriptor ring. This starts at the
+value set by the driver and wraps around when reaching the end of the
+ring.
+
+The \field{last_avail_wrap_counter} field indicates the last Driver Ring
+Wrap Counter that is observed by the device. This starts at the value
+set by the driver, and is flipped when reaching the end of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.
+
+\subsection{\field{Used State} Field}
+
+The \field{Used State} field is two bytes for the driver to set and
+get the state used by the virtqueue to make buffer used.
+
+When VIRTIO_F_RING_PACKED is not negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 16;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} where the device would write the next used
+descriptor head to the used ring (modulo the queue size). This starts
+at the value set by the driver, and increases. It is easy to see this
+is the initial value of the \field{idx} in the used ring.
+
+See also \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
+
+When VIRTIO_F_RING_PACKED is negotiated, the used state contains:
+
+\begin{lstlisting}
+le16 {
+        used_idx : 15;
+        used_wrap_counter : 1;
+} used_state;
+\end{lstlisting}
+
+The \field{used_idx} indicates where the device would write the next used
+descriptor head to the descriptor ring. This starts at the value set
+by the driver, and warps around when reaching the end of the ring.
+
+\field{used_wrap_counter} is the Device Ring Wrap Counter. This starts
Quoted 9 lines

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 469

Quoted 4 lines
content.tex +4 -0
+at the value set by the driver, and is flipped when reaching the end
+of the ring.
+
+See also \ref{sec:Packed Virtqueues / Driver and Device Ring Wrap Counters}.

Above only fully describes the vq state if descriptors
are used in order or at least all out of order descriptors are consumed at time of save.

I think that the most straightforward solution would be to add something similar to VHOST_USER_GET_INFLIGHT_FD, but without the _FD part.

Thanks!

Quoted 60 lines

Adding later option to devices such as net will need extra spec work.

Quoted 14 lines
content.tex +14 -0
+\drivernormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A driver MUST NOT set the virtqueue state before setting the
+  FEATURE_OK status bit.
+\item A driver MUST NOT set the virtqueue state after setting the
+  DRIVER_OK status bit.
+\end{itemize}
+
+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 494

Quoted 10 lines
content.tex +10 -0
+
+If VIRTIO_F_RING_STATE has been negotiated:
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its

may have?
should also go into a normative section

Quoted 29 lines
content.tex +10 -1
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.
+
  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
  We start with an overview of device initialization, then expand on the
@@ -420,6 +530,9 @@ \section{Device Initialization}\label{sec:General Initialization And Device Oper
     device, optional per-bus setup, reading and possibly writing the
     device's virtio configuration space, and population of virtqueues.
 
+\item\label{itm:General Initialization And Device Operation / Device
+  Initialization / Virtqueue State Setup} When VIRTIO_F_RING_STATE has been negotiated, perform virtqueue state setup, including the initialization of the per virtqueue available state, used state and the possible device specific virtqueue state.
+
  \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit.  At this point the device is
     ``live''.
  \end{enumerate}
@@ -6596,6 +6709,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
    transport specific.
    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_F_RING_STATE(40)] This feature indicates that the driver
+  can set and get the device internal virtqueue state.
+  See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
+
  \end{description}
 
  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
--

2.25.1

#12 lore raw

[virtio-comment] Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

In reply to Jason Wang
Quoted 24 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

This facility could be used by both migration and device diagnostic.

---
 content.tex | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)

+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}

content.tex +5 -0
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.
+
+If VIRTIO_F_RING_STATE has been negotiated:

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 496

Quoted 10 lines
content.tex +10 -0
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.

Maybe better

"If VIRTIO_F_RING_STATE has been negotiated, a device MAY provide a device-specific mechanism to set and get extra virtqueue states such as in flight reqeuests."

If a device type supports this facility, does it imply that it is always present when VIRTIO_RING_STATE has been negotiated? I guess it could define further device-specific features to make it more configurable.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#13 lore raw

[virtio-dev] Re: [PATCH V2 1/2] virtio: introduce virtqueue state as basic facility

Quoted 37 lines · Cornelia Huck
Quoted 23 lines · Jason Wang

This patch adds new device facility to save and restore virtqueue state. The virtqueue state is split into two parts:

- The available state: The state that is used for read the next available buffer.
- The used state: The state that is used for making buffer used.

Note that, there could be devices that is required to set and get the requests that are being processed by the device. I leave such API to be device specific.

This facility could be used by both migration and device diagnostic.

---
  content.tex | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 117 insertions(+)

+\devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}

content.tex +5 -0
+
+If VIRTIO_F_RING_STATE has not been negotiated, a device MUST ingore
+the read and write to the virtqueue state.
+
+If VIRTIO_F_RING_STATE has been negotiated:

content.tex \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo · line 496

Quoted 10 lines
content.tex +10 -0
+\begin{itemize}
+\item A device SHOULD ignore the write to the virtqueue state if the
+FEATURE_OK status bit is not set.
+\item A device SHOULD ignore the write to the virtqueue state if the
+DRIVER_OK status bit is set.
+\end{itemize}
+
+If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its
+device-specific way for the driver to set and get extra virtqueue
+states such as in flight requests.

Maybe better

Quoted 6 lines

"If VIRTIO_F_RING_STATE has been negotiated, a device MAY provide a device-specific mechanism to set and get extra virtqueue states such as in flight reqeuests."

If a device type supports this facility, does it imply that it is always present when VIRTIO_RING_STATE has been negotiated?

It's not, it's device specific.

Quoted 2 lines

I guess it could
define further device-specific features to make it more configurable.

Yes, I can clarify this in the next version.

Thanks

Quoted 1 line

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#14 lore raw

PATCH v2 2/2 virtio: introduce STOP status bit

In reply to Jason Wang

This patch introduces a new status bit STOP. This will be used by the driver to stop the device in order to safely fetch the device state (virtqueue state) from the device.

This is a must for supporting migration.

---
 content.tex | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 3 deletions(-)
content.tex +66 -3
index 8877b6f..284ead0 100644
@@ -47,6 +47,10 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
 \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
   drive the device.
 
+\item[STOP (32)] When VIRTIO_F_STOP is negotiated, indicates that the
+  device has been stopped by the driver. This status bit is different
+  from the reset since the device state is preserved.
+
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
   an error from which it can't recover.
 \end{description}
@@ -70,12 +74,38 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
 recover by issuing a reset.
 \end{note}
 
+If VIRTIO_F_STOP has been negotiated, the driver MUST NOT set STOP if
+DRIVER_OK is not set.
+
+If VIRTIO_F_STOP has been negotiated, to stop a device, after setting
+STOP, the driver MUST re-read the device status to ensure the STOP bit
+is set to synchronize with the device.
+
 \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field}
 The device MUST initialize \field{device status} to 0 upon reset.
 
 The device MUST NOT consume buffers or send any used buffer
 notifications to the driver before DRIVER_OK.
 
+If VIRTIO_F_STOP has not been negotiated, the device MUST ignore the
+write of STOP.
+
+If VIRTIO_F_STOP has been negotiated, after the driver writes STOP,
+the device MUST finish any pending operations like in flight requests
+or have its device specific way for driver to save the pending
+operations like in flight requests before setting the STOP status bit.
+
+If VIRTIO_F_STOP has been negotiated, the device MUST NOT consume
+buffers or send any used buffer notifications to the driver after
+STOP. The device MUST keep the configuration space unchanged and MUST
+NOT send configuration space change notification to the driver after
+STOP.
+
+If VIRTIO_F_STOP has been negotiated, after STOP, the device MUST
+preserve all the necessary state (the virtqueue states with the
+possible device specific states) that is required for restoring in the
+future.
+
 \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state
 that a reset is needed.  If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device
 MUST send a device configuration change notification to the driver.
@@ -474,8 +504,8 @@ \subsection{\field{Used State} Field}
 \begin{itemize}
 \item A driver MUST NOT set the virtqueue state before setting the
   FEATURE_OK status bit.
-\item A driver MUST NOT set the virtqueue state after setting the
-  DRIVER_OK status bit.
+\item A driver MUST NOT set the virtqueue state if DRIVER_OK status
+  bit is set without STOP status bit.
 \end{itemize}
 
 \devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
@@ -488,7 +518,7 @@ \subsection{\field{Used State} Field}
 \item A device SHOULD ignore the write to the virtqueue state if the
 FEATURE_OK status bit is not set.
 \item A device SHOULD ignore the write to the virtqueue state if the
-DRIVER_OK status bit is set.
+DRIVER_OK status bit is set without STOP status bit.
 \end{itemize}
 
 If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its
@@ -623,6 +653,36 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
 
 Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
 
+\section{Virtqueue State Saving}
+
+If both VIRTIO_F_RING_STATE and VIRTIO_F_STOP have been negotiated. A
+driver MAY save the internal virtqueue state.
+
+\drivernormative{\subsection}{Virtqueue State Saving}{General Initialization And Device Operation / Virtqueue State Saving}
+
+Assuming the device is 'live'. The driver MUST follow this sequence to
+stop the device and save the virtqueue state:
+
+\begin{enumerate}
+\item Set the STOP status bit.
+
+\item Re-read \field{device status} until the STOP bit is set to
+  synchronize with the device.
+
+\item Read \field{available state} and save it.
+
+\item Read \field{used state} and save it.
+
+\item Read device specific virtqueue states if needed.
+
+\item Reset the device.
+\end{enumerate}
+
+The driver MAY perform device specific steps to save device specific sate.
+
+The driver MAY 'resume' the device by redoing the device initialization
+with the saved virtqueue state. See ref\{sec:General Initialization and Device Operation / Device Initialization}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split
@@ -6713,6 +6773,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   can set and get the device internal virtqueue state.
   See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
 
+  \item[VIRTIO_F_STOP(41)] This feature indicates that the driver can
+  stop the device.
+  See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.25.1
#15 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 87 lines · * Jason Wang ([email protected]) wrote:

This patch introduces a new status bit STOP. This will be used by the driver to stop the device in order to safely fetch the device state (virtqueue state) from the device.

This is a must for supporting migration.

---
 content.tex | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 3 deletions(-)
content.tex +33 -3
index 8877b6f..284ead0 100644
@@ -47,6 +47,10 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
 \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
   drive the device.
 
+\item[STOP (32)] When VIRTIO_F_STOP is negotiated, indicates that the
+  device has been stopped by the driver. This status bit is different
+  from the reset since the device state is preserved.
+
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
   an error from which it can't recover.
 \end{description}
@@ -70,12 +74,38 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
 recover by issuing a reset.
 \end{note}
 
+If VIRTIO_F_STOP has been negotiated, the driver MUST NOT set STOP if
+DRIVER_OK is not set.
+
+If VIRTIO_F_STOP has been negotiated, to stop a device, after setting
+STOP, the driver MUST re-read the device status to ensure the STOP bit
+is set to synchronize with the device.
+
 \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field}
 The device MUST initialize \field{device status} to 0 upon reset.
 
 The device MUST NOT consume buffers or send any used buffer
 notifications to the driver before DRIVER_OK.
 
+If VIRTIO_F_STOP has not been negotiated, the device MUST ignore the
+write of STOP.
+
+If VIRTIO_F_STOP has been negotiated, after the driver writes STOP,
+the device MUST finish any pending operations like in flight requests
+or have its device specific way for driver to save the pending
+operations like in flight requests before setting the STOP status bit.
+
+If VIRTIO_F_STOP has been negotiated, the device MUST NOT consume
+buffers or send any used buffer notifications to the driver after
+STOP. The device MUST keep the configuration space unchanged and MUST
+NOT send configuration space change notification to the driver after
+STOP.
+
+If VIRTIO_F_STOP has been negotiated, after STOP, the device MUST
+preserve all the necessary state (the virtqueue states with the
+possible device specific states) that is required for restoring in the
+future.
+
 \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state
 that a reset is needed.  If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device
 MUST send a device configuration change notification to the driver.
@@ -474,8 +504,8 @@ \subsection{\field{Used State} Field}
 \begin{itemize}
 \item A driver MUST NOT set the virtqueue state before setting the
   FEATURE_OK status bit.
-\item A driver MUST NOT set the virtqueue state after setting the
-  DRIVER_OK status bit.
+\item A driver MUST NOT set the virtqueue state if DRIVER_OK status
+  bit is set without STOP status bit.
 \end{itemize}
 
 \devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
@@ -488,7 +518,7 @@ \subsection{\field{Used State} Field}
 \item A device SHOULD ignore the write to the virtqueue state if the
 FEATURE_OK status bit is not set.
 \item A device SHOULD ignore the write to the virtqueue state if the
-DRIVER_OK status bit is set.
+DRIVER_OK status bit is set without STOP status bit.
 \end{itemize}
 
 If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its
@@ -623,6 +653,36 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
 

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 663

Quoted 10 lines
content.tex +8 -0
 Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
 
+\section{Virtqueue State Saving}
+
+If both VIRTIO_F_RING_STATE and VIRTIO_F_STOP have been negotiated. A
+driver MAY save the internal virtqueue state.
+
+\drivernormative{\subsection}{Virtqueue State Saving}{General Initialization And Device Operation / Virtqueue State Saving}
+
+Assuming the device is 'live'.

Is that defined somewhere? If I understand correctly, this is all driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 670

Quoted 8 lines

The driver MUST follow this sequence to

content.tex +7 -0
+stop the device and save the virtqueue state:
+
+\begin{enumerate}
+\item Set the STOP status bit.
+
+\item Re-read \field{device status} until the STOP bit is set to
+  synchronize with the device.

At that point, is it guaranteed that the device has already stopped and that all outstanding transactions have finished?

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 678

Quoted 7 lines

+\item Read \field{available state} and save it.

content.tex +6 -0
+
+\item Read \field{used state} and save it.
+
+\item Read device specific virtqueue states if needed.
+
+\item Reset the device.

Say that a migration fails after this point; how does the source recover?
Why is the 'reset the device' there?

Quoted 40 lines

+\end{enumerate}

content.tex +9 -0
+
+The driver MAY perform device specific steps to save device specific sate.
+
+The driver MAY 'resume' the device by redoing the device initialization
+with the saved virtqueue state. See ref\{sec:General Initialization and Device Operation / Device Initialization}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split
@@ -6713,6 +6773,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   can set and get the device internal virtqueue state.
   See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
 
+  \item[VIRTIO_F_STOP(41)] This feature indicates that the driver can
+  stop the device.
+  See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 18 lines
2.25.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
author
#16 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 90 lines · Dr. David Alan Gilbert
Quoted 89 lines · * Jason Wang ([email protected]) wrote:

This patch introduces a new status bit STOP. This will be used by the driver to stop the device in order to safely fetch the device state (virtqueue state) from the device.

This is a must for supporting migration.

---
  content.tex | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++---
  1 file changed, 66 insertions(+), 3 deletions(-)
content.tex +33 -3
index 8877b6f..284ead0 100644
@@ -47,6 +47,10 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
  \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
    drive the device.
 
+\item[STOP (32)] When VIRTIO_F_STOP is negotiated, indicates that the
+  device has been stopped by the driver. This status bit is different
+  from the reset since the device state is preserved.
+
  \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
    an error from which it can't recover.
  \end{description}
@@ -70,12 +74,38 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
  recover by issuing a reset.
  \end{note}
 
+If VIRTIO_F_STOP has been negotiated, the driver MUST NOT set STOP if
+DRIVER_OK is not set.
+
+If VIRTIO_F_STOP has been negotiated, to stop a device, after setting
+STOP, the driver MUST re-read the device status to ensure the STOP bit
+is set to synchronize with the device.
+
  \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field}
  The device MUST initialize \field{device status} to 0 upon reset.
 
  The device MUST NOT consume buffers or send any used buffer
  notifications to the driver before DRIVER_OK.
 
+If VIRTIO_F_STOP has not been negotiated, the device MUST ignore the
+write of STOP.
+
+If VIRTIO_F_STOP has been negotiated, after the driver writes STOP,
+the device MUST finish any pending operations like in flight requests
+or have its device specific way for driver to save the pending
+operations like in flight requests before setting the STOP status bit.
+
+If VIRTIO_F_STOP has been negotiated, the device MUST NOT consume
+buffers or send any used buffer notifications to the driver after
+STOP. The device MUST keep the configuration space unchanged and MUST
+NOT send configuration space change notification to the driver after
+STOP.
+
+If VIRTIO_F_STOP has been negotiated, after STOP, the device MUST
+preserve all the necessary state (the virtqueue states with the
+possible device specific states) that is required for restoring in the
+future.
+
  \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state
  that a reset is needed.  If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device
  MUST send a device configuration change notification to the driver.
@@ -474,8 +504,8 @@ \subsection{\field{Used State} Field}
  \begin{itemize}
  \item A driver MUST NOT set the virtqueue state before setting the
    FEATURE_OK status bit.
-\item A driver MUST NOT set the virtqueue state after setting the
-  DRIVER_OK status bit.
+\item A driver MUST NOT set the virtqueue state if DRIVER_OK status
+  bit is set without STOP status bit.
  \end{itemize}
 
  \devicenormative{\subsection}{Virtqueue State}{Basic Facilities of a Virtio Device / Virtqueue State}
@@ -488,7 +518,7 @@ \subsection{\field{Used State} Field}
  \item A device SHOULD ignore the write to the virtqueue state if the
  FEATURE_OK status bit is not set.
  \item A device SHOULD ignore the write to the virtqueue state if the
-DRIVER_OK status bit is set.
+DRIVER_OK status bit is set without STOP status bit.
  \end{itemize}
 
  If VIRTIO_F_RING_STATE has been negotiated, a device MAY has its
@@ -623,6 +653,36 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
 
  Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
 
Quoted 9 lines

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /

Quoted 8 lines
content.tex +8 -0
+\section{Virtqueue State Saving}
+
+If both VIRTIO_F_RING_STATE and VIRTIO_F_STOP have been negotiated. A
+driver MAY save the internal virtqueue state.
+
+\drivernormative{\subsection}{Virtqueue State Saving}{General Initialization And Device Operation / Virtqueue State Saving}
+
+Assuming the device is 'live'.

Is that defined somewhere?

Probably not, but 'live' is used several times in the spec.

I can clarify that in the spec.

Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

Quoted 11 lines

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 670

Quoted 9 lines

The driver MUST follow this sequence to

content.tex +7 -0
+stop the device and save the virtqueue state:
+
+\begin{enumerate}
+\item Set the STOP status bit.
+
+\item Re-read \field{device status} until the STOP bit is set to
+  synchronize with the device.

At that point, is it guaranteed that the device has already stopped and that all outstanding transactions have finished?

No, the device may choose from one of the below:

1) all the pending transactions have finished

or

2) a device specific way for the driver to save and restore the pending transasctions

Quoted 10 lines

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 678

Quoted 8 lines

+\item Read \field{available state} and save it.

content.tex +6 -0
+
+\item Read \field{used state} and save it.
+
+\item Read device specific virtqueue states if needed.
+
+\item Reset the device.

Say that a migration fails after this point; how does the source recover?

Re-do the device initialization, and restore the state then.

Quoted 1 line

Why is the 'reset the device' there?

Because the current virtio device status machine is uni-directional. It's not allowed to clear a status bit. So the only way to recover is to reset and redo the initialization.

Thanks

Quoted 41 lines
Quoted 41 lines

+\end{enumerate}

content.tex +9 -0
+
+The driver MAY perform device specific steps to save device specific sate.
+
+The driver MAY 'resume' the device by redoing the device initialization
+with the saved virtqueue state. See ref\{sec:General Initialization and Device Operation / Device Initialization}
+
  \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
  Virtio can use various different buses, thus the standard is split
@@ -6713,6 +6773,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
    can set and get the device internal virtqueue state.
    See \ref{sec:Virtqueues / Virtqueue State}~\nameref{sec:Virtqueues / Virtqueue State}.
 
+  \item[VIRTIO_F_STOP(41)] This feature indicates that the driver can
+  stop the device.
+  See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}
  \end{description}
 
  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 18 lines
2.25.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#17 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 7 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!

#18 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 26 lines · Eugenio Perez Martin
Quoted 7 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.
Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

Signature · 1 line
MST
author
#19 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 25 lines · Michael S. Tsirkin
Quoted 24 lines · Eugenio Perez Martin
Quoted 6 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

Thanks

Quoted 1 line
#20 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 45 lines · Jason Wang
Quoted 25 lines · Michael S. Tsirkin
Quoted 24 lines · Eugenio Perez Martin
Quoted 6 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

Quoted 7 lines

It use the common technology like register shadowing without any further stuffs.

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time). It can be implemented concurrently (setting the STOP bit on all devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I'm a little worried about adding a new bit that requires busy waiting...

Stefan

author
#21 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 45 lines · Stefan Hajnoczi
Quoted 44 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 4 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 3 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Thanks

Quoted 2 lines

Stefan

#22 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 10 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

I don't think we can introduce arbitrary upper bounds here. At most, we can say that the device SHOULD try to set the STOP bit as early as possible (and make use of the mechanism to expose in-flight buffers.)

If we want to avoid polling for the STOP bit, we need some kind of notification mechanism, I guess. For ccw, I'd just use a channel command to stop the device; completion of that channel program would indicate that the device is done with the stop procedure. Not sure how well that translates to other transports.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#23 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 7 lines · Cornelia Huck
Quoted 5 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Quoted 7 lines
Quoted 4 lines

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

I don't think we can introduce arbitrary upper bounds here. At most, we can say that the device SHOULD try to set the STOP bit as early as possible (and make use of the mechanism to expose in-flight buffers.)

Yes, that's my understanding.

Quoted 5 lines

If we want to avoid polling for the STOP bit, we need some kind of notification mechanism, I guess. For ccw, I'd just use a channel command to stop the device; completion of that channel program would indicate that the device is done with the stop procedure.

A question, is interrupt used for such notification, or the VMM can choose to poll for the completion?

Quoted 2 lines

Not sure how
well that translates to other transports.

Actually, it's not necessarily a busy polling. VMM can schedule other process in and recheck the bit periodically.

Or as you mentioned before, we can use some kind of interrupt but it would be more complicated than the simple status bit. It's better to introduce the interrupt only if the status bit doesn't fit.

Thanks

Quoted 1 line
#24 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 21 lines · Jason Wang
Quoted 7 lines · Cornelia Huck
Quoted 5 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Quoted 7 lines
Quoted 4 lines

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

I don't think we can introduce arbitrary upper bounds here. At most, we can say that the device SHOULD try to set the STOP bit as early as possible (and make use of the mechanism to expose in-flight buffers.)

Yes, that's my understanding.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

If we want to avoid polling for the STOP bit, we need some kind of notification mechanism, I guess. For ccw, I'd just use a channel command to stop the device; completion of that channel program would indicate that the device is done with the stop procedure.

A question, is interrupt used for such notification, or the VMM can choose to poll for the completion?

You can poll for the subchannel to become status pending.

Quoted 6 lines
Quoted 4 lines

Not sure how
well that translates to other transports.

Quoted 6 lines

Actually, it's not necessarily a busy polling. VMM can schedule other process in and recheck the bit periodically.

Or as you mentioned before, we can use some kind of interrupt but it would be more complicated than the simple status bit. It's better to introduce the interrupt only if the status bit doesn't fit.

At least for ccw, waiting for the status bit to be set also involves an interrupt or polling (we use another channel program to retrieve the status.) A dedicated channel command would actually be better, as the interrupt/status pending would already inform us of success.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#25 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 36 lines · Cornelia Huck
Quoted 18 lines · Jason Wang
Quoted 6 lines · Cornelia Huck
Quoted 4 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Quoted 7 lines
Quoted 4 lines

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

I don't think we can introduce arbitrary upper bounds here. At most, we can say that the device SHOULD try to set the STOP bit as early as possible (and make use of the mechanism to expose in-flight buffers.)

Yes, that's my understanding.

Quoted 7 lines
Quoted 4 lines

If we want to avoid polling for the STOP bit, we need some kind of notification mechanism, I guess. For ccw, I'd just use a channel command to stop the device; completion of that channel program would indicate that the device is done with the stop procedure.

A question, is interrupt used for such notification, or the VMM can choose to poll for the completion?

You can poll for the subchannel to become status pending.

Quoted 7 lines
Quoted 3 lines

Not sure how
well that translates to other transports.

Actually, it's not necessarily a busy polling. VMM can schedule other process in and recheck the bit periodically.

Quoted 7 lines
Quoted 3 lines

Or as you mentioned before, we can use some kind of interrupt but it would be more complicated than the simple status bit. It's better to introduce the interrupt only if the status bit doesn't fit.

At least for ccw, waiting for the status bit to be set also involves an interrupt or polling (we use another channel program to retrieve the status.) A dedicated channel command would actually be better, as the interrupt/status pending would already inform us of success.

So it looks to me it doesn't conflict with this design: the device must wait for the device to be stopped to signal the success of the ccw command?

Thanks

Quoted 1 line

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#26 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 40 lines · Jason Wang
Quoted 39 lines · Cornelia Huck
Quoted 18 lines · Jason Wang
Quoted 6 lines · Cornelia Huck
Quoted 4 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Quoted 7 lines
Quoted 4 lines

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

I don't think we can introduce arbitrary upper bounds here. At most, we can say that the device SHOULD try to set the STOP bit as early as possible (and make use of the mechanism to expose in-flight buffers.)

Yes, that's my understanding.

Quoted 7 lines
Quoted 4 lines

If we want to avoid polling for the STOP bit, we need some kind of notification mechanism, I guess. For ccw, I'd just use a channel command to stop the device; completion of that channel program would indicate that the device is done with the stop procedure.

A question, is interrupt used for such notification, or the VMM can choose to poll for the completion?

You can poll for the subchannel to become status pending.

Quoted 10 lines
Quoted 3 lines

Not sure how
well that translates to other transports.

Actually, it's not necessarily a busy polling. VMM can schedule other process in and recheck the bit periodically.

Or as you mentioned before, we can use some kind of interrupt but it would be more complicated than the simple status bit. It's better to introduce the interrupt only if the status bit doesn't fit.

Quoted 8 lines
Quoted 4 lines

At least for ccw, waiting for the status bit to be set also involves an interrupt or polling (we use another channel program to retrieve the status.) A dedicated channel command would actually be better, as the interrupt/status pending would already inform us of success.

So it looks to me it doesn't conflict with this design: the device must wait for the device to be stopped to signal the success of the ccw command?

Yes, the difference is mainly how that information can be extracted.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#27 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 38 lines · Cornelia Huck
Quoted 36 lines · Jason Wang
Quoted 35 lines · Cornelia Huck
Quoted 17 lines · Jason Wang
Quoted 6 lines · Cornelia Huck
Quoted 4 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Quoted 7 lines
Quoted 4 lines

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

I don't think we can introduce arbitrary upper bounds here. At most, we can say that the device SHOULD try to set the STOP bit as early as possible (and make use of the mechanism to expose in-flight buffers.)

Yes, that's my understanding.

Quoted 6 lines
Quoted 4 lines

If we want to avoid polling for the STOP bit, we need some kind of notification mechanism, I guess. For ccw, I'd just use a channel command to stop the device; completion of that channel program would indicate that the device is done with the stop procedure.

A question, is interrupt used for such notification, or the VMM can choose to poll for the completion?

You can poll for the subchannel to become status pending.

Quoted 8 lines
Quoted 2 lines

Not sure how
well that translates to other transports.

Actually, it's not necessarily a busy polling. VMM can schedule other process in and recheck the bit periodically.

Or as you mentioned before, we can use some kind of interrupt but it would be more complicated than the simple status bit. It's better to introduce the interrupt only if the status bit doesn't fit.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

At least for ccw, waiting for the status bit to be set also involves an interrupt or polling (we use another channel program to retrieve the status.) A dedicated channel command would actually be better, as the interrupt/status pending would already inform us of success.

So it looks to me it doesn't conflict with this design: the device must wait for the device to be stopped to signal the success of the ccw command?

Yes, the difference is mainly how that information can be extracted.

So I had a look at how reset is described for ccw:

"

In order to reset a device, a driver sends the CCW_CMD_VDEV_RESET command.

"

This implies something similar, that is the success of the command means the success of the reset.

I wonder maybe I can remove the "re-read" from the basic facility and let the transport to decide what to do.

- for PCI, if a registers is used, we need re-read
- for CCW, follow the current implication, re-read is not needed and we can wait/poll for the success of the ccw command
- for admin virtqueue, it should be something similar to ccw, wait/poll for the success of the admin virtqueue command

Thanks

Quoted 1 line
#28 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 51 lines · Jason Wang
Quoted 38 lines · Cornelia Huck
Quoted 36 lines · Jason Wang
Quoted 35 lines · Cornelia Huck
Quoted 17 lines · Jason Wang
Quoted 6 lines · Cornelia Huck
Quoted 4 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Quoted 7 lines
Quoted 4 lines

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

I don't think we can introduce arbitrary upper bounds here. At most, we can say that the device SHOULD try to set the STOP bit as early as possible (and make use of the mechanism to expose in-flight buffers.)

Yes, that's my understanding.

Quoted 6 lines
Quoted 4 lines

If we want to avoid polling for the STOP bit, we need some kind of notification mechanism, I guess. For ccw, I'd just use a channel command to stop the device; completion of that channel program would indicate that the device is done with the stop procedure.

A question, is interrupt used for such notification, or the VMM can choose to poll for the completion?

You can poll for the subchannel to become status pending.

Quoted 8 lines
Quoted 2 lines

Not sure how
well that translates to other transports.

Actually, it's not necessarily a busy polling. VMM can schedule other process in and recheck the bit periodically.

Or as you mentioned before, we can use some kind of interrupt but it would be more complicated than the simple status bit. It's better to introduce the interrupt only if the status bit doesn't fit.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

At least for ccw, waiting for the status bit to be set also involves an interrupt or polling (we use another channel program to retrieve the status.) A dedicated channel command would actually be better, as the interrupt/status pending would already inform us of success.

So it looks to me it doesn't conflict with this design: the device must wait for the device to be stopped to signal the success of the ccw command?

Yes, the difference is mainly how that information can be extracted.

So I had a look at how reset is described for ccw:

Quoted 8 lines

"

In order to reset a device, a driver sends the CCW_CMD_VDEV_RESET command.

"

This implies something similar, that is the success of the command means the success of the reset.

Yes, indeed.

Quoted 7 lines

I wonder maybe I can remove the "re-read" from the basic facility and let the transport to decide what to do.

- for PCI, if a registers is used, we need re-read
- for CCW, follow the current implication, re-read is not needed and we can wait/poll for the success of the ccw command

If we are going with a status bit, it would be the same as for pci (we have WRITE_STATUS and READ_STATUS commands.) If we are going with a distinct command, we can skip the re-read. (I'd probably go with a more generic 'trigger an action' meta-command, but that would work just the same.)

Quoted 2 lines

- for admin virtqueue, it should be something similar to ccw, wait/poll for the success of the admin virtqueue command

Or we should maybe standardize on the admin virtqueue? That seems less confusing to me.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#29 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 61 lines · Cornelia Huck
Quoted 49 lines · Jason Wang
Quoted 38 lines · Cornelia Huck
Quoted 36 lines · Jason Wang
Quoted 35 lines · Cornelia Huck
Quoted 17 lines · Jason Wang
Quoted 6 lines · Cornelia Huck
Quoted 4 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Quoted 7 lines
Quoted 4 lines

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

I don't think we can introduce arbitrary upper bounds here. At most, we can say that the device SHOULD try to set the STOP bit as early as possible (and make use of the mechanism to expose in-flight buffers.)

Yes, that's my understanding.

Quoted 6 lines
Quoted 4 lines

If we want to avoid polling for the STOP bit, we need some kind of notification mechanism, I guess. For ccw, I'd just use a channel command to stop the device; completion of that channel program would indicate that the device is done with the stop procedure.

A question, is interrupt used for such notification, or the VMM can choose to poll for the completion?

You can poll for the subchannel to become status pending.

Quoted 8 lines
Quoted 2 lines

Not sure how
well that translates to other transports.

Actually, it's not necessarily a busy polling. VMM can schedule other process in and recheck the bit periodically.

Or as you mentioned before, we can use some kind of interrupt but it would be more complicated than the simple status bit. It's better to introduce the interrupt only if the status bit doesn't fit.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

At least for ccw, waiting for the status bit to be set also involves an interrupt or polling (we use another channel program to retrieve the status.) A dedicated channel command would actually be better, as the interrupt/status pending would already inform us of success.

So it looks to me it doesn't conflict with this design: the device must wait for the device to be stopped to signal the success of the ccw command?

Yes, the difference is mainly how that information can be extracted.

So I had a look at how reset is described for ccw:

Quoted 8 lines

"

In order to reset a device, a driver sends the CCW_CMD_VDEV_RESET command.

"

This implies something similar, that is the success of the command means the success of the reset.

Yes, indeed.

Quoted 8 lines
Quoted 6 lines

I wonder maybe I can remove the "re-read" from the basic facility and let the transport to decide what to do.

- for PCI, if a registers is used, we need re-read
- for CCW, follow the current implication, re-read is not needed and we can wait/poll for the success of the ccw command

If we are going with a status bit, it would be the same as for pci (we have WRITE_STATUS and READ_STATUS commands.)

So spec is unclear of the implications of the success of a command:

E.g for RESET (CCW_CMD_VDEV_RESET), the success of the command implies the success or the reset.

But for set_status (CCW_CMD_WRITE_STATUS), the success of the command does not imply the bit is set by the device.

If I understand this correctly, we still need re-read here.

Quoted 2 lines

If we are going with a
distinct command, we can skip the re-read.

Then it would be better to introduce the STOP as a dedicated device facility (as reset):

The device MUST present STOP bit after it has been stopped.

And for PCI:

- it was set via set the bit in the registers

for ccw:

- a distinct command (as reset) is introduced, and STOP is forbidden to set via device status?

Quoted 7 lines

(I'd probably go with a more
generic 'trigger an action' meta-command, but that would work just the same.)

Quoted 2 lines

- for admin virtqueue, it should be something similar to ccw, wait/poll for the success of the admin virtqueue command

Or we should maybe standardize on the admin virtqueue?

That's one way to go.

Quoted 2 lines

That seems less
confusing to me.

But it's just one of the possible interface to carry the commands. We still need to define the semantic or facility of "stop" first.

And we still need to clarify the implication for the success of each specific command as ccw. (E.g whether or not a re-read(get after set) is need)

The only difference is the transport: ccw command vs virtqqueue.

Thanks

Quoted 1 line
#30 lore raw

[virtio-dev] Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 20 lines · Jason Wang
Quoted 19 lines · Cornelia Huck
Quoted 10 lines · Jason Wang

So I had a look at how reset is described for ccw:

"

In order to reset a device, a driver sends the CCW_CMD_VDEV_RESET command.

"

This implies something similar, that is the success of the command means the success of the reset.

Yes, indeed.

Quoted 6 lines

I wonder maybe I can remove the "re-read" from the basic facility and let the transport to decide what to do.

- for PCI, if a registers is used, we need re-read
- for CCW, follow the current implication, re-read is not needed and we can wait/poll for the success of the ccw command

Quoted 8 lines
Quoted 2 lines

If we are going with a status bit, it would be the same as for pci (we have WRITE_STATUS and READ_STATUS commands.)

So spec is unclear of the implications of the success of a command:

E.g for RESET (CCW_CMD_VDEV_RESET), the success of the command implies the success or the reset.

Yes, sending RESET is basically the ccw equivalent of "write 0 to the status", and getting a status/interrupt that the command finished successfully is the equivalent of "get 0 when reading the status back". [We did not have a "read back status" command originally.]

Quoted 3 lines

But for set_status (CCW_CMD_WRITE_STATUS), the success of the command does not imply the bit is set by the device.

Yes, the success only indicates that the device has received the command successfully. It can still refuse to set some values, or only set them later.

Quoted 2 lines

If I understand this correctly, we still need re-read here.

Yes.

[Let me know if we can make this more clear in the spec!]

Quoted 11 lines
Quoted 4 lines

If we are going with a
distinct command, we can skip the re-read.

Then it would be better to introduce the STOP as a dedicated device facility (as reset):

The device MUST present STOP bit after it has been stopped.

Quoted 8 lines

And for PCI:

- it was set via set the bit in the registers

for ccw:

- a distinct command (as reset) is introduced, and STOP is forbidden to set via device status?

I think the situation for reset is different: a zero status is a natural way to express that a device is in its initial state. It does not really matter whether it is a freshly initialized device, or whether it has been reset by the driver, or which mechanism the driver is using.

For STOP, we'd end up indicating a certain status, with one way to actually write the status, and the other a dedicated command. I'd expect that the driver will still read the status to check whether the STOP bit is present, as it may take some time, regardless of the transport used (going by the Linux implementation, the various callbacks to interact with the device state are assumed to be synchronous, and we have to make the asynchronous ccw interactions synchronous beneath the covers; if we stick with that model for STOP, the asynchronous nature of ccw commands does not buy us anything.)

So, maybe using the same mechanism for every transport is better, if we end up reading the status back anyway.

Quoted 18 lines
Quoted 9 lines

(I'd probably go with a more
generic 'trigger an action' meta-command, but that would work just the same.)

Quoted 2 lines

- for admin virtqueue, it should be something similar to ccw, wait/poll for the success of the admin virtqueue command

Or we should maybe standardize on the admin virtqueue?

That's one way to go.

Quoted 2 lines

That seems less
confusing to me.

Quoted 8 lines

But it's just one of the possible interface to carry the commands. We still need to define the semantic or facility of "stop" first.

And we still need to clarify the implication for the success of each specific command as ccw. (E.g whether or not a re-read(get after set) is need)

The only difference is the transport: ccw command vs virtqqueue.

Historically, too many differences in how the transports implement device/driver interactions have lead to some awkwardness (see e.g. the might_sleep annotations, which are surprising to someone working with the pci transport.) So I think there's benefit in making the interactions either very similar, or so different that the transports can do their own things. (As said above, having an extra ccw command for STOP is probably only useful if generic code isn't polling the status for the bit to be set.)

So, maybe either/or

- write STOP to status, read it back (via already existing methods)
- use a virtqueue

The extra ccw command would only make sense if other transports implemented STOP via e.g. a new register (would that also be an option?)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#31 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 36 lines · Cornelia Huck
Quoted 20 lines · Jason Wang
Quoted 19 lines · Cornelia Huck
Quoted 10 lines · Jason Wang

So I had a look at how reset is described for ccw:

"

In order to reset a device, a driver sends the CCW_CMD_VDEV_RESET command.

"

This implies something similar, that is the success of the command means the success of the reset.

Yes, indeed.

Quoted 6 lines

I wonder maybe I can remove the "re-read" from the basic facility and let the transport to decide what to do.

- for PCI, if a registers is used, we need re-read
- for CCW, follow the current implication, re-read is not needed and we can wait/poll for the success of the ccw command

Quoted 7 lines
Quoted 2 lines

If we are going with a status bit, it would be the same as for pci (we have WRITE_STATUS and READ_STATUS commands.)

So spec is unclear of the implications of the success of a command:

E.g for RESET (CCW_CMD_VDEV_RESET), the success of the command implies the success or the reset.

Yes, sending RESET is basically the ccw equivalent of "write 0 to the status", and getting a status/interrupt that the command finished successfully is the equivalent of "get 0 when reading the status back". [We did not have a "read back status" command originally.]

Quoted 2 lines

But for set_status (CCW_CMD_WRITE_STATUS), the success of the command does not imply the bit is set by the device.

Quoted 8 lines

Yes, the success only indicates that the device has received the command successfully. It can still refuse to set some values, or only set them later.

Quoted 1 line

If I understand this correctly, we still need re-read here.

Yes.

[Let me know if we can make this more clear in the spec!]

We probably need to clarify them, I can only deduce those implications by reading the kernel driver codes.

Basically the subtle difference between VDEV_RESET and WRITE_STATUS.

Quoted 33 lines
Quoted 10 lines
Quoted 4 lines

If we are going with a
distinct command, we can skip the re-read.

Then it would be better to introduce the STOP as a dedicated device facility (as reset):

The device MUST present STOP bit after it has been stopped.

Quoted 8 lines

And for PCI:

- it was set via set the bit in the registers

for ccw:

- a distinct command (as reset) is introduced, and STOP is forbidden to set via device status?

I think the situation for reset is different: a zero status is a natural way to express that a device is in its initial state. It does not really matter whether it is a freshly initialized device, or whether it has been reset by the driver, or which mechanism the driver is using.

For STOP, we'd end up indicating a certain status, with one way to actually write the status, and the other a dedicated command. I'd expect that the driver will still read the status to check whether the STOP bit is present, as it may take some time, regardless of the transport used (going by the Linux implementation, the various callbacks to interact with the device state are assumed to be synchronous, and we have to make the asynchronous ccw interactions synchronous beneath the covers; if we stick with that model for STOP, the asynchronous nature of ccw commands does not buy us anything.)

Quoted 2 lines

So, maybe using the same mechanism for every transport is better, if we end up reading the status back anyway.

Ok.

Quoted 23 lines
Quoted 23 lines
Quoted 9 lines

(I'd probably go with a more
generic 'trigger an action' meta-command, but that would work just the same.)

Quoted 2 lines

- for admin virtqueue, it should be something similar to ccw, wait/poll for the success of the admin virtqueue command

Or we should maybe standardize on the admin virtqueue?

That's one way to go.

Quoted 2 lines

That seems less
confusing to me.

But it's just one of the possible interface to carry the commands. We still need to define the semantic or facility of "stop" first.

And we still need to clarify the implication for the success of each specific command as ccw. (E.g whether or not a re-read(get after set) is need)

Quoted 5 lines
Quoted 1 line

The only difference is the transport: ccw command vs virtqqueue.

Historically, too many differences in how the transports implement device/driver interactions have lead to some awkwardness (see e.g. the might_sleep annotations, which are surprising to someone working with the pci transport.)

Yes but it actually helps for the transport that depends on an extra layer to talk with the device. So it's not bad.

Quoted 5 lines

So I think there's benefit in making the
interactions either very similar, or so different that the transports can do their own things. (As said above, having an extra ccw command for STOP is probably only useful if generic code isn't polling the status for the bit to be set.)

I agree, but it will require the "general api" like admin virtqueue to carry not only the STOP stuffs but all the other configurations. Otherwise it's still a partial solution.

Quoted 8 lines

So, maybe either/or

- write STOP to status, read it back (via already existing methods)
- use a virtqueue

The extra ccw command would only make sense if other transports implemented STOP via e.g. a new register (would that also be an option?)

That's possible, but it's better then to introduce it as a basic facility and clarify how it interact with the existing device status.

Thanks

Quoted 1 line
#32 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Jason Wang
Quoted 103 lines · Jason Wang
Quoted 45 lines · Stefan Hajnoczi
Quoted 44 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 4 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 3 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated. In fact, the VMM would need to hide this bit and it's safer to keep it out-of-band instead of risking exposing it by accident.

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again. This isn't addressed in this patch series, but it's the next step and I think it's worth planning for it.

If all devices could stop very quickly and were stateless then I would agree that the STOP bit is an ideal solution. I think it will be necessary to support devices that don't behave like that, so the admin virtqueue approach seems worth exploring.

Stefan

author
#33 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 105 lines · Stefan Hajnoczi
Quoted 94 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 5 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 4 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

And with admin virtqueue, it's actually far more complicated e.g you need to define how to synchronize the concurrent access to the basic facilites.

Quoted 2 lines

This isn't addressed in this patch series, but it's the
next step and I think it's worth planning for it.

I agree, but for admin virtqueue, it's better to use it as a full transport instead of just use it for carrying part of the device basic facilities. Actually, as I said, I had patches to do that. But the motivation is not for live migration but for device slicing. I will post RFC before the KVM Forum this year (since I'm going to talk device slicing at virtio level). It does not conflict with Max's proposal, since migration part is not there.

Quoted 3 lines

If all devices could stop very quickly and were stateless then I would agree that the STOP bit is an ideal solution.

Note that in Max's proposal it also have something similar the "quiescence" and "freezed" state. It doesn't differ from STOP bit fundamentally. As Max suggested, we could introduce more status bit if necessary or even consider to unify Max's proposal with mine.

Quoted 3 lines

I think it will be
necessary to support devices that don't behave like that, so the admin virtqueue approach seems worth exploring.

Yes and as mentioned in another thread. I think the best way is to define the device specific state first and then consider how to implement the interface.

Admin virtqueue is worth to explore but should not be the only method. Device/transport are freed to implement it in many ways based on the actual hardware.

Thanks

Quoted 2 lines

Stefan

#34 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 163 lines · Jason Wang
Quoted 105 lines · Stefan Hajnoczi
Quoted 94 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 5 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

If we're going to need it for saving/loading device state anyway, then that's another reason to consider using a virtqueue for stopping the device, saving/loading virtqueue state, etc.

Quoted 2 lines

And with admin virtqueue, it's actually far more complicated e.g you need to define how to synchronize the concurrent access to the basic facilites.

I'm not sure I understand? Driver complexity? Device implementation complexity?

Quoted 10 lines
Quoted 2 lines

This isn't addressed in this patch series, but it's the
next step and I think it's worth planning for it.

I agree, but for admin virtqueue, it's better to use it as a full transport instead of just use it for carrying part of the device basic facilities. Actually, as I said, I had patches to do that. But the motivation is not for live migration but for device slicing. I will post RFC before the KVM Forum this year (since I'm going to talk device slicing at virtio level). It does not conflict with Max's proposal, since migration part is not there.

Great, I'm looking forward to your device slicing idea.

Quoted 15 lines
Quoted 2 lines

If all devices could stop very quickly and were stateless then I would agree that the STOP bit is an ideal solution.

Note that in Max's proposal it also have something similar the "quiescence" and "freezed" state. It doesn't differ from STOP bit fundamentally. As Max suggested, we could introduce more status bit if necessary or even consider to unify Max's proposal with mine.

Quoted 3 lines

I think it will be
necessary to support devices that don't behave like that, so the admin virtqueue approach seems worth exploring.

Quoted 7 lines

Yes and as mentioned in another thread. I think the best way is to define the device specific state first and then consider how to implement the interface.

Admin virtqueue is worth to explore but should not be the only method. Device/transport are freed to implement it in many ways based on the actual hardware.

What's the advantage of this proposal compared to an admin virtqueue? I see the admin virtqueue as a more general interface than this proposal and it can cover this use case.

Stefan

author
#35 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 158 lines · Stefan Hajnoczi
Quoted 157 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 90 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it? And DMA doesn't means a virtqueue, it could be a transport specific method.

I think we need to start from defining the state of one specific device and see what is the best interface.

Note that software can choose to intercept all the control commands, and shadow them. This means the best interface could be device specific.

Quoted 4 lines

If we're going to need it for saving/loading device state anyway, then that's another reason to consider using a virtqueue for stopping the device, saving/loading virtqueue state, etc.

It requires much more works than the simple virtqueue interface: (the main issues is that the function is not self-contained in a single function)

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device in the admin virtqueue? If yes, how to handle the concurrent access from PF and VF?
5) how it is expected to work with non-PCI virtio device?

And as I've stated several times, virtqueue is the interface or transport which carries the commands for implementing specific semantics. It doesn't conflict with what is proposed in this patch.

Quoted 5 lines
Quoted 3 lines

And with admin virtqueue, it's actually far more complicated e.g you need to define how to synchronize the concurrent access to the basic facilites.

I'm not sure I understand? Driver complexity? Device implementation complexity?

See the above questions.

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

This isn't addressed in this patch series, but it's the
next step and I think it's worth planning for it.

I agree, but for admin virtqueue, it's better to use it as a full transport instead of just use it for carrying part of the device basic facilities. Actually, as I said, I had patches to do that. But the motivation is not for live migration but for device slicing. I will post RFC before the KVM Forum this year (since I'm going to talk device slicing at virtio level). It does not conflict with Max's proposal, since migration part is not there.

Great, I'm looking forward to your device slicing idea.

Thanks

Quoted 14 lines
Quoted 14 lines
Quoted 3 lines

If all devices could stop very quickly and were stateless then I would agree that the STOP bit is an ideal solution.

Note that in Max's proposal it also have something similar the "quiescence" and "freezed" state. It doesn't differ from STOP bit fundamentally. As Max suggested, we could introduce more status bit if necessary or even consider to unify Max's proposal with mine.

Quoted 3 lines

I think it will be
necessary to support devices that don't behave like that, so the admin virtqueue approach seems worth exploring.

Quoted 8 lines
Quoted 7 lines

Yes and as mentioned in another thread. I think the best way is to define the device specific state first and then consider how to implement the interface.

Admin virtqueue is worth to explore but should not be the only method. Device/transport are freed to implement it in many ways based on the actual hardware.

What's the advantage of this proposal compared to an admin virtqueue?

See above. They are not contradict. Admin virtqueue could be used to transport the basic facility that is introduced in this patch.

Quoted 3 lines

I
see the admin virtqueue as a more general interface than this proposal and it can cover this use case.

Actually not, this proposal doesn't limit how it was actually implemented in the device or transport. So it is a much general proposal.

If you are talking about the patch of implementing it via PCI capability, it's just to demonstrate one of the possible implementation.

Thanks

Quoted 2 lines

Stefan

#36 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 163 lines · Jason Wang
Quoted 161 lines · Stefan Hajnoczi
Quoted 160 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 90 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Quoted 6 lines
Quoted 3 lines

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM. That's basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach. For example, some of the virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification. I think it makes sense to standardize it if it's possible to convey all necessary state and device implementors can easily implement this device state representation. If not, then device implementation-specific device state would be needed.

I think that's a larger discussion that deserves its own email thread.

Quoted 13 lines

Note that software can choose to intercept all the control commands, and shadow them. This means the best interface could be device specific.

Quoted 4 lines

If we're going to need it for saving/loading device state anyway, then that's another reason to consider using a virtqueue for stopping the device, saving/loading virtqueue state, etc.

It requires much more works than the simple virtqueue interface: (the main issues is that the function is not self-contained in a single function)

Quoted 7 lines

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device in the admin virtqueue? If yes, how to handle the concurrent access from PF and VF?
5) how it is expected to work with non-PCI virtio device?

I guess your device splitting proposal addresses some of these things?

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

Quoted 3 lines

And as I've stated several times, virtqueue is the interface or transport which carries the commands for implementing specific semantics. It doesn't conflict with what is proposed in this patch.

The abstract operations for stopping the device and fetching virtqueue state sound good to me, but I don't think a Device Status field STOP bit should be added. An out-of-band stop operation would support devices that take a long time to stop better.

Stefan

#37 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 10 lines · Stefan Hajnoczi
Quoted 10 lines

It requires much more works than the simple virtqueue interface: (the main issues is that the function is not self-contained in a single function)

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device in the admin virtqueue? If yes, how to handle the concurrent access from PF and VF?
5) how it is expected to work with non-PCI virtio device?

Quoted 7 lines

I guess your device splitting proposal addresses some of these things?

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

I think we mentioned this in the past and agreed that the only common entity between my solution for virtio VF migration to this proposal is the new admin control queue.

I can prepare some draft for this.

In our solution the PF will manage migration process for it's VFs using the PF admin queue. PF is not migratable.

I don't know who is using nested environments in production so don't know if it worth talking about that.

But, if you would like to implement it for testing, no problem. The VF in level n, probably seen as PF in level n+1. So it can manage the migration process for its nested VFs.

For question 5) what non-PCI devices are interesting in live migration ?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#38 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 20 lines · Max Gurtovoy
Quoted 10 lines · Stefan Hajnoczi
Quoted 10 lines

It requires much more works than the simple virtqueue interface: (the main issues is that the function is not self-contained in a single function)

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device in the admin virtqueue? If yes, how to handle the concurrent access from PF and VF?
5) how it is expected to work with non-PCI virtio device?

Quoted 7 lines

I guess your device splitting proposal addresses some of these things?

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

Quoted 5 lines

I think we mentioned this in the past and agreed that the only common entity between my solution for virtio VF migration to this proposal is the new admin control queue.

I can prepare some draft for this.

In our solution the PF will manage migration process for it's VFs using the PF admin queue. PF is not migratable.

That limits the use cases.

Quoted 2 lines

I don't know who is using nested environments in production so don't know if it worth talking about that.

There should be plenty users for the nested case.

Quoted 2 lines

But, if you would like to implement it for testing, no problem. The VF in level n, probably seen as PF in level n+1. So it can manage the migration process for its nested VFs.

The PF dependency makes the design almost impossible to be used in a nested environment.

Quoted 3 lines

For question 5) what non-PCI devices are interesting in live migration ?

Why not? Virtio support transport other than PCI (CCW, MMIO).

Thanks

#39 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 39 lines · Jason Wang
Quoted 23 lines · Max Gurtovoy
Quoted 13 lines · Stefan Hajnoczi
Quoted 4 lines

It requires much more works than the simple virtqueue interface:
(the main
issues is that the function is not self-contained in a single function)

Quoted 9 lines

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device
in the admin virtqueue? If yes, how to handle the concurrent access from PF
and VF?
5) how it is expected to work with non-PCI virtio device?

Quoted 7 lines

I guess your device splitting proposal addresses some of these things?

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

Quoted 8 lines

I think we mentioned this in the past and agreed that the only common entity between my solution for virtio VF migration to this proposal is the new admin control queue.

I can prepare some draft for this.

In our solution the PF will manage migration process for it's VFs using the PF admin queue. PF is not migratable.

That limits the use cases.

Quoted 1 line
Quoted 5 lines
Quoted 2 lines

I don't know who is using nested environments in production so don't know if it worth talking about that.

There should be plenty users for the nested case.

Yes, nested virtualization is becoming available in clouds, etc. I think nested virtualization support should be part of the design.

Quoted 8 lines
Quoted 4 lines

But, if you would like to implement it for testing, no problem. The VF in level n, probably seen as PF in level n+1. So it can manage the migration process for its nested VFs.

The PF dependency makes the design almost impossible to be used in a nested environment.

I'm not sure I understood Max's example, but first I want to check I understand yours:

A physical PF is passed through to an L1 guest. L2 guests are assigned VFs created by the L1 guest from the PF.

Now we want to live migrate the L1 guest to another host. We need to migrate the PF and its VFs are automatically included since there is no migration from the L2 perspective?

Quoted 5 lines
Quoted 3 lines

For question 5) what non-PCI devices are interesting in live migration ?

Why not? Virtio support transport other than PCI (CCW, MMIO).

Yes, VIRTIO isn't tied to PCI and the migration functionality should be mappable to other transports.

Luckily the admin virtqueue approach maps naturally to other transports. What requires more thought is how the admin virtqueue is enumerated/managed on those other transports.

Stefan

author
#40 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 51 lines · Stefan Hajnoczi
Quoted 31 lines · Jason Wang
Quoted 21 lines · Max Gurtovoy
Quoted 13 lines · Stefan Hajnoczi
Quoted 4 lines

It requires much more works than the simple virtqueue interface:
(the main
issues is that the function is not self-contained in a single function)

Quoted 9 lines

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device
in the admin virtqueue? If yes, how to handle the concurrent access from PF
and VF?
5) how it is expected to work with non-PCI virtio device?

Quoted 7 lines

I guess your device splitting proposal addresses some of these things?

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

Quoted 8 lines

I think we mentioned this in the past and agreed that the only common entity between my solution for virtio VF migration to this proposal is the new admin control queue.

I can prepare some draft for this.

In our solution the PF will manage migration process for it's VFs using the PF admin queue. PF is not migratable.

Quoted 7 lines

That limits the use cases.

Quoted 2 lines

I don't know who is using nested environments in production so don't know if it worth talking about that.

There should be plenty users for the nested case.

Yes, nested virtualization is becoming available in clouds, etc. I think nested virtualization support should be part of the design.

Quoted 6 lines
Quoted 3 lines

But, if you would like to implement it for testing, no problem. The VF in level n, probably seen as PF in level n+1. So it can manage the migration process for its nested VFs.

The PF dependency makes the design almost impossible to be used in a nested environment.

I'm not sure I understood Max's example, but first I want to check I understand yours:

Quoted 6 lines

A physical PF is passed through to an L1 guest. L2 guests are assigned VFs created by the L1 guest from the PF.

Now we want to live migrate the L1 guest to another host. We need to migrate the PF and its VFs are automatically included since there is no migration from the L2 perspective?

Yes, and I believe the more common case is.

PF is for L0, and we want to migrate L2 guest.

This can hardly work in the current design.

The reason is that the functions is not self contained in the VF.

Quoted 10 lines
Quoted 4 lines
Quoted 3 lines

For question 5) what non-PCI devices are interesting in live migration ?

Why not? Virtio support transport other than PCI (CCW, MMIO).

Yes, VIRTIO isn't tied to PCI and the migration functionality should be mappable to other transports.

Luckily the admin virtqueue approach maps naturally to other transports. What requires more thought is how the admin virtqueue is enumerated/managed on those other transports.

So admin virtqueue is really one way to go. But we can't mandate it in the spec. Sometime, it would be hard to define where the admin virtqueue needs to be located consider the transport may lack the concept of something like PF.

To me the most valuable part of the admin virtqueue is that it sits in the PF (or management device) where the DMA is naturally isolated.

Thanks

Quoted 2 lines

Stefan

#41 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 61 lines · Jason Wang
Quoted 51 lines · Stefan Hajnoczi
Quoted 31 lines · Jason Wang
Quoted 21 lines · Max Gurtovoy
Quoted 13 lines · Stefan Hajnoczi
Quoted 4 lines

It requires much more works than the simple virtqueue interface:
(the main
issues is that the function is not self-contained in a single function)

Quoted 9 lines

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device
in the admin virtqueue? If yes, how to handle the concurrent access from PF
and VF?
5) how it is expected to work with non-PCI virtio device?

Quoted 7 lines

I guess your device splitting proposal addresses some of these things?

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

Quoted 8 lines

I think we mentioned this in the past and agreed that the only common entity between my solution for virtio VF migration to this proposal is the new admin control queue.

I can prepare some draft for this.

In our solution the PF will manage migration process for it's VFs using the PF admin queue. PF is not migratable.

Quoted 7 lines

That limits the use cases.

Quoted 2 lines

I don't know who is using nested environments in production so don't know if it worth talking about that.

There should be plenty users for the nested case.

Yes, nested virtualization is becoming available in clouds, etc. I think nested virtualization support should be part of the design.

Quoted 6 lines
Quoted 3 lines

But, if you would like to implement it for testing, no problem. The VF in level n, probably seen as PF in level n+1. So it can manage the migration process for its nested VFs.

The PF dependency makes the design almost impossible to be used in a nested environment.

I'm not sure I understood Max's example, but first I want to check I understand yours:

Quoted 6 lines

A physical PF is passed through to an L1 guest. L2 guests are assigned VFs created by the L1 guest from the PF.

Now we want to live migrate the L1 guest to another host. We need to migrate the PF and its VFs are automatically included since there is no migration from the L2 perspective?

Quoted 7 lines

Yes, and I believe the more common case is.

PF is for L0, and we want to migrate L2 guest.

This can hardly work in the current design.

The reason is that the functions is not self contained in the VF.

Thanks for highlighting this case. It requires that the mechanism for stopping and saving/loading state comes with the VF so the L1 guest can perform live migration even though it does not have L0 PF access.

Stefan

author
#42 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 61 lines · Stefan Hajnoczi
Quoted 60 lines · Jason Wang
Quoted 48 lines · Stefan Hajnoczi
Quoted 28 lines · Jason Wang
Quoted 27 lines · Max Gurtovoy
Quoted 13 lines · Stefan Hajnoczi
Quoted 4 lines

It requires much more works than the simple virtqueue interface:
(the main
issues is that the function is not self-contained in a single function)

Quoted 9 lines

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device
in the admin virtqueue? If yes, how to handle the concurrent access from PF
and VF?
5) how it is expected to work with non-PCI virtio device?

Quoted 7 lines

I guess your device splitting proposal addresses some of these things?

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

I think we mentioned this in the past and agreed that the only common entity between my solution for virtio VF migration to this proposal is the new admin control queue.

I can prepare some draft for this.

Quoted 8 lines
Quoted 2 lines

In our solution the PF will manage migration process for it's VFs using the PF admin queue. PF is not migratable.

That limits the use cases.

Quoted 2 lines

I don't know who is using nested environments in production so don't know if it worth talking about that.

There should be plenty users for the nested case.

Yes, nested virtualization is becoming available in clouds, etc. I think nested virtualization support should be part of the design.

Quoted 5 lines
Quoted 3 lines

But, if you would like to implement it for testing, no problem. The VF in level n, probably seen as PF in level n+1. So it can manage the migration process for its nested VFs.

The PF dependency makes the design almost impossible to be used in a nested environment.

I'm not sure I understood Max's example, but first I want to check I understand yours:

Quoted 6 lines

A physical PF is passed through to an L1 guest. L2 guests are assigned VFs created by the L1 guest from the PF.

Now we want to live migrate the L1 guest to another host. We need to migrate the PF and its VFs are automatically included since there is no migration from the L2 perspective?

Yes, and I believe the more common case is.

PF is for L0, and we want to migrate L2 guest.

Quoted 8 lines
Quoted 3 lines

This can hardly work in the current design.

The reason is that the functions is not self contained in the VF.

Thanks for highlighting this case. It requires that the mechanism for stopping and saving/loading state comes with the VF so the L1 guest can perform live migration even though it does not have L0 PF access.

Stefan

Yes.

Thanks

#43 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Jason Wang
Quoted 47 lines · Jason Wang
Quoted 23 lines · Max Gurtovoy
Quoted 13 lines · Stefan Hajnoczi
Quoted 4 lines

It requires much more works than the simple virtqueue interface:
(the main
issues is that the function is not self-contained in a single function)

Quoted 9 lines

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device
in the admin virtqueue? If yes, how to handle the concurrent access from PF
and VF?
5) how it is expected to work with non-PCI virtio device?

Quoted 7 lines

I guess your device splitting proposal addresses some of these things?

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

Quoted 8 lines

I think we mentioned this in the past and agreed that the only common entity between my solution for virtio VF migration to this proposal is the new admin control queue.

I can prepare some draft for this.

In our solution the PF will manage migration process for it's VFs using the PF admin queue. PF is not migratable.

That limits the use cases.

Quoted 3 lines

I don't know who is using nested environments in production so don't know if it worth talking about that.

There should be plenty users for the nested case.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

But, if you would like to implement it for testing, no problem. The VF in level n, probably seen as PF in level n+1. So it can manage the migration process for its nested VFs.

The PF dependency makes the design almost impossible to be used in a nested environment.

Can't we have an emulated PF? This would be the most reasonable thing to do.

Quoted 9 lines
Quoted 4 lines

For question 5) what non-PCI devices are interesting in live migration ?

Why not? Virtio support transport other than PCI (CCW, MMIO).

Thanks

author
#44 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Jason Wang
Quoted 21 lines · Max Gurtovoy
Quoted 13 lines · Stefan Hajnoczi
Quoted 4 lines

It requires much more works than the simple virtqueue interface:
(the main
issues is that the function is not self-contained in a single function)

Quoted 9 lines

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device
in the admin virtqueue? If yes, how to handle the concurrent access from PF
and VF?
5) how it is expected to work with non-PCI virtio device?

Quoted 7 lines

I guess your device splitting proposal addresses some of these things?

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

Quoted 8 lines

I think we mentioned this in the past and agreed that the only common entity between my solution for virtio VF migration to this proposal is the new admin control queue.

I can prepare some draft for this.

In our solution the PF will manage migration process for it's VFs using the PF admin queue. PF is not migratable.

That limits the use cases.

Quoted 2 lines

I don't know who is using nested environments in production so don't know if it worth talking about that.

There should be plenty users for the nested case.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

But, if you would like to implement it for testing, no problem. The VF in level n, probably seen as PF in level n+1. So it can manage the migration process for its nested VFs.

The PF dependency makes the design almost impossible to be used in a nested environment.

Can't we have an emulated PF? This would be the most reasonable thing to do.

Isn't this much more complicated?

- VMM needs to emulate PF
- it means we need the support for PF live migration
- if we want to live migrate VF in L(x) we need PF(physical or emulated) from L0 to L(x-1)
- complicate the management stack

If we really want to go with admin + PF, it should be used with the per VF function as well. That is hide the admin vq + PF from any of the above layers.

L0 virtio device: admin + PF
L1 virtio device: L0 VMM presents the function that is self contained in a single function which is implemented vai (admin + PF)
...
Lx virtio device: L(x-1) VMM presents the function that is self contained in a single function

In another way, if all the function is self contained, we get even more simplified model

Lx virtio device: L(x-1) VMM presents the function that is self contained in a single function

But the OS should hide the implementation details on how the device is mediated and expose a unified uAPI for the userspace VMM(Qemu). That is what vDPA tries to achieve, provides a single device centric uAPI. This is another call for having a self contained per device(function) interface for all those stuffs. It would be very complicated to introduce uAPI for PF (or what ever it was called), and actually we can't mandate this even if we invent them. We've already had vendors that implements the virtqueue state via VF BAR(capability).

Thanks

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

For question 5) what non-PCI devices are interesting in live migration ?

Why not? Virtio support transport other than PCI (CCW, MMIO).

Thanks

author
#45 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Stefan Hajnoczi
Quoted 173 lines · Stefan Hajnoczi
Quoted 155 lines · Jason Wang
Quoted 154 lines · Stefan Hajnoczi
Quoted 153 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 90 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 7 lines
Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability
3) Not a proper uAPI desgin

Quoted 3 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

Quoted 6 lines

For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 3 lines

I think it makes sense to
standardize it if it's possible to convey all necessary state and device implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

Quoted 2 lines

If
not, then device implementation-specific device state would be needed.

Yes.

Quoted 2 lines

I think that's a larger discussion that deserves its own email thread.

I agree, but it doesn't prevent us from starting from simple device that virtqueue state is sufficient (e.g virtio-net).

Quoted 12 lines
Quoted 12 lines

Note that software can choose to intercept all the control commands, and shadow them. This means the best interface could be device specific.

Quoted 3 lines

If we're going to need it for saving/loading device state anyway, then that's another reason to consider using a virtqueue for stopping the device, saving/loading virtqueue state, etc.

It requires much more works than the simple virtqueue interface: (the main issues is that the function is not self-contained in a single function)

Quoted 8 lines
Quoted 7 lines

1) how to interact with the existing device status state machine?
2) how to make it work in a nested environment?
3) how to migrate the PF?
4) do we need to allow more control other than just stop/freeze the device in the admin virtqueue? If yes, how to handle the concurrent access from PF and VF?
5) how it is expected to work with non-PCI virtio device?

I guess your device splitting proposal addresses some of these things?

Note that the device facility doesn't limit how it is used. So the difference is per-function(VF) interface vs PF interface.

Per-function interface is self contained so it can address all the above issues:

1) STOP is the part of the device status state machine
2) it's self contained in the function, so it works in the nested environment by simply assign the function without any other dependency
3) for PF, those function is still self-contained, so it can be migrated
4) the problem doesn't exist since we have a single control path
5) non PCI device is freed to implement their own per device interface

And actually, the per function interface has already been done by some vendors.

Quoted 6 lines

Max probably has the most to say about these points.

If you want more input I can try to answer too, but I personally am not developing devices that need this right now, so I might not be the best person to propose solutions.

I think for us we should make sure the architecture is good enough to be not limited to any specific use cases.

Quoted 8 lines
Quoted 4 lines

And as I've stated several times, virtqueue is the interface or transport which carries the commands for implementing specific semantics. It doesn't conflict with what is proposed in this patch.

The abstract operations for stopping the device and fetching virtqueue state sound good to me, but I don't think a Device Status field STOP bit should be added. An out-of-band stop operation would support devices that take a long time to stop better.

So the long time request is not something that is introduced by the STOP bit. Spec already use that for reset.

Thanks

Quoted 2 lines

Stefan

#46 lore raw

[virtio-dev] Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 5 lines · Jason Wang
Quoted 4 lines · Stefan Hajnoczi
Quoted 3 lines · Jason Wang

And as I've stated several times, virtqueue is the interface or transport which carries the commands for implementing specific semantics. It doesn't conflict with what is proposed in this patch.

Quoted 8 lines
Quoted 4 lines

The abstract operations for stopping the device and fetching virtqueue state sound good to me, but I don't think a Device Status field STOP bit should be added. An out-of-band stop operation would support devices that take a long time to stop better.

So the long time request is not something that is introduced by the STOP bit. Spec already use that for reset.

Reset doesn't affect migration downtime. The register polling approach is problematic during migration downtime because it's difficult to stop multiple devices and do other downtime cleanup concurrently. Stopping devices sequentially increases migration downtime, so I think the interface should encourage concurrently stopping multiple devices.

I think you and Cornelia discussed that an interrupt could be added to solve this problem. That would address my concerns about the STOP bit.

Stefan

author
#47 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Jason Wang
Quoted 8 lines · Stefan Hajnoczi
Quoted 3 lines · Jason Wang

And as I've stated several times, virtqueue is the interface or transport which carries the commands for implementing specific semantics. It doesn't conflict with what is proposed in this patch.

The abstract operations for stopping the device and fetching virtqueue state sound good to me, but I don't think a Device Status field STOP bit should be added. An out-of-band stop operation would support devices that take a long time to stop better.

Quoted 5 lines
Quoted 2 lines

So the long time request is not something that is introduced by the STOP bit. Spec already use that for reset.

Reset doesn't affect migration downtime. The register polling approach is problematic during migration downtime because it's difficult to stop multiple devices and do other downtime cleanup concurrently.

This part I don't understand. We don't have an centralized control path that is used for each virtual functions.

VMM is free to stop multiple devices and poll for all those device status?

Quoted 6 lines

Stopping
devices sequentially increases migration downtime, so I think the interface should encourage concurrently stopping multiple devices.

I think you and Cornelia discussed that an interrupt could be added to solve this problem. That would address my concerns about the STOP bit.

The problems are:

1) if we generate an interrupt after STOP, it breaks the STOP semantic where the device should not generate any interrupt
2) if we generate an interrupt before STOP, we may end up with race conditions

Thanks

Quoted 2 lines

Stefan

#48 lore raw

Re: [virtio-dev] Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 20 lines · Jason Wang
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Jason Wang
Quoted 8 lines · Stefan Hajnoczi
Quoted 3 lines · Jason Wang

And as I've stated several times, virtqueue is the interface or transport which carries the commands for implementing specific semantics. It doesn't conflict with what is proposed in this patch.

The abstract operations for stopping the device and fetching virtqueue state sound good to me, but I don't think a Device Status field STOP bit should be added. An out-of-band stop operation would support devices that take a long time to stop better.

Quoted 5 lines
Quoted 2 lines

So the long time request is not something that is introduced by the STOP bit. Spec already use that for reset.

Reset doesn't affect migration downtime. The register polling approach is problematic during migration downtime because it's difficult to stop multiple devices and do other downtime cleanup concurrently.

Quoted 4 lines

This part I don't understand. We don't have an centralized control path that is used for each virtual functions.

VMM is free to stop multiple devices and poll for all those device status?

Yes, it's possible to do that but I think it's harder for VMMs to implement and consumes CPU (which competes with software devices that are trying to stop).

Stefan

author
#49 lore raw

Re: [virtio-dev] Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 18 lines · Stefan Hajnoczi
Quoted 17 lines · Jason Wang
Quoted 15 lines · Stefan Hajnoczi
Quoted 11 lines · Jason Wang
Quoted 8 lines · Stefan Hajnoczi
Quoted 3 lines · Jason Wang

And as I've stated several times, virtqueue is the interface or transport which carries the commands for implementing specific semantics. It doesn't conflict with what is proposed in this patch.

The abstract operations for stopping the device and fetching virtqueue state sound good to me, but I don't think a Device Status field STOP bit should be added. An out-of-band stop operation would support devices that take a long time to stop better.

So the long time request is not something that is introduced by the STOP bit. Spec already use that for reset.

Reset doesn't affect migration downtime. The register polling approach is problematic during migration downtime because it's difficult to stop multiple devices and do other downtime cleanup concurrently.

Quoted 7 lines
Quoted 4 lines

This part I don't understand. We don't have an centralized control path that is used for each virtual functions.

VMM is free to stop multiple devices and poll for all those device status?

Yes, it's possible to do that but I think it's harder for VMMs to implement and consumes CPU (which competes with software devices that are trying to stop).

Possibly, actually, there are two issues:

1) How to send the command to check the succeeding of the command
2) Whether or not we need a notification for the completion of the command

What is being proposed is 1), and 2) could be done via a transport specific way. (But requires more thought).

Actually, it's the software that choose the best way for them, it can do busy polling, timer etc.

Thanks

Quoted 2 lines

Stefan

#50 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Jason Wang
Quoted 9 lines · Jason Wang
Quoted 6 lines · Stefan Hajnoczi

Stopping
devices sequentially increases migration downtime, so I think the interface should encourage concurrently stopping multiple devices.

I think you and Cornelia discussed that an interrupt could be added to solve this problem. That would address my concerns about the STOP bit.

Quoted 6 lines

The problems are:

1) if we generate an interrupt after STOP, it breaks the STOP semantic where the device should not generate any interrupt
2) if we generate an interrupt before STOP, we may end up with race conditions

I think not all interrupts are created equal here.

For virtqueue notification interrupts, I agree. If the device is being stopped, no notification interrupts will be generated.

For device interrupts in the general sense, banning these would make it impossible to implement STOP for CCW, as any channel program (be it RESET, READ_STATUS, or any new one) is required to generate a status pending/interrupt when it is finished. I also don't see how that would create a race condition for CCW.

Why can't we simply have an interrupt indicating completion of the STOP request, and no further interrupts after that?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#51 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 18 lines · Cornelia Huck
Quoted 8 lines · Jason Wang
Quoted 6 lines · Stefan Hajnoczi

Stopping
devices sequentially increases migration downtime, so I think the interface should encourage concurrently stopping multiple devices.

I think you and Cornelia discussed that an interrupt could be added to solve this problem. That would address my concerns about the STOP bit.

Quoted 6 lines

The problems are:

1) if we generate an interrupt after STOP, it breaks the STOP semantic where the device should not generate any interrupt
2) if we generate an interrupt before STOP, we may end up with race conditions

I think not all interrupts are created equal here.

Quoted 7 lines

For virtqueue notification interrupts, I agree. If the device is being stopped, no notification interrupts will be generated.

For device interrupts in the general sense, banning these would make it impossible to implement STOP for CCW, as any channel program (be it RESET, READ_STATUS, or any new one) is required to generate a status pending/interrupt when it is finished.

So they are working at different levels. STOP is at the virtio level not the transport level.

For STOP, it means the device stop working at virtio level, it doesn't mean the device doesn't work at transport level. It means CCW can still have its interrupt if it's not generated from the virtiqueue or config.

So did RESET: I believe for both CCW and PCI, and virtio RESET doesn't meant a transport level reset.

Quoted 5 lines

I also don't see how that would
create a race condition for CCW.

Why can't we simply have an interrupt indicating completion of the STOP request, and no further interrupts after that?

So I think we had some discussion on this. I think STOP should work as RESET.

And if we want an indication of STOP, why don't we need it for RESET?

Thanks

Quoted 1 line
#52 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Jason Wang
Quoted 180 lines · Jason Wang
Quoted 178 lines · Stefan Hajnoczi
Quoted 155 lines · Jason Wang
Quoted 154 lines · Stefan Hajnoczi
Quoted 153 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 90 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 3 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work
the guest must be running and already have initialised the driver.

Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

Quoted 8 lines
Quoted 2 lines

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

Quoted 9 lines
Quoted 5 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.
2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 12 lines
Quoted 6 lines

For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 7 lines
Quoted 3 lines

I think it makes sense to
standardize it if it's possible to convey all necessary state and device implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Quoted 8 lines
Quoted 2 lines

If
not, then device implementation-specific device state would be needed.

Yes.

Quoted 1 line
Quoted 5 lines
Quoted 1 line

I think that's a larger discussion that deserves its own email thread.

I agree, but it doesn't prevent us from starting from simple device that virtqueue state is sufficient (e.g virtio-net).

Right, stateless devices don't need to save/load device state, at least when virtqueues are passed through selectively instead of full passsthrough.

Stefan

author
#53 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 182 lines · Stefan Hajnoczi
Quoted 181 lines · Jason Wang
Quoted 179 lines · Stefan Hajnoczi
Quoted 155 lines · Jason Wang
Quoted 154 lines · Stefan Hajnoczi
Quoted 153 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 90 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin
Quoted 5 lines

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work
the guest must be running and already have initialised the driver.
Yes.

As I see it, the feature can be driven entirely by the VMM as long as it intercept the relevant configuration space (PCI, MMIO, etc) from guest's reads and writes, and present it as coherent and transparent for the guest. Some use cases I can imagine with a physical device (or vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device. 2.1) The VMM stops the device to perform a live migration, and the guest does not write to STOP in any moment of the LM. It resets the destination device with the state, and then initializes the device. 2.2) The guest stops the device and, when STOP(32) is set, the source VMM migrates the device status. The destination VMM realizes the bit, so it sets the bit in the destination too after device initialization. 2.3) The device is not initialized by the guest so it doesn't matter what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 6 lines
Quoted 2 lines

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

Quoted 7 lines
Quoted 2 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 5 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

This brings troubles in the nested environment.

Thanks

Quoted 52 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 11 lines
Quoted 6 lines

For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 6 lines
Quoted 3 lines

I think it makes sense to
standardize it if it's possible to convey all necessary state and device implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Quoted 10 lines
Quoted 2 lines

If
not, then device implementation-specific device state would be needed.

Yes.

Quoted 1 line

I think that's a larger discussion that deserves its own email thread.

I agree, but it doesn't prevent us from starting from simple device that virtqueue state is sufficient (e.g virtio-net).

Right, stateless devices don't need to save/load device state, at least when virtqueues are passed through selectively instead of full passsthrough.

Stefan

author
#54 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 256 lines · Jason Wang
Quoted 186 lines · Stefan Hajnoczi
Quoted 185 lines · Jason Wang
Quoted 183 lines · Stefan Hajnoczi
Quoted 159 lines · Jason Wang
Quoted 158 lines · Stefan Hajnoczi
Quoted 157 lines · Jason Wang
Quoted 105 lines · Stefan Hajnoczi
Quoted 94 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin

        If I understand correctly, this is all
driven from the driver inside the guest, so for this to work
the guest must be running and already have initialised the driver.
Yes.

As I see it, the feature can be driven entirely by the VMM as long as
it intercept the relevant configuration space (PCI, MMIO, etc) from
guest's reads and writes, and present it as coherent and transparent
for the guest. Some use cases I can imagine with a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device.
2.1) The VMM stops the device to perform a live migration, and the
guest does not write to STOP in any moment of the LM. It resets the
destination device with the state, and then initializes the device.
2.2) The guest stops the device and, when STOP(32) is set, the source
VMM migrates the device status. The destination VMM realizes the bit,
so it sets the bit in the destination too after device initialization.
2.3) The device is not initialized by the guest so it doesn't matter
what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin
virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a
per function
interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all
the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin
virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

      It can be implemented concurrently (setting the STOP bit on all devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

     In fact, the VMM would need to hide this bit and it's safer to keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 6 lines
Quoted 2 lines

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

Quoted 7 lines
Quoted 2 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 5 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

Quoted 41 lines

This brings troubles in the nested environment.

Thanks

Quoted 36 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 11 lines
Quoted 6 lines

   For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 6 lines
Quoted 3 lines

   I think it makes sense to
standardize it if it's possible to convey all necessary state and device implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Having two ways will bring extra complexity. That why I suggest:

- to have general facility for the virtuqueue to be migrated
- leave the device specific state to be device specific. so device can choose what is convenient way or interface.

Thanks

Quoted 16 lines
Quoted 16 lines
Quoted 11 lines
Quoted 3 lines

If
not, then device implementation-specific device state would be needed.

Yes.

Quoted 1 line

I think that's a larger discussion that deserves its own email thread.

I agree, but it doesn't prevent us from starting from simple device that virtqueue state is sufficient (e.g virtio-net).

Right, stateless devices don't need to save/load device state, at least when virtqueues are passed through selectively instead of full passsthrough.

Stefan

#55 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 418 lines · Jason Wang
Quoted 372 lines · Jason Wang
Quoted 284 lines · Stefan Hajnoczi
Quoted 283 lines · Jason Wang
Quoted 281 lines · Stefan Hajnoczi
Quoted 258 lines · Jason Wang
Quoted 257 lines · Stefan Hajnoczi
Quoted 256 lines · Jason Wang
Quoted 187 lines · Stefan Hajnoczi
Quoted 170 lines · Jason Wang
Quoted 100 lines · Stefan Hajnoczi
Quoted 93 lines · Jason Wang
Quoted 72 lines · Michael S. Tsirkin

On Fri, Jul 09, 2021 at
07:23:33PM +0200, Eugenio
Perez Martin wrote:
        If I understand correctly, this is all
driven from the
driver inside
the guest, so
for this to work
the guest must
be running and
already have
initialised the
driver.
Yes.

As I see it, the feature
can be driven entirely
by the VMM as long as
it intercept the
relevant configuration
space (PCI, MMIO, etc)
from
guest's reads and
writes, and present it
as coherent and
transparent
for the guest. Some use
cases I can imagine with
a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not
to pass the feature
flag. The guest cannot
stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the
flag to the guest. The
guest can stop the
device.
2.1) The VMM stops the
device to perform a live
migration, and the
guest does not write to
STOP in any moment of
the LM. It resets the
destination device with
the state, and then
initializes the device.
2.2) The guest stops the
device and, when
STOP(32) is set, the
source
VMM migrates the device
status. The destination
VMM realizes the bit,
so it sets the bit in
the destination too
after device
initialization.
2.3) The device is not
initialized by the guest
so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's
all a lot of hoops to jump
through though.
It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard
to implement?

E.g for networking device, it should be sufficient to use this bit + the
virtqueue state.

Quoted 4 lines

Why don't we design the
feature in a way that is
useable by VMMs
and implementable by devices in a simple way?

It use the common technology like register shadowing without any further
stuffs.

Quoted 6 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through
those state registers).

Quoted 6 lines

If an admin virtqueue is used instead of the STOP Device Status field
bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with
admin virtqueue. Actually, admin
virtqueue could be used for carrying any
basic device facility like status
bit. E.g I'm going to post patches that
use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin
virtqueue, we still need a per function interface for this. This is a must for
nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no
need for the device to complete all
the consumed buffers, device can choose
to expose those inflight descriptors
in a device specific way and set the
STOP bit. This means, if we have the
device specific in-flight descriptor
reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device
facility but using the admin
virtqueue specific method, we still need
to clarify how it works with the
device status state machine, it will be
some kind of sub-states which looks
much more complicated than the current proposal.

Quoted 7 lines

When migrating a guest with many VIRTIO devices a busy waiting approach
extends downtime if implemented sequentially (stopping one device at a
time).

Well. You need some kinds of waiting for
sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio implementation which is hard to be
restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

Quoted 6 lines

      It can be implemented
concurrently (setting the STOP bit
on all
devices and then looping until all
their Device Status fields have the
bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common configuration structure layout

Quoted 7 lines

After writing 0 to device_status, the
driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit
is a small and clean VIRTIO
spec change. I like that.

On the other hand, devices need time to stop and that time can be unbounded. For example, software
virtio-blk/scsi implementations since
cannot immediately cancel in-flight I/O requests on Linux hosts.

Quoted 4 lines

The natural interface for long-running
operations is virtqueue requests.
That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said
before, admin virtqueue
could be used for carrying the device status bit.

Send a command to set STOP status bit to admin
virtqueue. Device will make
the command buffer used after it has
successfully stopped the device.

AFAIK, they are not mutually exclusive, since
they are trying to solve
different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way
to implement (part of) the
device facility

Quoted 5 lines

Although you mentioned that the stopped
state needs to be reflected in
the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit.
VMM will shadow the device
status bit in this case.

E.g with the current vhost-vDPA, vDPA behave
like a vhost device, guest is
unaware of the migration.

STOP status bit is set by Qemu to real virtio
hardware. But guest will only
see the DRIVER_OK without STOP.

It's not hard to implement the nested on top,
see the discussion initiated
by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 3 lines

     In fact, the VMM would need to hide
this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the
capability. It's useful for
migrating a nested guest.

If we design an interface that can be used in
the nested environment, it's
not an ideal interface.

Quoted 5 lines

In addition, stateful devices need to
load/save non-trivial amounts of
data. They need DMA to do this efficiently,
so an admin virtqueue is a
good fit again.

I don't get the point here. You still need to address the exact the similar
issues for admin virtqueue: the unbound time in freezing the device, the
interaction with the virtio device status state machine.

Quoted 4 lines
Quoted 3 lines

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements
it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 3 lines

I think we need to start from defining the state of one specific device and
see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 6 lines
Quoted 2 lines

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

That's
basically the difference between the vhost/vDPA's selective passthrough
approach and VFIO's full passthrough approach.

Quoted 8 lines
Quoted 3 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is
required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION
    to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

Quoted 42 lines

This brings troubles in the nested environment.

Thanks

Quoted 37 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 11 lines
Quoted 6 lines

   For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 7 lines
Quoted 4 lines

   I think it makes sense to
standardize it if it's possible to convey all necessary state and device
implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Quoted 5 lines

Having two ways will bring extra complexity. That why I suggest:

- to have general facility for the virtuqueue to be migrated
- leave the device specific state to be device specific. so device can choose what is convenient way or interface.

I don't think we have a choice. For stateful devices it can be impossible to define a standard device state representation.

Stefan

author
#56 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 404 lines · Stefan Hajnoczi
Quoted 403 lines · Jason Wang
Quoted 401 lines · Jason Wang
Quoted 283 lines · Stefan Hajnoczi
Quoted 282 lines · Jason Wang
Quoted 281 lines · Stefan Hajnoczi
Quoted 258 lines · Jason Wang
Quoted 257 lines · Stefan Hajnoczi
Quoted 256 lines · Jason Wang
Quoted 187 lines · Stefan Hajnoczi
Quoted 170 lines · Jason Wang
Quoted 100 lines · Stefan Hajnoczi
Quoted 99 lines · Jason Wang

在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at
07:23:33PM +0200, Eugenio
Perez Martin wrote:
        If I understand correctly, this is all
driven from the
driver inside
the guest, so
for this to work
the guest must
be running and
already have
initialised the
driver.
Yes.

As I see it, the feature
can be driven entirely
by the VMM as long as
it intercept the
relevant configuration
space (PCI, MMIO, etc)
from
guest's reads and
writes, and present it
as coherent and
transparent
for the guest. Some use
cases I can imagine with
a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not
to pass the feature
flag. The guest cannot
stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the
flag to the guest. The
guest can stop the
device.
2.1) The VMM stops the
device to perform a live
migration, and the
guest does not write to
STOP in any moment of
the LM. It resets the
destination device with
the state, and then
initializes the device.
2.2) The guest stops the
device and, when
STOP(32) is set, the
source
VMM migrates the device
status. The destination
VMM realizes the bit,
so it sets the bit in
the destination too
after device
initialization.
2.3) The device is not
initialized by the guest
so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's
all a lot of hoops to jump
through though.
It's also not easy for devices to implement.
It just requires a new status
bit. Anything that makes you
think it's hard
to implement?

E.g for networking device, it
should be sufficient to use this
bit + the
virtqueue state.

Why don't we design the
feature in a way that is
useable by VMMs
and implementable by devices in a simple way?
It use the common technology
like register shadowing without
any further
stuffs.

Or do you have any other ideas?

(I think we all know migration
will be very hard if we simply
pass through
those state registers).

Quoted 6 lines

If an admin virtqueue is used instead of the STOP Device Status field
bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with
admin virtqueue. Actually, admin
virtqueue could be used for carrying any
basic device facility like status
bit. E.g I'm going to post patches that
use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin
virtqueue, we still need a per function interface for this. This is a must for
nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no
need for the device to complete all
the consumed buffers, device can choose
to expose those inflight descriptors
in a device specific way and set the
STOP bit. This means, if we have the
device specific in-flight descriptor
reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device
facility but using the admin
virtqueue specific method, we still need
to clarify how it works with the
device status state machine, it will be
some kind of sub-states which looks
much more complicated than the current proposal.

Quoted 7 lines

When migrating a guest with many VIRTIO devices a busy waiting approach
extends downtime if implemented sequentially (stopping one device at a
time).

Well. You need some kinds of waiting for
sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio implementation which is hard to be
restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

Quoted 6 lines

      It can be implemented
concurrently (setting the STOP bit
on all
devices and then looping until all
their Device Status fields have the
bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common configuration structure layout

Quoted 7 lines

After writing 0 to device_status, the
driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit
is a small and clean VIRTIO
spec change. I like that.

On the other hand, devices need time to stop and that time can be unbounded. For example, software
virtio-blk/scsi implementations since
cannot immediately cancel in-flight I/O requests on Linux hosts.

Quoted 4 lines

The natural interface for long-running
operations is virtqueue requests.
That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said
before, admin virtqueue
could be used for carrying the device status bit.

Send a command to set STOP status bit to admin
virtqueue. Device will make
the command buffer used after it has
successfully stopped the device.

AFAIK, they are not mutually exclusive, since
they are trying to solve
different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way
to implement (part of) the
device facility

Quoted 5 lines

Although you mentioned that the stopped
state needs to be reflected in
the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit.
VMM will shadow the device
status bit in this case.

E.g with the current vhost-vDPA, vDPA behave
like a vhost device, guest is
unaware of the migration.

STOP status bit is set by Qemu to real virtio
hardware. But guest will only
see the DRIVER_OK without STOP.

It's not hard to implement the nested on top,
see the discussion initiated
by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 3 lines

     In fact, the VMM would need to hide
this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the
capability. It's useful for
migrating a nested guest.

If we design an interface that can be used in
the nested environment, it's
not an ideal interface.

Quoted 5 lines

In addition, stateful devices need to
load/save non-trivial amounts of
data. They need DMA to do this efficiently,
so an admin virtqueue is a
good fit again.

I don't get the point here. You still need to address the exact the similar
issues for admin virtqueue: the unbound time in freezing the device, the
interaction with the virtio device status state machine.

Quoted 4 lines
Quoted 3 lines

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements
it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 3 lines

I think we need to start from defining the state of one specific device and
see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That's
basically the difference between the vhost/vDPA's selective passthrough
approach and VFIO's full passthrough approach.

Quoted 8 lines
Quoted 3 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is
required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises
VFIO_REGION_TYPE_MIGRATION
    to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

This brings troubles in the nested environment.

Thanks

Quoted 34 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 10 lines
Quoted 6 lines

   For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 6 lines
Quoted 4 lines

   I think it makes sense to
standardize it if it's possible to convey all necessary state and device
implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Quoted 7 lines
Quoted 5 lines

Having two ways will bring extra complexity. That why I suggest:

- to have general facility for the virtuqueue to be migrated
- leave the device specific state to be device specific. so device can choose what is convenient way or interface.

I don't think we have a choice. For stateful devices it can be impossible to define a standard device state representation.

Let me clarify, I agree we can't have a standard device state for all kinds of device.

That's way I tend to leave them to be device specific. (but not implementation specific)

But we can generalize the virtqueue state for sure.

Thanks

Quoted 2 lines

Stefan

#57 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 415 lines · Jason Wang
Quoted 404 lines · Stefan Hajnoczi
Quoted 403 lines · Jason Wang
Quoted 401 lines · Jason Wang
Quoted 283 lines · Stefan Hajnoczi
Quoted 282 lines · Jason Wang
Quoted 281 lines · Stefan Hajnoczi
Quoted 258 lines · Jason Wang
Quoted 257 lines · Stefan Hajnoczi
Quoted 256 lines · Jason Wang
Quoted 187 lines · Stefan Hajnoczi
Quoted 170 lines · Jason Wang
Quoted 106 lines · Stefan Hajnoczi

On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at
07:23:33PM +0200, Eugenio
Perez Martin wrote:
        If I understand correctly, this is all
driven from the
driver inside
the guest, so
for this to work
the guest must
be running and
already have
initialised the
driver.
Yes.

As I see it, the feature
can be driven entirely
by the VMM as long as
it intercept the
relevant configuration
space (PCI, MMIO, etc)
from
guest's reads and
writes, and present it
as coherent and
transparent
for the guest. Some use
cases I can imagine with
a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not
to pass the feature
flag. The guest cannot
stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the
flag to the guest. The
guest can stop the
device.
2.1) The VMM stops the
device to perform a live
migration, and the
guest does not write to
STOP in any moment of
the LM. It resets the
destination device with
the state, and then
initializes the device.
2.2) The guest stops the
device and, when
STOP(32) is set, the
source
VMM migrates the device
status. The destination
VMM realizes the bit,
so it sets the bit in
the destination too
after device
initialization.
2.3) The device is not
initialized by the guest
so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's
all a lot of hoops to jump
through though.
It's also not easy for devices to implement.
It just requires a new status
bit. Anything that makes you
think it's hard
to implement?

E.g for networking device, it
should be sufficient to use this
bit + the
virtqueue state.

Why don't we design the
feature in a way that is
useable by VMMs
and implementable by devices in a simple way?
It use the common technology
like register shadowing without
any further
stuffs.

Or do you have any other ideas?

(I think we all know migration
will be very hard if we simply
pass through
those state registers).
If an admin virtqueue is used
instead of the STOP Device Status
field
bit then there's no need to re-read
the Device Status field in a loop
until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with
admin virtqueue. Actually, admin
virtqueue could be used for carrying any
basic device facility like status
bit. E.g I'm going to post patches that
use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin
virtqueue, we still need a per function interface for this. This is a must for
nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no
need for the device to complete all
the consumed buffers, device can choose
to expose those inflight descriptors
in a device specific way and set the
STOP bit. This means, if we have the
device specific in-flight descriptor
reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device
facility but using the admin
virtqueue specific method, we still need
to clarify how it works with the
device status state machine, it will be
some kind of sub-states which looks
much more complicated than the current proposal.

Quoted 7 lines

When migrating a guest with many
VIRTIO devices a busy waiting
approach
extends downtime if implemented
sequentially (stopping one device at
a
time).

Well. You need some kinds of waiting for
sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio implementation which is hard to be
restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

Quoted 6 lines

      It can be implemented
concurrently (setting the STOP bit
on all
devices and then looping until all
their Device Status fields have the
bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy
waiting...

Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common configuration structure layout

Quoted 7 lines

After writing 0 to device_status, the
driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit
is a small and clean VIRTIO
spec change. I like that.

On the other hand, devices need time to stop and that time can be unbounded. For example, software
virtio-blk/scsi implementations since
cannot immediately cancel in-flight I/O requests on Linux hosts.

Quoted 4 lines

The natural interface for long-running
operations is virtqueue requests.
That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said
before, admin virtqueue
could be used for carrying the device status bit.

Send a command to set STOP status bit to admin
virtqueue. Device will make
the command buffer used after it has
successfully stopped the device.

AFAIK, they are not mutually exclusive, since
they are trying to solve
different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way
to implement (part of) the
device facility

Quoted 5 lines

Although you mentioned that the stopped
state needs to be reflected in
the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit.
VMM will shadow the device
status bit in this case.

E.g with the current vhost-vDPA, vDPA behave
like a vhost device, guest is
unaware of the migration.

STOP status bit is set by Qemu to real virtio
hardware. But guest will only
see the DRIVER_OK without STOP.

It's not hard to implement the nested on top,
see the discussion initiated
by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 3 lines

     In fact, the VMM would need to hide
this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the
capability. It's useful for
migrating a nested guest.

If we design an interface that can be used in
the nested environment, it's
not an ideal interface.

Quoted 5 lines

In addition, stateful devices need to
load/save non-trivial amounts of
data. They need DMA to do this efficiently,
so an admin virtqueue is a
good fit again.

I don't get the point here. You still need to address the exact the similar
issues for admin virtqueue: the unbound time in freezing the device, the
interaction with the virtio device status state machine.

Quoted 4 lines
Quoted 3 lines

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements
it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 3 lines

I think we need to start from defining the state of one specific device and
see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That's
basically the difference between the vhost/vDPA's selective passthrough
approach and VFIO's full passthrough approach.

Quoted 8 lines
Quoted 3 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is
required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises
VFIO_REGION_TYPE_MIGRATION
    to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

This brings troubles in the nested environment.

Thanks

Quoted 34 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 10 lines
Quoted 6 lines

   For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 6 lines
Quoted 4 lines

   I think it makes sense to
standardize it if it's possible to convey all necessary state and device
implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Quoted 7 lines
Quoted 5 lines

Having two ways will bring extra complexity. That why I suggest:

- to have general facility for the virtuqueue to be migrated
- leave the device specific state to be device specific. so device can choose what is convenient way or interface.

I don't think we have a choice. For stateful devices it can be impossible to define a standard device state representation.

Quoted 5 lines

Let me clarify, I agree we can't have a standard device state for all kinds of device.

That's way I tend to leave them to be device specific. (but not implementation specific)

Unfortunately device state is sometimes implementation-specific. Not because the device is proprietary, but because the actual state is meaningless to other implementations.

I mentioned virtiofs as an example where file system backends can be implemented in completely different ways so the device state cannot be migrated between implementations.

Quoted 1 line

But we can generalize the virtqueue state for sure.

I agree and also that some device types can standardize their device state representations. But I think it's a technical requirement to support implementation-specific state for device types where cross-implementation migration is not possible.

I'm not saying the implementation-specific state representation has to be a binary blob. There could be an identifier registry to ensure live migration compatibility checks can be performed. There could also be a standard binary encoding for migration data. But the contents will be implementation-specific for some devices.

Stefan

#58 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 22 lines · Stefan Hajnoczi
Quoted 5 lines · Jason Wang

Let me clarify, I agree we can't have a standard device state for all kinds of device.

That's way I tend to leave them to be device specific. (but not implementation specific)

Unfortunately device state is sometimes implementation-specific. Not because the device is proprietary, but because the actual state is meaningless to other implementations.

I mentioned virtiofs as an example where file system backends can be implemented in completely different ways so the device state cannot be migrated between implementations.

Quoted 1 line

But we can generalize the virtqueue state for sure.

I agree and also that some device types can standardize their device state representations. But I think it's a technical requirement to support implementation-specific state for device types where cross-implementation migration is not possible.

Quoted 5 lines

I'm not saying the implementation-specific state representation has to be a binary blob. There could be an identifier registry to ensure live migration compatibility checks can be performed. There could also be a standard binary encoding for migration data. But the contents will be implementation-specific for some devices.

Can we at least put those implementation-specific states into some kind of structured, standardized form? E.g. something like

<type category: file system backend data>
<type identifier: file system foo>
<length>
<data>

so that we can at least do compat checks for "I know how to handle foo"?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#59 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 33 lines · Cornelia Huck
Quoted 22 lines · Stefan Hajnoczi
Quoted 5 lines · Jason Wang

Let me clarify, I agree we can't have a standard device state for all kinds of device.

That's way I tend to leave them to be device specific. (but not implementation specific)

Unfortunately device state is sometimes implementation-specific. Not because the device is proprietary, but because the actual state is meaningless to other implementations.

I mentioned virtiofs as an example where file system backends can be implemented in completely different ways so the device state cannot be migrated between implementations.

Quoted 1 line

But we can generalize the virtqueue state for sure.

I agree and also that some device types can standardize their device state representations. But I think it's a technical requirement to support implementation-specific state for device types where cross-implementation migration is not possible.

Quoted 5 lines

I'm not saying the implementation-specific state representation has to be a binary blob. There could be an identifier registry to ensure live migration compatibility checks can be performed. There could also be a standard binary encoding for migration data. But the contents will be implementation-specific for some devices.

Can we at least put those implementation-specific states into some kind of structured, standardized form? E.g. something like

Quoted 6 lines

<type category: file system backend data>
<type identifier: file system foo>
<length>
<data>

so that we can at least do compat checks for "I know how to handle foo"?

Yes, that's what I was trying to describe.

Stefan

author
#60 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Stefan Hajnoczi
Quoted 414 lines · Stefan Hajnoczi
Quoted 405 lines · Jason Wang
Quoted 404 lines · Stefan Hajnoczi
Quoted 403 lines · Jason Wang
Quoted 358 lines · Jason Wang
Quoted 283 lines · Stefan Hajnoczi
Quoted 282 lines · Jason Wang
Quoted 281 lines · Stefan Hajnoczi
Quoted 258 lines · Jason Wang
Quoted 257 lines · Stefan Hajnoczi
Quoted 256 lines · Jason Wang
Quoted 187 lines · Stefan Hajnoczi
Quoted 177 lines · Jason Wang

在 2021/7/12 下午5:57, Stefan Hajnoczi 写道:
On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at
07:23:33PM +0200, Eugenio
Perez Martin wrote:
        If I understand correctly, this is all
driven from the
driver inside
the guest, so
for this to work
the guest must
be running and
already have
initialised the
driver.
Yes.

As I see it, the feature
can be driven entirely
by the VMM as long as
it intercept the
relevant configuration
space (PCI, MMIO, etc)
from
guest's reads and
writes, and present it
as coherent and
transparent
for the guest. Some use
cases I can imagine with
a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not
to pass the feature
flag. The guest cannot
stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the
flag to the guest. The
guest can stop the
device.
2.1) The VMM stops the
device to perform a live
migration, and the
guest does not write to
STOP in any moment of
the LM. It resets the
destination device with
the state, and then
initializes the device.
2.2) The guest stops the
device and, when
STOP(32) is set, the
source
VMM migrates the device
status. The destination
VMM realizes the bit,
so it sets the bit in
the destination too
after device
initialization.
2.3) The device is not
initialized by the guest
so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's
all a lot of hoops to jump
through though.
It's also not easy for devices to implement.
It just requires a new status
bit. Anything that makes you
think it's hard
to implement?

E.g for networking device, it
should be sufficient to use this
bit + the
virtqueue state.

Why don't we design the
feature in a way that is
useable by VMMs
and implementable by devices in a simple way?
It use the common technology
like register shadowing without
any further
stuffs.

Or do you have any other ideas?

(I think we all know migration
will be very hard if we simply
pass through
those state registers).
If an admin virtqueue is used
instead of the STOP Device Status
field
bit then there's no need to re-read
the Device Status field in a loop
until the device has stopped.
Probably not. Let me to clarify several points:

- This proposal has nothing to do with
admin virtqueue. Actually, admin
virtqueue could be used for carrying any
basic device facility like status
bit. E.g I'm going to post patches that
use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin
virtqueue, we still need a per function
interface for this. This is a must for
nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no
need for the device to complete all
the consumed buffers, device can choose
to expose those inflight descriptors
in a device specific way and set the
STOP bit. This means, if we have the
device specific in-flight descriptor
reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device
facility but using the admin
virtqueue specific method, we still need
to clarify how it works with the
device status state machine, it will be
some kind of sub-states which looks
much more complicated than the current proposal.

When migrating a guest with many
VIRTIO devices a busy waiting
approach
extends downtime if implemented
sequentially (stopping one device at
a
time).
Well. You need some kinds of waiting for
sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio
implementation which is hard to be
restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

      It can be implemented
concurrently (setting the STOP bit
on all
devices and then looping until all
their Device Status fields have the
bit set), but this becomes more complex to implement.
I still don't get what kind of complexity did you worry here.

I'm a little worried about adding a new bit that requires busy
waiting...
Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common
configuration structure layout

After writing 0 to device_status, the
driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one
transport. We need do something
similar to when introducing basic facility.

Adding the STOP but as a Device Status bit
is a small and clean VIRTIO
spec change. I like that.

On the other hand, devices need time to stop and that time can be unbounded. For example, software
virtio-blk/scsi implementations since
cannot immediately cancel in-flight I/O requests on Linux hosts.

Quoted 4 lines

The natural interface for long-running
operations is virtqueue requests.
That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said
before, admin virtqueue
could be used for carrying the device status bit.

Send a command to set STOP status bit to admin
virtqueue. Device will make
the command buffer used after it has
successfully stopped the device.

AFAIK, they are not mutually exclusive, since
they are trying to solve
different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way
to implement (part of) the
device facility

Quoted 5 lines

Although you mentioned that the stopped
state needs to be reflected in
the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit.
VMM will shadow the device
status bit in this case.

E.g with the current vhost-vDPA, vDPA behave
like a vhost device, guest is
unaware of the migration.

STOP status bit is set by Qemu to real virtio
hardware. But guest will only
see the DRIVER_OK without STOP.

It's not hard to implement the nested on top,
see the discussion initiated
by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 3 lines

     In fact, the VMM would need to hide
this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the
capability. It's useful for
migrating a nested guest.

If we design an interface that can be used in
the nested environment, it's
not an ideal interface.

Quoted 5 lines

In addition, stateful devices need to
load/save non-trivial amounts of
data. They need DMA to do this efficiently,
so an admin virtqueue is a
good fit again.

I don't get the point here. You still need to address the exact the similar
issues for admin virtqueue: the unbound time in freezing the device, the
interaction with the virtio device status state machine.

Quoted 4 lines
Quoted 3 lines

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements
it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 3 lines

I think we need to start from defining the state of one specific device and
see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That's
basically the difference between the vhost/vDPA's selective passthrough
approach and VFIO's full passthrough approach.

Quoted 8 lines
Quoted 3 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is
required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises
VFIO_REGION_TYPE_MIGRATION
    to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

Quoted 39 lines

This brings troubles in the nested environment.

Thanks

Quoted 34 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 10 lines
Quoted 6 lines

   For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 6 lines
Quoted 4 lines

   I think it makes sense to
standardize it if it's possible to convey all necessary state and device
implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Having two ways will bring extra complexity. That why I suggest:

- to have general facility for the virtuqueue to be migrated
- leave the device specific state to be device specific. so device can choose what is convenient way or interface.

Quoted 8 lines
Quoted 2 lines

I don't think we have a choice. For stateful devices it can be impossible to define a standard device state representation.

Let me clarify, I agree we can't have a standard device state for all kinds of device.

That's way I tend to leave them to be device specific. (but not implementation specific)

Quoted 7 lines

Unfortunately device state is sometimes implementation-specific. Not because the device is proprietary, but because the actual state is meaningless to other implementations.

I mentioned virtiofs as an example where file system backends can be implemented in completely different ways so the device state cannot be migrated between implementations.

So let me clarify my understanding, we had two kinds of states:

1) implementation specific state that is not noticeable by the driver
2) device specific state that is noticeable by the driver

We don't have the interest in 1).

For 2) it's what needs to be defined in the spec. If we fail to generalize the device specific state, it can't be used by a standard virtio driver. Or maybe you can give a concrete example on how vitio-fs fail in doing this?

Quoted 6 lines
Quoted 2 lines

But we can generalize the virtqueue state for sure.

I agree and also that some device types can standardize their device state representations. But I think it's a technical requirement to support implementation-specific state for device types where cross-implementation migration is not possible.

A question here, if the driver depends on the implementation specific state, how can we make sure that driver can work for other implementation. If we're sure that a single driver can work for all kinds of implementations, it means the we had device specific state not implementation state.

Quoted 5 lines

I'm not saying the implementation-specific state representation has to be a binary blob. There could be an identifier registry to ensure live migration compatibility checks can be performed. There could also be a standard binary encoding for migration data.

Yes, such requirements has been well studied in the past. There should be plenty of protocols to do this.

Quoted 2 lines

But the contents will be
implementation-specific for some devices.

If we allow this, it breaks the spec effort for having a standard devices. And it will block the real customers.

Thanks

Quoted 2 lines

Stefan

#61 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 427 lines · Jason Wang
Quoted 414 lines · Stefan Hajnoczi
Quoted 405 lines · Jason Wang
Quoted 404 lines · Stefan Hajnoczi
Quoted 403 lines · Jason Wang
Quoted 358 lines · Jason Wang
Quoted 283 lines · Stefan Hajnoczi
Quoted 282 lines · Jason Wang
Quoted 281 lines · Stefan Hajnoczi
Quoted 258 lines · Jason Wang
Quoted 257 lines · Stefan Hajnoczi
Quoted 256 lines · Jason Wang
Quoted 191 lines · Stefan Hajnoczi

On Tue, Jul 13, 2021 at 11:27:03AM +0800, Jason Wang wrote:
在 2021/7/12 下午5:57, Stefan Hajnoczi 写道:
On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at
07:23:33PM +0200, Eugenio
Perez Martin wrote:
        If I understand correctly, this is all
driven from the
driver inside
the guest, so
for this to work
the guest must
be running and
already have
initialised the
driver.
Yes.

As I see it, the feature
can be driven entirely
by the VMM as long as
it intercept the
relevant configuration
space (PCI, MMIO, etc)
from
guest's reads and
writes, and present it
as coherent and
transparent
for the guest. Some use
cases I can imagine with
a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not
to pass the feature
flag. The guest cannot
stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the
flag to the guest. The
guest can stop the
device.
2.1) The VMM stops the
device to perform a live
migration, and the
guest does not write to
STOP in any moment of
the LM. It resets the
destination device with
the state, and then
initializes the device.
2.2) The guest stops the
device and, when
STOP(32) is set, the
source
VMM migrates the device
status. The destination
VMM realizes the bit,
so it sets the bit in
the destination too
after device
initialization.
2.3) The device is not
initialized by the guest
so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's
all a lot of hoops to jump
through though.
It's also not easy for devices to implement.
It just requires a new status
bit. Anything that makes you
think it's hard
to implement?

E.g for networking device, it
should be sufficient to use this
bit + the
virtqueue state.

Why don't we design the
feature in a way that is
useable by VMMs
and implementable by devices in a simple way?
It use the common technology
like register shadowing without
any further
stuffs.

Or do you have any other ideas?

(I think we all know migration
will be very hard if we simply
pass through
those state registers).
If an admin virtqueue is used
instead of the STOP Device Status
field
bit then there's no need to re-read
the Device Status field in a loop
until the device has stopped.
Probably not. Let me to clarify several points:

- This proposal has nothing to do with
admin virtqueue. Actually, admin
virtqueue could be used for carrying any
basic device facility like status
bit. E.g I'm going to post patches that
use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin
virtqueue, we still need a per function
interface for this. This is a must for
nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no
need for the device to complete all
the consumed buffers, device can choose
to expose those inflight descriptors
in a device specific way and set the
STOP bit. This means, if we have the
device specific in-flight descriptor
reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device
facility but using the admin
virtqueue specific method, we still need
to clarify how it works with the
device status state machine, it will be
some kind of sub-states which looks
much more complicated than the current proposal.

When migrating a guest with many
VIRTIO devices a busy waiting
approach
extends downtime if implemented
sequentially (stopping one device at
a
time).
Well. You need some kinds of waiting for
sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio
implementation which is hard to be
restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

      It can be implemented
concurrently (setting the STOP bit
on all
devices and then looping until all
their Device Status fields have the
bit set), but this becomes more complex to implement.
I still don't get what kind of complexity did you worry here.

I'm a little worried about adding a new bit that requires busy
waiting...
Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common
configuration structure layout

After writing 0 to device_status, the
driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one
transport. We need do something
similar to when introducing basic facility.
Adding the STOP but as a Device Status bit
is a small and clean VIRTIO
spec change. I like that.

On the other hand, devices need time to stop and that time can be
unbounded. For example, software
virtio-blk/scsi implementations since
cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running
operations is virtqueue requests.
That's why I mentioned the alternative of using an admin virtqueue
instead of a Device Status bit.

So I'm not against the admin virtqueue. As said
before, admin virtqueue
could be used for carrying the device status bit.

Send a command to set STOP status bit to admin
virtqueue. Device will make
the command buffer used after it has
successfully stopped the device.

AFAIK, they are not mutually exclusive, since
they are trying to solve
different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way
to implement (part of) the
device facility

Quoted 5 lines

Although you mentioned that the stopped
state needs to be reflected in
the Device Status field somehow, I'm not sure about that since the
driver typically doesn't need to know whether the device is being
migrated.

The guest won't see the real device status bit.
VMM will shadow the device
status bit in this case.

E.g with the current vhost-vDPA, vDPA behave
like a vhost device, guest is
unaware of the migration.

STOP status bit is set by Qemu to real virtio
hardware. But guest will only
see the DRIVER_OK without STOP.

It's not hard to implement the nested on top,
see the discussion initiated
by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 3 lines

     In fact, the VMM would need to hide
this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the
capability. It's useful for
migrating a nested guest.

If we design an interface that can be used in
the nested environment, it's
not an ideal interface.

Quoted 5 lines

In addition, stateful devices need to
load/save non-trivial amounts of
data. They need DMA to do this efficiently,
so an admin virtqueue is a
good fit again.

I don't get the point here. You still need to address the exact the similar
issues for admin virtqueue: the unbound time in freezing the device, the
interaction with the virtio device status state machine.

Quoted 4 lines
Quoted 3 lines

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements
it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 3 lines

I think we need to start from defining the state of one specific device and
see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That's
basically the difference between the vhost/vDPA's selective passthrough
approach and VFIO's full passthrough approach.

Quoted 8 lines
Quoted 3 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is
required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises
VFIO_REGION_TYPE_MIGRATION
    to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

Quoted 39 lines

This brings troubles in the nested environment.

Thanks

Quoted 34 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 10 lines
Quoted 6 lines

   For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 6 lines
Quoted 4 lines

   I think it makes sense to
standardize it if it's possible to convey all necessary state and device
implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Having two ways will bring extra complexity. That why I suggest:

- to have general facility for the virtuqueue to be migrated
- leave the device specific state to be device specific. so device can choose what is convenient way or interface.

Quoted 8 lines
Quoted 2 lines

I don't think we have a choice. For stateful devices it can be impossible to define a standard device state representation.

Let me clarify, I agree we can't have a standard device state for all kinds of device.

That's way I tend to leave them to be device specific. (but not implementation specific)

Quoted 7 lines

Unfortunately device state is sometimes implementation-specific. Not because the device is proprietary, but because the actual state is meaningless to other implementations.

I mentioned virtiofs as an example where file system backends can be implemented in completely different ways so the device state cannot be migrated between implementations.

So let me clarify my understanding, we had two kinds of states:

Quoted 8 lines

1) implementation specific state that is not noticeable by the driver
2) device specific state that is noticeable by the driver

We don't have the interest in 1).

For 2) it's what needs to be defined in the spec. If we fail to generalize the device specific state, it can't be used by a standard virtio driver. Or maybe you can give a concrete example on how vitio-fs fail in doing this?

2) is what I mean when I say a "stateful" device. I agree, 1) is not relevant to this discussion because we don't need to migrate internal device state that the driver cannot interact with.

The virtiofs device has an OPEN request for opening a file. Live migration must transfer the list of open files from the source to the destination device so the driver can continue accessing files it previously had open.

However, the result of the OPEN request is a number similar to a POSIX fd, not the full device-internal state associated with an open file. After migration the driver expects to continue using the number to operate on the file. We must transfer the open file state to the destination device.

Different device implementations may have completely different concepts of open file state:

- An in-memory file system. The list of open files is a list of in-memory inodes. We'll also need to transfer the entire contents of the files/directories since it's in-memory and not shared with the destination device.

- A passthrough Linux file system. We need to transfer the Linux file handles (see open_by_handle_at(2)) so the destination device can open the inodes on the underlying shared host file system.

- A distributed object storage API. We need to transfer the list of open object IDs so the destination device can perform I/O to the same objects.

- Anyone can create a custom virtiofs device implementation and it will rely on different open file state.

I imagine virtio-gpu and virtio-crypto might have similar situations where an object created through a virtqueue request has device-internal state associated with it that must be migrated.

Quoted 11 lines
Quoted 5 lines
Quoted 1 line

But we can generalize the virtqueue state for sure.

I agree and also that some device types can standardize their device state representations. But I think it's a technical requirement to support implementation-specific state for device types where cross-implementation migration is not possible.

A question here, if the driver depends on the implementation specific state, how can we make sure that driver can work for other implementation. If we're sure that a single driver can work for all kinds of implementations, it means the we had device specific state not implementation state.

I think this is confusing stateless and stateful devices. You are describing a stateless device here. I'll try to define a stateful device:

A stateful device maintains state that the driver operates on indirectly via standard requests. For example, the virtio-crypto device has CREATE_SESSION requests and a session ID is returned to the driver so further requests can be made on the session object. It may not be possible to replay, reconnect, or restart the device without losing state.

I hope that this description, together with the virtiofs specifics above, make the problem clearer.

Quoted 10 lines
Quoted 4 lines

I'm not saying the implementation-specific state representation has to be a binary blob. There could be an identifier registry to ensure live migration compatibility checks can be performed. There could also be a standard binary encoding for migration data.

Yes, such requirements has been well studied in the past. There should be plenty of protocols to do this.

Quoted 6 lines
Quoted 2 lines

But the contents will be
implementation-specific for some devices.

If we allow this, it breaks the spec effort for having a standard devices. And it will block the real customers.

If we forbid this then devices for which migration is technically possible will be unmigratable. Both users and device implementors will find other solutions, like VFIO, so I don't think we can stop them even if we tried.

I recognize that opaque device state poses a risk to migration compatibility, because device implementors may arbitrarily use opaque state when a standard is available.

However, the way to avoid this scenario is by:

1. Making the standard migration approach the easiest to implement because everything has been taken care of. It will save implementors the headache of defining and coding their own device state representations and versioning.

2. Educate users about migration compatibility so they can identify implementors are locking in their users.

Stefan

author
#62 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 460 lines · Stefan Hajnoczi
Quoted 424 lines · Jason Wang
Quoted 413 lines · Stefan Hajnoczi
Quoted 405 lines · Jason Wang
Quoted 404 lines · Stefan Hajnoczi
Quoted 403 lines · Jason Wang
Quoted 358 lines · Jason Wang
Quoted 283 lines · Stefan Hajnoczi
Quoted 282 lines · Jason Wang
Quoted 281 lines · Stefan Hajnoczi
Quoted 258 lines · Jason Wang
Quoted 257 lines · Stefan Hajnoczi
Quoted 256 lines · Jason Wang

在 2021/7/13 下午6:00, Stefan Hajnoczi 写道:
On Tue, Jul 13, 2021 at 11:27:03AM +0800, Jason Wang wrote:
在 2021/7/12 下午5:57, Stefan Hajnoczi 写道:
On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at
07:23:33PM +0200, Eugenio
Perez Martin wrote:
        If I understand correctly, this is all
driven from the
driver inside
the guest, so
for this to work
the guest must
be running and
already have
initialised the
driver.
Yes.

As I see it, the feature
can be driven entirely
by the VMM as long as
it intercept the
relevant configuration
space (PCI, MMIO, etc)
from
guest's reads and
writes, and present it
as coherent and
transparent
for the guest. Some use
cases I can imagine with
a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not
to pass the feature
flag. The guest cannot
stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the
flag to the guest. The
guest can stop the
device.
2.1) The VMM stops the
device to perform a live
migration, and the
guest does not write to
STOP in any moment of
the LM. It resets the
destination device with
the state, and then
initializes the device.
2.2) The guest stops the
device and, when
STOP(32) is set, the
source
VMM migrates the device
status. The destination
VMM realizes the bit,
so it sets the bit in
the destination too
after device
initialization.
2.3) The device is not
initialized by the guest
so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's
all a lot of hoops to jump
through though.
It's also not easy for devices to implement.
It just requires a new status
bit. Anything that makes you
think it's hard
to implement?

E.g for networking device, it
should be sufficient to use this
bit + the
virtqueue state.

Why don't we design the
feature in a way that is
useable by VMMs
and implementable by devices in a simple way?
It use the common technology
like register shadowing without
any further
stuffs.

Or do you have any other ideas?

(I think we all know migration
will be very hard if we simply
pass through
those state registers).
If an admin virtqueue is used
instead of the STOP Device Status
field
bit then there's no need to re-read
the Device Status field in a loop
until the device has stopped.
Probably not. Let me to clarify several points:

- This proposal has nothing to do with
admin virtqueue. Actually, admin
virtqueue could be used for carrying any
basic device facility like status
bit. E.g I'm going to post patches that
use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin
virtqueue, we still need a per function
interface for this. This is a must for
nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no
need for the device to complete all
the consumed buffers, device can choose
to expose those inflight descriptors
in a device specific way and set the
STOP bit. This means, if we have the
device specific in-flight descriptor
reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device
facility but using the admin
virtqueue specific method, we still need
to clarify how it works with the
device status state machine, it will be
some kind of sub-states which looks
much more complicated than the current proposal.

When migrating a guest with many
VIRTIO devices a busy waiting
approach
extends downtime if implemented
sequentially (stopping one device at
a
time).
Well. You need some kinds of waiting for
sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio
implementation which is hard to be
restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

      It can be implemented
concurrently (setting the STOP bit
on all
devices and then looping until all
their Device Status fields have the
bit set), but this becomes more complex to implement.
I still don't get what kind of complexity did you worry here.

I'm a little worried about adding a new bit that requires busy
waiting...
Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common
configuration structure layout

After writing 0 to device_status, the
driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one
transport. We need do something
similar to when introducing basic facility.
Adding the STOP but as a Device Status bit
is a small and clean VIRTIO
spec change. I like that.

On the other hand, devices need time to stop and that time can be
unbounded. For example, software
virtio-blk/scsi implementations since
cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running
operations is virtqueue requests.
That's why I mentioned the alternative of using an admin virtqueue
instead of a Device Status bit.
So I'm not against the admin virtqueue. As said
before, admin virtqueue
could be used for carrying the device status bit.

Send a command to set STOP status bit to admin
virtqueue. Device will make
the command buffer used after it has
successfully stopped the device.

AFAIK, they are not mutually exclusive, since
they are trying to solve
different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way
to implement (part of) the
device facility

Although you mentioned that the stopped
state needs to be reflected in
the Device Status field somehow, I'm not sure about that since the
driver typically doesn't need to know whether the device is being
migrated.
The guest won't see the real device status bit.
VMM will shadow the device
status bit in this case.

E.g with the current vhost-vDPA, vDPA behave
like a vhost device, guest is
unaware of the migration.

STOP status bit is set by Qemu to real virtio
hardware. But guest will only
see the DRIVER_OK without STOP.

It's not hard to implement the nested on top,
see the discussion initiated
by Eugenio about how expose VIRTIO_F_STOP to guest for nested live
migration.

     In fact, the VMM would need to hide
this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.
See above, VMM may choose to hide or expose the
capability. It's useful for
migrating a nested guest.

If we design an interface that can be used in
the nested environment, it's
not an ideal interface.

In addition, stateful devices need to
load/save non-trivial amounts of
data. They need DMA to do this efficiently,
so an admin virtqueue is a
good fit again.
I don't get the point here. You still need to
address the exact the similar
issues for admin virtqueue: the unbound time in
freezing the device, the
interaction with the virtio device status state machine.

Quoted 4 lines
Quoted 3 lines

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements
it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 3 lines

I think we need to start from defining the state of one specific device and
see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That's
basically the difference between the vhost/vDPA's selective passthrough
approach and VFIO's full passthrough approach.

Quoted 8 lines
Quoted 3 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is
required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises
VFIO_REGION_TYPE_MIGRATION
    to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

Quoted 39 lines

This brings troubles in the nested environment.

Thanks

Quoted 34 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 10 lines
Quoted 6 lines

   For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 6 lines
Quoted 4 lines

   I think it makes sense to
standardize it if it's possible to convey all necessary state and device
implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Having two ways will bring extra complexity. That why I suggest:

- to have general facility for the virtuqueue to be migrated
- leave the device specific state to be device specific. so device can choose what is convenient way or interface.

Quoted 7 lines
Quoted 2 lines

I don't think we have a choice. For stateful devices it can be impossible to define a standard device state representation.

Let me clarify, I agree we can't have a standard device state for all kinds of device.

That's way I tend to leave them to be device specific. (but not implementation specific)

Quoted 7 lines

Unfortunately device state is sometimes implementation-specific. Not because the device is proprietary, but because the actual state is meaningless to other implementations.

I mentioned virtiofs as an example where file system backends can be implemented in completely different ways so the device state cannot be migrated between implementations.

So let me clarify my understanding, we had two kinds of states:

Quoted 8 lines

1) implementation specific state that is not noticeable by the driver
2) device specific state that is noticeable by the driver

We don't have the interest in 1).

For 2) it's what needs to be defined in the spec. If we fail to generalize the device specific state, it can't be used by a standard virtio driver. Or maybe you can give a concrete example on how vitio-fs fail in doing this?

2) is what I mean when I say a "stateful" device. I agree, 1) is not relevant to this discussion because we don't need to migrate internal device state that the driver cannot interact with.

The virtiofs device has an OPEN request for opening a file. Live migration must transfer the list of open files from the source to the destination device so the driver can continue accessing files it previously had open.

However, the result of the OPEN request is a number similar to a POSIX fd, not the full device-internal state associated with an open file. After migration the driver expects to continue using the number to operate on the file. We must transfer the open file state to the destination device.

Different device implementations may have completely different concepts of open file state:

- An in-memory file system. The list of open files is a list of in-memory inodes. We'll also need to transfer the entire contents of the files/directories since it's in-memory and not shared with the destination device.

- A passthrough Linux file system. We need to transfer the Linux file handles (see open_by_handle_at(2)) so the destination device can open the inodes on the underlying shared host file system.

Quoted 6 lines

- A distributed object storage API. We need to transfer the list of open object IDs so the destination device can perform I/O to the same objects.

- Anyone can create a custom virtiofs device implementation and it will rely on different open file state.

So it looks to me you want to propose migration drivers for different implementation. It looks to me we it's better to go with something more easier:

1) Having a common driver visible state defined in the spec, and use them for migration
2) It's the charge of the device or backend to "map" the driver visible state to its implementation specific state.

If 1) is in-sufficient, we should extend it until the it satisfies 2)

For virtio-fs, it looks like the issue is that the implementation needs to associate objects in different namespaces (guest vs host).

For the above cases:

1) For the in-memory file system, if I understand correctly, it can be accessed directly by guest via a transport specific way (e.g BAR). In this case, it's driver noticeable state so all the memory must be migrated to the destination.
2) For the passthrough, it's the charge of the implementation to do the map. I think it doesn't differ a lot with the current migration among shared storage of block devices. (Note that open_by_handle_at() requires CAP_DAC_READ_SEARCH which I'm not sure it can used)
3) For distributed storage, the implementation should implement the association between driver object and distributed storage object (I think each such API should has something like UUID for a global namespace) and provide a way for reverse lookup.
4) For other implementation, it should be the same as 3)

It's the management layer to decide whether we can do cross implementation migration, but qemu should migrate the common driver visible state instead of implementation specific state.

We can have a dedicated feature flag for this and block the migration for the devices without this feature.

I think we don't want to end up with several migration drivers for virtio-fs.

Quoted 3 lines

I imagine virtio-gpu and virtio-crypto might have similar situations where an object created through a virtqueue request has device-internal state associated with it that must be migrated.

So the point stands still. The device internal state should be restored from the device specific state defined in the spec. Spec would guarantee that the minimal part of the device specific state, implementation should use those to restore implementation specific state.

Quoted 15 lines
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines

But we can generalize the virtqueue state for sure.

I agree and also that some device types can standardize their device state representations. But I think it's a technical requirement to support implementation-specific state for device types where cross-implementation migration is not possible.

A question here, if the driver depends on the implementation specific state, how can we make sure that driver can work for other implementation. If we're sure that a single driver can work for all kinds of implementations, it means the we had device specific state not implementation state.

I think this is confusing stateless and stateful devices. You are describing a stateless device here. I'll try to define a stateful device:

Quoted 6 lines

A stateful device maintains state that the driver operates on indirectly via standard requests. For example, the virtio-crypto device has CREATE_SESSION requests and a session ID is returned to the driver so further requests can be made on the session object. It may not be possible to replay, reconnect, or restart the device without losing state.

If it's impossible to do all of these. These state should be noticeable by the driver and they are not implementation specific but device specific which could be defined in the spec.

Quoted 3 lines

I hope that this description, together with the virtiofs specifics above, make the problem clearer.

Yes.

Quoted 13 lines
Quoted 13 lines
Quoted 5 lines

I'm not saying the implementation-specific state representation has to be a binary blob. There could be an identifier registry to ensure live migration compatibility checks can be performed. There could also be a standard binary encoding for migration data.

Yes, such requirements has been well studied in the past. There should be plenty of protocols to do this.

Quoted 2 lines

But the contents will be
implementation-specific for some devices.

Quoted 6 lines
Quoted 2 lines

If we allow this, it breaks the spec effort for having a standard devices. And it will block the real customers.

If we forbid this then devices for which migration is technically possible will be unmigratable. Both users and device implementors will find other solutions, like VFIO, so I don't think we can stop them even if we tried.

The difference is standard device vs vendor specific device. We can't avoid the implementation specific state for vendor specific hardware.

But for standard device, we don't want to end up with migration drivers. (Do we want vendor to ship the vendor specific migration drivers for NVM(e) devices?)

Quoted 7 lines

I recognize that opaque device state poses a risk to migration compatibility, because device implementors may arbitrarily use opaque state when a standard is available.

However, the way to avoid this scenario is by:

Quoted 7 lines

1. Making the standard migration approach the easiest to implement because everything has been taken care of. It will save implementors the headache of defining and coding their own device state representations and versioning.

2. Educate users about migration compatibility so they can identify implementors are locking in their users.

For vendor specific device, this may work. But for standard devices like virtio, we should go further.

The device states should be defined in the spec clearly. We should re-visit the design if those states contains anything that is implementation specific.

Thanks

Quoted 2 lines

Stefan

#63 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 484 lines · Jason Wang
Quoted 460 lines · Stefan Hajnoczi
Quoted 424 lines · Jason Wang
Quoted 413 lines · Stefan Hajnoczi
Quoted 405 lines · Jason Wang
Quoted 404 lines · Stefan Hajnoczi
Quoted 403 lines · Jason Wang
Quoted 358 lines · Jason Wang
Quoted 283 lines · Stefan Hajnoczi
Quoted 282 lines · Jason Wang
Quoted 281 lines · Stefan Hajnoczi
Quoted 258 lines · Jason Wang
Quoted 257 lines · Stefan Hajnoczi

On Tue, Jul 13, 2021 at 08:16:35PM +0800, Jason Wang wrote:
在 2021/7/13 下午6:00, Stefan Hajnoczi 写道:
On Tue, Jul 13, 2021 at 11:27:03AM +0800, Jason Wang wrote:
在 2021/7/12 下午5:57, Stefan Hajnoczi 写道:
On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at
07:23:33PM +0200, Eugenio
Perez Martin wrote:
        If I understand correctly, this is all
driven from the
driver inside
the guest, so
for this to work
the guest must
be running and
already have
initialised the
driver.
Yes.

As I see it, the feature
can be driven entirely
by the VMM as long as
it intercept the
relevant configuration
space (PCI, MMIO, etc)
from
guest's reads and
writes, and present it
as coherent and
transparent
for the guest. Some use
cases I can imagine with
a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not
to pass the feature
flag. The guest cannot
stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the
flag to the guest. The
guest can stop the
device.
2.1) The VMM stops the
device to perform a live
migration, and the
guest does not write to
STOP in any moment of
the LM. It resets the
destination device with
the state, and then
initializes the device.
2.2) The guest stops the
device and, when
STOP(32) is set, the
source
VMM migrates the device
status. The destination
VMM realizes the bit,
so it sets the bit in
the destination too
after device
initialization.
2.3) The device is not
initialized by the guest
so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's
all a lot of hoops to jump
through though.
It's also not easy for devices to implement.
It just requires a new status
bit. Anything that makes you
think it's hard
to implement?

E.g for networking device, it
should be sufficient to use this
bit + the
virtqueue state.

Why don't we design the
feature in a way that is
useable by VMMs
and implementable by devices in a simple way?
It use the common technology
like register shadowing without
any further
stuffs.

Or do you have any other ideas?

(I think we all know migration
will be very hard if we simply
pass through
those state registers).
If an admin virtqueue is used
instead of the STOP Device Status
field
bit then there's no need to re-read
the Device Status field in a loop
until the device has stopped.
Probably not. Let me to clarify several points:

- This proposal has nothing to do with
admin virtqueue. Actually, admin
virtqueue could be used for carrying any
basic device facility like status
bit. E.g I'm going to post patches that
use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin
virtqueue, we still need a per function
interface for this. This is a must for
nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no
need for the device to complete all
the consumed buffers, device can choose
to expose those inflight descriptors
in a device specific way and set the
STOP bit. This means, if we have the
device specific in-flight descriptor
reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device
facility but using the admin
virtqueue specific method, we still need
to clarify how it works with the
device status state machine, it will be
some kind of sub-states which looks
much more complicated than the current proposal.

When migrating a guest with many
VIRTIO devices a busy waiting
approach
extends downtime if implemented
sequentially (stopping one device at
a
time).
Well. You need some kinds of waiting for
sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio
implementation which is hard to be
restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

      It can be implemented
concurrently (setting the STOP bit
on all
devices and then looping until all
their Device Status fields have the
bit set), but this becomes more complex to implement.
I still don't get what kind of complexity did you worry here.

I'm a little worried about adding a new bit that requires busy
waiting...
Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common
configuration structure layout

After writing 0 to device_status, the
driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one
transport. We need do something
similar to when introducing basic facility.
Adding the STOP but as a Device Status bit
is a small and clean VIRTIO
spec change. I like that.

On the other hand, devices need time to stop and that time can be
unbounded. For example, software
virtio-blk/scsi implementations since
cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running
operations is virtqueue requests.
That's why I mentioned the alternative of using an admin virtqueue
instead of a Device Status bit.
So I'm not against the admin virtqueue. As said
before, admin virtqueue
could be used for carrying the device status bit.

Send a command to set STOP status bit to admin
virtqueue. Device will make
the command buffer used after it has
successfully stopped the device.

AFAIK, they are not mutually exclusive, since
they are trying to solve
different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way
to implement (part of) the
device facility

Although you mentioned that the stopped
state needs to be reflected in
the Device Status field somehow, I'm not sure about that since the
driver typically doesn't need to know whether the device is being
migrated.
The guest won't see the real device status bit.
VMM will shadow the device
status bit in this case.

E.g with the current vhost-vDPA, vDPA behave
like a vhost device, guest is
unaware of the migration.

STOP status bit is set by Qemu to real virtio
hardware. But guest will only
see the DRIVER_OK without STOP.

It's not hard to implement the nested on top,
see the discussion initiated
by Eugenio about how expose VIRTIO_F_STOP to guest for nested live
migration.

     In fact, the VMM would need to hide
this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.
See above, VMM may choose to hide or expose the
capability. It's useful for
migrating a nested guest.

If we design an interface that can be used in
the nested environment, it's
not an ideal interface.

In addition, stateful devices need to
load/save non-trivial amounts of
data. They need DMA to do this efficiently,
so an admin virtqueue is a
good fit again.
I don't get the point here. You still need to
address the exact the similar
issues for admin virtqueue: the unbound time in
freezing the device, the
interaction with the virtio device status state machine.

Quoted 4 lines
Quoted 3 lines

Device state state can be large so a register interface would be a
bottleneck. DMA is needed. I think a virtqueue is a good fit for
saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements
it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 3 lines

I think we need to start from defining the state of one specific device and
see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That's
basically the difference between the vhost/vDPA's selective passthrough
approach and VFIO's full passthrough approach.

Quoted 8 lines
Quoted 3 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is
required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises
VFIO_REGION_TYPE_MIGRATION
    to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

Quoted 39 lines

This brings troubles in the nested environment.

Thanks

Quoted 34 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 10 lines
Quoted 6 lines

   For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 6 lines
Quoted 4 lines

   I think it makes sense to
standardize it if it's possible to convey all necessary state and device
implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Having two ways will bring extra complexity. That why I suggest:

- to have general facility for the virtuqueue to be migrated
- leave the device specific state to be device specific. so device can choose what is convenient way or interface.

Quoted 7 lines
Quoted 2 lines

I don't think we have a choice. For stateful devices it can be impossible to define a standard device state representation.

Let me clarify, I agree we can't have a standard device state for all kinds of device.

That's way I tend to leave them to be device specific. (but not implementation specific)

Quoted 7 lines

Unfortunately device state is sometimes implementation-specific. Not because the device is proprietary, but because the actual state is meaningless to other implementations.

I mentioned virtiofs as an example where file system backends can be implemented in completely different ways so the device state cannot be migrated between implementations.

So let me clarify my understanding, we had two kinds of states:

Quoted 8 lines

1) implementation specific state that is not noticeable by the driver
2) device specific state that is noticeable by the driver

We don't have the interest in 1).

For 2) it's what needs to be defined in the spec. If we fail to generalize the device specific state, it can't be used by a standard virtio driver. Or maybe you can give a concrete example on how vitio-fs fail in doing this?

2) is what I mean when I say a "stateful" device. I agree, 1) is not relevant to this discussion because we don't need to migrate internal device state that the driver cannot interact with.

The virtiofs device has an OPEN request for opening a file. Live migration must transfer the list of open files from the source to the destination device so the driver can continue accessing files it previously had open.

However, the result of the OPEN request is a number similar to a POSIX fd, not the full device-internal state associated with an open file. After migration the driver expects to continue using the number to operate on the file. We must transfer the open file state to the destination device.

Different device implementations may have completely different concepts of open file state:

- An in-memory file system. The list of open files is a list of in-memory inodes. We'll also need to transfer the entire contents of the files/directories since it's in-memory and not shared with the destination device.

- A passthrough Linux file system. We need to transfer the Linux file handles (see open_by_handle_at(2)) so the destination device can open the inodes on the underlying shared host file system.

Quoted 6 lines

- A distributed object storage API. We need to transfer the list of open object IDs so the destination device can perform I/O to the same objects.

- Anyone can create a custom virtiofs device implementation and it will rely on different open file state.

So it looks to me you want to propose migration drivers for different implementation. It looks to me we it's better to go with something more easier:

1) Having a common driver visible state defined in the spec, and use them for migration
2) It's the charge of the device or backend to "map" the driver visible state to its implementation specific state.

If 1) is in-sufficient, we should extend it until the it satisfies 2)

For virtio-fs, it looks like the issue is that the implementation needs to associate objects in different namespaces (guest vs host).

Quoted 6 lines

For the above cases:

1) For the in-memory file system, if I understand correctly, it can be accessed directly by guest via a transport specific way (e.g BAR). In this case, it's driver noticeable state so all the memory must be migrated to the destination.

The virtiofs DAX Window feature is optional. Also, the DAX Window BAR only exposes file contents, not file system metadata. The guest driver or the BAR contents are not enough to migrate the state of the device.

Quoted 4 lines

2) For the passthrough, it's the charge of the implementation to do the map. I think it doesn't differ a lot with the current migration among shared storage of block devices. (Note that open_by_handle_at() requires CAP_DAC_READ_SEARCH which I'm not sure it can used)

It's different from block devices because the FUSE protocol is stateful. Commands like LOOKUP, OPEN, OPENDIR create objects that are referred to by a temporary number (similar to POSIX fds). Due to POSIX file system semantics there's no way to recreate the mapping correctly without Linux file handles (because files can be deleted, renamed, moved while they are still open or their inode is known). Shared block devices do not have this issue.

Quoted 108 lines

3) For distributed storage, the implementation should implement the association between driver object and distributed storage object (I think each such API should has something like UUID for a global namespace) and provide a way for reverse lookup.
4) For other implementation, it should be the same as 3)

It's the management layer to decide whether we can do cross implementation migration, but qemu should migrate the common driver visible state instead of implementation specific state.

We can have a dedicated feature flag for this and block the migration for the devices without this feature.

I think we don't want to end up with several migration drivers for virtio-fs.

Quoted 3 lines

I imagine virtio-gpu and virtio-crypto might have similar situations where an object created through a virtqueue request has device-internal state associated with it that must be migrated.

So the point stands still. The device internal state should be restored from the device specific state defined in the spec. Spec would guarantee that the minimal part of the device specific state, implementation should use those to restore implementation specific state.

Quoted 15 lines
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines

But we can generalize the virtqueue state for sure.

I agree and also that some device types can standardize their device state representations. But I think it's a technical requirement to support implementation-specific state for device types where cross-implementation migration is not possible.

A question here, if the driver depends on the implementation specific state, how can we make sure that driver can work for other implementation. If we're sure that a single driver can work for all kinds of implementations, it means the we had device specific state not implementation state.

I think this is confusing stateless and stateful devices. You are describing a stateless device here. I'll try to define a stateful device:

Quoted 6 lines

A stateful device maintains state that the driver operates on indirectly via standard requests. For example, the virtio-crypto device has CREATE_SESSION requests and a session ID is returned to the driver so further requests can be made on the session object. It may not be possible to replay, reconnect, or restart the device without losing state.

If it's impossible to do all of these. These state should be noticeable by the driver and they are not implementation specific but device specific which could be defined in the spec.

Quoted 3 lines

I hope that this description, together with the virtiofs specifics above, make the problem clearer.

Yes.

Quoted 13 lines
Quoted 13 lines
Quoted 5 lines

I'm not saying the implementation-specific state representation has to be a binary blob. There could be an identifier registry to ensure live migration compatibility checks can be performed. There could also be a standard binary encoding for migration data.

Yes, such requirements has been well studied in the past. There should be plenty of protocols to do this.

Quoted 2 lines

But the contents will be
implementation-specific for some devices.

Quoted 6 lines
Quoted 2 lines

If we allow this, it breaks the spec effort for having a standard devices. And it will block the real customers.

If we forbid this then devices for which migration is technically possible will be unmigratable. Both users and device implementors will find other solutions, like VFIO, so I don't think we can stop them even if we tried.

The difference is standard device vs vendor specific device. We can't avoid the implementation specific state for vendor specific hardware.

But for standard device, we don't want to end up with migration drivers. (Do we want vendor to ship the vendor specific migration drivers for NVM(e) devices?)

Quoted 7 lines

I recognize that opaque device state poses a risk to migration compatibility, because device implementors may arbitrarily use opaque state when a standard is available.

However, the way to avoid this scenario is by:

Quoted 7 lines

1. Making the standard migration approach the easiest to implement because everything has been taken care of. It will save implementors the headache of defining and coding their own device state representations and versioning.

2. Educate users about migration compatibility so they can identify implementors are locking in their users.

Quoted 6 lines

For vendor specific device, this may work. But for standard devices like virtio, we should go further.

The device states should be defined in the spec clearly. We should re-visit the design if those states contains anything that is implementation specific.

Can you describe how migrating virtiofs devices should work? I think that might be quicker than if I reply to each of your points because our views are still quite far apart.

Stefan

author
#64 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 484 lines · Stefan Hajnoczi
Quoted 483 lines · Jason Wang
Quoted 459 lines · Stefan Hajnoczi
Quoted 423 lines · Jason Wang
Quoted 413 lines · Stefan Hajnoczi
Quoted 405 lines · Jason Wang
Quoted 404 lines · Stefan Hajnoczi
Quoted 403 lines · Jason Wang
Quoted 358 lines · Jason Wang
Quoted 283 lines · Stefan Hajnoczi
Quoted 282 lines · Jason Wang
Quoted 281 lines · Stefan Hajnoczi
Quoted 262 lines · Jason Wang

在 2021/7/14 下午5:53, Stefan Hajnoczi 写道:
On Tue, Jul 13, 2021 at 08:16:35PM +0800, Jason Wang wrote:
在 2021/7/13 下午6:00, Stefan Hajnoczi 写道:
On Tue, Jul 13, 2021 at 11:27:03AM +0800, Jason Wang wrote:
在 2021/7/12 下午5:57, Stefan Hajnoczi 写道:
On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at
07:23:33PM +0200, Eugenio
Perez Martin wrote:
        If I understand correctly, this is all
driven from the
driver inside
the guest, so
for this to work
the guest must
be running and
already have
initialised the
driver.
Yes.

As I see it, the feature
can be driven entirely
by the VMM as long as
it intercept the
relevant configuration
space (PCI, MMIO, etc)
from
guest's reads and
writes, and present it
as coherent and
transparent
for the guest. Some use
cases I can imagine with
a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not
to pass the feature
flag. The guest cannot
stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the
flag to the guest. The
guest can stop the
device.
2.1) The VMM stops the
device to perform a live
migration, and the
guest does not write to
STOP in any moment of
the LM. It resets the
destination device with
the state, and then
initializes the device.
2.2) The guest stops the
device and, when
STOP(32) is set, the
source
VMM migrates the device
status. The destination
VMM realizes the bit,
so it sets the bit in
the destination too
after device
initialization.
2.3) The device is not
initialized by the guest
so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's
all a lot of hoops to jump
through though.
It's also not easy for devices to implement.
It just requires a new status
bit. Anything that makes you
think it's hard
to implement?

E.g for networking device, it
should be sufficient to use this
bit + the
virtqueue state.

Why don't we design the
feature in a way that is
useable by VMMs
and implementable by devices in a simple way?
It use the common technology
like register shadowing without
any further
stuffs.

Or do you have any other ideas?

(I think we all know migration
will be very hard if we simply
pass through
those state registers).
If an admin virtqueue is used
instead of the STOP Device Status
field
bit then there's no need to re-read
the Device Status field in a loop
until the device has stopped.
Probably not. Let me to clarify several points:

- This proposal has nothing to do with
admin virtqueue. Actually, admin
virtqueue could be used for carrying any
basic device facility like status
bit. E.g I'm going to post patches that
use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin
virtqueue, we still need a per function
interface for this. This is a must for
nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no
need for the device to complete all
the consumed buffers, device can choose
to expose those inflight descriptors
in a device specific way and set the
STOP bit. This means, if we have the
device specific in-flight descriptor
reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device
facility but using the admin
virtqueue specific method, we still need
to clarify how it works with the
device status state machine, it will be
some kind of sub-states which looks
much more complicated than the current proposal.

When migrating a guest with many
VIRTIO devices a busy waiting
approach
extends downtime if implemented
sequentially (stopping one device at
a
time).
Well. You need some kinds of waiting for
sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio
implementation which is hard to be
restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

      It can be implemented
concurrently (setting the STOP bit
on all
devices and then looping until all
their Device Status fields have the
bit set), but this becomes more complex to implement.
I still don't get what kind of complexity did you worry here.

I'm a little worried about adding a new bit that requires busy
waiting...
Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common
configuration structure layout

After writing 0 to device_status, the
driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one
transport. We need do something
similar to when introducing basic facility.
Adding the STOP but as a Device Status bit
is a small and clean VIRTIO
spec change. I like that.

On the other hand, devices need time to stop and that time can be
unbounded. For example, software
virtio-blk/scsi implementations since
cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running
operations is virtqueue requests.
That's why I mentioned the alternative of using an admin virtqueue
instead of a Device Status bit.
So I'm not against the admin virtqueue. As said
before, admin virtqueue
could be used for carrying the device status bit.

Send a command to set STOP status bit to admin
virtqueue. Device will make
the command buffer used after it has
successfully stopped the device.

AFAIK, they are not mutually exclusive, since
they are trying to solve
different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way
to implement (part of) the
device facility

Although you mentioned that the stopped
state needs to be reflected in
the Device Status field somehow, I'm not sure about that since the
driver typically doesn't need to know whether the device is being
migrated.
The guest won't see the real device status bit.
VMM will shadow the device
status bit in this case.

E.g with the current vhost-vDPA, vDPA behave
like a vhost device, guest is
unaware of the migration.

STOP status bit is set by Qemu to real virtio
hardware. But guest will only
see the DRIVER_OK without STOP.

It's not hard to implement the nested on top,
see the discussion initiated
by Eugenio about how expose VIRTIO_F_STOP to guest for nested live
migration.

     In fact, the VMM would need to hide
this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.
See above, VMM may choose to hide or expose the
capability. It's useful for
migrating a nested guest.

If we design an interface that can be used in
the nested environment, it's
not an ideal interface.

In addition, stateful devices need to
load/save non-trivial amounts of
data. They need DMA to do this efficiently,
so an admin virtqueue is a
good fit again.
I don't get the point here. You still need to
address the exact the similar
issues for admin virtqueue: the unbound time in
freezing the device, the
interaction with the virtio device status state machine.
Device state state can be large so a register interface would be a
bottleneck. DMA is needed. I think a virtqueue is a good fit for
saving/loading device state.
So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements
it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 3 lines

I think we need to start from defining the state of one
specific device and
see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?
2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

We should standardize everything that is visible by the driver to be a standard device. That's the power of virtio.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That's
basically the difference between the vhost/vDPA's selective passthrough
approach and VFIO's full passthrough approach.

Quoted 8 lines
Quoted 3 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is
required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

So I think it still depend on the driver to implement migrate state which is vendor specific.

Note that it's just an uAPI definition not something defined in the PCI spec.

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises
VFIO_REGION_TYPE_MIGRATION
    to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

Quoted 39 lines

This brings troubles in the nested environment.

Thanks

Quoted 34 lines

Changpeng Liu originally mentioned the idea of defining a migration PCI Capability.

Quoted 10 lines
Quoted 6 lines

   For example, some of the
virtio-net state is available to the VMM with vhost/vDPA because it intercepts the virtio-net control virtqueue.

Also, we need to decide to what degree the device state representation is standardized in the VIRTIO specification.

I think all the states must be defined in the spec otherwise the device can't claim it supports migration at virtio level.

Quoted 6 lines
Quoted 4 lines

   I think it makes sense to
standardize it if it's possible to convey all necessary state and device
implementors can easily implement this device state representation.

I doubt it's high device specific. E.g can we standardize device(GPU) memory?

For devices that have little internal state it's possible to define a standard device state representation.

For other devices, like virtio-crypto, virtio-fs, etc it becomes difficult because the device implementation contains state that will be needed but is very specific to the implementation. These devices *are* migratable but they don't have standard state. Even here there is a spectrum:
- Host OS-specific state (e.g. Linux struct file_handles)
- Library-specific state (e.g. crypto library state)
- Implementation-specific state (e.g. sshfs inode state for virtio-fs)

This is why I think it's necessary to support both standard device state representations and implementation-specific device state representations.

Having two ways will bring extra complexity. That why I suggest:

- to have general facility for the virtuqueue to be migrated
- leave the device specific state to be device specific. so device can choose what is convenient way or interface.

Quoted 7 lines
Quoted 2 lines

I don't think we have a choice. For stateful devices it can be impossible to define a standard device state representation.

Let me clarify, I agree we can't have a standard device state for all kinds of device.

That's way I tend to leave them to be device specific. (but not implementation specific)

Quoted 7 lines

Unfortunately device state is sometimes implementation-specific. Not because the device is proprietary, but because the actual state is meaningless to other implementations.

I mentioned virtiofs as an example where file system backends can be implemented in completely different ways so the device state cannot be migrated between implementations.

So let me clarify my understanding, we had two kinds of states:

Quoted 8 lines

1) implementation specific state that is not noticeable by the driver
2) device specific state that is noticeable by the driver

We don't have the interest in 1).

For 2) it's what needs to be defined in the spec. If we fail to generalize the device specific state, it can't be used by a standard virtio driver. Or maybe you can give a concrete example on how vitio-fs fail in doing this?

2) is what I mean when I say a "stateful" device. I agree, 1) is not relevant to this discussion because we don't need to migrate internal device state that the driver cannot interact with.

The virtiofs device has an OPEN request for opening a file. Live migration must transfer the list of open files from the source to the destination device so the driver can continue accessing files it previously had open.

However, the result of the OPEN request is a number similar to a POSIX fd, not the full device-internal state associated with an open file. After migration the driver expects to continue using the number to operate on the file. We must transfer the open file state to the destination device.

Different device implementations may have completely different concepts of open file state:

- An in-memory file system. The list of open files is a list of in-memory inodes. We'll also need to transfer the entire contents of the files/directories since it's in-memory and not shared with the destination device.

- A passthrough Linux file system. We need to transfer the Linux file handles (see open_by_handle_at(2)) so the destination device can open the inodes on the underlying shared host file system.

Quoted 6 lines

- A distributed object storage API. We need to transfer the list of open object IDs so the destination device can perform I/O to the same objects.

- Anyone can create a custom virtiofs device implementation and it will rely on different open file state.

So it looks to me you want to propose migration drivers for different implementation. It looks to me we it's better to go with something more easier:

1) Having a common driver visible state defined in the spec, and use them for migration
2) It's the charge of the device or backend to "map" the driver visible state to its implementation specific state.

If 1) is in-sufficient, we should extend it until the it satisfies 2)

For virtio-fs, it looks like the issue is that the implementation needs to associate objects in different namespaces (guest vs host).

For the above cases:

Quoted 7 lines
Quoted 4 lines

1) For the in-memory file system, if I understand correctly, it can be accessed directly by guest via a transport specific way (e.g BAR). In this case, it's driver noticeable state so all the memory must be migrated to the destination.

The virtiofs DAX Window feature is optional. Also, the DAX Window BAR only exposes file contents, not file system metadata. The guest driver or the BAR contents are not enough to migrate the state of the device.

Yes, so the point is if the state is noticeable, it need to be migrated.

And another thing that may cause confusing is that, we probably need to clarify what DAX is in the spec or avoid using Linux specific terminology as virtio can live without it.

Quoted 5 lines
Quoted 5 lines

2) For the passthrough, it's the charge of the implementation to do the map. I think it doesn't differ a lot with the current migration among shared storage of block devices. (Note that open_by_handle_at() requires CAP_DAC_READ_SEARCH which I'm not sure it can used)

Quoted 7 lines

It's different from block devices because the FUSE protocol is stateful. Commands like LOOKUP, OPEN, OPENDIR create objects that are referred to by a temporary number (similar to POSIX fds). Due to POSIX file system semantics there's no way to recreate the mapping correctly without Linux file handles (because files can be deleted, renamed, moved while they are still open or their inode is known). Shared block devices do not have this issue.

See below, I would like to have a look at how Linux file handle is expected to work.

Quoted 97 lines
Quoted 97 lines

3) For distributed storage, the implementation should implement the association between driver object and distributed storage object (I think each such API should has something like UUID for a global namespace) and provide a way for reverse lookup.
4) For other implementation, it should be the same as 3)

It's the management layer to decide whether we can do cross implementation migration, but qemu should migrate the common driver visible state instead of implementation specific state.

We can have a dedicated feature flag for this and block the migration for the devices without this feature.

I think we don't want to end up with several migration drivers for virtio-fs.

Quoted 3 lines

I imagine virtio-gpu and virtio-crypto might have similar situations where an object created through a virtqueue request has device-internal state associated with it that must be migrated.

So the point stands still. The device internal state should be restored from the device specific state defined in the spec. Spec would guarantee that the minimal part of the device specific state, implementation should use those to restore implementation specific state.

Quoted 13 lines
Quoted 9 lines
Quoted 5 lines
Quoted 1 line

But we can generalize the virtqueue state for sure.

I agree and also that some device types can standardize their device state representations. But I think it's a technical requirement to support implementation-specific state for device types where cross-implementation migration is not possible.

A question here, if the driver depends on the implementation specific state, how can we make sure that driver can work for other implementation. If we're sure that a single driver can work for all kinds of implementations, it means the we had device specific state not implementation state.

I think this is confusing stateless and stateful devices. You are describing a stateless device here. I'll try to define a stateful device:

Quoted 6 lines

A stateful device maintains state that the driver operates on indirectly via standard requests. For example, the virtio-crypto device has CREATE_SESSION requests and a session ID is returned to the driver so further requests can be made on the session object. It may not be possible to replay, reconnect, or restart the device without losing state.

If it's impossible to do all of these. These state should be noticeable by the driver and they are not implementation specific but device specific which could be defined in the spec.

Quoted 2 lines

I hope that this description, together with the virtiofs specifics above, make the problem clearer.

Yes.

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

I'm not saying the implementation-specific state representation has to be a binary blob. There could be an identifier registry to ensure live migration compatibility checks can be performed. There could also be a standard binary encoding for migration data.

Yes, such requirements has been well studied in the past. There should be plenty of protocols to do this.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

But the contents will be
implementation-specific for some devices.

If we allow this, it breaks the spec effort for having a standard devices. And it will block the real customers.

If we forbid this then devices for which migration is technically possible will be unmigratable. Both users and device implementors will find other solutions, like VFIO, so I don't think we can stop them even if we tried.

The difference is standard device vs vendor specific device. We can't avoid the implementation specific state for vendor specific hardware.

But for standard device, we don't want to end up with migration drivers. (Do we want vendor to ship the vendor specific migration drivers for NVM(e) devices?)

Quoted 13 lines

I recognize that opaque device state poses a risk to migration compatibility, because device implementors may arbitrarily use opaque state when a standard is available.

However, the way to avoid this scenario is by:

1. Making the standard migration approach the easiest to implement because everything has been taken care of. It will save implementors the headache of defining and coding their own device state representations and versioning.

2. Educate users about migration compatibility so they can identify implementors are locking in their users.

Quoted 7 lines
Quoted 6 lines

For vendor specific device, this may work. But for standard devices like virtio, we should go further.

The device states should be defined in the spec clearly. We should re-visit the design if those states contains anything that is implementation specific.

Can you describe how migrating virtiofs devices should work?

I need to learn more virtio-fs before answering this question.

Actually, it would be faster if I can see a prototype of the migration support for virtio-fs and start from there (as I've suggested this in another thread).

Quoted 3 lines

I think
that might be quicker than if I reply to each of your points because our views are still quite far apart.

Yes, it would be quicker if we can start from a prototype.

Thanks

Quoted 2 lines

Stefan

#65 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 36 lines · Jason Wang
Quoted 18 lines · Stefan Hajnoczi
Quoted 17 lines · Jason Wang
Quoted 15 lines · Stefan Hajnoczi
Quoted 1 line · Jason Wang

在 2021/7/20 下午4:50, Stefan Hajnoczi 写道:

I recognize that opaque device state poses a risk to migration compatibility, because device implementors may arbitrarily use opaque state when a standard is available.

However, the way to avoid this scenario is by:

1. Making the standard migration approach the easiest to implement because everything has been taken care of. It will save implementors the headache of defining and coding their own device state representations and versioning.

2. Educate users about migration compatibility so they can identify implementors are locking in their users.

Quoted 7 lines
Quoted 6 lines

For vendor specific device, this may work. But for standard devices like virtio, we should go further.

The device states should be defined in the spec clearly. We should re-visit the design if those states contains anything that is implementation specific.

Can you describe how migrating virtiofs devices should work?

I need to learn more virtio-fs before answering this question.

Actually, it would be faster if I can see a prototype of the migration support for virtio-fs and start from there (as I've suggested this in another thread).

Quoted 6 lines
Quoted 3 lines

I think
that might be quicker than if I reply to each of your points because our views are still quite far apart.

Yes, it would be quicker if we can start from a prototype.

I have CCed Max Reitz to check whether a prototype of virtiofs migration might be available soon?

But I can describe the key state that needs to be migrated:

- FUSE nodeid -> host inode mappings. The driver uses nodeid numbers in the FUSE protocol and the device maps them to actual inodes on the passthrough file system.
- FUSE fh -> open fd mappings. The driver uses fh numbers in the FUSE protocol and the device maps them to actual file descriptors on the host.
- FUSE fh -> open dir fd mappings. The driver uses fh numbers in the FUSE protocol and the device maps them to actual O_DIRECTORY file descriptors on the host.

The driver expects to be able to continue using nodeid and fh numbers across migration. Let's look at just the open fds for a moment:

The OPEN command opens the file for a given nodeid and returns its fh. Due to POSIX file system semantics there is no reliable way to reopen the same file from just the filename. The problem is that a file can be renamed or deleted (but still accessible until the last fd is closed).

Linux file handles (open_by_handle_at(2) and name_to_handle_at(2)) make it possible to reopen the exact same file using a struct file_handle instead of a filename. So the virtiofs device could transfer the Linux file handles to the destination where the fd -> open fd mappings can be restored.

The problem is that Linux file handles are an implementation-specific solution to this problem. On non-Linux hosts there may be other solutions that userspace file systems use to solve this problem. Or a virtiofs device may not implement a passthrough host file system and have a completely different concept of what an inode is.

This means only a subset of virtiofs implementations can use Linux file handles as part of their device state. There is no way for the driver or device to recreate or restore the necessary information without implementation-specific device state like Linux file handles, though.

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Stefan

#66 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 34 lines · Stefan Hajnoczi
Quoted 33 lines · Jason Wang
Quoted 17 lines · Stefan Hajnoczi
Quoted 16 lines · Jason Wang
Quoted 15 lines · Stefan Hajnoczi
Quoted 1 line · Jason Wang

在 2021/7/20 下午4:50, Stefan Hajnoczi 写道:

I recognize that opaque device state poses a risk to migration compatibility, because device implementors may arbitrarily use opaque state when a standard is available.

However, the way to avoid this scenario is by:

1. Making the standard migration approach the easiest to implement because everything has been taken care of. It will save implementors the headache of defining and coding their own device state representations and versioning.

2. Educate users about migration compatibility so they can identify implementors are locking in their users.

Quoted 7 lines
Quoted 6 lines

For vendor specific device, this may work. But for standard devices like virtio, we should go further.

The device states should be defined in the spec clearly. We should re-visit the design if those states contains anything that is implementation specific.

Can you describe how migrating virtiofs devices should work?

I need to learn more virtio-fs before answering this question.

Actually, it would be faster if I can see a prototype of the migration support for virtio-fs and start from there (as I've suggested this in another thread).

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I think
that might be quicker than if I reply to each of your points because our views are still quite far apart.

Yes, it would be quicker if we can start from a prototype.

I have CCed Max Reitz to check whether a prototype of virtiofs migration might be available soon?

No, I don’t think so.

I hope I can start with looking into it soon (where "soon" already means one or two months), but I can make absolutely no predictions on when something usable even as just a prototype might come out of it.  (And I find that to be so many conditionals that I don’t think there’s going to be one soon.)

Quoted 32 lines

But I can describe the key state that needs to be migrated:

- FUSE nodeid -> host inode mappings. The driver uses nodeid numbers in the FUSE protocol and the device maps them to actual inodes on the passthrough file system.
- FUSE fh -> open fd mappings. The driver uses fh numbers in the FUSE protocol and the device maps them to actual file descriptors on the host.
- FUSE fh -> open dir fd mappings. The driver uses fh numbers in the FUSE protocol and the device maps them to actual O_DIRECTORY file descriptors on the host.

The driver expects to be able to continue using nodeid and fh numbers across migration. Let's look at just the open fds for a moment:

The OPEN command opens the file for a given nodeid and returns its fh. Due to POSIX file system semantics there is no reliable way to reopen the same file from just the filename. The problem is that a file can be renamed or deleted (but still accessible until the last fd is closed).

Linux file handles (open_by_handle_at(2) and name_to_handle_at(2)) make it possible to reopen the exact same file using a struct file_handle instead of a filename. So the virtiofs device could transfer the Linux file handles to the destination where the fd -> open fd mappings can be restored.

The problem is that Linux file handles are an implementation-specific solution to this problem. On non-Linux hosts there may be other solutions that userspace file systems use to solve this problem. Or a virtiofs device may not implement a passthrough host file system and have a completely different concept of what an inode is.

Quoted 4 lines

This means only a subset of virtiofs implementations can use Linux file handles as part of their device state. There is no way for the driver or device to recreate or restore the necessary information without implementation-specific device state like Linux file handles, though.

(For the record, I agree with this summary.)

Max

Quoted 5 lines

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Stefan

author
#67 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Stefan Hajnoczi
Quoted 62 lines · Stefan Hajnoczi
Quoted 33 lines · Jason Wang
Quoted 17 lines · Stefan Hajnoczi
Quoted 16 lines · Jason Wang
Quoted 15 lines · Stefan Hajnoczi
Quoted 1 line · Jason Wang

在 2021/7/20 下午4:50, Stefan Hajnoczi 写道:

I recognize that opaque device state poses a risk to migration compatibility, because device implementors may arbitrarily use opaque state when a standard is available.

However, the way to avoid this scenario is by:

1. Making the standard migration approach the easiest to implement because everything has been taken care of. It will save implementors the headache of defining and coding their own device state representations and versioning.

2. Educate users about migration compatibility so they can identify implementors are locking in their users.

Quoted 7 lines
Quoted 6 lines

For vendor specific device, this may work. But for standard devices like virtio, we should go further.

The device states should be defined in the spec clearly. We should re-visit the design if those states contains anything that is implementation specific.

Can you describe how migrating virtiofs devices should work?

I need to learn more virtio-fs before answering this question.

Actually, it would be faster if I can see a prototype of the migration support for virtio-fs and start from there (as I've suggested this in another thread).

Quoted 5 lines
Quoted 3 lines

I think
that might be quicker than if I reply to each of your points because our views are still quite far apart.

Yes, it would be quicker if we can start from a prototype.

I have CCed Max Reitz to check whether a prototype of virtiofs migration might be available soon?

But I can describe the key state that needs to be migrated:

- FUSE nodeid -> host inode mappings. The driver uses nodeid numbers in the FUSE protocol and the device maps them to actual inodes on the passthrough file system.
- FUSE fh -> open fd mappings. The driver uses fh numbers in the FUSE protocol and the device maps them to actual file descriptors on the host.
- FUSE fh -> open dir fd mappings. The driver uses fh numbers in the FUSE protocol and the device maps them to actual O_DIRECTORY file descriptors on the host.

The driver expects to be able to continue using nodeid and fh numbers across migration. Let's look at just the open fds for a moment:

The OPEN command opens the file for a given nodeid and returns its fh. Due to POSIX file system semantics there is no reliable way to reopen the same file from just the filename. The problem is that a file can be renamed or deleted (but still accessible until the last fd is closed).

Quoted 8 lines

Linux file handles (open_by_handle_at(2) and name_to_handle_at(2)) make it possible to reopen the exact same file using a struct file_handle instead of a filename. So the virtiofs device could transfer the Linux file handles to the destination where the fd -> open fd mappings can be restored.

The problem is that Linux file handles are an implementation-specific solution to this problem.

Yes according to the manpage, it not a part of the uABI, so it's not guaranteed to work on the destination if I understand it correctly.

Quoted 4 lines

On non-Linux hosts there may be other
solutions that userspace file systems use to solve this problem. Or a virtiofs device may not implement a passthrough host file system and have a completely different concept of what an inode is.

The situation is somehow similar to device pass-through which makes it very hard to have a general way to migrate.

Quoted 5 lines

This means only a subset of virtiofs implementations can use Linux file handles as part of their device state. There is no way for the driver or device to recreate or restore the necessary information without implementation-specific device state like Linux file handles, though.

So my understanding is that even the linux file handle is not a general solution:

- not a part of uABI (not guaranteed to work on the destination)
- depends on the kernel version and a specific Kconfig (CONFIG_FHANDLE)

Quoted 4 lines

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

Thanks

Quoted 2 lines

Stefan

#68 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 9 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles. So they need to be transferred to the destination (or stored on a shared file system as you suggested), regardless of whether they are part of the VIRTIO spec's device state or not.

Implementation-specific state can be considered outside the scope of the VIRTIO spec. In other words, we could exclude it from the VIRTIO-level device state that save/load operate on. This does not solve the problem, it just shifts the responsibility to the virtualization stack to migrate this state.

The Linux file handles or other virtiofsd implementation-specific state would be migrated separately (e.g. using DBus VMstate) so that by the time the destination device does a VIRTIO load operation, it has the necessary implementation-specific state ready.

I prefer to support in-band migration of implementation-specific state because it's less complex to have a single device state instead of splitting it.

Is this the direction you were thinking in?

Stefan

author
#69 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 11 lines · Stefan Hajnoczi
Quoted 8 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

Assuming this is true. Can we store those mapping (e.g fuse inode -> host inode) in a known path/file in the passthrough filesystem itself and hide that file from the guest?

The destination can simply open this unkown file and do the lookup the mapping and reopen the file if necessary.

Then we don't need the Linux file handle.

Quoted 11 lines

So they need to be transferred to the
destination (or stored on a shared file system as you suggested), regardless of whether they are part of the VIRTIO spec's device state or not.

Implementation-specific state can be considered outside the scope of the VIRTIO spec. In other words, we could exclude it from the VIRTIO-level device state that save/load operate on. This does not solve the problem, it just shifts the responsibility to the virtualization stack to migrate this state.

Quoted 4 lines

The Linux file handles or other virtiofsd implementation-specific state would be migrated separately (e.g. using DBus VMstate) so that by the time the destination device does a VIRTIO load operation, it has the necessary implementation-specific state ready.

That may work but I want to get rid of the implementation specific stuffs like linux handles completely.

Quoted 4 lines

I prefer to support in-band migration of implementation-specific state because it's less complex to have a single device state instead of splitting it.

I wonder how to deal with migration compatibility in this case.

Quoted 2 lines

Is this the direction you were thinking in?

Somehow.

Thanks

Quoted 2 lines

Stefan

#70 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 15 lines · * Jason Wang ([email protected]) wrote:
Quoted 11 lines · Stefan Hajnoczi
Quoted 8 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

Quoted 6 lines

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

Assuming this is true. Can we store those mapping (e.g fuse inode -> host inode) in a known path/file in the passthrough filesystem itself and hide that file from the guest?

That's pretty dangerous; it assumes that the filesystem is only used together with virtiofs; as a *shared* filesystem it's possible that it's being used directly by normal NFS clients as well.
It's also very racy; trying to make sure those mappings reflect the *current* meaning of inodes even while they're changing under your feet is non-trivial.

Quoted 17 lines

The destination can simply open this unkown file and do the lookup the mapping and reopen the file if necessary.

Then we don't need the Linux file handle.

Quoted 11 lines

So they need to be transferred to the
destination (or stored on a shared file system as you suggested), regardless of whether they are part of the VIRTIO spec's device state or not.

Implementation-specific state can be considered outside the scope of the VIRTIO spec. In other words, we could exclude it from the VIRTIO-level device state that save/load operate on. This does not solve the problem, it just shifts the responsibility to the virtualization stack to migrate this state.

Quoted 8 lines
Quoted 4 lines

The Linux file handles or other virtiofsd implementation-specific state would be migrated separately (e.g. using DBus VMstate) so that by the time the destination device does a VIRTIO load operation, it has the necessary implementation-specific state ready.

That may work but I want to get rid of the implementation specific stuffs like linux handles completely.

I'm not sure how much implementation specific you can get rid of; but you should be able to comparmentalise it, and you should be able to make it so that common things can be shared; i.e. if I have two implementations of virtiofs, both running on Linux, then it might be good if we can live migrate between them, and standardise the format.

So, I'd expect the core virtiofs data to be standardised globally, then I'd expect how Linux implementations work to be standardised.

Dave

Quoted 22 lines
Quoted 5 lines

I prefer to support in-band migration of implementation-specific state because it's less complex to have a single device state instead of splitting it.

I wonder how to deal with migration compatibility in this case.

Quoted 2 lines

Is this the direction you were thinking in?

Somehow.

Thanks

Quoted 2 lines

Stefan

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
author
#71 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 19 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Jason Wang ([email protected]) wrote:
Quoted 10 lines · Stefan Hajnoczi
Quoted 7 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

Quoted 6 lines

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

Assuming this is true. Can we store those mapping (e.g fuse inode -> host inode) in a known path/file in the passthrough filesystem itself and hide that file from the guest?

Quoted 6 lines

That's pretty dangerous; it assumes that the filesystem is only used together with virtiofs; as a *shared* filesystem it's possible that it's being used directly by normal NFS clients as well.
It's also very racy; trying to make sure those mappings reflect the *current* meaning of inodes even while they're changing under your feet is non-trivial.

Right, it's just a thought to avoid migrating implementation specific stuffs.

Quoted 23 lines
Quoted 23 lines

The destination can simply open this unkown file and do the lookup the mapping and reopen the file if necessary.

Then we don't need the Linux file handle.

Quoted 15 lines

So they need to be transferred to the
destination (or stored on a shared file system as you suggested), regardless of whether they are part of the VIRTIO spec's device state or not.

Implementation-specific state can be considered outside the scope of the VIRTIO spec. In other words, we could exclude it from the VIRTIO-level device state that save/load operate on. This does not solve the problem, it just shifts the responsibility to the virtualization stack to migrate this state.

The Linux file handles or other virtiofsd implementation-specific state would be migrated separately (e.g. using DBus VMstate) so that by the time the destination device does a VIRTIO load operation, it has the necessary implementation-specific state ready.

Quoted 5 lines
Quoted 2 lines

That may work but I want to get rid of the implementation specific stuffs like linux handles completely.

I'm not sure how much implementation specific you can get rid of; but you should be able to comparmentalise it, and you should be able to make it so that common things can be shared;

Yes, that's is the way we need to go.

Quoted 3 lines

i.e. if I have two
implementations of virtiofs, both running on Linux, then it might be good if we can live migrate between them, and standardise the format.

As replied in the previous version, I'm not sure how hard it is consider the file_handle mentioned by Stefan is not a part of uABI and it depends on specific kernel config to work.

Quoted 2 lines

So, I'd expect the core virtiofs data to be standardised globally,

Yes, maybe start at the FUSE level.

Quoted 2 lines

then
I'd expect how Linux implementations work to be standardised.

Does it mean we need:

1) port virtiofsd to multiple platforms
2) only support live migration among virtiofds

?

Thanks

Quoted 18 lines

Dave

Quoted 15 lines
Quoted 3 lines

I prefer to support in-band migration of implementation-specific state because it's less complex to have a single device state instead of splitting it.

I wonder how to deal with migration compatibility in this case.

Quoted 1 line

Is this the direction you were thinking in?

Somehow.

Thanks

Quoted 1 line

Stefan

#72 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 87 lines · * Jason Wang ([email protected]) wrote:
Quoted 19 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Jason Wang ([email protected]) wrote:
Quoted 10 lines · Stefan Hajnoczi
Quoted 7 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

Quoted 6 lines

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

Assuming this is true. Can we store those mapping (e.g fuse inode -> host inode) in a known path/file in the passthrough filesystem itself and hide that file from the guest?

Quoted 6 lines

That's pretty dangerous; it assumes that the filesystem is only used together with virtiofs; as a *shared* filesystem it's possible that it's being used directly by normal NFS clients as well.
It's also very racy; trying to make sure those mappings reflect the *current* meaning of inodes even while they're changing under your feet is non-trivial.

Right, it's just a thought to avoid migrating implementation specific stuffs.

Quoted 23 lines
Quoted 23 lines

The destination can simply open this unkown file and do the lookup the mapping and reopen the file if necessary.

Then we don't need the Linux file handle.

Quoted 15 lines

So they need to be transferred to the
destination (or stored on a shared file system as you suggested), regardless of whether they are part of the VIRTIO spec's device state or not.

Implementation-specific state can be considered outside the scope of the VIRTIO spec. In other words, we could exclude it from the VIRTIO-level device state that save/load operate on. This does not solve the problem, it just shifts the responsibility to the virtualization stack to migrate this state.

The Linux file handles or other virtiofsd implementation-specific state would be migrated separately (e.g. using DBus VMstate) so that by the time the destination device does a VIRTIO load operation, it has the necessary implementation-specific state ready.

Quoted 5 lines
Quoted 2 lines

That may work but I want to get rid of the implementation specific stuffs like linux handles completely.

I'm not sure how much implementation specific you can get rid of; but you should be able to comparmentalise it, and you should be able to make it so that common things can be shared;

Yes, that's is the way we need to go.

Quoted 3 lines

i.e. if I have two
implementations of virtiofs, both running on Linux, then it might be good if we can live migrate between them, and standardise the format.

As replied in the previous version, I'm not sure how hard it is consider the file_handle mentioned by Stefan is not a part of uABI and it depends on specific kernel config to work.

Quoted 2 lines

So, I'd expect the core virtiofs data to be standardised globally,

Yes, maybe start at the FUSE level.

Quoted 2 lines

then
I'd expect how Linux implementations work to be standardised.

Quoted 6 lines

Does it mean we need:

1) port virtiofsd to multiple platforms
2) only support live migration among virtiofds

?

Not necessarily; I mean that we have layers:
a) Virtio
b) Virtio-fs
c]
c1) virtio-fs backed by a Linux filesystem
c2) virtio-fs backed by some object store
c3) virito-fs backed by something else

(a) is standardised
The migration data for (b) can be standardised
We can also standardise c1, c2 (not that we've made a c2) and we could expect migration between different implementations all that are backed by a Linux filesystem (if that file handle stuff is portable); but we wouldn't expect a migration between c1 and c2 to work. (and c2 might get split if there are different types of object store).

So, just because there are different types of backends, doesn't mean we have to give up standardisation; we just have to acknowledge there's a range of backends and standardisethe bits we can.

Dave

Quoted 21 lines
Quoted 20 lines

Dave

Quoted 15 lines
Quoted 3 lines

I prefer to support in-band migration of implementation-specific state because it's less complex to have a single device state instead of splitting it.

I wonder how to deal with migration compatibility in this case.

Quoted 1 line

Is this the direction you were thinking in?

Somehow.

Thanks

Quoted 1 line

Stefan

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
author
#73 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 92 lines · Dr. David Alan Gilbert
Quoted 77 lines · * Jason Wang ([email protected]) wrote:
Quoted 18 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Jason Wang ([email protected]) wrote:
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

Quoted 8 lines
Quoted 2 lines

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

Assuming this is true. Can we store those mapping (e.g fuse inode -> host inode) in a known path/file in the passthrough filesystem itself and hide that file from the guest?

Quoted 6 lines

That's pretty dangerous; it assumes that the filesystem is only used together with virtiofs; as a *shared* filesystem it's possible that it's being used directly by normal NFS clients as well.
It's also very racy; trying to make sure those mappings reflect the *current* meaning of inodes even while they're changing under your feet is non-trivial.

Right, it's just a thought to avoid migrating implementation specific stuffs.

Quoted 21 lines
Quoted 21 lines

The destination can simply open this unkown file and do the lookup the mapping and reopen the file if necessary.

Then we don't need the Linux file handle.

Quoted 15 lines

So they need to be transferred to the
destination (or stored on a shared file system as you suggested), regardless of whether they are part of the VIRTIO spec's device state or not.

Implementation-specific state can be considered outside the scope of the VIRTIO spec. In other words, we could exclude it from the VIRTIO-level device state that save/load operate on. This does not solve the problem, it just shifts the responsibility to the virtualization stack to migrate this state.

The Linux file handles or other virtiofsd implementation-specific state would be migrated separately (e.g. using DBus VMstate) so that by the time the destination device does a VIRTIO load operation, it has the necessary implementation-specific state ready.

Quoted 5 lines
Quoted 2 lines

That may work but I want to get rid of the implementation specific stuffs like linux handles completely.

I'm not sure how much implementation specific you can get rid of; but you should be able to comparmentalise it, and you should be able to make it so that common things can be shared;

Yes, that's is the way we need to go.

Quoted 3 lines

i.e. if I have two
implementations of virtiofs, both running on Linux, then it might be good if we can live migrate between them, and standardise the format.

As replied in the previous version, I'm not sure how hard it is consider the file_handle mentioned by Stefan is not a part of uABI and it depends on specific kernel config to work.

Quoted 1 line

So, I'd expect the core virtiofs data to be standardised globally,

Yes, maybe start at the FUSE level.

Quoted 2 lines

then
I'd expect how Linux implementations work to be standardised.

Quoted 6 lines

Does it mean we need:

1) port virtiofsd to multiple platforms
2) only support live migration among virtiofds

?

Not necessarily; I mean that we have layers:
a) Virtio
b) Virtio-fs
c]
c1) virtio-fs backed by a Linux filesystem
c2) virtio-fs backed by some object store
c3) virito-fs backed by something else

Quoted 2 lines

(a) is standardised
The migration data for (b) can be standardised

That would be good.

Quoted 5 lines

We can also standardise c1, c2 (not that we've made a c2) and we could expect migration between different implementations all that are backed by a Linux filesystem (if that file handle stuff is portable); but we wouldn't expect a migration between c1 and c2 to work. (and c2 might get split if there are different types of object store).

If I understand this correctly, this requires the management layer to know the details of the backend before trying to live migrate the guest. Or do we need different feature bits for the above three types of the virtio-fs device?

Quoted 4 lines

So, just because there are different types of backends, doesn't mean we have to give up standardisation; we just have to acknowledge there's a range of backends and standardisethe bits we can.

Right.

Thanks

Quoted 19 lines

Dave

Quoted 16 lines
Quoted 16 lines

Dave

Quoted 13 lines
Quoted 3 lines

I prefer to support in-band migration of implementation-specific state because it's less complex to have a single device state instead of splitting it.

I wonder how to deal with migration compatibility in this case.

Quoted 1 line

Is this the direction you were thinking in?

Somehow.

Thanks

Quoted 1 line

Stefan

#74 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 108 lines · * Jason Wang ([email protected]) wrote:
Quoted 92 lines · Dr. David Alan Gilbert
Quoted 77 lines · * Jason Wang ([email protected]) wrote:
Quoted 18 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Jason Wang ([email protected]) wrote:
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

Quoted 8 lines
Quoted 2 lines

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

Assuming this is true. Can we store those mapping (e.g fuse inode -> host inode) in a known path/file in the passthrough filesystem itself and hide that file from the guest?

Quoted 6 lines

That's pretty dangerous; it assumes that the filesystem is only used together with virtiofs; as a *shared* filesystem it's possible that it's being used directly by normal NFS clients as well.
It's also very racy; trying to make sure those mappings reflect the *current* meaning of inodes even while they're changing under your feet is non-trivial.

Right, it's just a thought to avoid migrating implementation specific stuffs.

Quoted 21 lines
Quoted 21 lines

The destination can simply open this unkown file and do the lookup the mapping and reopen the file if necessary.

Then we don't need the Linux file handle.

Quoted 15 lines

So they need to be transferred to the
destination (or stored on a shared file system as you suggested), regardless of whether they are part of the VIRTIO spec's device state or not.

Implementation-specific state can be considered outside the scope of the VIRTIO spec. In other words, we could exclude it from the VIRTIO-level device state that save/load operate on. This does not solve the problem, it just shifts the responsibility to the virtualization stack to migrate this state.

The Linux file handles or other virtiofsd implementation-specific state would be migrated separately (e.g. using DBus VMstate) so that by the time the destination device does a VIRTIO load operation, it has the necessary implementation-specific state ready.

Quoted 5 lines
Quoted 2 lines

That may work but I want to get rid of the implementation specific stuffs like linux handles completely.

I'm not sure how much implementation specific you can get rid of; but you should be able to comparmentalise it, and you should be able to make it so that common things can be shared;

Yes, that's is the way we need to go.

Quoted 3 lines

i.e. if I have two
implementations of virtiofs, both running on Linux, then it might be good if we can live migrate between them, and standardise the format.

As replied in the previous version, I'm not sure how hard it is consider the file_handle mentioned by Stefan is not a part of uABI and it depends on specific kernel config to work.

Quoted 1 line

So, I'd expect the core virtiofs data to be standardised globally,

Yes, maybe start at the FUSE level.

Quoted 2 lines

then
I'd expect how Linux implementations work to be standardised.

Quoted 6 lines

Does it mean we need:

1) port virtiofsd to multiple platforms
2) only support live migration among virtiofds

?

Not necessarily; I mean that we have layers:
a) Virtio
b) Virtio-fs
c]
c1) virtio-fs backed by a Linux filesystem
c2) virtio-fs backed by some object store
c3) virito-fs backed by something else

Quoted 2 lines

(a) is standardised
The migration data for (b) can be standardised

That would be good.

Quoted 5 lines

We can also standardise c1, c2 (not that we've made a c2) and we could expect migration between different implementations all that are backed by a Linux filesystem (if that file handle stuff is portable); but we wouldn't expect a migration between c1 and c2 to work. (and c2 might get split if there are different types of object store).

Quoted 4 lines

If I understand this correctly, this requires the management layer to know the details of the backend before trying to live migrate the guest. Or do we need different feature bits for the above three types of the virtio-fs device?

Yep, something would need to know the details of the backend; but that's true of most existing backends anyway; e.g. in virtio-net the management layer understands the underlying network and what it has to setup on the destination to ensure the network on both sides looks the same; it's got different implications but it does still need to know it.

Dave

Quoted 32 lines
Quoted 5 lines

So, just because there are different types of backends, doesn't mean we have to give up standardisation; we just have to acknowledge there's a range of backends and standardisethe bits we can.

Right.

Thanks

Quoted 19 lines

Dave

Quoted 16 lines
Quoted 16 lines

Dave

Quoted 13 lines
Quoted 3 lines

I prefer to support in-band migration of implementation-specific state because it's less complex to have a single device state instead of splitting it.

I wonder how to deal with migration compatibility in this case.

Quoted 1 line

Is this the direction you were thinking in?

Somehow.

Thanks

Quoted 1 line

Stefan

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
author
#75 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 105 lines · Dr. David Alan Gilbert
Quoted 100 lines · * Jason Wang ([email protected]) wrote:
Quoted 87 lines · Dr. David Alan Gilbert
Quoted 70 lines · * Jason Wang ([email protected]) wrote:
Quoted 18 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Jason Wang ([email protected]) wrote:
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

Quoted 8 lines
Quoted 2 lines

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

Assuming this is true. Can we store those mapping (e.g fuse inode -> host inode) in a known path/file in the passthrough filesystem itself and hide that file from the guest?

Quoted 6 lines

That's pretty dangerous; it assumes that the filesystem is only used together with virtiofs; as a *shared* filesystem it's possible that it's being used directly by normal NFS clients as well.
It's also very racy; trying to make sure those mappings reflect the *current* meaning of inodes even while they're changing under your feet is non-trivial.

Right, it's just a thought to avoid migrating implementation specific stuffs.

Quoted 21 lines
Quoted 21 lines

The destination can simply open this unkown file and do the lookup the mapping and reopen the file if necessary.

Then we don't need the Linux file handle.

Quoted 15 lines

So they need to be transferred to the
destination (or stored on a shared file system as you suggested), regardless of whether they are part of the VIRTIO spec's device state or not.

Implementation-specific state can be considered outside the scope of the VIRTIO spec. In other words, we could exclude it from the VIRTIO-level device state that save/load operate on. This does not solve the problem, it just shifts the responsibility to the virtualization stack to migrate this state.

The Linux file handles or other virtiofsd implementation-specific state would be migrated separately (e.g. using DBus VMstate) so that by the time the destination device does a VIRTIO load operation, it has the necessary implementation-specific state ready.

Quoted 5 lines
Quoted 2 lines

That may work but I want to get rid of the implementation specific stuffs like linux handles completely.

I'm not sure how much implementation specific you can get rid of; but you should be able to comparmentalise it, and you should be able to make it so that common things can be shared;

Yes, that's is the way we need to go.

Quoted 3 lines

i.e. if I have two
implementations of virtiofs, both running on Linux, then it might be good if we can live migrate between them, and standardise the format.

As replied in the previous version, I'm not sure how hard it is consider the file_handle mentioned by Stefan is not a part of uABI and it depends on specific kernel config to work.

Quoted 1 line

So, I'd expect the core virtiofs data to be standardised globally,

Yes, maybe start at the FUSE level.

Quoted 8 lines
Quoted 2 lines

then
I'd expect how Linux implementations work to be standardised.

Does it mean we need:

1) port virtiofsd to multiple platforms
2) only support live migration among virtiofds

?

Not necessarily; I mean that we have layers:
a) Virtio
b) Virtio-fs
c]
c1) virtio-fs backed by a Linux filesystem
c2) virtio-fs backed by some object store
c3) virito-fs backed by something else

Quoted 2 lines

(a) is standardised
The migration data for (b) can be standardised

That would be good.

Quoted 5 lines

We can also standardise c1, c2 (not that we've made a c2) and we could expect migration between different implementations all that are backed by a Linux filesystem (if that file handle stuff is portable); but we wouldn't expect a migration between c1 and c2 to work. (and c2 might get split if there are different types of object store).

Quoted 4 lines

If I understand this correctly, this requires the management layer to know the details of the backend before trying to live migrate the guest. Or do we need different feature bits for the above three types of the virtio-fs device?

Quoted 5 lines

Yep, something would need to know the details of the backend; but that's true of most existing backends anyway; e.g. in virtio-net the management layer understands the underlying network and what it has to setup on the destination to ensure the network on both sides looks the same; it's got different implications but it does still need to know it.

I think it's different. E.g for the case of virtio-net though setup on the destination doesn't depends on the device state.

E.g technically, we can do cross backend live migration. E.g from qemu virtio-net to a vhost-user backend.

Thanks

Quoted 29 lines

Dave

Quoted 26 lines
Quoted 3 lines

So, just because there are different types of backends, doesn't mean we have to give up standardisation; we just have to acknowledge there's a range of backends and standardisethe bits we can.

Right.

Thanks

Quoted 17 lines

Dave

Quoted 15 lines
Quoted 15 lines

Dave

Quoted 13 lines
Quoted 3 lines

I prefer to support in-band migration of implementation-specific state because it's less complex to have a single device state instead of splitting it.

I wonder how to deal with migration compatibility in this case.

Quoted 1 line

Is this the direction you were thinking in?

Somehow.

Thanks

Quoted 1 line

Stefan

#76 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 114 lines · Jason Wang
Quoted 105 lines · Dr. David Alan Gilbert
Quoted 100 lines · * Jason Wang ([email protected]) wrote:
Quoted 87 lines · Dr. David Alan Gilbert
Quoted 70 lines · * Jason Wang ([email protected]) wrote:
Quoted 18 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Jason Wang ([email protected]) wrote:
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

Quoted 8 lines
Quoted 2 lines

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

Assuming this is true. Can we store those mapping (e.g fuse inode -> host inode) in a known path/file in the passthrough filesystem itself and hide that file from the guest?

Quoted 6 lines

That's pretty dangerous; it assumes that the filesystem is only used together with virtiofs; as a *shared* filesystem it's possible that it's being used directly by normal NFS clients as well.
It's also very racy; trying to make sure those mappings reflect the *current* meaning of inodes even while they're changing under your feet is non-trivial.

Right, it's just a thought to avoid migrating implementation specific stuffs.

Quoted 21 lines
Quoted 21 lines

The destination can simply open this unkown file and do the lookup the mapping and reopen the file if necessary.

Then we don't need the Linux file handle.

Quoted 15 lines

     So they need to be transferred to the
destination (or stored on a shared file system as you suggested), regardless of whether they are part of the VIRTIO spec's device state or not.

Implementation-specific state can be considered outside the scope of the VIRTIO spec. In other words, we could exclude it from the VIRTIO-level device state that save/load operate on. This does not solve the problem, it just shifts the responsibility to the virtualization stack to migrate this state.

The Linux file handles or other virtiofsd implementation-specific state would be migrated separately (e.g. using DBus VMstate) so that by the time the destination device does a VIRTIO load operation, it has the necessary implementation-specific state ready.

Quoted 5 lines
Quoted 2 lines

That may work but I want to get rid of the implementation specific stuffs like linux handles completely.

I'm not sure how much implementation specific you can get rid of; but you should be able to comparmentalise it, and you should be able to make it so that common things can be shared;

Yes, that's is the way we need to go.

Quoted 3 lines

    i.e. if I have two
implementations of virtiofs, both running on Linux, then it might be good if we can live migrate between them, and standardise the format.

As replied in the previous version, I'm not sure how hard it is consider the file_handle mentioned by Stefan is not a part of uABI and it depends on specific kernel config to work.

Quoted 1 line

So, I'd expect the core virtiofs data to be standardised globally,

Yes, maybe start at the FUSE level.

Quoted 8 lines
Quoted 2 lines

    then
I'd expect how Linux implementations work to be standardised.

Does it mean we need:

1) port virtiofsd to multiple platforms
2) only support live migration among virtiofds

?

Not necessarily; I mean that we have layers:
    a) Virtio
    b) Virtio-fs
    c]
      c1) virtio-fs backed by a Linux filesystem
      c2) virtio-fs backed by some object store
      c3) virito-fs backed by something else

Quoted 2 lines

(a) is standardised
The migration data for (b) can be standardised

That would be good.

Quoted 5 lines

We can also standardise c1, c2 (not that we've made a c2) and we could expect migration between different implementations all that are backed by a Linux filesystem (if that file handle stuff is portable); but we wouldn't expect a migration between c1 and c2 to work. (and c2 might get split if there are different types of object store).

Quoted 4 lines

If I understand this correctly, this requires the management layer to know the details of the backend before trying to live migrate the guest. Or do we need different feature bits for the above three types of the virtio-fs device?

Quoted 5 lines

Yep, something would need to know the details of the backend; but that's true of most existing backends anyway; e.g. in virtio-net the management layer understands the underlying network and what it has to setup on the destination to ensure the network on both sides looks the same; it's got different implications but it does still need to know it.

Quoted 3 lines

I think it's different. E.g for the case of virtio-net though setup on the destination doesn't depends on the device state.

E.g technically, we can do cross backend live migration. E.g from qemu virtio-net to a vhost-user backend.

virtio-blk also need to have management to control backends.

Let's say the storage is backed by remote nvmf target so you must to create the same connection in the destination as well.

This should be done by some sys-admin.

Quoted 34 lines

Thanks

Quoted 29 lines

Dave

Quoted 26 lines
Quoted 3 lines

So, just because there are different types of backends, doesn't mean we have to give up standardisation; we just have to acknowledge there's a range of backends and standardisethe bits we can.

Right.

Thanks

Quoted 17 lines

Dave

Quoted 15 lines
Quoted 15 lines

Dave

Quoted 13 lines
Quoted 3 lines

I prefer to support in-band migration of implementation-specific state because it's less complex to have a single device state instead of splitting it.

I wonder how to deal with migration compatibility in this case.

Quoted 1 line

Is this the direction you were thinking in?

Somehow.

Thanks

Quoted 1 line

Stefan

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#77 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Jason Wang
Quoted 11 lines · Jason Wang
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 3 lines · * Jason Wang ([email protected]) wrote:
Quoted 2 lines · Stefan Hajnoczi
Quoted 1 line · Jason Wang

在 2021/7/26 下午11:07, Stefan Hajnoczi 写道:

I'd expect how Linux implementations work to be standardised.

Does it mean we need:

1) port virtiofsd to multiple platforms

Correct migration requires a non-POSIX mechanism to reopen files (saving inode numbers as you've suggested isn't enough). If that's unavailable then it won't be possible to migrate safely.

Quoted 1 line

2) only support live migration among virtiofds

We can standardize the device state representation for Linux passthrough file systems and implement it in QEMU's virtiofsd and virtiofsd-rs.

However, it's technically possible for other virtiofsd implementations to migrate too and they shouldn't be second-class citizens. QEMU's virtiofsd isn't special and Linux passthrough file systems aren't special.

Some device state representations will apply to one specific virtiofs implementation, so the value of standardizing it beyond choosing a unique identifier to prevent collisions is questionable. Does the VIRTIO TC want to spend time reviewing implementation-specific device state representations?

What I suggest is to allow in-band implementation-specific device state with a unique identifier that prevents migration between incompatible implementations. Standardize device state representations that are actually worth standardizing (like the Linux passthrough file system where there are multiple implementations): implementors benefit from using the standard because it saves them time and ensures migration compatibility.

Stefan

author
#78 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 6 lines · Stefan Hajnoczi
Quoted 5 lines · Jason Wang
Quoted 4 lines · Dr. David Alan Gilbert
Quoted 3 lines · * Jason Wang ([email protected]) wrote:
Quoted 2 lines · Stefan Hajnoczi
Quoted 1 line · Jason Wang

在 2021/7/26 下午11:07, Stefan Hajnoczi 写道:

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I'd expect how Linux implementations work to be standardised.

Does it mean we need:

1) port virtiofsd to multiple platforms

Correct migration requires a non-POSIX mechanism to reopen files (saving inode numbers as you've suggested isn't enough). If that's unavailable then it won't be possible to migrate safely.

Ok.

Quoted 9 lines
Quoted 2 lines

2) only support live migration among virtiofds

We can standardize the device state representation for Linux passthrough file systems and implement it in QEMU's virtiofsd and virtiofsd-rs.

However, it's technically possible for other virtiofsd implementations to migrate too and they shouldn't be second-class citizens. QEMU's virtiofsd isn't special and Linux passthrough file systems aren't special.

So the migration compatibility is still a problem for those backends.

Quoted 4 lines

Some device state representations will apply to one specific virtiofs implementation, so the value of standardizing it beyond choosing a unique identifier to prevent collisions is questionable.

As replied in another thread, could we categorize the different types of backend with different feature bits. Then we can start think of how to standardize the state of each?

Quoted 3 lines

Does the VIRTIO
TC want to spend time reviewing implementation-specific device state representations?

If it's implementation specific not virtio specific, I guess not. But if we use feature bits for identify the backend types, do we have the chance to make it virtio specific instead of implementation specific?

Quoted 4 lines

What I suggest is to allow in-band implementation-specific device state with a unique identifier that prevents migration between incompatible implementations.

Does this mean we can only know it's impossible to migrate after a migration failure?

Quoted 5 lines

Standardize device state representations that are
actually worth standardizing (like the Linux passthrough file system where there are multiple implementations): implementors benefit from using the standard because it saves them time and ensures migration compatibility.

Yes.

Thanks

Quoted 2 lines

Stefan

#79 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Dr. David Alan Gilbert
Quoted 23 lines · Dr. David Alan Gilbert
Quoted 15 lines · * Jason Wang ([email protected]) wrote:
Quoted 11 lines · Stefan Hajnoczi
Quoted 8 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

Quoted 6 lines

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

Assuming this is true. Can we store those mapping (e.g fuse inode -> host inode) in a known path/file in the passthrough filesystem itself and hide that file from the guest?

Quoted 6 lines

That's pretty dangerous; it assumes that the filesystem is only used together with virtiofs; as a *shared* filesystem it's possible that it's being used directly by normal NFS clients as well.
It's also very racy; trying to make sure those mappings reflect the *current* meaning of inodes even while they're changing under your feet is non-trivial.

Right, it's impossible to guarantee that you will reopen or even be able to find the same inode for the reasons I've mentioned (deleting and renaming files) plus more (inode number reuse).

Stefan

#80 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Jason Wang
Quoted 11 lines · Jason Wang
Quoted 9 lines · Stefan Hajnoczi
Quoted 8 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

Quoted 6 lines
Quoted 2 lines

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

The virtiofs device is not tied to any particular file system backend. The file system could be shared (available from both the source and destination) or local. It might be a passthrough file system or something else (an in-memory file system similar to Linux tmpfs, a non-POSIX network storage like a REST object storage API, etc).

Stefan

author
#81 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 13 lines · Stefan Hajnoczi
Quoted 12 lines · Jason Wang
Quoted 10 lines · Stefan Hajnoczi
Quoted 7 lines · Jason Wang
Quoted 3 lines · Stefan Hajnoczi

I guess this is just a summary of what we've already discussed and not new information. I think an implementation today would use DBus VMState to transfer implementation-specific device state (an opaque blob).

Instead of trying to migrate those opaque stuffs which is kind of tricky, I wonder if we can avoid them by recording the mapping in the shared filesystem itself.

The problem is that virtiofsd has no way of reopening the exact same files without Linux file handles.

Quoted 7 lines
Quoted 2 lines

I believe if we want to support live migration of the passthrough filesystem. The filesystem itself must be shared? (like NFS)

The virtiofs device is not tied to any particular file system backend. The file system could be shared (available from both the source and destination) or local. It might be a passthrough file system or something else (an in-memory file system similar to Linux tmpfs, a non-POSIX network storage like a REST object storage API, etc).

Yes, I meant for the case of passthrough file system, it must be shared in order to be migrated.

Or can we live migrate the whole passthrough file system to the destination as block device?

Thanks

Quoted 2 lines

Stefan

#82 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

In reply to Jason Wang
Quoted 200 lines · Jason Wang
Quoted 182 lines · Stefan Hajnoczi
Quoted 181 lines · Jason Wang
Quoted 179 lines · Stefan Hajnoczi
Quoted 155 lines · Jason Wang
Quoted 154 lines · Stefan Hajnoczi
Quoted 153 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 90 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Eugenio Perez Martin

If I understand correctly, this is all
driven from the driver inside the guest, so for this to work
the guest must be running and already have initialised the driver.
Yes.

As I see it, the feature can be driven entirely by the VMM as long as
it intercept the relevant configuration space (PCI, MMIO, etc) from
guest's reads and writes, and present it as coherent and transparent
for the guest. Some use cases I can imagine with a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device.
2.1) The VMM stops the device to perform a live migration, and the
guest does not write to STOP in any moment of the LM. It resets the
destination device with the state, and then initializes the device.
2.2) The guest stops the device and, when STOP(32) is set, the source
VMM migrates the device status. The destination VMM realizes the bit,
so it sets the bit in the destination too after device initialization.
2.3) The device is not initialized by the guest so it doesn't matter
what bit has the HW, but the VM can be migrated.

Quoted 5 lines
Quoted 3 lines

Am I missing something?

Thanks!

It's doable like this. It's all a lot of hoops to jump through though. It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

Quoted 7 lines

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

My virtiofs device implementation is backed by an in-memory file system. The device state includes the contents of each file.

Your virtiofs device implementation uses Linux file handles to keep track of open files. The device state includes Linux file handles (but not the contents of each file) so the destination host can access the same files on shared storage.

Cornelia's virtiofs device implementation is backed by an object storage HTTP API. The device state includes API object IDs.

The device state is implementation-dependent. There is no standard representation and it's not possible to migrate between device implementations. How are they supposed to migrate?

This is why I think it's necessarily to allow implementation-specific device state representations.

Quoted 12 lines
Quoted 6 lines
Quoted 2 lines

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

Quoted 5 lines

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?

See above. Stateful devices may require an implementation-defined device state representation.

Quoted 2 lines

2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

Opaque data like D-Bus VMState:
https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

Quoted 7 lines
Quoted 2 lines

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

Quoted 2 lines

So I think it still depend on the driver to implement migrate state which is vendor specific.

The current VFIO migration interface depends on a device-specific software mdev driver but here I'm showing that the physical device can implement the migration interface so that no device-specific driver code is needed.

Quoted 3 lines

Note that it's just an uAPI definition not something defined in the PCI spec.

Yes, that's why I mentioned Changpeng Liu's idea to turn the uAPI into a standard PCI Capability to eliminate the need for device-specific drivers.

Quoted 3 lines

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

I agree. It would have helped to have a complete vision for how live migration should work along with demos. I don't see any migration code in samples/vfio-mdev/ :(.

Quoted 8 lines
Quoted 5 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

This brings troubles in the nested environment.

It depends on the device splitting/management design. If L0 wishes to let L1 manage the VFs then it would need to expose a management device. Since the migration interface is generic (not device-specific) a generic management device solves this for all devices.

Stefan

author
#83 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 213 lines · Stefan Hajnoczi
Quoted 197 lines · Jason Wang
Quoted 181 lines · Stefan Hajnoczi
Quoted 180 lines · Jason Wang
Quoted 179 lines · Stefan Hajnoczi
Quoted 155 lines · Jason Wang
Quoted 154 lines · Stefan Hajnoczi
Quoted 153 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 90 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang
Quoted 29 lines · Michael S. Tsirkin

On Fri, Jul 09, 2021 at 07:23:33PM +0200, Eugenio Perez Martin wrote:
If I understand correctly, this is all
driven from the driver inside the guest, so for this to work
the guest must be running and already have initialised the driver.
Yes.

As I see it, the feature can be driven entirely by the VMM as long as
it intercept the relevant configuration space (PCI, MMIO, etc) from
guest's reads and writes, and present it as coherent and transparent
for the guest. Some use cases I can imagine with a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device.
2.1) The VMM stops the device to perform a live migration, and the
guest does not write to STOP in any moment of the LM. It resets the
destination device with the state, and then initializes the device.
2.2) The guest stops the device and, when STOP(32) is set, the source
VMM migrates the device status. The destination VMM realizes the bit,
so it sets the bit in the destination too after device initialization.
2.3) The device is not initialized by the guest so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's all a lot of hoops to jump through though.
It's also not easy for devices to implement.

It just requires a new status bit. Anything that makes you think it's hard to implement?

E.g for networking device, it should be sufficient to use this bit + the virtqueue state.

Quoted 2 lines

Why don't we design the feature in a way that is useable by VMMs
and implementable by devices in a simple way?

It use the common technology like register shadowing without any further stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

Quoted 7 lines

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

My virtiofs device implementation is backed by an in-memory file system. The device state includes the contents of each file.

Your virtiofs device implementation uses Linux file handles to keep track of open files. The device state includes Linux file handles (but not the contents of each file) so the destination host can access the same files on shared storage.

Quoted 6 lines

Cornelia's virtiofs device implementation is backed by an object storage HTTP API. The device state includes API object IDs.

The device state is implementation-dependent. There is no standard representation and it's not possible to migrate between device implementations. How are they supposed to migrate?

So if I understand correclty, virtio-fs is not desigined to be migrate-able?

(Having a check on the current virtio-fs support in qemu, it looks to me it has a migration blocker).

Quoted 3 lines

This is why I think it's necessarily to allow implementation-specific device state representations.

Or you probably mean you don't support cross backend migration. This sounds like a drawback and it's actually not a standard device but a vendor/implementation specific device.

It would bring a lot of troubles, not only for the implementation but for the management. Maybe we can start from adding the support of migration for some specific backend and start from there.

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

Quoted 7 lines
Quoted 5 lines

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

1) why do we still offer opaque migration state to userspace?

See above. Stateful devices may require an implementation-defined device state representation.

So my point stand still, it's not a standard device if we do this.

Quoted 5 lines
Quoted 3 lines

2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

Opaque data like D-Bus VMState:
https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html

Actually, I meant how to keep the opaque state which is compatible with all the existing device that can do migration.

E.g we want to live migration virtio-blk among any backends (from a hardware device to a software backend).

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

Quoted 6 lines
Quoted 2 lines

So I think it still depend on the driver to implement migrate state which is vendor specific.

The current VFIO migration interface depends on a device-specific software mdev driver but here I'm showing that the physical device can implement the migration interface so that no device-specific driver code is needed.

This is not what I read from the patch:

 * device_state: (read/write)
 *      - The user application writes to this field to inform the vendor driver *        about the device state to be transitioned to.
 *      - The vendor driver should take the necessary actions to change the *        device state. After successful transition to a given state, the *        vendor driver should return success on write(device_state, state) *        system call. If the device state transition fails, the vendor driver *        should return an appropriate -errno for the fault condition.

Vendor driver need to mediate between the uAPI and the actual device.

Quoted 6 lines
Quoted 3 lines

Note that it's just an uAPI definition not something defined in the PCI spec.

Yes, that's why I mentioned Changpeng Liu's idea to turn the uAPI into a standard PCI Capability to eliminate the need for device-specific drivers.

Ok.

Quoted 6 lines
Quoted 3 lines

Out of curiosity, the patch is merged without any real users in the Linux. This is very bad since we lose the change to audit the whole design.

I agree. It would have helped to have a complete vision for how live migration should work along with demos. I don't see any migration code in samples/vfio-mdev/ :(.

Right.

Quoted 11 lines
Quoted 7 lines
Quoted 5 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

This brings troubles in the nested environment.

It depends on the device splitting/management design. If L0 wishes to let L1 manage the VFs then it would need to expose a management device. Since the migration interface is generic (not device-specific) a generic management device solves this for all devices.

Right, but it's a burden to expose the management device or it may just won't work.

Thanks

Quoted 2 lines

Stefan

#84 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 223 lines · Jason Wang
Quoted 213 lines · Stefan Hajnoczi
Quoted 197 lines · Jason Wang
Quoted 181 lines · Stefan Hajnoczi
Quoted 180 lines · Jason Wang
Quoted 179 lines · Stefan Hajnoczi
Quoted 155 lines · Jason Wang
Quoted 154 lines · Stefan Hajnoczi
Quoted 153 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 90 lines · Jason Wang
Quoted 43 lines · Stefan Hajnoczi
Quoted 42 lines · Jason Wang

在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at 07:23:33PM +0200, Eugenio Perez Martin wrote:
If I understand correctly, this is all
driven from the driver inside the guest, so for this to work
the guest must be running and already have initialised the driver.
Yes.

As I see it, the feature can be driven entirely by the VMM as long as
it intercept the relevant configuration space (PCI, MMIO, etc) from
guest's reads and writes, and present it as coherent and transparent
for the guest. Some use cases I can imagine with a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device.
2.1) The VMM stops the device to perform a live migration, and the
guest does not write to STOP in any moment of the LM. It resets the
destination device with the state, and then initializes the device.
2.2) The guest stops the device and, when STOP(32) is set, the source
VMM migrates the device status. The destination VMM realizes the bit,
so it sets the bit in the destination too after device initialization.
2.3) The device is not initialized by the guest so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's all a lot of hoops to jump through though.
It's also not easy for devices to implement.
It just requires a new status bit. Anything that makes you think it's hard
to implement?

E.g for networking device, it should be sufficient to use this bit + the
virtqueue state.

Why don't we design the feature in a way that is useable by VMMs
and implementable by devices in a simple way?
It use the common technology like register shadowing without any further
stuffs.

Quoted 7 lines
Quoted 4 lines

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through
those state registers).

If an admin virtqueue is used instead of the STOP Device Status field bit then there's no need to re-read the Device Status field in a loop until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach extends downtime if implemented sequentially (stopping one device at a time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

Quoted 7 lines

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

My virtiofs device implementation is backed by an in-memory file system. The device state includes the contents of each file.

Your virtiofs device implementation uses Linux file handles to keep track of open files. The device state includes Linux file handles (but not the contents of each file) so the destination host can access the same files on shared storage.

Quoted 6 lines

Cornelia's virtiofs device implementation is backed by an object storage HTTP API. The device state includes API object IDs.

The device state is implementation-dependent. There is no standard representation and it's not possible to migrate between device implementations. How are they supposed to migrate?

Quoted 4 lines

So if I understand correclty, virtio-fs is not desigined to be migrate-able?

(Having a check on the current virtio-fs support in qemu, it looks to me it has a migration blocker).

The code does not support live migration but it's on the roadmap. Max Reitz added Linux file handle support to virtiofsd. That was the first step towards being able to migrate the device's state.

Quoted 11 lines
Quoted 2 lines

This is why I think it's necessarily to allow implementation-specific device state representations.

Or you probably mean you don't support cross backend migration. This sounds like a drawback and it's actually not a standard device but a vendor/implementation specific device.

It would bring a lot of troubles, not only for the implementation but for the management. Maybe we can start from adding the support of migration for some specific backend and start from there.

Yes, it's complicated. Some implementations could be compatible, but others can never be compatible because they have completely different state.

The virtiofsd implementation is the main one for virtiofs and the device state representation can be published, even standardized. Others can implement it to achieve migration compatibility.

But it must be possible for implementations that have completely different state to migrate too. virtiofsd isn't special.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

1) why do we still offer opaque migration state to userspace?

See above. Stateful devices may require an implementation-defined device state representation.

So my point stand still, it's not a standard device if we do this.

These "non-standard devices" still need to be able to migrate. How should we do that?

Quoted 11 lines
Quoted 4 lines
Quoted 2 lines

2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

Opaque data like D-Bus VMState:
https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html

Actually, I meant how to keep the opaque state which is compatible with all the existing device that can do migration.

E.g we want to live migration virtio-blk among any backends (from a hardware device to a software backend).

There was a series of email threads last year where migration compatibility was discussed:

https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg02620.html

I proposed an algorithm for checking migration compatibility between devices. The source and destination device can have different implementations (e.g. hardware, software, etc).

It involves picking an identifier like virtio-spec.org/pci/virtio-net for the device state representation and device parameters for aspects of the device that vary between instances (e.g. tso=on|off).

It's more complex than today's live migration approach in libvirt and QEMU. Today libvirt configures the source and destination in a compatible manner (thanks to knowledge of the device implementation) and then QEMU transfers the device state.

Part of the point of defining a migration compatibility algorithm is that it's possible to lift the assumptions out of libvirt so that arbitrary device implementations can be supported (hardware, software, etc) without putting knowledge about every device/VMM implementation into libvirt.

(The other advantage is that this allows orchestration software to determine migration compatibility before starting a migration.)

Quoted 32 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

Quoted 6 lines
Quoted 2 lines

So I think it still depend on the driver to implement migrate state which is vendor specific.

The current VFIO migration interface depends on a device-specific software mdev driver but here I'm showing that the physical device can implement the migration interface so that no device-specific driver code is needed.

This is not what I read from the patch:

 * device_state: (read/write)
 *      - The user application writes to this field to inform the vendor driver
 *        about the device state to be transitioned to.
 *      - The vendor driver should take the necessary actions to change the *        device state. After successful transition to a given state, the *        vendor driver should return success on write(device_state, state) *        system call. If the device state transition fails, the vendor driver
 *        should return an appropriate -errno for the fault condition.

Quoted 1 line

Vendor driver need to mediate between the uAPI and the actual device.

Yes, that's the current state of VFIO migration. If a hardware interface (e.g. PCI Capability) is defined that maps to this API then no device-specific drivers would be necessary because core VFIO PCI code can implement the uAPI by talking to the hardware.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

This brings troubles in the nested environment.

Quoted 8 lines
Quoted 4 lines

It depends on the device splitting/management design. If L0 wishes to let L1 manage the VFs then it would need to expose a management device. Since the migration interface is generic (not device-specific) a generic management device solves this for all devices.

Right, but it's a burden to expose the management device or it may just won't work.

A single generic management device is not a huge burden and it may turn out that keeping the management device out-of-band is actually a desirable feature if the device owner does not wish to expose the stop/save/load functionality for some reason.

Stefan

author
#85 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 221 lines · Stefan Hajnoczi
Quoted 220 lines · Jason Wang
Quoted 218 lines · Stefan Hajnoczi
Quoted 196 lines · Jason Wang
Quoted 181 lines · Stefan Hajnoczi
Quoted 180 lines · Jason Wang
Quoted 179 lines · Stefan Hajnoczi
Quoted 155 lines · Jason Wang
Quoted 154 lines · Stefan Hajnoczi
Quoted 153 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 90 lines · Jason Wang
Quoted 50 lines · Stefan Hajnoczi

On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at 07:23:33PM +0200, Eugenio Perez Martin wrote:
If I understand correctly, this is all
driven from the driver inside the guest, so for this to work
the guest must be running and already have initialised the driver.
Yes.

As I see it, the feature can be driven entirely by the VMM as long as
it intercept the relevant configuration space (PCI, MMIO, etc) from
guest's reads and writes, and present it as coherent and transparent
for the guest. Some use cases I can imagine with a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device.
2.1) The VMM stops the device to perform a live migration, and the
guest does not write to STOP in any moment of the LM. It resets the
destination device with the state, and then initializes the device.
2.2) The guest stops the device and, when STOP(32) is set, the source
VMM migrates the device status. The destination VMM realizes the bit,
so it sets the bit in the destination too after device initialization.
2.3) The device is not initialized by the guest so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's all a lot of hoops to jump through though.
It's also not easy for devices to implement.
It just requires a new status bit. Anything that makes you think it's hard
to implement?

E.g for networking device, it should be sufficient to use this bit + the
virtqueue state.

Why don't we design the feature in a way that is useable by VMMs
and implementable by devices in a simple way?
It use the common technology like register shadowing without any further
stuffs.

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through
those state registers).
If an admin virtqueue is used instead of the STOP Device Status field
bit then there's no need to re-read the Device Status field in a loop
until the device has stopped.

Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin virtqueue could be used for carrying any basic device facility like status bit. E.g I'm going to post patches that use admin virtqueue as a "transport" for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function interface for this. This is a must for nested virtualization, we can't always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all the consumed buffers, device can choose to expose those inflight descriptors in a device specific way and set the STOP bit. This means, if we have the device specific in-flight descriptor reporting facility, the device can almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin virtqueue specific method, we still need to clarify how it works with the device status state machine, it will be some kind of sub-states which looks much more complicated than the current proposal.

Quoted 3 lines

When migrating a guest with many VIRTIO devices a busy waiting approach
extends downtime if implemented sequentially (stopping one device at a
time).

Well. You need some kinds of waiting for sure, the device/DMA needs sometime to be stopped. The downtime is determined by a specific virtio implementation which is hard to be restricted at the spec level. We can clarify that the device must set the STOP bit in e.g 100ms.

Quoted 3 lines

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the
bit set), but this becomes more complex to implement.

I still don't get what kind of complexity did you worry here.

Quoted 2 lines

I'm a little worried about adding a new bit that requires busy
waiting...

Busy wait is not something that is introduced in this patch:

Quoted 7 lines

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

Quoted 7 lines

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

My virtiofs device implementation is backed by an in-memory file system. The device state includes the contents of each file.

Your virtiofs device implementation uses Linux file handles to keep track of open files. The device state includes Linux file handles (but not the contents of each file) so the destination host can access the same files on shared storage.

Cornelia's virtiofs device implementation is backed by an object storage HTTP API. The device state includes API object IDs.

The device state is implementation-dependent. There is no standard representation and it's not possible to migrate between device implementations. How are they supposed to migrate?

Quoted 7 lines
Quoted 4 lines

So if I understand correclty, virtio-fs is not desigined to be migrate-able?

(Having a check on the current virtio-fs support in qemu, it looks to me it has a migration blocker).

The code does not support live migration but it's on the roadmap. Max Reitz added Linux file handle support to virtiofsd. That was the first step towards being able to migrate the device's state.

A dumb question, how do qemu know it is connected to virtiofsd?

Quoted 35 lines
Quoted 11 lines
Quoted 3 lines

This is why I think it's necessarily to allow implementation-specific device state representations.

Or you probably mean you don't support cross backend migration. This sounds like a drawback and it's actually not a standard device but a vendor/implementation specific device.

It would bring a lot of troubles, not only for the implementation but for the management. Maybe we can start from adding the support of migration for some specific backend and start from there.

Yes, it's complicated. Some implementations could be compatible, but others can never be compatible because they have completely different state.

The virtiofsd implementation is the main one for virtiofs and the device state representation can be published, even standardized. Others can implement it to achieve migration compatibility.

But it must be possible for implementations that have completely different state to migrate too. virtiofsd isn't special.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

1) why do we still offer opaque migration state to userspace?

See above. Stateful devices may require an implementation-defined device state representation.

So my point stand still, it's not a standard device if we do this.

These "non-standard devices" still need to be able to migrate.

See other thread, it breaks the effort of having a spec.

Quoted 2 lines

How
should we do that?

I think the main issue is that, to me it's not a virtio device but a device that is using virtio queue with implementation specific state. So it can't be migrated by the virtio subsystem but through a vendor/implementation specific migration driver.

Quoted 29 lines
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines

2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

Opaque data like D-Bus VMState:
https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html

Actually, I meant how to keep the opaque state which is compatible with all the existing device that can do migration.

E.g we want to live migration virtio-blk among any backends (from a hardware device to a software backend).

There was a series of email threads last year where migration compatibility was discussed:

https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg02620.html

I proposed an algorithm for checking migration compatibility between devices. The source and destination device can have different implementations (e.g. hardware, software, etc).

It involves picking an identifier like virtio-spec.org/pci/virtio-net for the device state representation and device parameters for aspects of the device that vary between instances (e.g. tso=on|off).

It's more complex than today's live migration approach in libvirt and QEMU. Today libvirt configures the source and destination in a compatible manner (thanks to knowledge of the device implementation) and then QEMU transfers the device state.

Quoted 8 lines

Part of the point of defining a migration compatibility algorithm is that it's possible to lift the assumptions out of libvirt so that arbitrary device implementations can be supported (hardware, software, etc) without putting knowledge about every device/VMM implementation into libvirt.

(The other advantage is that this allows orchestration software to determine migration compatibility before starting a migration.)

This looks like another independent issues and I fully agree to have a better migration protocol. But using that means we break the migration compatibility with the existing device which is used for more than 10 years. We still need to make the migration from/to the existing virtio device to work.

Quoted 31 lines
Quoted 31 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

Quoted 6 lines
Quoted 2 lines

So I think it still depend on the driver to implement migrate state which is vendor specific.

The current VFIO migration interface depends on a device-specific software mdev driver but here I'm showing that the physical device can implement the migration interface so that no device-specific driver code is needed.

This is not what I read from the patch:

 * device_state: (read/write)
 *      - The user application writes to this field to inform the vendor
driver
 *        about the device state to be transitioned to.
 *      - The vendor driver should take the necessary actions to change the *        device state. After successful transition to a given state, the *        vendor driver should return success on write(device_state, state) *        system call. If the device state transition fails, the vendor
driver
 *        should return an appropriate -errno for the fault condition.

Quoted 5 lines
Quoted 1 line

Vendor driver need to mediate between the uAPI and the actual device.

Yes, that's the current state of VFIO migration. If a hardware interface (e.g. PCI Capability) is defined that maps to this API then no device-specific drivers would be necessary because core VFIO PCI code can implement the uAPI by talking to the hardware.

As we discussed, it would be very hard. The device state is implementation specific which may not fit for the Capability. (PCIE has already had VF migration state in the SR-IOV extended capability).

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 7 lines
Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

This brings troubles in the nested environment.

It depends on the device splitting/management design. If L0 wishes to let L1 manage the VFs then it would need to expose a management device. Since the migration interface is generic (not device-specific) a generic management device solves this for all devices.

Quoted 6 lines
Quoted 2 lines

Right, but it's a burden to expose the management device or it may just won't work.

A single generic management device is not a huge burden and it may turn out that keeping the management device out-of-band is actually a desirable feature if the device owner does not wish to expose the stop/save/load functionality for some reason.

VMM are free to hide those features from guest. Management can just do -device virtio-pci,state=false

Having management device works for L0 but not suitable for L(x>0). A per function device interface is a must for the nested virt to work in a simple and easy way.

Thanks

Quoted 2 lines

Stefan

#86 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 225 lines · Jason Wang
Quoted 223 lines · Stefan Hajnoczi
Quoted 222 lines · Jason Wang
Quoted 212 lines · Stefan Hajnoczi
Quoted 196 lines · Jason Wang
Quoted 181 lines · Stefan Hajnoczi
Quoted 180 lines · Jason Wang
Quoted 179 lines · Stefan Hajnoczi
Quoted 155 lines · Jason Wang
Quoted 154 lines · Stefan Hajnoczi
Quoted 153 lines · Jason Wang
Quoted 101 lines · Stefan Hajnoczi
Quoted 97 lines · Jason Wang

在 2021/7/12 下午5:57, Stefan Hajnoczi 写道:
On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at 07:23:33PM +0200, Eugenio Perez Martin wrote:
If I understand correctly, this is all
driven from the driver inside the guest, so for this to work
the guest must be running and already have initialised the driver.
Yes.

As I see it, the feature can be driven entirely by the VMM as long as
it intercept the relevant configuration space (PCI, MMIO, etc) from
guest's reads and writes, and present it as coherent and transparent
for the guest. Some use cases I can imagine with a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device.
2.1) The VMM stops the device to perform a live migration, and the
guest does not write to STOP in any moment of the LM. It resets the
destination device with the state, and then initializes the device.
2.2) The guest stops the device and, when STOP(32) is set, the source
VMM migrates the device status. The destination VMM realizes the bit,
so it sets the bit in the destination too after device initialization.
2.3) The device is not initialized by the guest so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's all a lot of hoops to jump through though.
It's also not easy for devices to implement.
It just requires a new status bit. Anything that makes you think it's hard
to implement?

E.g for networking device, it should be sufficient to use this bit + the
virtqueue state.

Why don't we design the feature in a way that is useable by VMMs
and implementable by devices in a simple way?
It use the common technology like register shadowing without any further
stuffs.

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through
those state registers).
If an admin virtqueue is used instead of the STOP Device Status field
bit then there's no need to re-read the Device Status field in a loop
until the device has stopped.
Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin
virtqueue could be used for carrying any basic device facility like status
bit. E.g I'm going to post patches that use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function
interface for this. This is a must for nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all
the consumed buffers, device can choose to expose those inflight descriptors
in a device specific way and set the STOP bit. This means, if we have the
device specific in-flight descriptor reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin
virtqueue specific method, we still need to clarify how it works with the
device status state machine, it will be some kind of sub-states which looks
much more complicated than the current proposal.

When migrating a guest with many VIRTIO devices a busy waiting approach
extends downtime if implemented sequentially (stopping one device at a
time).
Well. You need some kinds of waiting for sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio
implementation which is hard to be restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the
bit set), but this becomes more complex to implement.
I still don't get what kind of complexity did you worry here.

I'm a little worried about adding a new bit that requires busy
waiting...
Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something
similar to when introducing basic facility.

Adding the STOP but as a Device Status bit is a small and clean VIRTIO spec change. I like that.

Quoted 7 lines

On the other hand, devices need time to stop and that time can be unbounded. For example, software virtio-blk/scsi implementations since cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests. That's why I mentioned the alternative of using an admin virtqueue instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in the Device Status field somehow, I'm not sure about that since the driver typically doesn't need to know whether the device is being migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of data. They need DMA to do this efficiently, so an admin virtqueue is a good fit again.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

Quoted 7 lines

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

My virtiofs device implementation is backed by an in-memory file system. The device state includes the contents of each file.

Your virtiofs device implementation uses Linux file handles to keep track of open files. The device state includes Linux file handles (but not the contents of each file) so the destination host can access the same files on shared storage.

Quoted 6 lines

Cornelia's virtiofs device implementation is backed by an object storage HTTP API. The device state includes API object IDs.

The device state is implementation-dependent. There is no standard representation and it's not possible to migrate between device implementations. How are they supposed to migrate?

So if I understand correclty, virtio-fs is not desigined to be migrate-able?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

(Having a check on the current virtio-fs support in qemu, it looks to me it has a migration blocker).

The code does not support live migration but it's on the roadmap. Max Reitz added Linux file handle support to virtiofsd. That was the first step towards being able to migrate the device's state.

A dumb question, how do qemu know it is connected to virtiofsd?

virtiofsd is a vhost-user-fs device. QEMU doesn't know if it's connected to virtiofsd or another implementation.

Quoted 44 lines
Quoted 34 lines
Quoted 10 lines
Quoted 2 lines

This is why I think it's necessarily to allow implementation-specific device state representations.

Or you probably mean you don't support cross backend migration. This sounds like a drawback and it's actually not a standard device but a vendor/implementation specific device.

It would bring a lot of troubles, not only for the implementation but for the management. Maybe we can start from adding the support of migration for some specific backend and start from there.

Yes, it's complicated. Some implementations could be compatible, but others can never be compatible because they have completely different state.

The virtiofsd implementation is the main one for virtiofs and the device state representation can be published, even standardized. Others can implement it to achieve migration compatibility.

But it must be possible for implementations that have completely different state to migrate too. virtiofsd isn't special.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

1) why do we still offer opaque migration state to userspace?

See above. Stateful devices may require an implementation-defined device state representation.

So my point stand still, it's not a standard device if we do this.

These "non-standard devices" still need to be able to migrate.

See other thread, it breaks the effort of having a spec.

Quoted 8 lines
Quoted 2 lines

How
should we do that?

I think the main issue is that, to me it's not a virtio device but a device that is using virtio queue with implementation specific state. So it can't be migrated by the virtio subsystem but through a vendor/implementation specific migration driver.

Okay. Are you thinking about a separate set of vDPA APIs and vhost ioctls so the VMM can save/load implementation-specific device state? These separate APIs just need to be called as part of the standard VIRTIO stop and vq save/load lifecycle.

Quoted 39 lines
Quoted 29 lines
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines

2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

Opaque data like D-Bus VMState:
https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html

Actually, I meant how to keep the opaque state which is compatible with all the existing device that can do migration.

E.g we want to live migration virtio-blk among any backends (from a hardware device to a software backend).

There was a series of email threads last year where migration compatibility was discussed:

https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg02620.html

I proposed an algorithm for checking migration compatibility between devices. The source and destination device can have different implementations (e.g. hardware, software, etc).

It involves picking an identifier like virtio-spec.org/pci/virtio-net for the device state representation and device parameters for aspects of the device that vary between instances (e.g. tso=on|off).

It's more complex than today's live migration approach in libvirt and QEMU. Today libvirt configures the source and destination in a compatible manner (thanks to knowledge of the device implementation) and then QEMU transfers the device state.

Quoted 8 lines

Part of the point of defining a migration compatibility algorithm is that it's possible to lift the assumptions out of libvirt so that arbitrary device implementations can be supported (hardware, software, etc) without putting knowledge about every device/VMM implementation into libvirt.

(The other advantage is that this allows orchestration software to determine migration compatibility before starting a migration.)

Quoted 5 lines

This looks like another independent issues and I fully agree to have a better migration protocol. But using that means we break the migration compatibility with the existing device which is used for more than 10 years. We still need to make the migration from/to the existing virtio device to work.

I agree that migrating to/from existing devices needs to work. It should be possible to transition without breaking migration.

Quoted 62 lines
Quoted 30 lines
Quoted 30 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

Quoted 6 lines
Quoted 2 lines

So I think it still depend on the driver to implement migrate state which is vendor specific.

The current VFIO migration interface depends on a device-specific software mdev driver but here I'm showing that the physical device can implement the migration interface so that no device-specific driver code is needed.

This is not what I read from the patch:

 * device_state: (read/write)
 *      - The user application writes to this field to inform the vendor
driver
 *        about the device state to be transitioned to.
 *      - The vendor driver should take the necessary actions to change the *        device state. After successful transition to a given state, the *        vendor driver should return success on write(device_state, state) *        system call. If the device state transition fails, the vendor
driver
 *        should return an appropriate -errno for the fault condition.

Quoted 5 lines
Quoted 1 line

Vendor driver need to mediate between the uAPI and the actual device.

Yes, that's the current state of VFIO migration. If a hardware interface (e.g. PCI Capability) is defined that maps to this API then no device-specific drivers would be necessary because core VFIO PCI code can implement the uAPI by talking to the hardware.

As we discussed, it would be very hard. The device state is implementation specific which may not fit for the Capability. (PCIE has already had VF migration state in the SR-IOV extended capability).

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 7 lines
Quoted 6 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

This brings troubles in the nested environment.

It depends on the device splitting/management design. If L0 wishes to let L1 manage the VFs then it would need to expose a management device. Since the migration interface is generic (not device-specific) a generic management device solves this for all devices.

Quoted 6 lines
Quoted 2 lines

Right, but it's a burden to expose the management device or it may just won't work.

A single generic management device is not a huge burden and it may turn out that keeping the management device out-of-band is actually a desirable feature if the device owner does not wish to expose the stop/save/load functionality for some reason.

Quoted 6 lines

VMM are free to hide those features from guest. Management can just do -device virtio-pci,state=false

Having management device works for L0 but not suitable for L(x>0). A per function device interface is a must for the nested virt to work in a simple and easy way.

You are right, a per function interface is simplest. I'm not experienced enough with SR-IOV and nested virtualization to have a strong opinion in this area.

Stefan

author
#87 lore raw

Re: [virtio-comment] [PATCH V2 2/2] virtio: introduce STOP status bit

Quoted 226 lines · Stefan Hajnoczi
Quoted 225 lines · Jason Wang
Quoted 224 lines · Stefan Hajnoczi
Quoted 223 lines · Jason Wang
Quoted 212 lines · Stefan Hajnoczi
Quoted 196 lines · Jason Wang
Quoted 181 lines · Stefan Hajnoczi
Quoted 180 lines · Jason Wang
Quoted 179 lines · Stefan Hajnoczi
Quoted 155 lines · Jason Wang
Quoted 154 lines · Stefan Hajnoczi
Quoted 153 lines · Jason Wang
Quoted 108 lines · Stefan Hajnoczi

On Tue, Jul 13, 2021 at 11:27:03AM +0800, Jason Wang wrote:
在 2021/7/12 下午5:57, Stefan Hajnoczi 写道:
On Mon, Jul 12, 2021 at 12:00:39PM +0800, Jason Wang wrote:
在 2021/7/11 上午4:36, Michael S. Tsirkin 写道:
On Fri, Jul 09, 2021 at 07:23:33PM +0200, Eugenio Perez Martin wrote:
If I understand correctly, this is all
driven from the driver inside the guest, so for this to work
the guest must be running and already have initialised the driver.
Yes.

As I see it, the feature can be driven entirely by the VMM as long as
it intercept the relevant configuration space (PCI, MMIO, etc) from
guest's reads and writes, and present it as coherent and transparent
for the guest. Some use cases I can imagine with a physical device (or
vp_vpda device) with VIRTIO_F_STOP:

1) The VMM chooses not to pass the feature flag. The guest cannot stop
the device, so any write to this flag is an error/undefined.
2) The VMM passes the flag to the guest. The guest can stop the device.
2.1) The VMM stops the device to perform a live migration, and the
guest does not write to STOP in any moment of the LM. It resets the
destination device with the state, and then initializes the device.
2.2) The guest stops the device and, when STOP(32) is set, the source
VMM migrates the device status. The destination VMM realizes the bit,
so it sets the bit in the destination too after device initialization.
2.3) The device is not initialized by the guest so it doesn't matter
what bit has the HW, but the VM can be migrated.

Am I missing something?

Thanks!
It's doable like this. It's all a lot of hoops to jump through though.
It's also not easy for devices to implement.
It just requires a new status bit. Anything that makes you think it's hard
to implement?

E.g for networking device, it should be sufficient to use this bit + the
virtqueue state.

Why don't we design the feature in a way that is useable by VMMs
and implementable by devices in a simple way?
It use the common technology like register shadowing without any further
stuffs.

Or do you have any other ideas?

(I think we all know migration will be very hard if we simply pass through
those state registers).
If an admin virtqueue is used instead of the STOP Device Status field
bit then there's no need to re-read the Device Status field in a loop
until the device has stopped.
Probably not. Let me to clarify several points:

- This proposal has nothing to do with admin virtqueue. Actually, admin
virtqueue could be used for carrying any basic device facility like status
bit. E.g I'm going to post patches that use admin virtqueue as a "transport"
for device slicing at virtio level.
- Even if we had introduced admin virtqueue, we still need a per function
interface for this. This is a must for nested virtualization, we can't
always expect things like PF can be assigned to L1 guest.
- According to the proposal, there's no need for the device to complete all
the consumed buffers, device can choose to expose those inflight descriptors
in a device specific way and set the STOP bit. This means, if we have the
device specific in-flight descriptor reporting facility, the device can
almost set the STOP bit immediately.
- If we don't go with the basic device facility but using the admin
virtqueue specific method, we still need to clarify how it works with the
device status state machine, it will be some kind of sub-states which looks
much more complicated than the current proposal.

When migrating a guest with many VIRTIO devices a busy waiting approach
extends downtime if implemented sequentially (stopping one device at a
time).
Well. You need some kinds of waiting for sure, the device/DMA needs sometime
to be stopped. The downtime is determined by a specific virtio
implementation which is hard to be restricted at the spec level. We can
clarify that the device must set the STOP bit in e.g 100ms.

It can be implemented concurrently (setting the STOP bit on all
devices and then looping until all their Device Status fields have the
bit set), but this becomes more complex to implement.
I still don't get what kind of complexity did you worry here.

I'm a little worried about adding a new bit that requires busy
waiting...
Busy wait is not something that is introduced in this patch:

4.1.4.3.2 Driver Requirements: Common configuration structure layout

After writing 0 to device_status, the driver MUST wait for a read of
device_status to return 0 before reinitializing the device.

Since it was required for at least one transport. We need do something
similar to when introducing basic facility.
Adding the STOP but as a Device Status bit is a small and clean VIRTIO
spec change. I like that.

On the other hand, devices need time to stop and that time can be
unbounded. For example, software virtio-blk/scsi implementations since
cannot immediately cancel in-flight I/O requests on Linux hosts.

The natural interface for long-running operations is virtqueue requests.
That's why I mentioned the alternative of using an admin virtqueue
instead of a Device Status bit.

So I'm not against the admin virtqueue. As said before, admin virtqueue could be used for carrying the device status bit.

Send a command to set STOP status bit to admin virtqueue. Device will make the command buffer used after it has successfully stopped the device.

AFAIK, they are not mutually exclusive, since they are trying to solve different problems.

Device status - basic device facility

Admin virtqueue - transport/device specific way to implement (part of) the device facility

Quoted 4 lines

Although you mentioned that the stopped state needs to be reflected in
the Device Status field somehow, I'm not sure about that since the
driver typically doesn't need to know whether the device is being
migrated.

The guest won't see the real device status bit. VMM will shadow the device status bit in this case.

E.g with the current vhost-vDPA, vDPA behave like a vhost device, guest is unaware of the migration.

STOP status bit is set by Qemu to real virtio hardware. But guest will only see the DRIVER_OK without STOP.

It's not hard to implement the nested on top, see the discussion initiated by Eugenio about how expose VIRTIO_F_STOP to guest for nested live migration.

Quoted 2 lines

In fact, the VMM would need to hide this bit and it's safer to
keep it out-of-band instead of risking exposing it by accident.

See above, VMM may choose to hide or expose the capability. It's useful for migrating a nested guest.

If we design an interface that can be used in the nested environment, it's not an ideal interface.

Quoted 3 lines

In addition, stateful devices need to load/save non-trivial amounts of
data. They need DMA to do this efficiently, so an admin virtqueue is a
good fit again.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I don't get the point here. You still need to address the exact the similar issues for admin virtqueue: the unbound time in freezing the device, the interaction with the virtio device status state machine.

Device state state can be large so a register interface would be a bottleneck. DMA is needed. I think a virtqueue is a good fit for saving/loading device state.

So this patch doesn't mandate a register interface, isn't it?

You're right, not this patch. I mentioned it because your other patch series ("[PATCH] virtio-pci: implement VIRTIO_F_RING_STATE") implements it a register interface.

Quoted 2 lines

And DMA
doesn't means a virtqueue, it could be a transport specific method.

Yes, although virtqueues are a pretty good interface that works across transports (PCI/MMIO/etc) thanks to the standard vring memory layout.

Quoted 2 lines

I think we need to start from defining the state of one specific device and see what is the best interface.

virtio-blk might be the simplest. I think virtio-net has more device state and virtio-scsi is definitely more complext than virtio-blk.

First we need agreement on whether "device state" encompasses the full state of the device or just state that is unknown to the VMM.

Quoted 7 lines
Quoted 4 lines

I think we've discussed this in the past. It can't work since:

1) The state and its format must be clearly defined in the spec
2) We need to maintain migration compatibility and debug-ability

Some devices need implementation-specific state. They should still be able to live migrate even if it means cross-implementation migration and debug-ability is not possible.

I think we need to re-visit this conclusion. Migration compatibility is pretty important, especially consider the software stack has spent a huge mount of effort in maintaining them.

Say a virtio hardware would break this, this mean we will lose all the advantages of being a standard device.

Quoted 7 lines

If we can't do live migration among:

1) different backends, e.g migrate from virtio hardware to migrate software
2) different vendors

We failed to say as a standard device and the customer is in fact locked by the vendor implicitly.

My virtiofs device implementation is backed by an in-memory file system. The device state includes the contents of each file.

Your virtiofs device implementation uses Linux file handles to keep track of open files. The device state includes Linux file handles (but not the contents of each file) so the destination host can access the same files on shared storage.

Quoted 6 lines

Cornelia's virtiofs device implementation is backed by an object storage HTTP API. The device state includes API object IDs.

The device state is implementation-dependent. There is no standard representation and it's not possible to migrate between device implementations. How are they supposed to migrate?

So if I understand correclty, virtio-fs is not desigined to be migrate-able?

(Having a check on the current virtio-fs support in qemu, it looks to me it has a migration blocker).

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

The code does not support live migration but it's on the roadmap. Max Reitz added Linux file handle support to virtiofsd. That was the first step towards being able to migrate the device's state.

A dumb question, how do qemu know it is connected to virtiofsd?

virtiofsd is a vhost-user-fs device. QEMU doesn't know if it's connected to virtiofsd or another implementation.

That's my understanding. So this answers my questions basically: there could be a common migration implementation for each virtio-fs device which implies that we only need to migrate the common device specific state but not implementation specific state.

Quoted 45 lines
Quoted 45 lines
Quoted 34 lines
Quoted 10 lines
Quoted 3 lines

This is why I think it's necessarily to allow implementation-specific device state representations.

Or you probably mean you don't support cross backend migration. This sounds like a drawback and it's actually not a standard device but a vendor/implementation specific device.

It would bring a lot of troubles, not only for the implementation but for the management. Maybe we can start from adding the support of migration for some specific backend and start from there.

Yes, it's complicated. Some implementations could be compatible, but others can never be compatible because they have completely different state.

The virtiofsd implementation is the main one for virtiofs and the device state representation can be published, even standardized. Others can implement it to achieve migration compatibility.

But it must be possible for implementations that have completely different state to migrate too. virtiofsd isn't special.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Quoted 1 line

3) Not a proper uAPI desgin

I never understood this argument. The Linux uAPI passes through lots of opaque data from devices to userspace. Allowing an implementation-specific device state representation is nothing new. VFIO already does it.

I think we've already had a lots of discussion for VFIO but without a conclusion. Maybe we need the verdict from Linus or Greg (not sure if it's too late). But that's not related to virito and this thread.

What you propose here is kind of conflict with the efforts of virtio. I think we all aggree that we should define the state in the spec. Assuming this is correct:

Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

1) why do we still offer opaque migration state to userspace?

See above. Stateful devices may require an implementation-defined device state representation.

So my point stand still, it's not a standard device if we do this.

These "non-standard devices" still need to be able to migrate.

See other thread, it breaks the effort of having a spec.

Quoted 2 lines

How
should we do that?

Quoted 6 lines
Quoted 4 lines

I think the main issue is that, to me it's not a virtio device but a device that is using virtio queue with implementation specific state. So it can't be migrated by the virtio subsystem but through a vendor/implementation specific migration driver.

Okay. Are you thinking about a separate set of vDPA APIs and vhost ioctls so the VMM can save/load implementation-specific device state?

Probably not, I think the question is can we define the virtio-fs device state in the spec? If yes (and I think the answer is yes), we're fine. If not, it looks like we need to improve the spec or design.

Quoted 2 lines

These separate APIs just need to be called as part of the standard VIRTIO stop and vq save/load lifecycle.

Yes, if they are virtio standard, we need to invent them.

Quoted 105 lines
Quoted 34 lines
Quoted 34 lines
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

2) how can it be integrated into the current VMM (Qemu) virtio devices' migration bytes stream?

Opaque data like D-Bus VMState:
https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html

Actually, I meant how to keep the opaque state which is compatible with all the existing device that can do migration.

E.g we want to live migration virtio-blk among any backends (from a hardware device to a software backend).

There was a series of email threads last year where migration compatibility was discussed:

https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg02620.html

I proposed an algorithm for checking migration compatibility between devices. The source and destination device can have different implementations (e.g. hardware, software, etc).

It involves picking an identifier like virtio-spec.org/pci/virtio-net for the device state representation and device parameters for aspects of the device that vary between instances (e.g. tso=on|off).

It's more complex than today's live migration approach in libvirt and QEMU. Today libvirt configures the source and destination in a compatible manner (thanks to knowledge of the device implementation) and then QEMU transfers the device state.

Part of the point of defining a migration compatibility algorithm is that it's possible to lift the assumptions out of libvirt so that arbitrary device implementations can be supported (hardware, software, etc) without putting knowledge about every device/VMM implementation into libvirt.

Quoted 8 lines
Quoted 2 lines

(The other advantage is that this allows orchestration software to determine migration compatibility before starting a migration.)

This looks like another independent issues and I fully agree to have a better migration protocol. But using that means we break the migration compatibility with the existing device which is used for more than 10 years. We still need to make the migration from/to the existing virtio device to work.

I agree that migrating to/from existing devices needs to work. It should be possible to transition without breaking migration.

Quoted 60 lines
Quoted 29 lines
Quoted 29 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

That's
basically the difference between the vhost/vDPA's selective passthrough approach and VFIO's full passthrough approach.

We can't do VFIO full pasthrough for migration anyway, some kind of mdev is required but it's duplicated with the current vp_vdpa driver.

I'm not sure that's true. Generic VFIO PCI migration can probably be achieved without mdev:
1. Define a migration PCI Capability that indicates support for VFIO_REGION_TYPE_MIGRATION. This allows the PCI device to implement the migration interface in hardware instead of an mdev driver.

Quoted 6 lines
Quoted 2 lines

So I think it still depend on the driver to implement migrate state which is vendor specific.

The current VFIO migration interface depends on a device-specific software mdev driver but here I'm showing that the physical device can implement the migration interface so that no device-specific driver code is needed.

This is not what I read from the patch:

 * device_state: (read/write)
 *      - The user application writes to this field to inform the vendor
driver
 *        about the device state to be transitioned to.
 *      - The vendor driver should take the necessary actions to change the *        device state. After successful transition to a given state, the *        vendor driver should return success on write(device_state, state) *        system call. If the device state transition fails, the vendor
driver
 *        should return an appropriate -errno for the fault condition.

Quoted 5 lines
Quoted 1 line

Vendor driver need to mediate between the uAPI and the actual device.

Yes, that's the current state of VFIO migration. If a hardware interface (e.g. PCI Capability) is defined that maps to this API then no device-specific drivers would be necessary because core VFIO PCI code can implement the uAPI by talking to the hardware.

As we discussed, it would be very hard. The device state is implementation specific which may not fit for the Capability. (PCIE has already had VF migration state in the SR-IOV extended capability).

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Quoted 5 lines

2. The VMM either uses the migration PCI Capability directly from userspace or core VFIO PCI code advertises VFIO_REGION_TYPE_MIGRATION to userspace so migration can proceed in the same way as with VFIO/mdev drivers.
3. The PCI Capability is not passed through to the guest.

This brings troubles in the nested environment.

It depends on the device splitting/management design. If L0 wishes to let L1 manage the VFs then it would need to expose a management device. Since the migration interface is generic (not device-specific) a generic management device solves this for all devices.

Quoted 6 lines
Quoted 2 lines

Right, but it's a burden to expose the management device or it may just won't work.

A single generic management device is not a huge burden and it may turn out that keeping the management device out-of-band is actually a desirable feature if the device owner does not wish to expose the stop/save/load functionality for some reason.

VMM are free to hide those features from guest. Management can just do -device virtio-pci,state=false

Quoted 6 lines
Quoted 3 lines

Having management device works for L0 but not suitable for L(x>0). A per function device interface is a must for the nested virt to work in a simple and easy way.

You are right, a per function interface is simplest. I'm not experienced enough with SR-IOV and nested virtualization to have a strong opinion in this area.

Yes, and they can co-exist, the admin virtqueue works for L0, but we need hide those via per function API for nested.

That's why I start from proposing the basic facility instead of an actual transport (PCI or admin virtqueue) implementation.

Thanks

Quoted 2 lines

Stefan