Skip to contentSkip to conversation

virtio: introduce SUSPEND bit in device status

PATCH In review Zhu Lingshan 5 participants 69 messages Aug 1, 2024 – Oct 17, 2024

lore
Share & tools
lore
#0 lore raw

PATCH v7 1/1 virtio: introduce SUSPEND bit in device status

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

---
 content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

content.tex +65 -10
index 0a62dce..2d1bee8 100644
@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
     this bit.  For example, under Linux, drivers can be loadable modules.
   \end{note}
 
-\item[FAILED (128)] Indicates that something went wrong in the guest,
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason, or
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
+  drive the device.
 
 \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the
   features it understands, and feature negotiation is complete.
 
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
-  drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
   an error from which it can't recover.
+
+\item[FAILED (128)] Indicates that something went wrong in the guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason, or
+  even a fatal error during device operation.
 \end{description}
 
 The \field{device status} field starts out as 0, and is reinitialized to 0 by
@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
 initialization sequence specified in
 \ref{sec:General Initialization And Device Operation / Device
 Initialization}.
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other than SUSPEND
+except when setting \field{device status} to 0 as a transport-specific way to
+initiate a reset. If the driver sets the FAILED bit,
 the driver MUST later reset the device before attempting to re-initialize.
 
 The driver SHOULD NOT rely on completion of operations of a
@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
 
 Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
 
+\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the
+SUSPEND bit in \field{device status} to suspend a device, and can
+clear the SUSPEND bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.
+
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the device.
+\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues.
+\item The driver MUST NOT access Device Configuration Space.
+\end{itemize}
+
+\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the Configuration Space.
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in
+its Configuration Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit is set,
+the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification after the SUSPEND bit has been cleared.
+
+When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend.
+
+If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,
+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET.
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}:
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues
+\item Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split
@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.45.2
#1 lore raw

Hi Lingshan, David,

Quoted 122 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

---
 content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++-----
--
 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

content.tex +0 -0
index 0a62dce..2d1bee8 100644
@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic

Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +10 -5
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason, or
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
+  drive the device.
 
 \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the
   features it understands, and feature negotiation is complete.
 
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
-  drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that
+the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
   an error from which it can't recover.
+
+\item[FAILED (128)] Indicates that something went wrong in the guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason, or
+  even a fatal error during device operation.
 \end{description}
 
 The \field{device status} field starts out as 0, and is reinitialized to 0 by @@ -

60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}.

content.tex +5 -2
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other than
+SUSPEND except when setting \field{device status} to 0 as a
+transport-specific way to initiate a reset. If the driver sets the
+FAILED bit,
 the driver MUST later reset the device before attempting to re-initialize.
 
 The driver SHOULD NOT rely on completion of operations of a @@ -99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device

/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device type

-\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset) before
removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And Device

content.tex +10 -0
+Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the SUSPEND bit
+in \field{device status} to suspend a device, and can clear the SUSPEND
+bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General Initialization
+And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the
Quoted 7 lines

SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Do we agree that
a. suspending a device is non frequent operation (in order of N operations/sec, where N is roughly in range of 10 or 100) per device? b. A software-based device may not always want to force VM_EXIT on read and write on the device_status register?

Quoted 55 lines

+\devicenormative{\subsection}{Device Suspend}{General Initialization

content.tex +3 -0
+And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +3 -0
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the Configuration

Space.

content.tex +8 -0
+
+A device MUST NOT send any notifications for any virtqeuues, access any
+virtqueues, or modify any fields in its Configuration Space while
+suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit is
+set, the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification after the SUSPEND bit has

been cleared.

content.tex +2 -0
+
+When the driver sets SUSPEND, the device MUST either suspend itself or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver clears
+SUSPEND, the device MUST either resume normal operation or set

DEVICE_NEEDS_RESET.

content.tex +3 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +8 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item Wait until
+all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature

Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}
for

content.tex +4 -1
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
--

2.45.2

#2 lore raw
Quoted 123 lines · Parav Pandit

Hi Lingshan, David,

Quoted 121 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

---
 content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++-----
--
 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

content.tex +0 -0
index 0a62dce..2d1bee8 100644
@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic

Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +10 -5
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason, or
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
+  drive the device.
 
 \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the
   features it understands, and feature negotiation is complete.
 
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
-  drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that
+the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
   an error from which it can't recover.
+
+\item[FAILED (128)] Indicates that something went wrong in the guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason, or
+  even a fatal error during device operation.
 \end{description}
 
 The \field{device status} field starts out as 0, and is reinitialized to 0 by @@ -

60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}.

content.tex +5 -2
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other than
+SUSPEND except when setting \field{device status} to 0 as a
+transport-specific way to initiate a reset. If the driver sets the
+FAILED bit,
 the driver MUST later reset the device before attempting to re-initialize.
 
 The driver SHOULD NOT rely on completion of operations of a @@ -99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device

/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device type

-\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset) before
removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And Device

content.tex +10 -0
+Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the SUSPEND bit
+in \field{device status} to suspend a device, and can clear the SUSPEND
+bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General Initialization
+And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +3 -0
+
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
Quoted 8 lines
Quoted 8 lines

+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 2 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N operations/sec, where N is roughly in range of 10 or 100) per device?

Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.

Quoted 1 line

b. A software-based device may not always want to force VM_EXIT on read and write on the device_status register?

Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Thanks
Zhu Lingshan

Quoted 57 lines
Quoted 57 lines

+\devicenormative{\subsection}{Device Suspend}{General Initialization

content.tex +3 -0
+And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +3 -0
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the Configuration

Space.

content.tex +8 -0
+
+A device MUST NOT send any notifications for any virtqeuues, access any
+virtqueues, or modify any fields in its Configuration Space while
+suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit is
+set, the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification after the SUSPEND bit has

been cleared.

content.tex +2 -0
+
+When the driver sets SUSPEND, the device MUST either suspend itself or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver clears
+SUSPEND, the device MUST either resume normal operation or set

DEVICE_NEEDS_RESET.

content.tex +3 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +8 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item Wait until
+all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature

Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}
for

content.tex +4 -1
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
--

2.45.2

#3 lore raw
Quoted 155 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 17 lines · Parav Pandit

Hi Lingshan, David,

Quoted 15 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

---
 content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++-

----

Quoted 41 lines
Quoted 41 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +3 -3
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason, or
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and ready
+to
+  drive the device.
Quoted 1 line
Quoted 1 line

\item[FEATURES_OK (8)] Indicates that the driver has acknowledged all

the

Quoted 10 lines

content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev

Quoted 10 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to

content.tex +3 -1
-  drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 9 lines
Quoted 9 lines

an error from which it can't recover.

content.tex +6 -0
+
+\item[FAILED (128)] Indicates that something went wrong in the
+guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason, or
+  even a fatal error during device operation.
 \end{description}
Quoted 5 lines
Quoted 5 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 46 lines
Quoted 46 lines
content.tex +6 -3
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other than
+SUSPEND except when setting \field{device status} to 0 as a
+transport-specific way to initiate a reset. If the driver sets the
+FAILED bit,
 the driver MUST later reset the device before attempting to re-initialize.
 
 The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
+Virtio Device

/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device
type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue +and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset)
before removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And Device

content.tex +10 -0
+Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the SUSPEND
+bit in \field{device status} to suspend a device, and can clear the
+SUSPEND bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General Initialization
+And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
Quoted 8 lines
Quoted 8 lines

VIRTIO_F_SUSPEND is not negotiated.

content.tex +5 -0
+
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify
+whether the

SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 6 lines
Quoted 2 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

Quoted 5 lines
Quoted 1 line

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Sure, I didn’t suggest to put such things in the spec.
My question is, whether to trap and emulate or not is a choice of the software. Do you agree?

Quoted 66 lines

Thanks
Zhu Lingshan

Quoted 64 lines
Quoted 64 lines

+\devicenormative{\subsection}{Device Suspend}{General Initialization

content.tex +3 -0
+And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the
+Configuration

Space.

content.tex +9 -0
+
+A device MUST NOT send any notifications for any virtqeuues, access
+any virtqueues, or modify any fields in its Configuration Space
+while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit is
+set, the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification after the SUSPEND bit
+has

been cleared.

content.tex +3 -0
+
+When the driver sets SUSPEND, the device MUST either suspend itself
+or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver clears
+SUSPEND, the device MUST either resume normal operation or set

DEVICE_NEEDS_RESET.

content.tex +4 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting
+that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item Wait
+until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport

Options}

Virtio can use various different buses, thus the standard is split
@@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature
Bits} for

content.tex +4 -0
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature

Bits}
--
2.45.2

#4 lore raw
Quoted 154 lines · Parav Pandit
Quoted 154 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 17 lines · Parav Pandit

Hi Lingshan, David,

Quoted 15 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

---
 content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++-

----

Quoted 41 lines
Quoted 41 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +3 -3
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason, or
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and ready
+to
+  drive the device.
Quoted 1 line
Quoted 1 line

\item[FEATURES_OK (8)] Indicates that the driver has acknowledged all

the

Quoted 10 lines

content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev

Quoted 10 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to

content.tex +3 -1
-  drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 9 lines
Quoted 9 lines

an error from which it can't recover.

content.tex +6 -0
+
+\item[FAILED (128)] Indicates that something went wrong in the
+guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason, or
+  even a fatal error during device operation.
 \end{description}
Quoted 5 lines
Quoted 5 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 46 lines
Quoted 46 lines
content.tex +6 -3
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other than
+SUSPEND except when setting \field{device status} to 0 as a
+transport-specific way to initiate a reset. If the driver sets the
+FAILED bit,
 the driver MUST later reset the device before attempting to re-initialize.
 
 The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
+Virtio Device

/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device
type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue +and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset)
before removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And Device

content.tex +10 -0
+Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the SUSPEND
+bit in \field{device status} to suspend a device, and can clear the
+SUSPEND bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General Initialization
+And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
Quoted 8 lines
Quoted 8 lines

VIRTIO_F_SUSPEND is not negotiated.

content.tex +5 -0
+
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify
+whether the

SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

I think we don't have an intention of the frequency in the spec.
The spec only provides generic mechanisms and interfaces.
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Sure, I didn’t suggest to put such things in the spec.
My question is, whether to trap and emulate or not is a choice of the software. Do you agree?

The device emulator does not know anything about whether trapped or not. Trapping and Emulation is a hypervisor thing.

If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.

this is out of the spec anyway.

Thanks
Zhu Lingshan

Quoted 67 lines
Quoted 67 lines

Thanks
Zhu Lingshan

Quoted 63 lines
Quoted 63 lines

+\devicenormative{\subsection}{Device Suspend}{General Initialization

content.tex +3 -0
+And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the
+Configuration

Space.

content.tex +9 -0
+
+A device MUST NOT send any notifications for any virtqeuues, access
+any virtqueues, or modify any fields in its Configuration Space
+while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit is
+set, the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification after the SUSPEND bit
+has

been cleared.

content.tex +3 -0
+
+When the driver sets SUSPEND, the device MUST either suspend itself
+or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver clears
+SUSPEND, the device MUST either resume normal operation or set

DEVICE_NEEDS_RESET.

content.tex +4 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting
+that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item Wait
+until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport

Options}

Virtio can use various different buses, thus the standard is split
@@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature
Bits} for

content.tex +4 -0
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature

Bits}
--
2.45.2

#5 lore raw
Quoted 5 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AM

I removed the unreachable intel email id as every single email is bouncing from it. Please consider dropping that email from v8 as it will cause all reviewers email to bounce.

Quoted 163 lines · Parav Pandit
Quoted 16 lines
Quoted 16 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 11 lines · Parav Pandit

Hi Lingshan, David,

Quoted 9 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
---
 content.tex | 75

++++++++++++++++++++++++++++++++++++++++++++++-

Quoted 46 lines
Quoted 46 lines

----

Quoted 42 lines
Quoted 42 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

content.tex index 0a62dce..2d1bee8 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +3 -3
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason, or
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and ready
+to
+  drive the device.
Quoted 2 lines
Quoted 2 lines

\item[FEATURES_OK (8)] Indicates that the driver has acknowledged
all

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated,

indicates

Quoted 20 lines
Quoted 20 lines
Quoted 4 lines

content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev

Quoted 4 lines
content.tex +2 -0
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 9 lines
Quoted 9 lines

an error from which it can't recover.

content.tex +6 -0
+
+\item[FAILED (128)] Indicates that something went wrong in the
+guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason, or
+  even a fatal error during device operation.
 \end{description}
Quoted 5 lines
Quoted 5 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

content.tex

Quoted 7 lines
content.tex +4 -2
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other than
+SUSPEND except when setting \field{device status} to 0 as a
+transport-specific way to initiate a reset. If the driver sets the
+FAILED bit,
 the driver MUST later reset the device before attempting to re-

initialize.

Quoted 58 lines
Quoted 58 lines
Quoted 40 lines
Quoted 40 lines

The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device
type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue +and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset)
before removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And

content.tex +10 -0
+Device Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the
+SUSPEND bit in \field{device status} to suspend a device, and can
+clear the SUSPEND bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
Quoted 8 lines
Quoted 8 lines

VIRTIO_F_SUSPEND is not negotiated.

content.tex +5 -0
+
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify
+whether the

SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 2 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

I think we don't have an intention of the frequency in the spec.

Sure.

Quoted 1 line

The spec only provides generic mechanisms and interfaces.

Sure.

Quoted 2 lines

Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.

As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is < 50nsec of time.
The suspend operation for a net device to store hundreds of queues, RSS table, flow filters, takes plenty of time (at least more than 50nsec :) ). Similarly for the GPU to store some MBs of memory takes more than 50nsec of time, for example to store in a file for a software-based GPU device. So a device cannot respond back suspend=true in next 50nsec time.

More below.

Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Sure, I didn’t suggest to put such things in the spec.
My question is, whether to trap and emulate or not is a choice of the

Quoted 9 lines

software.

Quoted 1 line

Do you agree?

The device emulator does not know anything about whether trapped or not. Trapping and Emulation is a hypervisor thing.

If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.

Right. So a software wants to implement device_status as pure MMIO writes. (and not VM_EXIT). And prefer to returning SUSPEND=true at slow pace.
This means, the device implementation cannot immediately return suspend=true right after it was written. A MMIO read will read it back, as suspend=true.

An alternative would be, to forward CPU loads and CPU stores to different address. However, this does not work for the hw based devices.

That means, PCI HW needs to return suspend=0, until the device is not suspended.
In this example, the device cannot build special circuitry to answer suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is the default value.
3. When the device completes suspend, it changes the polarity of suspend_status=true.

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and does not have to force VM_EXITs.

[B] It also enables hw based devices to not build special circuitry to answer within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

Quoted 74 lines

this is out of the spec anyway.

Thanks
Zhu Lingshan

Quoted 14 lines
Quoted 14 lines

Thanks
Zhu Lingshan

Quoted 10 lines
Quoted 10 lines

+\devicenormative{\subsection}{Device Suspend}{General

content.tex +3 -0
+Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the
+Configuration
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines

Space.

content.tex +7 -0
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in its Configuration
+Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit
+is set, the device MUST NOT send any configuration change

notifications.

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
content.tex +2 -0
+Instead, the device MUST send the notification after the SUSPEND
+bit has

been cleared.

content.tex +3 -0
+
+When the driver sets SUSPEND, the device MUST either suspend
+itself or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver clears
+SUSPEND, the device MUST either resume normal operation or set

DEVICE_NEEDS_RESET.

content.tex +4 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting
+that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item Wait
+until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport

Options}

Virtio can use various different buses, thus the standard is split
@@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver

can

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex +3 -0
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature

Bits}
--
2.45.2

#6 lore raw
Quoted 180 lines · Parav Pandit
Quoted 4 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AM

I removed the unreachable intel email id as every single email is bouncing from it. Please consider dropping that email from v8 as it will cause all reviewers email to bounce.

Quoted 161 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 11 lines · Parav Pandit

Hi Lingshan, David,

Quoted 9 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
---
 content.tex | 75

++++++++++++++++++++++++++++++++++++++++++++++-

Quoted 46 lines
Quoted 46 lines

----

Quoted 42 lines
Quoted 42 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

content.tex index 0a62dce..2d1bee8 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +3 -3
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason, or
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and ready
+to
+  drive the device.
Quoted 2 lines
Quoted 2 lines

\item[FEATURES_OK (8)] Indicates that the driver has acknowledged
all

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated,

indicates

Quoted 20 lines
Quoted 20 lines
Quoted 4 lines

content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev

Quoted 4 lines
content.tex +2 -0
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 9 lines
Quoted 9 lines

an error from which it can't recover.

content.tex +6 -0
+
+\item[FAILED (128)] Indicates that something went wrong in the
+guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason, or
+  even a fatal error during device operation.
 \end{description}
Quoted 5 lines
Quoted 5 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

content.tex

Quoted 7 lines
content.tex +4 -2
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other than
+SUSPEND except when setting \field{device status} to 0 as a
+transport-specific way to initiate a reset. If the driver sets the
+FAILED bit,
 the driver MUST later reset the device before attempting to re-

initialize.

Quoted 57 lines
Quoted 57 lines
Quoted 39 lines
Quoted 39 lines

The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device
type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue +and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset)
before removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And

content.tex +10 -0
+Device Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the
+SUSPEND bit in \field{device status} to suspend a device, and can
+clear the SUSPEND bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
Quoted 8 lines
Quoted 8 lines

VIRTIO_F_SUSPEND is not negotiated.

content.tex +5 -0
+
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify
+whether the

SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 2 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

I think we don't have an intention of the frequency in the spec.

Sure.

Quoted 1 line

The spec only provides generic mechanisms and interfaces.

Sure.

Quoted 7 lines
Quoted 2 lines

Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.

As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is < 50nsec of time.
The suspend operation for a net device to store hundreds of queues, RSS table, flow filters, takes plenty of time (at least more than 50nsec :) ). Similarly for the GPU to store some MBs of memory takes more than 50nsec of time, for example to store in a file for a software-based GPU device. So a device cannot respond back suspend=true in next 50nsec time.

It's OK for the device to take longer time to respond, the driver simply re-reads device status.

Quoted 13 lines

More below.

Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Sure, I didn’t suggest to put such things in the spec.
My question is, whether to trap and emulate or not is a choice of the

software.

Quoted 1 line

Do you agree?

Quoted 8 lines
Quoted 7 lines

The device emulator does not know anything about whether trapped or not. Trapping and Emulation is a hypervisor thing.

If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.

Right. So a software wants to implement device_status as pure MMIO writes. (and not VM_EXIT).

This is not always true, there can be VM_EXIT of pure emulated devices.  The HW registers are sensitive resource and any access to them need to be trapped and emulated.

Quoted 9 lines

And prefer to returning SUSPEND=true at slow pace.
This means, the device implementation cannot immediately return suspend=true right after it was written. A MMIO read will read it back, as suspend=true.

An alternative would be, to forward CPU loads and CPU stores to different address. However, this does not work for the hw based devices.

That means, PCI HW needs to return suspend=0, until the device is not suspended.
In this example, the device cannot build special circuitry to answer suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.

Quoted 5 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

Quoted 8 lines

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is the default value.
3. When the device completes suspend, it changes the polarity of suspend_status=true.

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and does not have to force VM_EXITs.

[B] It also enables hw based devices to not build special circuitry to answer within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I think we have already discussed on this before in V5, and Jason has some insightful comments

https://lore.kernel.org/virtio-comment/[email protected]/T/#mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9

Quoted 73 lines
Quoted 73 lines

this is out of the spec anyway.

Thanks
Zhu Lingshan

Quoted 12 lines
Quoted 12 lines

Thanks
Zhu Lingshan

Quoted 9 lines
Quoted 9 lines

+\devicenormative{\subsection}{Device Suspend}{General

content.tex +3 -0
+Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the
+Configuration
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines

Space.

content.tex +7 -0
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in its Configuration
+Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit
+is set, the device MUST NOT send any configuration change

notifications.

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
content.tex +2 -0
+Instead, the device MUST send the notification after the SUSPEND
+bit has

been cleared.

content.tex +3 -0
+
+When the driver sets SUSPEND, the device MUST either suspend
+itself or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver clears
+SUSPEND, the device MUST either resume normal operation or set

DEVICE_NEEDS_RESET.

content.tex +4 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting
+that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item Wait
+until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport

Options}

Virtio can use various different buses, thus the standard is split
@@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver

can

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex +3 -0
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature

Bits}
--
2.45.2

#7 lore raw
Quoted 193 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PM

Quoted 5 lines · Parav Pandit
Quoted 4 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AM

I removed the unreachable intel email id as every single email is bouncing

from it.

Quoted 1 line

Please consider dropping that email from v8 as it will cause all reviewers

email to bounce.

Quoted 174 lines
Quoted 165 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 11 lines · Parav Pandit

Hi Lingshan, David,

Quoted 9 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
---
 content.tex | 75

++++++++++++++++++++++++++++++++++++++++++++++-

Quoted 47 lines
Quoted 47 lines

----

Quoted 43 lines
Quoted 43 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

content.tex index 0a62dce..2d1bee8 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +0 -2
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason,

or

content.tex +3 -1
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and
+ready to
+  drive the device.
Quoted 2 lines
Quoted 2 lines

\item[FEATURES_OK (8)] Indicates that the driver has
acknowledged all

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated,

indicates

Quoted 22 lines
Quoted 22 lines
Quoted 4 lines

content.tex

Quoted 4 lines
content.tex +2 -0
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 10 lines
Quoted 10 lines

an error from which it can't recover.

content.tex +7 -0
+
+\item[FAILED (128)] Indicates that something went wrong in the
+guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason,
+or
+  even a fatal error during device operation.
 \end{description}
Quoted 6 lines
Quoted 6 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

content.tex

Quoted 7 lines
content.tex +4 -2
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other than
+SUSPEND except when setting \field{device status} to 0 as a
+transport-specific way to initiate a reset. If the driver sets
+the FAILED bit,
 the driver MUST later reset the device before attempting to re-

initialize.

Quoted 57 lines
Quoted 57 lines
Quoted 39 lines
Quoted 39 lines

The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And

content.tex +10 -0
+Device Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the
+SUSPEND bit in \field{device status} to suspend a device, and
+can clear the SUSPEND bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
Quoted 8 lines
Quoted 8 lines

VIRTIO_F_SUSPEND is not negotiated.

content.tex +5 -0
+
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify
+whether the

SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 2 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

I think we don't have an intention of the frequency in the spec.

Sure.

Quoted 7 lines
Quoted 1 line

The spec only provides generic mechanisms and interfaces.

Sure.

Quoted 2 lines

Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.

As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <

Quoted 8 lines

50nsec of time.

Quoted 1 line

The suspend operation for a net device to store hundreds of queues, RSS

table, flow filters, takes plenty of time (at least more than 50nsec :) ).

Quoted 1 line

Similarly for the GPU to store some MBs of memory takes more than 50nsec

of time, for example to store in a file for a software-based GPU device.

Quoted 1 line

So a device cannot respond back suspend=true in next 50nsec time.

It's OK for the device to take longer time to respond, the driver simply re-reads device status.

Sure, the issue is, when the driver re-reads, the device must present suspend=false within 50nsec. (because device didn't suspend it yet).

As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly differently.

Quoted 18 lines
Quoted 15 lines

More below.

Quoted 12 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the

software.

Quoted 1 line

Do you agree?

The device emulator does not know anything about whether trapped or

not.

Quoted 2 lines
Quoted 2 lines

Trapping and Emulation is a hypervisor thing.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.

Right. So a software wants to implement device_status as pure MMIO

writes. (and not VM_EXIT).
This is not always true, there can be VM_EXIT of pure emulated devices.

I am not denying that VM_EXIT can/cannot be there.
I am saying, the proposal forces VM_EXIT based approach in my understanding of this patch. If that is not true, may be can you please explain how this can be implemented without VM_EXIT?

We should have an interface that can be done with and without VM_EXIT method at least for any new additions.

Quoted 3 lines

The
HW registers are sensitive resource and any access to them need to be trapped and emulated.

This does not apply to PCI PFs and VFs which are HW devices (mainly PFs). so this trap + emulation is narrow view that we better avoid.

If you think this is the way forward, you should put forward in patch as MUST requirement. and that does not look right to me.
I hope you also don't mean to force this method to device implementations. Right?

Quoted 9 lines
Quoted 2 lines

And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return

suspend=true right after it was written.

Quoted 3 lines

A MMIO read will read it back, as suspend=true.

An alternative would be, to forward CPU loads and CPU stores to different

address.

Quoted 2 lines

However, this does not work for the hw based devices.

Quoted 7 lines
Quoted 1 line

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 1 line

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.

When driver wrote, it wrote suspend=true,
And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec or less to answer suspend=false.

And even worst, it needs to suspend=true when unsuspending within 50nsec when resuming is ongoing.

Quoted 13 lines
Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

Quoted 8 lines

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful comments

Unfortunately, not. His comment was that it is not specific to suspend.
But here we are introducing a new interface and functionality that does not need to suffer or follow anything that may not be efficient.

Quoted 77 lines

https://lore.kernel.org/virtio-comment/[email protected]/T/#mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9

Quoted 75 lines
Quoted 75 lines

this is out of the spec anyway.

Thanks
Zhu Lingshan

Quoted 12 lines
Quoted 12 lines

Thanks
Zhu Lingshan

Quoted 9 lines
Quoted 9 lines

+\devicenormative{\subsection}{Device Suspend}{General

content.tex +3 -0
+Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space
+while suspended, except for \field{device status} if it is part
+of the Configuration
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines

Space.

content.tex +7 -0
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in its Configuration
+Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND
+bit is set, the device MUST NOT send any configuration change

notifications.

Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
content.tex +2 -0
+Instead, the device MUST send the notification after the SUSPEND
+bit has

been cleared.

content.tex +3 -0
+
+When the driver sets SUSPEND, the device MUST either suspend
+itself or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +4 -0
+
+If SUSPEND is set in \field{device status}, when the driver
+clears SUSPEND, the device MUST either resume normal operation
+or set

DEVICE_NEEDS_RESET.

content.tex +4 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before
+presenting that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item Wait
+until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport

Options}

Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the
+ driver

can

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex +3 -0
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved

Feature Bits}
--
2.45.2

#8 lore raw
Quoted 196 lines · Parav Pandit
Quoted 196 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PM

Quoted 4 lines · Parav Pandit
Quoted 3 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AM

I removed the unreachable intel email id as every single email is bouncing

from it.

Quoted 1 line

Please consider dropping that email from v8 as it will cause all reviewers

email to bounce.

Quoted 173 lines
Quoted 164 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 11 lines · Parav Pandit

Hi Lingshan, David,

Quoted 9 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
---
 content.tex | 75

++++++++++++++++++++++++++++++++++++++++++++++-

Quoted 47 lines
Quoted 47 lines

----

Quoted 43 lines
Quoted 43 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

content.tex index 0a62dce..2d1bee8 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +0 -2
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason,

or

content.tex +3 -1
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and
+ready to
+  drive the device.
Quoted 2 lines
Quoted 2 lines

\item[FEATURES_OK (8)] Indicates that the driver has
acknowledged all

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated,

indicates

Quoted 22 lines
Quoted 22 lines
Quoted 4 lines

content.tex

Quoted 4 lines
content.tex +2 -0
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 10 lines
Quoted 10 lines

an error from which it can't recover.

content.tex +7 -0
+
+\item[FAILED (128)] Indicates that something went wrong in the
+guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason,
+or
+  even a fatal error during device operation.
 \end{description}
Quoted 6 lines
Quoted 6 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

content.tex

Quoted 7 lines
content.tex +4 -2
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other than
+SUSPEND except when setting \field{device status} to 0 as a
+transport-specific way to initiate a reset. If the driver sets
+the FAILED bit,
 the driver MUST later reset the device before attempting to re-

initialize.

Quoted 57 lines
Quoted 57 lines
Quoted 39 lines
Quoted 39 lines

The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And

content.tex +10 -0
+Device Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the
+SUSPEND bit in \field{device status} to suspend a device, and
+can clear the SUSPEND bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
Quoted 8 lines
Quoted 8 lines

VIRTIO_F_SUSPEND is not negotiated.

content.tex +5 -0
+
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify
+whether the

SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 2 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

I think we don't have an intention of the frequency in the spec.

Sure.

Quoted 7 lines
Quoted 1 line

The spec only provides generic mechanisms and interfaces.

Sure.

Quoted 2 lines

Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.

As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <

50nsec of time.

Quoted 1 line

The suspend operation for a net device to store hundreds of queues, RSS

table, flow filters, takes plenty of time (at least more than 50nsec :) ).

Quoted 1 line

Similarly for the GPU to store some MBs of memory takes more than 50nsec

of time, for example to store in a file for a software-based GPU device.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

So a device cannot respond back suspend=true in next 50nsec time.

It's OK for the device to take longer time to respond, the driver simply re-reads device status.

Sure, the issue is, when the driver re-reads, the device must present suspend=false within 50nsec. (because device didn't suspend it yet).

As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly differently.

why there is a constraint condition of the time? Are there any similar constraining for other states like RESET or DRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.

Quoted 22 lines
Quoted 22 lines
Quoted 15 lines

More below.

Quoted 12 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the

software.

Quoted 1 line

Do you agree?

The device emulator does not know anything about whether trapped or

not.

Quoted 6 lines
Quoted 6 lines

Trapping and Emulation is a hypervisor thing.

If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Right. So a software wants to implement device_status as pure MMIO

writes. (and not VM_EXIT).
This is not always true, there can be VM_EXIT of pure emulated devices.

I am not denying that VM_EXIT can/cannot be there.
I am saying, the proposal forces VM_EXIT based approach in my understanding of this patch. If that is not true, may be can you please explain how this can be implemented without VM_EXIT?

We should have an interface that can be done with and without VM_EXIT method at least for any new additions.

VM_EXIT is out of spec, it is a hypervisor and the processor thing.
In non-pass-through case, any register access are sensitive and will trigger VM_EXIT. Like RESET or DRIVER_OK needs to access device_status, nothing different.

Quoted 6 lines
Quoted 4 lines

The
HW registers are sensitive resource and any access to them need to be trapped and emulated.

This does not apply to PCI PFs and VFs which are HW devices (mainly PFs). so this trap + emulation is narrow view that we better avoid.

This is how *basic* virtualization work, once access sensitive resource, trap it.

Quoted 5 lines

If you think this is the way forward, you should put forward in patch as MUST requirement. and that does not look right to me.
I hope you also don't mean to force this method to device implementations. Right?

Again, VM_EXIT is a hypervisor thing,  out of spec. Whether there is a VM_EXIT when setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different from DRIVER_OK.

Means, if your virtualization needs to trap SUSPEND, it also needs to trap DRIVER_OK, and don't assume DRIVER_OK is faster than SUSPEND.

Quoted 17 lines
Quoted 10 lines
Quoted 3 lines

And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return

suspend=true right after it was written.

Quoted 3 lines

A MMIO read will read it back, as suspend=true.

An alternative would be, to forward CPU loads and CPU stores to different

address.

Quoted 2 lines

However, this does not work for the hw based devices.

Quoted 7 lines
Quoted 1 line

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 1 line

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.

Quoted 5 lines

When driver wrote, it wrote suspend=true,
And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec or less to answer suspend=false.

And even worst, it needs to suspend=true when unsuspending within 50nsec when resuming is ongoing.

again, there is not a 50nsec constraining and please take a reference of how DRIVER_OK work with virtualization.

Quoted 96 lines
Quoted 10 lines
Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 9 lines
Quoted 2 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 1 line

[B] It also enables hw based devices to not build special circuitry to answer

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful comments

Unfortunately, not. His comment was that it is not specific to suspend.
But here we are introducing a new interface and functionality that does not need to suffer or follow anything that may not be efficient.

Quoted 74 lines

https://lore.kernel.org/virtio-comment/[email protected]/T/#mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9

Quoted 73 lines
Quoted 73 lines

this is out of the spec anyway.

Thanks
Zhu Lingshan

Quoted 11 lines
Quoted 11 lines

Thanks
Zhu Lingshan

Quoted 9 lines
Quoted 9 lines

+\devicenormative{\subsection}{Device Suspend}{General

content.tex +3 -0
+Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space
+while suspended, except for \field{device status} if it is part
+of the Configuration
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines

Space.

content.tex +7 -0
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in its Configuration
+Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND
+bit is set, the device MUST NOT send any configuration change

notifications.

Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
content.tex +2 -0
+Instead, the device MUST send the notification after the SUSPEND
+bit has

been cleared.

content.tex +3 -0
+
+When the driver sets SUSPEND, the device MUST either suspend
+itself or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +4 -0
+
+If SUSPEND is set in \field{device status}, when the driver
+clears SUSPEND, the device MUST either resume normal operation
+or set

DEVICE_NEEDS_RESET.

content.tex +4 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before
+presenting that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item Wait
+until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport

Options}

Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the
+ driver

can

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex +3 -0
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved

Feature Bits}
--
2.45.2

#9 lore raw
Quoted 216 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:02 AM

Quoted 61 lines · Parav Pandit
Quoted 61 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PM

Quoted 5 lines · Parav Pandit
Quoted 3 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AM

I removed the unreachable intel email id as every single email is bouncing

from it.

Quoted 2 lines

Please consider dropping that email from v8 as it will cause all reviewers

email to bounce.

Quoted 48 lines
Quoted 48 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 11 lines · Parav Pandit

Hi Lingshan, David,

Quoted 9 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
---
 content.tex | 75

++++++++++++++++++++++++++++++++++++++++++++++-

Quoted 25 lines
Quoted 25 lines

----

Quoted 24 lines
Quoted 24 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
content.tex index 0a62dce..2d1bee8 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable

modules.

Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
   \end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +0 -2
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason,

or

content.tex +3 -1
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and
+ready to
+  drive the device.
 
 \item[FEATURES_OK (8)] Indicates that the driver has

acknowledged all

Quoted 7 lines
Quoted 7 lines

the

Quoted 6 lines
Quoted 6 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated,

indicates

Quoted 22 lines
Quoted 22 lines
Quoted 4 lines

content.tex

Quoted 4 lines
content.tex +2 -0
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 10 lines
Quoted 10 lines

an error from which it can't recover.

content.tex +7 -0
+
+\item[FAILED (128)] Indicates that something went wrong in the
+guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason,
+or
+  even a fatal error during device operation.
 \end{description}
Quoted 6 lines
Quoted 6 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex +4 -2
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other
+than SUSPEND except when setting \field{device status} to 0 as
+a transport-specific way to initiate a reset. If the driver
+sets the FAILED bit,
 the driver MUST later reset the device before attempting to

re-

initialize.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The driver SHOULD NOT rely on completion of operations of a

@@

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of +a Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

content.tex \section{Device

Quoted 2 lines

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device

Cleanup}\label{sec:General

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+\section{Device Suspend}\label{sec:General Initialization And

content.tex +5 -0
+Device Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the
+SUSPEND bit in \field{device status} to suspend a device, and
+can clear the SUSPEND bit to resume a suspended device.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines
content.tex +5 -0
+
+\drivernormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +2 -0
+
+Once the driver sets SUSPEND to \field{device status} of the

device:

Quoted 43 lines
Quoted 43 lines
Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
Quoted 6 lines

+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify +whether the
SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to +the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 2 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

I think we don't have an intention of the frequency in the spec.

Sure.

Quoted 8 lines
Quoted 1 line

The spec only provides generic mechanisms and interfaces.

Sure.

Quoted 2 lines

Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.

As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <

50nsec of time.

Quoted 2 lines

The suspend operation for a net device to store hundreds of queues, RSS

table, flow filters, takes plenty of time (at least more than 50nsec :) ).

Quoted 2 lines

Similarly for the GPU to store some MBs of memory takes more than 50nsec

of time, for example to store in a file for a software-based GPU device.

Quoted 1 line

So a device cannot respond back suspend=true in next 50nsec time.

It's OK for the device to take longer time to respond, the driver simply re-reads device status.

Quoted 8 lines
Quoted 1 line

Sure, the issue is, when the driver re-reads, the device must present

suspend=false within 50nsec.

Quoted 4 lines

(because device didn't suspend it yet).

As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly

differently.
why there is a constraint condition of the time?

Because this is what driver does as you explained in [1] indicating "we must be able to handle ..."

[1] https://lore.kernel.org/virtio-comment/[email protected]/T/#m6f081f96ef9dcea29c64a88b633eb21d50e8c410

Quoted 3 lines

Are there any similar
constraining for other states like RESET or DRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.

DRIVER_OK does not suffer from it because it is async notification. A device may start slow after DRIVER_OK.
SUSPEND operation cannot rely on such async behavior.

RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.

Here a new functionality is being proposed and it has a chance for efficient device implementation. Therefore the request is to improve it.

Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

More below.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the

software.

Quoted 1 line

Do you agree?

The device emulator does not know anything about whether trapped

or

Quoted 7 lines
Quoted 7 lines

not.

Quoted 6 lines
Quoted 6 lines

Trapping and Emulation is a hypervisor thing.

If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Right. So a software wants to implement device_status as pure MMIO

writes. (and not VM_EXIT).
This is not always true, there can be VM_EXIT of pure emulated devices.

I am not denying that VM_EXIT can/cannot be there. I am saying, the proposal forces VM_EXIT based approach in my

understanding of this patch.

[MARKER_1]

Quoted 6 lines
Quoted 1 line

If that is not true, may be can you please explain how this can be

implemented without VM_EXIT?

Quoted 2 lines

We should have an interface that can be done with and without VM_EXIT

method at least for any new additions.
VM_EXIT is out of spec, it is a hypervisor and the processor thing.

You keep repeating VM_EXIST is out of spec.
I already replied at [MARKER_1], sure it is out of spec, the current approach forces one to do VM_EXIT based approach. And if not, please explain, how can it be achieved?

Quoted 3 lines

In non-pass-through case, any register access are sensitive and will trigger VM_EXIT. Like RESET or DRIVER_OK needs to access device_status, nothing different.

I don't think you understood the point.
Let me repeat, The question is, if the device implementation wants to achieve the functionality without VM_EXIT, what is the way?

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

The
HW registers are sensitive resource and any access to them need to be trapped and emulated.

This does not apply to PCI PFs and VFs which are HW devices (mainly PFs). so this trap + emulation is narrow view that we better avoid.

This is how *basic* virtualization work, once access sensitive resource, trap it.

Quoted 2 lines

If you think this is the way forward, you should put forward in patch as

Quoted 9 lines

MUST requirement.

Quoted 2 lines

and that does not look right to me.
I hope you also don't mean to force this method to device

implementations.

Quoted 1 line

Right?

Again, VM_EXIT is a hypervisor thing,  out of spec. Whether there is a VM_EXIT when setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different from DRIVER_OK.

Please avoid repeating the point that VM_EXIT is hypervosor thing.
No one asked to put this in spec. Please re-read [MARKER1]. You probably missed that.

SUSPEND is different than DRIVER_OK. I explained the timing constraints and the required circuitry needed to fulfill the proposal. And with the additional register, such complicated circuitry can be easily avoided.

Quoted 2 lines

Means, if your virtualization needs to trap SUSPEND, it also needs to trap DRIVER_OK, and don't assume DRIVER_OK is faster than SUSPEND.

DRIVER_OK is by law of physics is faster than SUSPEND because it does not demand the driver of reading back. There is no driver side loop to check if the device accepted DRIVER_OK or not.
Agree?

Quoted 22 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines

And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return

suspend=true right after it was written.

Quoted 4 lines

A MMIO read will read it back, as suspend=true.

An alternative would be, to forward CPU loads and CPU stores to different

address.

Quoted 4 lines

However, this does not work for the hw based devices.

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation. why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.

When driver wrote, it wrote suspend=true, And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec

Quoted 6 lines

or less to answer suspend=false.

Quoted 2 lines

And even worst, it needs to suspend=true when unsuspending within

50nsec when resuming is ongoing.
again, there is not a 50nsec constraining and please take a reference of how DRIVER_OK work with virtualization.

There is. The device is expected to return back the desired value to indicate driver that suspend is ongoing. It is different than DRIVER_OK.

Quoted 121 lines
Quoted 17 lines
Quoted 17 lines
Quoted 8 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 7 lines
Quoted 5 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful comments

Unfortunately, not. His comment was that it is not specific to suspend. But here we are introducing a new interface and functionality that does not

need to suffer or follow anything that may not be efficient.

Quoted 2 lines
Quoted 2 lines

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor

Quoted 1 line
Quoted 1 line

e.kernel.org%2Fvirtio-comment%2F20240612082055-mutt-send-email-

mst%40

Quoted 1 line
Quoted 1 line

kernel.org%2FT%2F%23mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9&dat a=05

Quoted 1 line
Quoted 1 line

%7C02%7Cparav%40nvidia.com%7C619c82b60b824ca03f8808dcc89beb4a%7 C43083

Quoted 1 line
Quoted 1 line

d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638605819247985989%7CUn known%7C

Quoted 1 line
Quoted 1 line

TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL CJXV

Quoted 1 line
Quoted 1 line

CI6Mn0%3D%7C0%7C%7C%7C&sdata=VqmJmqt3k5tf3x4ihjE5Bd7u59GadOn OaOfJ5lvG

Quoted 25 lines
Quoted 25 lines

1DE%3D&reserved=0

Quoted 24 lines
Quoted 24 lines

this is out of the spec anyway.

Thanks
Zhu Lingshan

Quoted 11 lines
Quoted 11 lines

Thanks
Zhu Lingshan

Quoted 9 lines
Quoted 9 lines

+\devicenormative{\subsection}{Device Suspend}{General

content.tex +3 -0
+Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space
+while suspended, except for \field{device status} if it is
+part of the Configuration
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines

Space.

content.tex +7 -0
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in its
+Configuration Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND
+bit is set, the device MUST NOT send any configuration change

notifications.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

content.tex

Quoted 10 lines
content.tex +2 -0
+Instead, the device MUST send the notification after the
+SUSPEND bit has

been cleared.

content.tex +3 -0
+
+When the driver sets SUSPEND, the device MUST either suspend
+itself or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver
+clears SUSPEND, the device MUST either resume normal

operation

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

+or set
DEVICE_NEEDS_RESET.

content.tex +4 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before
+presenting that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item
+Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport

Options}

Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the
+ driver

can

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
content.tex +2 -0
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status

Field}.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+
\end{description}

\drivernormative{\section}{Reserved Feature Bits}{Reserved
Feature Bits}
--
2.45.2

#10 lore raw
Quoted 219 lines · Parav Pandit
Quoted 219 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:02 AM

Quoted 61 lines · Parav Pandit
Quoted 61 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PM

Quoted 5 lines · Parav Pandit
Quoted 3 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AM

I removed the unreachable intel email id as every single email is bouncing

from it.

Quoted 2 lines

Please consider dropping that email from v8 as it will cause all reviewers

email to bounce.

Quoted 48 lines
Quoted 48 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 11 lines · Parav Pandit

Hi Lingshan, David,

Quoted 9 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
---
 content.tex | 75

++++++++++++++++++++++++++++++++++++++++++++++-

Quoted 25 lines
Quoted 25 lines

----

Quoted 24 lines
Quoted 24 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
content.tex index 0a62dce..2d1bee8 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable

modules.

Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
   \end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +0 -2
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason,

or

content.tex +3 -1
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and
+ready to
+  drive the device.
 
 \item[FEATURES_OK (8)] Indicates that the driver has

acknowledged all

Quoted 7 lines
Quoted 7 lines

the

Quoted 6 lines
Quoted 6 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated,

indicates

Quoted 22 lines
Quoted 22 lines
Quoted 4 lines

content.tex

Quoted 4 lines
content.tex +2 -0
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 10 lines
Quoted 10 lines

an error from which it can't recover.

content.tex +7 -0
+
+\item[FAILED (128)] Indicates that something went wrong in the
+guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason,
+or
+  even a fatal error during device operation.
 \end{description}
Quoted 6 lines
Quoted 6 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex +4 -2
-The driver MUST NOT clear a
-\field{device status} bit.  If the driver sets the FAILED bit,
+The driver MUST NOT clear a \field{device status} bit other
+than SUSPEND except when setting \field{device status} to 0 as
+a transport-specific way to initiate a reset. If the driver
+sets the FAILED bit,
 the driver MUST later reset the device before attempting to

re-

initialize.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The driver SHOULD NOT rely on completion of operations of a

@@

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of +a Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

content.tex \section{Device

Quoted 2 lines

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device

Cleanup}\label{sec:General

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+\section{Device Suspend}\label{sec:General Initialization And

content.tex +5 -0
+Device Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the
+SUSPEND bit in \field{device status} to suspend a device, and
+can clear the SUSPEND bit to resume a suspended device.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines
content.tex +5 -0
+
+\drivernormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +2 -0
+
+Once the driver sets SUSPEND to \field{device status} of the

device:

Quoted 36 lines
Quoted 36 lines
Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
Quoted 6 lines

+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify +whether the
SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to +the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 2 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

I think we don't have an intention of the frequency in the spec.

Sure.

Quoted 8 lines
Quoted 1 line

The spec only provides generic mechanisms and interfaces.

Sure.

Quoted 2 lines

Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.

As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <

50nsec of time.

Quoted 2 lines

The suspend operation for a net device to store hundreds of queues, RSS

Quoted 8 lines
Quoted 7 lines

table, flow filters, takes plenty of time (at least more than 50nsec :) ).

Quoted 2 lines

Similarly for the GPU to store some MBs of memory takes more than 50nsec

of time, for example to store in a file for a software-based GPU device.

Quoted 1 line

So a device cannot respond back suspend=true in next 50nsec time.

It's OK for the device to take longer time to respond, the driver simply re-reads device status.

Sure, the issue is, when the driver re-reads, the device must present

suspend=false within 50nsec.

Quoted 2 lines

(because device didn't suspend it yet).

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly

differently.
why there is a constraint condition of the time?

Because this is what driver does as you explained in [1] indicating "we must be able to handle ..."

[1] https://lore.kernel.org/virtio-comment/[email protected]/T/#m6f081f96ef9dcea29c64a88b633eb21d50e8c410

The sentence is "we must be able to handle the scenario in which SUSPENDING is often", where do you see any constraint conditions of time?

Quoted 7 lines
Quoted 4 lines

Are there any similar
constraining for other states like RESET or DRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.

DRIVER_OK does not suffer from it because it is async notification. A device may start slow after DRIVER_OK.
SUSPEND operation cannot rely on such async behavior.

The driver can set suspend and re-read & wait. This is a common routine in the driver.

Quoted 6 lines

RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.

Here a new functionality is being proposed and it has a chance for efficient device implementation. Therefore the request is to improve it.

sure, as long as we confirm this new register apply for all device status transitions, not only for SUSPEND.

Quoted 32 lines
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

More below.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the

software.

Quoted 1 line

Do you agree?

The device emulator does not know anything about whether trapped

or

Quoted 7 lines
Quoted 7 lines

not.

Quoted 6 lines
Quoted 6 lines

Trapping and Emulation is a hypervisor thing.

If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Right. So a software wants to implement device_status as pure MMIO

writes. (and not VM_EXIT).
This is not always true, there can be VM_EXIT of pure emulated devices.

I am not denying that VM_EXIT can/cannot be there. I am saying, the proposal forces VM_EXIT based approach in my

understanding of this patch.

[MARKER_1]

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

If that is not true, may be can you please explain how this can be

implemented without VM_EXIT?

Quoted 1 line

We should have an interface that can be done with and without VM_EXIT

method at least for any new additions.
VM_EXIT is out of spec, it is a hypervisor and the processor thing.

You keep repeating VM_EXIST is out of spec.
I already replied at [MARKER_1], sure it is out of spec, the current approach forces one to do VM_EXIT based approach. And if not, please explain, how can it be achieved?

It is not the current solution force the hypervisor & guest to perform VM_EXIT.
In basic virtualization, any access to hypervisor emulated HW registers would be Traped and Emulated, means a VM_EXIT and a VM_ENTRY.

Quoted 6 lines
Quoted 4 lines

In non-pass-through case, any register access are sensitive and will trigger VM_EXIT. Like RESET or DRIVER_OK needs to access device_status, nothing different.

I don't think you understood the point.
Let me repeat, The question is, if the device implementation wants to achieve the functionality without VM_EXIT, what is the way?

Map the register to guest address space, out of the spec.

Quoted 17 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

The
HW registers are sensitive resource and any access to them need to be trapped and emulated.

This does not apply to PCI PFs and VFs which are HW devices (mainly PFs). so this trap + emulation is narrow view that we better avoid.

This is how *basic* virtualization work, once access sensitive resource, trap it.

Quoted 1 line

If you think this is the way forward, you should put forward in patch as

Quoted 9 lines

MUST requirement.

Quoted 2 lines

and that does not look right to me.
I hope you also don't mean to force this method to device

implementations.

Quoted 1 line

Right?

Again, VM_EXIT is a hypervisor thing,  out of spec. Whether there is a VM_EXIT when setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different from DRIVER_OK.

Quoted 5 lines

Please avoid repeating the point that VM_EXIT is hypervosor thing.
No one asked to put this in spec. Please re-read [MARKER1]. You probably missed that.

SUSPEND is different than DRIVER_OK. I explained the timing constraints and the required circuitry needed to fulfill the proposal. And with the additional register, such complicated circuitry can be easily avoided.

Please read QEMU code, that can help you find how the trap - emulate mechanism work for guest when it tries to access sensitive resources.

Quoted 6 lines
Quoted 3 lines

Means, if your virtualization needs to trap SUSPEND, it also needs to trap DRIVER_OK, and don't assume DRIVER_OK is faster than SUSPEND.

DRIVER_OK is by law of physics is faster than SUSPEND because it does not demand the driver of reading back. There is no driver side loop to check if the device accepted DRIVER_OK or not.
Agree?

Why do you assume so? In some vendor implementation, DRIVER_OK can be slow. Can DRIVER_OK implement a deferred initialization even deferred resource allocation? Does the spec say that DRIVER_OK must be faster?

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 3 lines

And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return

suspend=true right after it was written.

Quoted 4 lines

A MMIO read will read it back, as suspend=true.

An alternative would be, to forward CPU loads and CPU stores to different

address.

Quoted 2 lines

However, this does not work for the hw based devices.

Quoted 8 lines
Quoted 2 lines

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 1 line

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation. why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.

When driver wrote, it wrote suspend=true, And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec

Quoted 7 lines
Quoted 5 lines

or less to answer suspend=false.

Quoted 1 line

And even worst, it needs to suspend=true when unsuspending within

50nsec when resuming is ongoing.
again, there is not a 50nsec constraining and please take a reference of how DRIVER_OK work with virtualization.

There is. The device is expected to return back the desired value to indicate driver that suspend is ongoing. It is different than DRIVER_OK.

where does the spec or any code say 50nsec?

Quoted 114 lines
Quoted 114 lines
Quoted 17 lines
Quoted 17 lines
Quoted 8 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 7 lines
Quoted 5 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful comments

Unfortunately, not. His comment was that it is not specific to suspend. But here we are introducing a new interface and functionality that does not

need to suffer or follow anything that may not be efficient.
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor

Quoted 1 line
Quoted 1 line

e.kernel.org%2Fvirtio-comment%2F20240612082055-mutt-send-email-

mst%40 kernel.org%2FT%2F%23mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9&dat a=05 %7C02%7Cparav%40nvidia.com%7C619c82b60b824ca03f8808dcc89beb4a%7 C43083 d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638605819247985989%7CUn known%7C TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL CJXV CI6Mn0%3D%7C0%7C%7C%7C&sdata=VqmJmqt3k5tf3x4ihjE5Bd7u59GadOn OaOfJ5lvG

Quoted 25 lines
Quoted 25 lines

1DE%3D&reserved=0

Quoted 24 lines
Quoted 24 lines

this is out of the spec anyway.

Thanks
Zhu Lingshan

Quoted 11 lines
Quoted 11 lines

Thanks
Zhu Lingshan

Quoted 9 lines
Quoted 9 lines

+\devicenormative{\subsection}{Device Suspend}{General

content.tex +3 -0
+Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space
+while suspended, except for \field{device status} if it is
+part of the Configuration
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines

Space.

content.tex +7 -0
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in its
+Configuration Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND
+bit is set, the device MUST NOT send any configuration change

notifications.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

content.tex

Quoted 10 lines
content.tex +2 -0
+Instead, the device MUST send the notification after the
+SUSPEND bit has

been cleared.

content.tex +3 -0
+
+When the driver sets SUSPEND, the device MUST either suspend
+itself or set

DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver
+clears SUSPEND, the device MUST either resume normal

operation

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

+or set
DEVICE_NEEDS_RESET.

content.tex +4 -0
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before
+presenting that

the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item
+Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport

Options}

Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the
+ driver

can

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
content.tex +2 -0
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status

Field}.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+
\end{description}

\drivernormative{\section}{Reserved Feature Bits}{Reserved
Feature Bits}
--
2.45.2

#11 lore raw
Quoted 7 lines · Zhu Lingshan
Quoted 6 lines
Quoted 3 lines

Are there any similar
constraining for other states like RESET or DRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.

DRIVER_OK does not suffer from it because it is async notification. A device may start slow after DRIVER_OK.
SUSPEND operation cannot rely on such async behavior.

The driver can set suspend and re-read & wait. This is a common routine in the driver.

I don't buy all this talk about special machinery, some 10 nor gates, the cost is in the noise - we are talking about cards with several ARM processors on chip.

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

Quoted 6 lines
Quoted 5 lines

RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.

Here a new functionality is being proposed and it has a chance for efficient device implementation. Therefore the request is to improve it.

sure, as long as we confirm this new register apply for all device status transitions, not only for SUSPEND.

#12 lore raw
Quoted 12 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, September 3, 2024 3:16 PM

Quoted 6 lines · Zhu Lingshan
Quoted 6 lines
Quoted 3 lines

Are there any similar
constraining for other states like RESET or DRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.

DRIVER_OK does not suffer from it because it is async notification. A device may start slow after DRIVER_OK.
SUSPEND operation cannot rely on such async behavior.

Quoted 6 lines
Quoted 1 line

The driver can set suspend and re-read & wait. This is a common routine in

the driver.

I don't buy all this talk about special machinery, some 10 nor gates, the cost is in the noise - we are talking about cards with several ARM processors on chip.

They may not respond back under 50nsec.
How many cores would you like to dedicate and for how many PCI functions? And how they can be easily underutilized...
So, it is not a noise.

Circuity is the device abstraction that we keep instead of digging down on gates.

Also can you please explain how can it work without VM_EXIT? You have better insight to this than me..

Quoted 9 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

#13 lore raw
Quoted 18 lines · Parav Pandit
Quoted 12 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, September 3, 2024 3:16 PM

Quoted 6 lines · Zhu Lingshan
Quoted 6 lines
Quoted 3 lines

Are there any similar
constraining for other states like RESET or DRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.

DRIVER_OK does not suffer from it because it is async notification. A device may start slow after DRIVER_OK.
SUSPEND operation cannot rely on such async behavior.

Quoted 6 lines
Quoted 1 line

The driver can set suspend and re-read & wait. This is a common routine in

the driver.

I don't buy all this talk about special machinery, some 10 nor gates, the cost is in the noise - we are talking about cards with several ARM processors on chip.

Quoted 8 lines

They may not respond back under 50nsec.
How many cores would you like to dedicate and for how many PCI functions? And how they can be easily underutilized...
So, it is not a noise.

Circuity is the device abstraction that we keep instead of digging down on gates.

Also can you please explain how can it work without VM_EXIT? You have better insight to this than me..

I don't know which interface we are arguing about here, when there's a specific proposal, I can comment.

Quoted 10 lines
Quoted 9 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

#14 lore raw
Quoted 7 lines · Michael S. Tsirkin
Quoted 7 lines
Quoted 7 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status. If we need to reflect and control device status, we need something else.

#15 lore raw
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Michael S. Tsirkin
Quoted 7 lines
Quoted 7 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others. Anyhow driver know its own status.

Thanks

Quoted 3 lines

If we need
to reflect and control device status, we need something else.

#16 lore raw
Quoted 15 lines · Jason Wang
Quoted 14 lines · Michael S. Tsirkin
Quoted 12 lines · Michael S. Tsirkin
Quoted 10 lines
Quoted 9 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

#17 lore raw
Quoted 24 lines · Michael S. Tsirkin
Quoted 15 lines · Jason Wang
Quoted 14 lines · Michael S. Tsirkin
Quoted 12 lines · Michael S. Tsirkin
Quoted 10 lines
Quoted 9 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

Quoted 15 lines
Quoted 5 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#18 lore raw
Quoted 35 lines · Jason Wang
Quoted 17 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Thanks
Zhu Lingshan

Quoted 15 lines
Quoted 13 lines
Quoted 5 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#19 lore raw
Quoted 47 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 1 line · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>

Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to

control thing is elegant.

Quoted 26 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

Quoted 7 lines
Quoted 4 lines

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Sounds good to me as long as new functionality does not force the device to react in 50nsec. In the current state it forces the device so please upgrade the proposal as Michael guided. Thanks.

#20 lore raw
Quoted 49 lines · Parav Pandit
Quoted 49 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 1 line · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>

Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to

control thing is elegant.

Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

Quoted 5 lines
Quoted 3 lines

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Sounds good to me as long as new functionality does not force the device to react in 50nsec. In the current state it forces the device so please upgrade the proposal as Michael guided.

Trap and emulate is very often in virtualization, of course the sooner the better for emulation. But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

Quoted 1 line

Thanks.

#21 lore raw
Quoted 59 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PM

Quoted 53 lines · Parav Pandit
Quoted 53 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 1 line · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>

Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to

control thing is elegant.

Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 9 lines
Quoted 1 line

Sounds good to me as long as new functionality does not force the device

to react in 50nsec.

Quoted 1 line

In the current state it forces the device so please upgrade the proposal as

Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

Quoted 1 line

Thanks.

You are completely ignoring the comments.
Good luck..

#22 lore raw
Quoted 61 lines · Parav Pandit
Quoted 61 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PM

Quoted 46 lines · Parav Pandit
Quoted 46 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 1 line · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>

Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to

control thing is elegant.

Quoted 25 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Sounds good to me as long as new functionality does not force the device

to react in 50nsec.

Quoted 1 line

In the current state it forces the device so please upgrade the proposal as

Quoted 8 lines
Quoted 6 lines

Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

Quoted 1 line

Thanks.

You are completely ignoring the comments.
Good luck..

VM_EXIT and a VM_ENTRY work in a pair.

Quoted 1 line
#23 lore raw
Quoted 78 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:00 PM

Quoted 17 lines · Parav Pandit
Quoted 17 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PM

Quoted 12 lines · Parav Pandit
Quoted 12 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 2 lines · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>

Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for

debuggability.

Quoted 51 lines
Quoted 51 lines
Quoted 38 lines
Quoted 38 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to

control thing is elegant.

Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

Quoted 5 lines
Quoted 3 lines

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Sounds good to me as long as new functionality does not force the device

to react in 50nsec.

Quoted 2 lines

In the current state it forces the device so please upgrade the proposal as

Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

Thanks.

You are completely ignoring the comments.
Good luck..

VM_EXIT and a VM_ENTRY work in a pair.

Quoted 1 line

For hw pci pf where there is no virtualization used? How?

#24 lore raw
Quoted 77 lines · Parav Pandit
Quoted 77 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:00 PM

Quoted 17 lines · Parav Pandit
Quoted 17 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PM

Quoted 12 lines · Parav Pandit
Quoted 12 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 2 lines · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>

Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

But I don't like it that looking at the registers, one does
not know the device state. Hidden state is bad for

debuggability.

Quoted 51 lines
Quoted 51 lines
Quoted 38 lines
Quoted 38 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to

control thing is elegant.

Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

Quoted 5 lines
Quoted 3 lines

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Sounds good to me as long as new functionality does not force the device

to react in 50nsec.

Quoted 2 lines

In the current state it forces the device so please upgrade the proposal as

Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

Thanks.

You are completely ignoring the comments.
Good luck..

VM_EXIT and a VM_ENTRY work in a pair.

For hw pci pf where there is no virtualization used? How?

you bring the VM_EXIT problem to me, if without virtualization, there is no VM_EXIT.

Quoted 1 line
#25 lore raw
Quoted 78 lines · Zhu Lingshan
Quoted 77 lines · Parav Pandit
Quoted 77 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:00 PM

Quoted 17 lines · Parav Pandit
Quoted 17 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PM

Quoted 12 lines · Parav Pandit
Quoted 12 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 2 lines · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>

Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

But I don't like it that looking at the registers, one does
not know the device state. Hidden state is bad for

debuggability.

Quoted 51 lines
Quoted 51 lines
Quoted 38 lines
Quoted 38 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control
register to

control thing is elegant.

Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

Quoted 5 lines
Quoted 3 lines

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Sounds good to me as long as new functionality does not force the device

to react in 50nsec.

Quoted 2 lines

In the current state it forces the device so please upgrade the proposal as

Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

Thanks.

You are completely ignoring the comments.
Good luck..

VM_EXIT and a VM_ENTRY work in a pair.

For hw pci pf where there is no virtualization used? How?

you bring the VM_EXIT problem to me, if without virtualization, there is no VM_EXIT.

The problem Parav is raising is that the single bit approach effectively requires that the device handles updates to the status field synchronously. In a virtualization context with VM_EXIT/VM_ENTRY, this synchronous handling is easy. However, it's not necessarily easy with bare metal.

Even in the virtualization context, it's entirely possible to implement a virtualized device where the status field is plain memory that is periodically checked by the device. If the device only checks the status field once per second, then it's pretty clear that the single suspend bit approach won't work at all. I don't know why someone would implement a device like this, but it's not something that the virtio spec should rule out.

-David

#26 lore raw
Quoted 100 lines

From: David Stevens <[email protected]>
Sent: Thursday, September 5, 2024 2:12 PM

Quoted 79 lines · Zhu Lingshan
Quoted 78 lines · Parav Pandit
Quoted 78 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:00 PM

Quoted 18 lines · Parav Pandit
Quoted 18 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PM

Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 2 lines · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

But I don't like it that looking at the registers, one
does not know the device state. Hidden state is bad for

debuggability.

Quoted 51 lines
Quoted 51 lines
Quoted 38 lines
Quoted 38 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and
control register to

control thing is elegant.

Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

Quoted 5 lines
Quoted 3 lines

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Sounds good to me as long as new functionality does not force the device

to react in 50nsec.

Quoted 2 lines

In the current state it forces the device so please upgrade the proposal as

Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

Thanks.

You are completely ignoring the comments.
Good luck..

VM_EXIT and a VM_ENTRY work in a pair.

For hw pci pf where there is no virtualization used? How?

you bring the VM_EXIT problem to me, if without virtualization, there is no

VM_EXIT.

The problem Parav is raising is that the single bit approach effectively requires that the device handles updates to the status field synchronously. In a virtualization context with VM_EXIT/VM_ENTRY, this synchronous handling is easy. However, it's not necessarily easy with bare metal.

Quoted 5 lines

Even in the virtualization context, it's entirely possible to implement a virtualized device where the status field is plain memory that is periodically checked by the device. If the device only checks the status field once per second, then it's pretty clear that the single suspend bit approach won't work at all.

Thanks a lot David. You captured the summary precisely.

Quoted 4 lines

I don't know why someone would implement a device like this, but it's not something that the virtio spec should rule out.

-David

#27 lore raw
In reply to Zhu Lingshan
Quoted 52 lines · Zhu Lingshan
Quoted 49 lines · Parav Pandit
Quoted 49 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 1 line · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>

Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to

control thing is elegant.

Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Sounds good to me as long as new functionality does not force the device to react in 50nsec. In the current state it forces the device so please upgrade the proposal as Michael guided.

Trap and emulate is very often in virtualization, of course the sooner the better for emulation. But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

The spec should not assume devices are virtual, there are baremetal deployments of virtio. In such setups, there is no one to trap or
emulate: driver must do everything itself.

Quoted 1 line
Quoted 1 line

Thanks.

#28 lore raw
Quoted 55 lines · Michael S. Tsirkin
Quoted 54 lines · Zhu Lingshan
Quoted 52 lines · Parav Pandit
Quoted 52 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 1 line · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>

Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to

control thing is elegant.

Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Sounds good to me as long as new functionality does not force the device to react in 50nsec. In the current state it forces the device so please upgrade the proposal as Michael guided.

Trap and emulate is very often in virtualization, of course the sooner the better for emulation. But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

The spec should not assume devices are virtual, there are baremetal deployments of virtio. In such setups, there is no one to trap or
emulate: driver must do everything itself.

sure thing, I am also confused why VM_EXIT & VM_ENTRY and is a problem here...

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Thanks.

#29 lore raw
Quoted 71 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:01 PM

Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Zhu Lingshan
Quoted 12 lines · Parav Pandit
Quoted 12 lines

From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PM

Quoted 2 lines · Jason Wang

On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines

On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>

Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for

debuggability.

Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to

control thing is elegant.

Quoted 25 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Sounds good to me as long as new functionality does not force the

device to react in 50nsec.

Quoted 1 line
Quoted 1 line
Quoted 1 line

In the current state it forces the device so please upgrade the proposal

as Michael guided.

Quoted 1 line
Quoted 1 line

Trap and emulate is very often in virtualization, of course the sooner the

Quoted 8 lines

better for emulation.

Quoted 5 lines
Quoted 2 lines

But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time

The spec should not assume devices are virtual, there are baremetal deployments of virtio. In such setups, there is no one to trap or
emulate: driver must do everything itself.

sure thing, I am also confused why VM_EXIT & VM_ENTRY and is a problem here...

How does proposed VM_EXIT and VM_ENTRY work for basic PCI PF hw device?
How can one build sw without VM_EXIT and VM_ENTRY restriction if they want to?
Apart from other timing restrictions imposed that makes this very high complexity device for rare event. Please list down the assumptions and limitations so that it becomes clear the value of this bit..

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Thanks.

#30 lore raw
In reply to Zhu Lingshan
Quoted 46 lines · Zhu Lingshan
Quoted 35 lines · Jason Wang
Quoted 17 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

Quoted 5 lines

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 15 lines
Quoted 15 lines
Quoted 13 lines
Quoted 5 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#31 lore raw
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 7 lines
Quoted 2 lines

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

I guess the driver can reset the device even after DRIVER_OK. IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK. So we don't need a feature bit, we may just place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!

Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.

Thanks

Quoted 19 lines

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#32 lore raw
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

I guess the driver can reset the device even after DRIVER_OK.

in fact, unlike suspend - at any point at all.

Quoted 3 lines

IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK.

I don't know what kind of transition is there for DRIVER/DRIVER_OK. FEATURES_OK brings with it more issues, e.g. it can fail.

Quoted 3 lines

So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!

I have to say, this project is really ballooning out ;)

Quoted 5 lines

Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.

Thanks

or just a register for suspend transitions, worry about extending it later/

Quoted 19 lines
Quoted 19 lines

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#33 lore raw
Quoted 72 lines · Michael S. Tsirkin
Quoted 49 lines · Zhu Lingshan
Quoted 47 lines · Michael S. Tsirkin
Quoted 46 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

I guess the driver can reset the device even after DRIVER_OK.

in fact, unlike suspend - at any point at all.

Quoted 3 lines

IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK.

I don't know what kind of transition is there for DRIVER/DRIVER_OK. FEATURES_OK brings with it more issues, e.g. it can fail.

Quoted 3 lines

So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!

I have to say, this project is really ballooning out ;)

Quoted 7 lines
Quoted 5 lines

Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.

Thanks

or just a register for suspend transitions, worry about extending it later/

OK, will implement it in V8

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#34 lore raw
Quoted 9 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 2:40 PM

Quoted 3 lines
Quoted 1 line

Thanks

or just a register for suspend transitions, worry about extending it later/

OK, will implement it in V8

Thanks Zhu Lingshan.

#35 lore raw
In reply to Michael S. Tsirkin
Quoted 78 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

I guess the driver can reset the device even after DRIVER_OK.

in fact, unlike suspend - at any point at all.

Quoted 3 lines

IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK.

I don't know what kind of transition is there for DRIVER/DRIVER_OK. FEATURES_OK brings with it more issues, e.g. it can fail.

Quoted 3 lines

So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!

I have to say, this project is really ballooning out ;)

Quoted 8 lines
Quoted 5 lines

Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.

Thanks

or just a register for suspend transitions, worry about extending it later/

Is this for PCI only? Another question is that, if suspend needs that, reset would also want. Or it doesn't justify itself as reset needs to take longer time than reset.

Thanks

Quoted 21 lines
Quoted 20 lines
Quoted 20 lines

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#36 lore raw
Quoted 77 lines · Jason Wang
Quoted 76 lines · Michael S. Tsirkin
Quoted 49 lines · Zhu Lingshan
Quoted 47 lines · Michael S. Tsirkin
Quoted 46 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

I guess the driver can reset the device even after DRIVER_OK.

in fact, unlike suspend - at any point at all.

Quoted 3 lines

IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK.

I don't know what kind of transition is there for DRIVER/DRIVER_OK. FEATURES_OK brings with it more issues, e.g. it can fail.

Quoted 3 lines

So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!

I have to say, this project is really ballooning out ;)

Quoted 4 lines

Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Thanks

or just a register for suspend transitions, worry about extending it later/

Is this for PCI only? Another question is that, if suspend needs that, reset would also want. Or it doesn't justify itself as reset needs to take longer time than reset.

Ping Michael.

I think Jason's question is critical and we should address it before we proceed v8

Thanks

Quoted 22 lines

Thanks

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#37 lore raw
In reply to Jason Wang
Quoted 83 lines · Jason Wang
Quoted 82 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

I guess the driver can reset the device even after DRIVER_OK.

in fact, unlike suspend - at any point at all.

Quoted 3 lines

IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK.

I don't know what kind of transition is there for DRIVER/DRIVER_OK. FEATURES_OK brings with it more issues, e.g. it can fail.

Quoted 3 lines

So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!

I have to say, this project is really ballooning out ;)

Quoted 4 lines

Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thanks

or just a register for suspend transitions, worry about extending it later/

Is this for PCI only?

It's best to add it to all transports. Not rocket science at all.

Quoted 2 lines

Another question is that, if suspend needs that,
reset would also want.

I don't know what does "reset would also want" means. Unlike suspend, reset is not a special state, so it does not really require a spcial register to track that state.

Quoted 5 lines

Or it doesn't justify itself as reset needs to
take longer time than reset.

Thanks

Don't know what this means, either.

Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 20 lines

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#38 lore raw
Quoted 85 lines · Michael S. Tsirkin
Quoted 83 lines · Jason Wang
Quoted 82 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device
state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

I guess the driver can reset the device even after DRIVER_OK.

in fact, unlike suspend - at any point at all.

Quoted 3 lines

IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK.

I don't know what kind of transition is there for DRIVER/DRIVER_OK. FEATURES_OK brings with it more issues, e.g. it can fail.

Quoted 3 lines

So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!

I have to say, this project is really ballooning out ;)

Quoted 4 lines

Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thanks

or just a register for suspend transitions, worry about extending it later/

Is this for PCI only?

It's best to add it to all transports. Not rocket science at all.

I'm asking since I wonder if the "issue" exists only for PCI. If yes, solving the PCI transport (registers) issue at the basic facility level seems like a layer violation. And I still don't see why introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions. As mentioned before, re-read has been used for both FEAUTRES_OK and reset.

Quoted 5 lines
Quoted 3 lines

Another question is that, if suspend needs that,
reset would also want.

I don't know what does "reset would also want" means.

I meant we have already had a state transition like reset.

For example, you worries about:

suspending->suspended->resuming->resumed

But we've already had

setting_features -> feature_ok(or not) -> resetting -> reseted

Quoted 1 line

Unlike suspend, reset is not a special state,

It depends but we had a dedicated chapter in the basic facility to describe the reset state.

Quoted 3 lines

so it
does not really require a spcial register to track that state.

This seems to be PCI specific. Note that ccw has a dedicated command for reset:

#define CCW_CMD_VDEV_RESET 0x33
...
#define CCW_CMD_WRITE_STATUS 0x31

Quoted 8 lines
Quoted 6 lines

Or it doesn't justify itself as reset needs to
take longer time than reset.

Thanks

Don't know what this means, either.

I meant

1) driver knows its own status and it can read device status
2) reset may take much longer time than suspend

If we really need a new status register (I don't see why so far), we should not only suspend.

Thanks

Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#39 lore raw
Quoted 91 lines · Jason Wang
Quoted 90 lines · Michael S. Tsirkin
Quoted 88 lines · Jason Wang
Quoted 86 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines

But I don't like it that looking at the registers, one does not know the device
state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.

No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

I guess the driver can reset the device even after DRIVER_OK.

in fact, unlike suspend - at any point at all.

Quoted 3 lines

IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK.

I don't know what kind of transition is there for DRIVER/DRIVER_OK. FEATURES_OK brings with it more issues, e.g. it can fail.

Quoted 3 lines

So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!

I have to say, this project is really ballooning out ;)

Quoted 5 lines

Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.

Thanks

or just a register for suspend transitions, worry about extending it later/

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Is this for PCI only?

It's best to add it to all transports. Not rocket science at all.

I'm asking since I wonder if the "issue" exists only for PCI. If yes,
solving the PCI transport (registers) issue at the basic facility level seems like a layer violation.

I don't think so. Suspend is fundamentally a slow operation for any transport.

Quoted 3 lines

And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.

reset is not a state.

Quoted 3 lines

As
mentioned before, re-read has been used for both FEAUTRES_OK and reset.

FEAUTRES_OK is driver state, so can take place immediately

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Another question is that, if suspend needs that,
reset would also want.

I don't know what does "reset would also want" means.

I meant we have already had a state transition like reset.

Quoted 7 lines

For example, you worries about:

suspending->suspended->resuming->resumed

But we've already had

setting_features -> feature_ok(or not) -> resetting -> reseted

features ok just has to validate the bitmap, so it can take place immediately. no intermediate state.

Quoted 4 lines
Quoted 1 line

Unlike suspend, reset is not a special state,

It depends but we had a dedicated chapter in the basic facility to describe the reset state.

this is two different meanings of the same word. reset is not a special state, device operates normally.

Quoted 9 lines
Quoted 3 lines

so it
does not really require a spcial register to track that state.

This seems to be PCI specific. Note that ccw has a dedicated command for reset:

#define CCW_CMD_VDEV_RESET 0x33
...
#define CCW_CMD_WRITE_STATUS 0x31

maybe reusing state 0 to reset was a bad idea.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Or it doesn't justify itself as reset needs to
take longer time than reset.

Thanks

Quoted 6 lines
Quoted 1 line

Don't know what this means, either.

I meant

1) driver knows its own status and it can read device status
2) reset may take much longer time than suspend

that is why driver re-reads status to check for reset.

Quoted 5 lines

If we really need a new status register (I don't see why so far), we should not only suspend.

Thanks

it's not a new status register, it's a suspend register used for PM.

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#40 lore raw
Quoted 97 lines · Michael S. Tsirkin
Quoted 95 lines · Jason Wang
Quoted 93 lines · Michael S. Tsirkin
Quoted 83 lines · Jason Wang
Quoted 82 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin

But I don't like it that looking at the registers, one does not know the device
state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

we could steal 2 bits from status but it seems a bit much.

This is why letting the status tell the status and control register to control thing is elegant.
No argument here.

Or, to be more precise, our status is driver status.

It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.

Quoted 9 lines

The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.

We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:

        while (vp_modern_get_status(mdev))
                msleep(1);

We can do the same for suspending.

Quoted 8 lines

The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.

But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?

I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Thanks
Zhu Lingshan

If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.

I guess the driver can reset the device even after DRIVER_OK.

in fact, unlike suspend - at any point at all.

Quoted 3 lines

IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK.

I don't know what kind of transition is there for DRIVER/DRIVER_OK. FEATURES_OK brings with it more issues, e.g. it can fail.

Quoted 3 lines

So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!

I have to say, this project is really ballooning out ;)

Quoted 4 lines

Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thanks

or just a register for suspend transitions, worry about extending it later/

Is this for PCI only?

It's best to add it to all transports. Not rocket science at all.

Quoted 8 lines
Quoted 4 lines

I'm asking since I wonder if the "issue" exists only for PCI. If yes,
solving the PCI transport (registers) issue at the basic facility level seems like a layer violation.

I don't think so. Suspend is fundamentally a slow operation for any transport.

Probably, but how such slowness makes a new register a must? And my understanding is, reset should be even slower than suspend.

Quoted 7 lines
Quoted 5 lines

And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.

reset is not a state.

Well, I'm not sure I get this. Each value read from status should represent a device state. And 0 is a valid status which means device is resetted:

"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""

Quoted 6 lines
Quoted 4 lines

As
mentioned before, re-read has been used for both FEAUTRES_OK and reset.

FEAUTRES_OK is driver state, so can take place immediately

I'm not sure how to define immediately, but suspend could be very fast depending on the implementation.

Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Quoted 4 lines

Another question is that, if suspend needs that,
reset would also want.

I don't know what does "reset would also want" means.

I meant we have already had a state transition like reset.

For example, you worries about:

suspending->suspended->resuming->resumed

Quoted 7 lines
Quoted 3 lines

But we've already had

setting_features -> feature_ok(or not) -> resetting -> reseted

features ok just has to validate the bitmap, so it can take place immediately. no intermediate state.

True but there's an intermediate state between resetting and reseted. When is the state when the driver has written 0 to status but not read 0 from status.

Quoted 14 lines
Quoted 5 lines
Quoted 2 lines

Unlike suspend, reset is not a special state,

It depends but we had a dedicated chapter in the basic facility to describe the reset state.

this is two different meanings of the same word. reset is not a special state, device operates normally.

Quoted 4 lines
Quoted 3 lines

so it
does not really require a spcial register to track that state.

Quoted 8 lines
Quoted 5 lines

This seems to be PCI specific. Note that ccw has a dedicated command for reset:

#define CCW_CMD_VDEV_RESET 0x33
...
#define CCW_CMD_WRITE_STATUS 0x31

maybe reusing state 0 to reset was a bad idea.

Not sure, but it's too late to change that.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Or it doesn't justify itself as reset needs to
take longer time than reset.

Thanks

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Don't know what this means, either.

I meant

1) driver knows its own status and it can read device status
2) reset may take much longer time than suspend

that is why driver re-reads status to check for reset.

Yes, but this is exactly how suspend is supposed to work for this patch.

Reset has one intermediate state and drivers need to poll for the completion. Suspending has two intermediate states, drivers need to poll the completion for them.

Quoted 9 lines
Quoted 7 lines

If we really need a new status register (I don't see why so far), we should not only suspend.

Thanks

it's not a new status register, it's a suspend register used for PM.

I think power management should belong to the transport layer, not the virtio layer.

Thanks

Quoted 26 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 23 lines

Regrettably, reset will have to still be supported through clearing status just because this is always there.

Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines

Anyhow driver know
its own status.

Thanks

indeed, the status register is there to inform the device about the driver status.

Quoted 3 lines
Quoted 3 lines

If we need
to reflect and control device status, we need something else.

Thanks

#41 lore raw
Quoted 9 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines

And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.

reset is not a state.

Well, I'm not sure I get this. Each value read from status should represent a device state.

What makes you say so? status represents driver state, not device state. In fact, driver writes there and does not double check with the sole exception of reset.

Quoted 7 lines

And 0 is a valid status which means device
is resetted:

"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""

It means all registers have their reset values, but it is not a special state in that device behaves normally in this state, it does not need to be handles in any special way.

Contrast with suspend where device does not respond and must be first resumed.

Signature · 1 line
MST
#42 lore raw
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines

And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.

reset is not a state.

Quoted 7 lines
Quoted 2 lines

Well, I'm not sure I get this. Each value read from status should represent a device state.

What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.

For example, spec said

"""

2.1 Device Status Field

During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.

The device status field provides a simple low-level indication of the completed steps of this sequence. It’s most useful to imagine it hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):

"""

Driver knows its own state, so there's no need for the driver to read it from the device.

Quoted 10 lines
Quoted 8 lines

And 0 is a valid status which means device
is resetted:

"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""

It means all registers have their reset values,

This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.

Quoted 6 lines

but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.

Contrast with suspend where device does not respond and must be first resumed.

I think not. Driver can choose to reset the device when the device is suspended.

Thanks

Quoted 7 lines

--
MST

#43 lore raw
Quoted 19 lines · Jason Wang
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines

And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.

reset is not a state.

Quoted 7 lines
Quoted 2 lines

Well, I'm not sure I get this. Each value read from status should represent a device state.

What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.

Quoted 5 lines

For example, spec said

"""

2.1 Device Status Field

yes, this is confusingly written

Quoted 5 lines

During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.

The device status field provides a simple low-level indication of the completed steps of this sequence.

And note the sequence is driver sequence.

Quoted 5 lines

It’s most useful to imagine it
hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):

Quoted 7 lines

"""

Driver knows its own state, so there's no need for the driver to read it from the device.

That is why it never reads status.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

And 0 is a valid status which means device
is resetted:

"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""

Quoted 4 lines
Quoted 1 line

It means all registers have their reset values,

This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.

yes but otherwise, all registers respond normally device is assumed to be fully operational, and it is assumed that no time at all is needed to get from that to DRIVER.

Quoted 10 lines
Quoted 6 lines

but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.

Contrast with suspend where device does not respond and must be first resumed.

I think not. Driver can choose to reset the device when the device is suspended.

Thanks

Yes, reset should take device out of suspend.

Quoted 7 lines
Quoted 7 lines

--
MST

#44 lore raw
Quoted 21 lines · Michael S. Tsirkin
Quoted 19 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 9 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines

And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.

reset is not a state.

Well, I'm not sure I get this. Each value read from status should represent a device state.

What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.

Quoted 7 lines
Quoted 5 lines

For example, spec said

"""

2.1 Device Status Field

yes, this is confusingly written

Anything makes you think it is confusing?

Quoted 8 lines
Quoted 6 lines

During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.

The device status field provides a simple low-level indication of the completed steps of this sequence.

And note the sequence is driver sequence.

Right but it means the driver needs to know the status of the device in order to proceed.

Quoted 11 lines
Quoted 7 lines

It’s most useful to imagine it
hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):

"""

Quoted 4 lines
Quoted 2 lines

Driver knows its own state, so there's no need for the driver to read it from the device.

That is why it never reads status.

Well, looking at the current Linux drivers there're plenty (in virtio core, there could be other in the transport)

drivers/virtio/virtio.c: return sysfs_emit(buf, "0x%08x\n",
dev->config->get_status(dev));
drivers/virtio/virtio.c: dev->config->set_status(dev,
dev->config->get_status(dev) | status);

drivers/virtio/virtio.c:        status = dev->config->get_status(dev);
drivers/virtio/virtio.c:        if (!(dev->config->get_status(dev) &
VIRTIO_CONFIG_S_DRIVER_OK))
drivers/virtio/virtio.c:        WARN_ON_ONCE(dev->config->get_status(dev));
drivers/virtio/virtio.c:        dev->failed =

dev->config->get_status(dev) & VIRTIO_CONFIG_S_FAILED;
drivers/virtio/virtio.c: if (!(dev->config->get_status(dev) &
VIRTIO_CONFIG_S_DRIVER_OK))

Quoted 26 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

And 0 is a valid status which means device
is resetted:

"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""

Quoted 4 lines
Quoted 1 line

It means all registers have their reset values,

This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.

yes but otherwise, all registers respond normally device is assumed to be fully operational, and it is assumed that no time at all is needed to get from that to DRIVER.

Quoted 4 lines
Quoted 4 lines

but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Contrast with suspend where device does not respond and must be first resumed.

I think not. Driver can choose to reset the device when the device is suspended.

Thanks

Yes, reset should take device out of suspend.

Thanks

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

--
MST

#45 lore raw
Quoted 24 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Jason Wang
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines

And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.

reset is not a state.

Quoted 7 lines
Quoted 2 lines

Well, I'm not sure I get this. Each value read from status should represent a device state.

What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.

For example, spec said

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

"""

2.1 Device Status Field

yes, this is confusingly written

Anything makes you think it is confusing?

It says Device Status but most of it reflects the driver probe progress. Compare to pci RO status and RW control.

Quoted 11 lines
Quoted 8 lines
Quoted 6 lines

During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.

The device status field provides a simple low-level indication of the completed steps of this sequence.

And note the sequence is driver sequence.

Right but it means the driver needs to know the status of the device in order to proceed.

again this is not device status at all.

Quoted 23 lines
Quoted 11 lines
Quoted 7 lines

It’s most useful to imagine it
hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):

"""

Quoted 4 lines
Quoted 2 lines

Driver knows its own state, so there's no need for the driver to read it from the device.

That is why it never reads status.

Well, looking at the current Linux drivers there're plenty (in virtio core, there could be other in the transport)

drivers/virtio/virtio.c: return sysfs_emit(buf, "0x%08x\n",
dev->config->get_status(dev));
drivers/virtio/virtio.c: dev->config->set_status(dev,
dev->config->get_status(dev) | status);

Quoted 8 lines
drivers/virtio/virtio.c:        status = dev->config->get_status(dev);
drivers/virtio/virtio.c:        if (!(dev->config->get_status(dev) &
VIRTIO_CONFIG_S_DRIVER_OK))
drivers/virtio/virtio.c:        WARN_ON_ONCE(dev->config->get_status(dev));
drivers/virtio/virtio.c:        dev->failed =

dev->config->get_status(dev) & VIRTIO_CONFIG_S_FAILED;
drivers/virtio/virtio.c: if (!(dev->config->get_status(dev) &
VIRTIO_CONFIG_S_DRIVER_OK))

It's done like this for robustness and simplicity. But we could easily cache it in the driver on write.

Quoted 46 lines
Quoted 26 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

And 0 is a valid status which means device
is resetted:

"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""

Quoted 4 lines
Quoted 1 line

It means all registers have their reset values,

This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.

yes but otherwise, all registers respond normally device is assumed to be fully operational, and it is assumed that no time at all is needed to get from that to DRIVER.

Quoted 4 lines
Quoted 4 lines

but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Contrast with suspend where device does not respond and must be first resumed.

I think not. Driver can choose to reset the device when the device is suspended.

Thanks

Yes, reset should take device out of suspend.

Thanks

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

--
MST

#46 lore raw
Quoted 30 lines · Michael S. Tsirkin
Quoted 28 lines · Jason Wang
Quoted 27 lines · Michael S. Tsirkin
Quoted 24 lines · Jason Wang
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines

And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.

reset is not a state.

Quoted 7 lines
Quoted 2 lines

Well, I'm not sure I get this. Each value read from status should represent a device state.

What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.

For example, spec said

"""

2.1 Device Status Field

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

yes, this is confusingly written

Anything makes you think it is confusing?

It says Device Status but most of it reflects the driver probe progress. Compare to pci RO status and RW control.

My understanding is that

1) any value that is read from the device status reflects a device status.
2) any value write from the driver for the device status reflects a device status that driver want to go

Here are some examples:

1) DRIVER_OK:

"""
The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK.
"""

It's a device status that "Device knows driver is OK"

2) FEATURES_OK

"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the FEATURES_OK device status bit when the driver writes it. """

Device tells the driver that it doesn't support some of the features.

3) NEEDS_RESET:

"""
The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.
"""

This is a device status but not a driver.

And usually, device status is a bitmask of several bits of status defined in the spec.

Quoted 49 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

The device status field provides a simple low-level indication of the completed steps of this sequence.

And note the sequence is driver sequence.

Right but it means the driver needs to know the status of the device in order to proceed.

again this is not device status at all.

Quoted 23 lines
Quoted 11 lines
Quoted 7 lines

It’s most useful to imagine it
hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):

"""

Quoted 4 lines
Quoted 2 lines

Driver knows its own state, so there's no need for the driver to read it from the device.

That is why it never reads status.

Well, looking at the current Linux drivers there're plenty (in virtio core, there could be other in the transport)

drivers/virtio/virtio.c: return sysfs_emit(buf, "0x%08x\n",
dev->config->get_status(dev));
drivers/virtio/virtio.c: dev->config->set_status(dev,
dev->config->get_status(dev) | status);

Quoted 8 lines
drivers/virtio/virtio.c:        status = dev->config->get_status(dev);
drivers/virtio/virtio.c:        if (!(dev->config->get_status(dev) &
VIRTIO_CONFIG_S_DRIVER_OK))
drivers/virtio/virtio.c:        WARN_ON_ONCE(dev->config->get_status(dev));
drivers/virtio/virtio.c:        dev->failed =

dev->config->get_status(dev) & VIRTIO_CONFIG_S_FAILED;
drivers/virtio/virtio.c: if (!(dev->config->get_status(dev) &
VIRTIO_CONFIG_S_DRIVER_OK))

Quoted 2 lines

It's done like this for robustness and simplicity. But we could easily cache it in the driver on write.

Probably not, one example:

        virtio_add_status(dev, VIRTIO_CONFIG_S_FEATURES_OK);
        status = dev->config->get_status(dev);
        if (!(status & VIRTIO_CONFIG_S_FEATURES_OK)) {
                dev_err(&dev->dev, "virtio: device refuses features: %x\n",
                        status);
                return -ENODEV;
        }

Devices may refuse the features.

Thanks

Quoted 49 lines
Quoted 48 lines
Quoted 28 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

And 0 is a valid status which means device
is resetted:

"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""

Quoted 4 lines
Quoted 1 line

It means all registers have their reset values,

This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.

yes but otherwise, all registers respond normally device is assumed to be fully operational, and it is assumed that no time at all is needed to get from that to DRIVER.

Quoted 4 lines
Quoted 4 lines

but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Contrast with suspend where device does not respond and must be first resumed.

I think not. Driver can choose to reset the device when the device is suspended.

Thanks

Yes, reset should take device out of suspend.

Thanks

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

--
MST

#47 lore raw
In reply to Zhu Lingshan
Quoted 259 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, September 3, 2024 2:36 PM

Quoted 25 lines · Parav Pandit
Quoted 25 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:02 AM

Quoted 20 lines · Parav Pandit
Quoted 20 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PM

Quoted 5 lines · Parav Pandit
Quoted 3 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AM

I removed the unreachable intel email id as every single email is bouncing

from it.

Quoted 2 lines

Please consider dropping that email from v8 as it will cause all reviewers

email to bounce.

Quoted 7 lines
Quoted 7 lines · Parav Pandit
Quoted 6 lines
Quoted 6 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 2 lines · Parav Pandit

Hi Lingshan, David,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit

VIRTIO_F_SUSPEND

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

which indicating whether the device support SUSPEND.

---
 content.tex | 75

++++++++++++++++++++++++++++++++++++++++++++++-

Quoted 17 lines
Quoted 17 lines

----

Quoted 16 lines
Quoted 16 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the

device.

Quoted 97 lines
Quoted 97 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

- disallow access to config space while suspended.

content.tex index 0a62dce..2d1bee8 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be
loadable

modules.

Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
   \end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,
- and it has given up on the device. This could be an internal
- error, or the driver didn't like the device for some reason, or

content.tex +3 -1
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and
+ready to
+  drive the device.
 
 \item[FEATURES_OK (8)] Indicates that the driver has

acknowledged all

Quoted 7 lines
Quoted 7 lines

the

Quoted 6 lines
Quoted 6 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated,

indicates

Quoted 23 lines
Quoted 23 lines
Quoted 4 lines

content.tex

Quoted 4 lines
content.tex +2 -0
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 11 lines
Quoted 11 lines

an error from which it can't recover.

content.tex +8 -0
+
+\item[FAILED (128)] Indicates that something went wrong in
+the guest,
+  and it has given up on the device. This could be an
+internal
+  error, or the driver didn't like the device for some
+reason, or
+  even a fatal error during device operation.
 \end{description}
Quoted 6 lines
Quoted 6 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

-The driver MUST NOT clear a
-\field{device status} bit. If the driver sets the FAILED bit,

content.tex +4 -0
+The driver MUST NOT clear a \field{device status} bit other
+than SUSPEND except when setting \field{device status} to 0
+as a transport-specific way to initiate a reset. If the
+driver sets the FAILED bit,
 the driver MUST later reset the device before attempting to

re-

initialize.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The driver SHOULD NOT rely on completion of operations of a

@@

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities +of a Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

content.tex \section{Device

Quoted 2 lines

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device

Cleanup}\label{sec:General

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

content.tex

Quoted 11 lines

+\section{Device Suspend}\label{sec:General Initialization

content.tex +10 -0
+And Device Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the
+SUSPEND bit in \field{device status} to suspend a device,
+and can clear the SUSPEND bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set

or

Quoted 18 lines
Quoted 18 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

content.tex

Quoted 3 lines

VIRTIO_F_SUSPEND is not negotiated.

content.tex +2 -0
+
+Once the driver sets SUSPEND to \field{device status} of the

device:

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines

+\begin{itemize}
+\item The driver MUST re-read \field{device status} to +verify whether the
SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to +the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

operations/sec, where N is roughly in range of 10 or 100) per

device?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is

often.

Quoted 26 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

I think we don't have an intention of the frequency in the spec.

Sure.

Quoted 8 lines
Quoted 1 line

The spec only provides generic mechanisms and interfaces.

Sure.

Quoted 2 lines

Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.

As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <

50nsec of time.

Quoted 2 lines

The suspend operation for a net device to store hundreds of queues, RSS

table, flow filters, takes plenty of time (at least more than 50nsec :) ).

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Similarly for the GPU to store some MBs of memory takes more than 50nsec

of time, for example to store in a file for a software-based GPU device.

Quoted 1 line

So a device cannot respond back suspend=true in next 50nsec time.

It's OK for the device to take longer time to respond, the driver simply re-reads device status.

Sure, the issue is, when the driver re-reads, the device must present

suspend=false within 50nsec.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

(because device didn't suspend it yet).

As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly

differently.
why there is a constraint condition of the time?

Because this is what driver does as you explained in [1] indicating "we must

be able to handle ..."

Quoted 3 lines

[1]

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore

Quoted 1 line

.kernel.org%2Fvirtio-comment%2Fc4d5eed3-774b-4d35-a007-

f9dff28ce884%40

Quoted 1 line

amd.com%2FT%2F%23m6f081f96ef9dcea29c64a88b633eb21d50e8c410&dat a=05%7C0

Quoted 1 line

2%7Cparav%40nvidia.com%7C4a248d6982514660015808dccbf79c55%7C430 83d1572

Quoted 1 line

7340c1b7db39efd9ccc17a%7C0%7C0%7C638609511605195425%7CUnknown %7CTWFpbG

Quoted 1 line
Quoted 8 lines

Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 Mn0%

Quoted 1 line

3D%7C0%7C%7C%7C&sdata=Gj4oG8Z1O7wEl%2B7Mbud%2BbP31aMDMAFA H9ElHJSvBTUA%

Quoted 1 line

3D&reserved=0

The sentence is "we must be able to handle the scenario in which SUSPENDING is often", where do you see any constraint conditions of time?

If the device takes X usec, how can the device inform that, it suspending in progress without flipping the polarity?

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Are there any similar
constraining for other states like RESET or DRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.

DRIVER_OK does not suffer from it because it is async notification. A device may start slow after DRIVER_OK.
SUSPEND operation cannot rely on such async behavior.

The driver can set suspend and re-read & wait. This is a common routine in the driver.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.

Here a new functionality is being proposed and it has a chance for efficient

device implementation.

Quoted 1 line

Therefore the request is to improve it.

sure, as long as we confirm this new register apply for all device status transitions, not only for SUSPEND.

Alright, please do it as you suggested here.

Quoted 36 lines
Quoted 36 lines
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

More below.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the

software.

Quoted 1 line

Do you agree?

The device emulator does not know anything about whether trapped

or

Quoted 7 lines
Quoted 7 lines

not.

Quoted 6 lines
Quoted 6 lines

Trapping and Emulation is a hypervisor thing.

If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Right. So a software wants to implement device_status as pure MMIO

writes. (and not VM_EXIT).
This is not always true, there can be VM_EXIT of pure emulated devices.

I am not denying that VM_EXIT can/cannot be there. I am saying, the proposal forces VM_EXIT based approach in my

understanding of this patch.

[MARKER_1]

Quoted 4 lines
Quoted 1 line

If that is not true, may be can you please explain how this can be

implemented without VM_EXIT?

Quoted 2 lines

We should have an interface that can be done with and without VM_EXIT

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

method at least for any new additions.
VM_EXIT is out of spec, it is a hypervisor and the processor thing.

You keep repeating VM_EXIST is out of spec.
I already replied at [MARKER_1], sure it is out of spec, the current approach

forces one to do VM_EXIT based approach.

Quoted 1 line

And if not, please explain, how can it be achieved?

It is not the current solution force the hypervisor & guest to perform VM_EXIT.

I read it few times but couldn't parse it.
Can you please explain how your proposal works WITHOUT VM_EXIT? If not, you can say, that current proposal is limited to force VM_EXIT. Please explain.

Quoted 2 lines

In basic virtualization, any access to hypervisor emulated HW registers would be Traped and Emulated, means a VM_EXIT and a VM_ENTRY.

That was not my question, my question is how to achieve without VM_EXIT? You are suggesting a spec that forces VM_EXIT.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

In non-pass-through case, any register access are sensitive and will trigger VM_EXIT. Like RESET or DRIVER_OK needs to access device_status, nothing different.

I don't think you understood the point.
Let me repeat, The question is, if the device implementation wants to

achieve the functionality without VM_EXIT, what is the way? Map the register to guest address space, out of the spec.

And which component will flip the bit of suspend before driver reads it to indicate that device is still_suspending?

Quoted 22 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

The
HW registers are sensitive resource and any access to them need to be trapped and emulated.

This does not apply to PCI PFs and VFs which are HW devices (mainly

PFs).

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

so this trap + emulation is narrow view that we better avoid.

This is how *basic* virtualization work, once access sensitive resource,

trap it.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

If you think this is the way forward, you should put forward in patch as

MUST requirement.

Quoted 2 lines

and that does not look right to me.
I hope you also don't mean to force this method to device

Quoted 8 lines
Quoted 6 lines

implementations.

Quoted 1 line

Right?

Again, VM_EXIT is a hypervisor thing,  out of spec. Whether there is a VM_EXIT when setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different from DRIVER_OK.

Please avoid repeating the point that VM_EXIT is hypervosor thing. No one asked to put this in spec. Please re-read [MARKER1]. You probably

Quoted 8 lines

missed that.

Quoted 2 lines

SUSPEND is different than DRIVER_OK. I explained the timing constraints

and the required circuitry needed to fulfill the proposal.

Quoted 1 line

And with the additional register, such complicated circuitry can be easily

avoided.
Please read QEMU code, that can help you find how the trap - emulate mechanism work for guest when it tries to access sensitive resources.

I am not suggesting trap and emulation. Please come out of this looping thoughts.

I repeat my question, please explain how does suggested solution works without VM_EXIT? From your response, I derive it cannot. Can you please confirm?

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Means, if your virtualization needs to trap SUSPEND, it also needs to trap DRIVER_OK, and don't assume DRIVER_OK is faster than SUSPEND.

DRIVER_OK is by law of physics is faster than SUSPEND because it does not

demand the driver of reading back.

Quoted 1 line

There is no driver side loop to check if the device accepted DRIVER_OK or

not.

Quoted 1 line

Agree?

Why do you assume so?

I am not assuming. It is the current spec. :)
Please read the device initialization sequence, that does NOT say that driver must READ DRIVER_OK. Hence by law of physics it is faster.

Quoted 3 lines

In some vendor implementation, DRIVER_OK can be
slow. Can DRIVER_OK implement a deferred initialization even deferred resource allocation? Does the spec say that DRIVER_OK must be faster?

No. it does not. But your suspend proposal says so that device must respond back before the next read arrives from the driver.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
Quoted 3 lines

And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return

suspend=true right after it was written.

Quoted 4 lines

A MMIO read will read it back, as suspend=true.

An alternative would be, to forward CPU loads and CPU stores to different

address.

Quoted 4 lines

However, this does not work for the hw based devices.

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 2 lines

In this example, the device cannot build special circuitry to answer

Quoted 8 lines
Quoted 4 lines

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation. why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.

When driver wrote, it wrote suspend=true, And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec

or less to answer suspend=false.

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

And even worst, it needs to suspend=true when unsuspending within

50nsec when resuming is ongoing.
again, there is not a 50nsec constraining and please take a reference of how DRIVER_OK work with virtualization.

There is. The device is expected to return back the desired value to indicate

driver that suspend is ongoing.

Quoted 1 line

It is different than DRIVER_OK.

where does the spec or any code say 50nsec?

Your spec and your previous comment implies that when device is busy suspending, it must returned the toggled value than what driver set it. Isn't it? It means device needs to react before the next read arrives from the driver, isn't it?
I took that read as 50nsec example as you explained previously that driver will read very fast.

Quoted 137 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful comments

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Unfortunately, not. His comment was that it is not specific to suspend. But here we are introducing a new interface and functionality that does not

need to suffer or follow anything that may not be efficient.

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor

Quoted 1 line
Quoted 1 line

%2F&data=05%7C02%7Cparav%40nvidia.com%7C4a248d6982514660015808 dccbf79

Quoted 1 line
Quoted 1 line

c55%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6386095116052 07133%7

Quoted 1 line
Quoted 1 line

CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB TiI6Ik

Quoted 1 line
Quoted 1 line

1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VGOfVe0deSDilzufY7NH gU5RmciNh

Quoted 4 lines
Quoted 4 lines

dXMMQiVUSiP%2BKM%3D&reserved=0

Quoted 1 line
Quoted 1 line

e.kernel.org%2Fvirtio-comment%2F20240612082055-mutt-send-email-

mst%40

kernel.org%2FT%2F%23mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9&dat

Quoted 2 lines
Quoted 2 lines

a=05

%7C02%7Cparav%40nvidia.com%7C619c82b60b824ca03f8808dcc89beb4a%7

Quoted 2 lines
Quoted 2 lines

C43083

d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638605819247985989%7CUn

Quoted 2 lines
Quoted 2 lines

known%7C

TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL

Quoted 2 lines
Quoted 2 lines

CJXV

CI6Mn0%3D%7C0%7C%7C%7C&sdata=VqmJmqt3k5tf3x4ihjE5Bd7u59GadOn

Quoted 27 lines
Quoted 27 lines

OaOfJ5lvG

Quoted 26 lines
Quoted 26 lines

1DE%3D&reserved=0

Quoted 25 lines
Quoted 25 lines

this is out of the spec anyway.

Thanks
Zhu Lingshan

Quoted 12 lines
Quoted 12 lines

Thanks
Zhu Lingshan

Quoted 10 lines
Quoted 10 lines

+\devicenormative{\subsection}{Device Suspend}{General

content.tex +3 -0
+Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space
+while suspended, except for \field{device status} if it is
+part of the Configuration

Space.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines
content.tex +8 -0
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in its
+Configuration Space while suspended.
+
+If changes occur in the Configuration Space while the
+SUSPEND bit is set, the device MUST NOT send any
+configuration change

notifications.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

content.tex

Quoted 5 lines
content.tex +2 -0
+Instead, the device MUST send the notification after the
+SUSPEND bit has

been cleared.

content.tex +2 -0
+
+When the driver sets SUSPEND, the device MUST either

suspend

Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

content.tex

Quoted 5 lines

+itself or set
DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver
+clears SUSPEND, the device MUST either resume normal

operation

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

content.tex

Quoted 4 lines

+or set
DEVICE_NEEDS_RESET.

content.tex +2 -0
+
+When the driver sets SUSPEND, the device SHOULD perform

the

Quoted 36 lines
Quoted 36 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

+following actions before presenting that
the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item
+Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio

Transport Options}

Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that
+ the driver

can

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
content.tex +3 -0
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device
+ Status

Field}.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+
\end{description}

\drivernormative{\section}{Reserved Feature Bits}{Reserved
Feature Bits}
--
2.45.2

#48 lore raw
Quoted 256 lines · Parav Pandit
Quoted 256 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, September 3, 2024 2:36 PM

Quoted 24 lines · Parav Pandit
Quoted 24 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:02 AM

Quoted 20 lines · Parav Pandit
Quoted 20 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PM

Quoted 5 lines · Parav Pandit
Quoted 3 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AM

I removed the unreachable intel email id as every single email is bouncing

from it.

Quoted 2 lines

Please consider dropping that email from v8 as it will cause all reviewers

email to bounce.

Quoted 7 lines
Quoted 7 lines · Parav Pandit
Quoted 6 lines
Quoted 6 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AM

Quoted 2 lines · Parav Pandit

Hi Lingshan, David,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PM

This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.

This commit also introduce a new feature bit

VIRTIO_F_SUSPEND

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

which indicating whether the device support SUSPEND.

---
 content.tex | 75

++++++++++++++++++++++++++++++++++++++++++++++-

Quoted 17 lines
Quoted 17 lines

----

Quoted 16 lines
Quoted 16 lines

--

 1 file changed, 65 insertions(+), 10 deletions(-)

Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.

Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs

Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the

device.

Quoted 97 lines
Quoted 97 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

- disallow access to config space while suspended.

content.tex index 0a62dce..2d1bee8 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -36,19 +36,22 @@ \section{\field{Device Status}

Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be
loadable

modules.

Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
   \end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,
- and it has given up on the device. This could be an internal
- error, or the driver didn't like the device for some reason, or

content.tex +3 -1
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and
+ready to
+  drive the device.
 
 \item[FEATURES_OK (8)] Indicates that the driver has

acknowledged all

Quoted 7 lines
Quoted 7 lines

the

Quoted 6 lines
Quoted 6 lines

features it understands, and feature negotiation is complete.

-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated,

indicates

Quoted 23 lines
Quoted 23 lines
Quoted 4 lines

content.tex

Quoted 4 lines
content.tex +2 -0
+that the
+  device has been suspended by the driver.
 
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has

experienced

Quoted 11 lines
Quoted 11 lines

an error from which it can't recover.

content.tex +8 -0
+
+\item[FAILED (128)] Indicates that something went wrong in
+the guest,
+  and it has given up on the device. This could be an
+internal
+  error, or the driver didn't like the device for some
+reason, or
+  even a fatal error during device operation.
 \end{description}
Quoted 6 lines
Quoted 6 lines

The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device

Initialization}.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

-The driver MUST NOT clear a
-\field{device status} bit. If the driver sets the FAILED bit,

content.tex +4 -0
+The driver MUST NOT clear a \field{device status} bit other
+than SUSPEND except when setting \field{device status} to 0
+as a transport-specific way to initiate a reset. If the
+driver sets the FAILED bit,
 the driver MUST later reset the device before attempting to

re-

initialize.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The driver SHOULD NOT rely on completion of operations of a

@@

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities +of a Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type

-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms

-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

content.tex \section{Device

Quoted 2 lines

\begin{note}

content.tex +0 -0
@@ -629,6 +633,53 @@ \section{Device

Cleanup}\label{sec:General

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

content.tex

Quoted 11 lines

+\section{Device Suspend}\label{sec:General Initialization

content.tex +10 -0
+And Device Operation / Device Suspend}
+
+When VIRTIO_F_SUSPEND is negotiated, the driver can set the
+SUSPEND bit in \field{device status} to suspend a device,
+and can clear the SUSPEND bit to resume a suspended device.
+
+\drivernormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set

or

Quoted 18 lines
Quoted 18 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

content.tex

Quoted 3 lines

VIRTIO_F_SUSPEND is not negotiated.

content.tex +2 -0
+
+Once the driver sets SUSPEND to \field{device status} of the

device:

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines

+\begin{itemize}
+\item The driver MUST re-read \field{device status} to +verify whether the
SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to +the

device.

Quoted 6 lines
Quoted 6 lines

+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+

Hi Parva

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Do we agree that
a. suspending a device is non frequent operation (in order of N

operations/sec, where N is roughly in range of 10 or 100) per

device?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is

often.

Quoted 26 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?

I think we don't have an intention of the frequency in the spec.

Sure.

Quoted 8 lines
Quoted 1 line

The spec only provides generic mechanisms and interfaces.

Sure.

Quoted 2 lines

Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.

As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <

50nsec of time.

Quoted 2 lines

The suspend operation for a net device to store hundreds of queues, RSS

table, flow filters, takes plenty of time (at least more than 50nsec :) ).

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Similarly for the GPU to store some MBs of memory takes more than 50nsec

of time, for example to store in a file for a software-based GPU device.

Quoted 1 line

So a device cannot respond back suspend=true in next 50nsec time.

It's OK for the device to take longer time to respond, the driver simply re-reads device status.

Sure, the issue is, when the driver re-reads, the device must present

suspend=false within 50nsec.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

(because device didn't suspend it yet).

As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly

differently.
why there is a constraint condition of the time?

Because this is what driver does as you explained in [1] indicating "we must

be able to handle ..."

Quoted 2 lines

[1]

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore

Quoted 1 line

.kernel.org%2Fvirtio-comment%2Fc4d5eed3-774b-4d35-a007-

f9dff28ce884%40 amd.com%2FT%2F%23m6f081f96ef9dcea29c64a88b633eb21d50e8c410&dat a=05%7C0 2%7Cparav%40nvidia.com%7C4a248d6982514660015808dccbf79c55%7C430 83d1572 7340c1b7db39efd9ccc17a%7C0%7C0%7C638609511605195425%7CUnknown %7CTWFpbG Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 Mn0% 3D%7C0%7C%7C%7C&sdata=Gj4oG8Z1O7wEl%2B7Mbud%2BbP31aMDMAFA H9ElHJSvBTUA%

Quoted 4 lines
Quoted 3 lines
Quoted 1 line

3D&reserved=0

The sentence is "we must be able to handle the scenario in which SUSPENDING is often", where do you see any constraint conditions of time?

If the device takes X usec, how can the device inform that, it suspending in progress without flipping the polarity?

Oh, I see your puzzle. The device does not inform that. It is VM_EXIT and VM_ENTRY, once VM_ENTRY, the guest vcpu aka the driver resume running

Quoted 58 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

Are there any similar
constraining for other states like RESET or DRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.

DRIVER_OK does not suffer from it because it is async notification. A device may start slow after DRIVER_OK.
SUSPEND operation cannot rely on such async behavior.

The driver can set suspend and re-read & wait. This is a common routine in the driver.

Quoted 8 lines
Quoted 4 lines

RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.

Here a new functionality is being proposed and it has a chance for efficient

device implementation.

Quoted 1 line

Therefore the request is to improve it.

sure, as long as we confirm this new register apply for all device status transitions, not only for SUSPEND.

Alright, please do it as you suggested here.

Quoted 39 lines
Quoted 39 lines
Quoted 22 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

More below.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the

software.

Quoted 1 line

Do you agree?

The device emulator does not know anything about whether trapped

or

Quoted 7 lines
Quoted 7 lines

not.

Quoted 6 lines
Quoted 6 lines

Trapping and Emulation is a hypervisor thing.

If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Right. So a software wants to implement device_status as pure MMIO

writes. (and not VM_EXIT).
This is not always true, there can be VM_EXIT of pure emulated devices.

I am not denying that VM_EXIT can/cannot be there. I am saying, the proposal forces VM_EXIT based approach in my

understanding of this patch.

[MARKER_1]

Quoted 6 lines
Quoted 1 line

If that is not true, may be can you please explain how this can be

implemented without VM_EXIT?

Quoted 2 lines

We should have an interface that can be done with and without VM_EXIT

method at least for any new additions.
VM_EXIT is out of spec, it is a hypervisor and the processor thing.

You keep repeating VM_EXIST is out of spec.
I already replied at [MARKER_1], sure it is out of spec, the current approach

Quoted 8 lines
Quoted 4 lines

forces one to do VM_EXIT based approach.

Quoted 1 line

And if not, please explain, how can it be achieved?

It is not the current solution force the hypervisor & guest to perform VM_EXIT.

I read it few times but couldn't parse it.
Can you please explain how your proposal works WITHOUT VM_EXIT? If not, you can say, that current proposal is limited to force VM_EXIT. Please explain.

It needs VM_EXIT, not only SUSPEND, all sensitive resource, like the config space access should be trapped.

Quoted 5 lines
Quoted 3 lines

In basic virtualization, any access to hypervisor emulated HW registers would be Traped and Emulated, means a VM_EXIT and a VM_ENTRY.

That was not my question, my question is how to achieve without VM_EXIT? You are suggesting a spec that forces VM_EXIT.

It needs VM_EXIT, and there is a pairing VM_ENTRY

Quoted 228 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

In non-pass-through case, any register access are sensitive and will trigger VM_EXIT. Like RESET or DRIVER_OK needs to access device_status, nothing different.

I don't think you understood the point.
Let me repeat, The question is, if the device implementation wants to

achieve the functionality without VM_EXIT, what is the way? Map the register to guest address space, out of the spec.

And which component will flip the bit of suspend before driver reads it to indicate that device is still_suspending?

Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

The
HW registers are sensitive resource and any access to them need to be trapped and emulated.

This does not apply to PCI PFs and VFs which are HW devices (mainly

PFs).

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

so this trap + emulation is narrow view that we better avoid.

This is how *basic* virtualization work, once access sensitive resource,

trap it.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

If you think this is the way forward, you should put forward in patch as

MUST requirement.

Quoted 2 lines

and that does not look right to me.
I hope you also don't mean to force this method to device

Quoted 8 lines
Quoted 6 lines

implementations.

Quoted 1 line

Right?

Again, VM_EXIT is a hypervisor thing,  out of spec. Whether there is a VM_EXIT when setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different from DRIVER_OK.

Please avoid repeating the point that VM_EXIT is hypervosor thing. No one asked to put this in spec. Please re-read [MARKER1]. You probably

Quoted 7 lines

missed that.

Quoted 1 line

SUSPEND is different than DRIVER_OK. I explained the timing constraints

and the required circuitry needed to fulfill the proposal.

Quoted 1 line

And with the additional register, such complicated circuitry can be easily

avoided.
Please read QEMU code, that can help you find how the trap - emulate mechanism work for guest when it tries to access sensitive resources.

I am not suggesting trap and emulation. Please come out of this looping thoughts.

I repeat my question, please explain how does suggested solution works without VM_EXIT? From your response, I derive it cannot. Can you please confirm?

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Means, if your virtualization needs to trap SUSPEND, it also needs to trap DRIVER_OK, and don't assume DRIVER_OK is faster than SUSPEND.

DRIVER_OK is by law of physics is faster than SUSPEND because it does not

demand the driver of reading back.

Quoted 1 line

There is no driver side loop to check if the device accepted DRIVER_OK or

not.

Quoted 1 line

Agree?

Why do you assume so?

I am not assuming. It is the current spec. :)
Please read the device initialization sequence, that does NOT say that driver must READ DRIVER_OK. Hence by law of physics it is faster.

Quoted 3 lines

In some vendor implementation, DRIVER_OK can be
slow. Can DRIVER_OK implement a deferred initialization even deferred resource allocation? Does the spec say that DRIVER_OK must be faster?

No. it does not. But your suspend proposal says so that device must respond back before the next read arrives from the driver.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
Quoted 2 lines

And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return

suspend=true right after it was written.

Quoted 4 lines

A MMIO read will read it back, as suspend=true.

An alternative would be, to forward CPU loads and CPU stores to different

address.

Quoted 4 lines

However, this does not work for the hw based devices.

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 2 lines

In this example, the device cannot build special circuitry to answer

Quoted 8 lines
Quoted 4 lines

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation. why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.

When driver wrote, it wrote suspend=true, And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec

or less to answer suspend=false.

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

And even worst, it needs to suspend=true when unsuspending within

50nsec when resuming is ongoing.
again, there is not a 50nsec constraining and please take a reference of how DRIVER_OK work with virtualization.

There is. The device is expected to return back the desired value to indicate

driver that suspend is ongoing.

Quoted 1 line

It is different than DRIVER_OK.

where does the spec or any code say 50nsec?

Your spec and your previous comment implies that when device is busy suspending, it must returned the toggled value than what driver set it. Isn't it? It means device needs to react before the next read arrives from the driver, isn't it?
I took that read as 50nsec example as you explained previously that driver will read very fast.

Quoted 132 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful comments

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Unfortunately, not. His comment was that it is not specific to suspend. But here we are introducing a new interface and functionality that does not

need to suffer or follow anything that may not be efficient.

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
%2F&data=05%7C02%7Cparav%40nvidia.com%7C4a248d6982514660015808 dccbf79 c55%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6386095116052 07133%7 CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB TiI6Ik 1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VGOfVe0deSDilzufY7NH gU5RmciNh

Quoted 4 lines
Quoted 4 lines

dXMMQiVUSiP%2BKM%3D&reserved=0

Quoted 1 line
Quoted 1 line

e.kernel.org%2Fvirtio-comment%2F20240612082055-mutt-send-email-

mst%40

kernel.org%2FT%2F%23mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9&dat

Quoted 2 lines
Quoted 2 lines

a=05

%7C02%7Cparav%40nvidia.com%7C619c82b60b824ca03f8808dcc89beb4a%7

Quoted 2 lines
Quoted 2 lines

C43083

d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638605819247985989%7CUn

Quoted 2 lines
Quoted 2 lines

known%7C

TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL

Quoted 2 lines
Quoted 2 lines

CJXV

CI6Mn0%3D%7C0%7C%7C%7C&sdata=VqmJmqt3k5tf3x4ihjE5Bd7u59GadOn

Quoted 27 lines
Quoted 27 lines

OaOfJ5lvG

Quoted 26 lines
Quoted 26 lines

1DE%3D&reserved=0

Quoted 25 lines
Quoted 25 lines

this is out of the spec anyway.

Thanks
Zhu Lingshan

Quoted 12 lines
Quoted 12 lines

Thanks
Zhu Lingshan

Quoted 10 lines
Quoted 10 lines

+\devicenormative{\subsection}{Device Suspend}{General

content.tex +3 -0
+Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+The device MUST ignore all access to its Configuration Space
+while suspended, except for \field{device status} if it is
+part of the Configuration

Space.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

content.tex

Quoted 8 lines
content.tex +8 -0
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in its
+Configuration Space while suspended.
+
+If changes occur in the Configuration Space while the
+SUSPEND bit is set, the device MUST NOT send any
+configuration change

notifications.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

content.tex

Quoted 5 lines
content.tex +2 -0
+Instead, the device MUST send the notification after the
+SUSPEND bit has

been cleared.

content.tex +2 -0
+
+When the driver sets SUSPEND, the device MUST either

suspend

Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

content.tex

Quoted 5 lines

+itself or set
DEVICE_NEEDS_RESET if failed to suspend.

content.tex +3 -0
+
+If SUSPEND is set in \field{device status}, when the driver
+clears SUSPEND, the device MUST either resume normal

operation

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

content.tex

Quoted 4 lines

+or set
DEVICE_NEEDS_RESET.

content.tex +2 -0
+
+When the driver sets SUSPEND, the device SHOULD perform

the

Quoted 36 lines
Quoted 36 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

+following actions before presenting that
the SUSPEND bit is set to 1 in the \field{device status}:

content.tex +7 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item
+Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio

Transport Options}

Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that
+ the driver

can

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
content.tex +3 -0
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device
+ Status

Field}.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+
\end{description}

\drivernormative{\section}{Reserved Feature Bits}{Reserved
Feature Bits}
--
2.45.2

#49 lore raw
In reply to Zhu Lingshan
Quoted 2 lines · Zhu Lingshan
Quoted 1 line

So a device cannot respond back suspend=true in next 50nsec time.

It's OK for the device to take longer time to respond, the driver simply re-reads device status.

You didn't bother saying in the spec that it must, though.

Signature · 1 line
MST
#50 lore raw
Quoted 4 lines · Michael S. Tsirkin
Quoted 2 lines · Zhu Lingshan
Quoted 1 line

So a device cannot respond back suspend=true in next 50nsec time.

It's OK for the device to take longer time to respond, the driver simply re-reads device status.

You didn't bother saying in the spec that it must, though.

I can add this re-read like how we handle FEATURE_OK

Quoted 1 line
#51 lore raw
In reply to Parav Pandit
Quoted 7 lines · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not suspended.
In this example, the device cannot build special circuitry to answer suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

Quoted 8 lines

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is the default value.
3. When the device completes suspend, it changes the polarity of suspend_status=true.

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and does not have to force VM_EXITs.

[B] It also enables hw based devices to not build special circuitry to answer within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Signature · 1 line
MST
#52 lore raw
Quoted 24 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 1 line · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 1 line

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

Quoted 9 lines

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this operation.

#53 lore raw
Quoted 27 lines · Parav Pandit
Quoted 27 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 1 line · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 1 line

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 8 lines
Quoted 6 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this operation.

Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress

What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)

How does this sound?

Signature · 1 line
MST
#54 lore raw
Quoted 65 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 8:38 PM

Quoted 32 lines · Parav Pandit
Quoted 32 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 2 lines · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 2 lines

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

Quoted 8 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives
not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 4 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 3 lines

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 8 lines
Quoted 6 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this

operation.

Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress

What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)

Quoted 2 lines

How does this sound?

Yes.
Just that having command in different register than the status is better approach to me. As it allows steering command to different piece of hw from the status.
1. separate status register can be optimized by the device as its largely serving the reads.
2. modular
3. simplicity and clarity to developers

cmd_register:
0 = resume
1 = suspend

status:
0 = resumed.
1 = suspended

cmd status
0 0 operational (resumed), typical default.
1 0 suspend in progress (initiated)
1 1 suspended
0 1 resume initiated (still suspended)

Quoted 2 lines

--
MST

#55 lore raw
In reply to Michael S. Tsirkin
Quoted 49 lines · Michael S. Tsirkin
Quoted 23 lines · Parav Pandit
Quoted 23 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 1 line · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 1 line

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

Quoted 6 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 2 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 1 line

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 8 lines
Quoted 6 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this operation.

Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress

What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)

Quoted 1 line

How does this sound?

This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device.
When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.

But if you think we should add a new register which applying for all device_status transitions, NOT only for SUSPEND. we can surely do that.

Thanks

Quoted 1 line
#56 lore raw
Quoted 68 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AM

Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Parav Pandit
Quoted 28 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 2 lines · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 2 lines

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this

operation.

Quoted 17 lines

Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress

What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)

Quoted 1 line

How does this sound?

This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.

Quoted 5 lines

But if you think we should add a new register which applying for all device_status transitions, NOT only for SUSPEND. we can surely do that.

Thanks

Quoted 1 line

New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.

#57 lore raw
Quoted 65 lines · Parav Pandit
Quoted 65 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AM

Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Parav Pandit
Quoted 27 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 2 lines · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 2 lines

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this

operation.

Quoted 16 lines

Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress

What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)

Quoted 1 line

How does this sound?

This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.

Quoted 6 lines
Quoted 4 lines

But if you think we should add a new register which applying for all device_status transitions, NOT only for SUSPEND. we can surely do that.

Thanks

New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.

There should be no difference in how the driver handles SUSPEND and other device status like RESET.

If we want to add a new register, then it is not only for SUSPEND, but for all status transitions.
We need Michael to confirm we should implement this new register that apply to all device_status transitions for common interests.

Thanks

Quoted 1 line
#58 lore raw
Quoted 74 lines

From: Zhu Lingshan <[email protected]>
Sent: Tuesday, September 3, 2024 2:21 PM

Quoted 10 lines · Parav Pandit
Quoted 10 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AM

Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Parav Pandit
Quoted 3 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Parav Pandit
Quoted 2 lines

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 2 lines

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow

operation.

Quoted 48 lines
Quoted 48 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation. And device_status new bit that Lingshan defined returns the status of this

operation.

Quoted 16 lines

Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress

What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)

Quoted 1 line

How does this sound?

This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

But if you think we should add a new register which applying for all device_status transitions, NOT only for SUSPEND. we can surely do that.

Thanks

New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.

There should be no difference in how the driver handles SUSPEND and other device status like RESET.

You still miss the fundamental point that I explained many times that is to avoid special time sensitive circuitry in the device by means of a new register. (not the driver).

Quoted 3 lines

If we want to add a new register, then it is not only for SUSPEND, but for all status transitions. We need Michael to confirm we should implement this new register that apply to all device_status transitions for common interests.

Sure. Look forward to his opinion.

#59 lore raw
In reply to Zhu Lingshan
Quoted 77 lines · Zhu Lingshan
Quoted 65 lines · Parav Pandit
Quoted 65 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AM

Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Parav Pandit
Quoted 27 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 2 lines · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 2 lines

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this

operation.

Quoted 16 lines

Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress

What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)

Quoted 1 line

How does this sound?

This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.

Quoted 6 lines
Quoted 4 lines

But if you think we should add a new register which applying for all device_status transitions, NOT only for SUSPEND. we can surely do that.

Thanks

New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.

There should be no difference in how the driver handles SUSPEND and other device status like RESET.

Quoted 6 lines

If we want to add a new register, then it is not only for SUSPEND, but for all status transitions.
We need Michael to confirm we should implement this new register that apply to all device_status transitions for common interests.

Thanks

There is a difference between SUSPEND and RESET. RESET is not a state. Thus a single bit is enough to signal "reset in progress".

I don't really see any other transitions that can take a long time. We can start with just suspend, and extend it later if appropriate.

Quoted 1 line
Quoted 1 line
#60 lore raw
Quoted 80 lines · Michael S. Tsirkin
Quoted 79 lines · Zhu Lingshan
Quoted 63 lines · Parav Pandit
Quoted 63 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AM

Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Parav Pandit
Quoted 27 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 2 lines · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 2 lines

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this

operation.

Quoted 15 lines

Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress

What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)

Quoted 1 line

How does this sound?

This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.

Quoted 6 lines
Quoted 4 lines

But if you think we should add a new register which applying for all device_status transitions, NOT only for SUSPEND. we can surely do that.

Thanks

New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.

There should be no difference in how the driver handles SUSPEND and other device status like RESET.

If we want to add a new register, then it is not only for SUSPEND, but for all status transitions.
We need Michael to confirm we should implement this new register that apply to all device_status transitions for common interests.

Quoted 8 lines
Quoted 1 line

Thanks

There is a difference between SUSPEND and RESET. RESET is not a state. Thus a single bit is enough to signal "reset in progress".

I don't really see any other transitions that can take a long time. We can start with just suspend, and extend it later if appropriate.

Yes, that is what I mean, the register should not only work for SUSPEND. To be more specific, the definition should be:

// The device_status is still in a status transition #define DEVICE_STATUS_TRANSITION_IN_PROGRESS     0 // device status transition is done
#define DEVICE_STATUS_TRANSITION_DONE                     1

They should not be defined as:
#define DEVICE_STATUS_SUSPEND_IN_PROGRESS     0

Thanks

Quoted 3 lines
#61 lore raw
Quoted 99 lines · Zhu Lingshan
Quoted 80 lines · Michael S. Tsirkin
Quoted 79 lines · Zhu Lingshan
Quoted 63 lines · Parav Pandit
Quoted 63 lines

From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AM

Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Parav Pandit
Quoted 27 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PM

Quoted 2 lines · Parav Pandit

That means, PCI HW needs to return suspend=0, until the device is not

suspended.

Quoted 2 lines

In this example, the device cannot build special circuitry to answer

suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.

Quoted 7 lines

If this understanding of burden is clear,

The proposal is, can you please extend the interface such that,

1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is

the default value.

Quoted 1 line

3. When the device completes suspend, it changes the polarity of

suspend_status=true.

Quoted 3 lines

This has two main benefits:
[A] This will enable software-based devices to write data to slow files and

does not have to force VM_EXITs.

Quoted 2 lines

[B] It also enables hw based devices to not build special circuitry to answer

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.

I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?

Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this

operation.

Quoted 15 lines

Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress

What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)

Quoted 1 line

How does this sound?

This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.

Quoted 6 lines
Quoted 4 lines

But if you think we should add a new register which applying for all device_status transitions, NOT only for SUSPEND. we can surely do that.

Thanks

New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.

There should be no difference in how the driver handles SUSPEND and other device status like RESET.

If we want to add a new register, then it is not only for SUSPEND, but for all status transitions.
We need Michael to confirm we should implement this new register that apply to all device_status transitions for common interests.

Quoted 8 lines
Quoted 1 line

Thanks

There is a difference between SUSPEND and RESET. RESET is not a state. Thus a single bit is enough to signal "reset in progress".

I don't really see any other transitions that can take a long time. We can start with just suspend, and extend it later if appropriate.

Yes, that is what I mean, the register should not only work for SUSPEND. To be more specific, the definition should be:

// The device_status is still in a status transition #define DEVICE_STATUS_TRANSITION_IN_PROGRESS     0 // device status transition is done
#define DEVICE_STATUS_TRANSITION_DONE                     1

Quoted 5 lines

They should not be defined as:
#define DEVICE_STATUS_SUSPEND_IN_PROGRESS     0

Thanks

in case we add more commands down the line? ok, sure.

Quoted 2 lines
Quoted 2 lines
#62 lore raw
In reply to Parav Pandit
Quoted 2 lines · Parav Pandit

a. suspending a device is non frequent operation (in order of N operations/sec, where N is roughly in range of 10 or 100) per device? b. A software-based device may not always want to force VM_EXIT on read and write on the device_status register?

I just see where this is going. Parav, if you want a device with as little as possible memory on it, using DMA as much as possible, what you want is develop an alternative transport
that will let you control status without using memory accesses. This will also, with time, allow doing this for existing config space. We aren't blocking all spec development until you do, and the tradeoff of easy to debug, robust MMIO control versus complex but maybe cheaper for some hardware vendors DMA based access is up to the device.

Signature · 1 line
MST
#63 lore raw
Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 13, 2024 1:22 PM

Removing intel email id again to avoid constant bouncing messages.

Quoted 10 lines · Parav Pandit
Quoted 1 line

a. suspending a device is non frequent operation (in order of N

operations/sec, where N is roughly in range of 10 or 100) per device?

Quoted 1 line

b. A software-based device may not always want to force VM_EXIT on read

and write on the device_status register?

I just see where this is going. Parav, if you want a device with as little as possible memory on it, using DMA as much as possible, what you want is develop an alternative transport that will let you control status without using memory accesses.

No, there is no need of DMA here.
This is not about less memory either.

As I explained, the challenge is implementing suspend bits to operate at sub-microsecond level granularity.

Quoted 5 lines

This will also, with time, allow doing this for existing config space. We aren't blocking all spec development until you do, and the tradeoff of easy to debug, robust MMIO control versus complex but maybe cheaper for some hardware vendors DMA based access is up to the device.

No one is blocking the spec etc. Not sure why you say that.
The idea is to have suspend as command register, and its status to reflect in the existing device_status.

Last time you recommended to only make design suggestion and not go down the exact details of the bit and let author to fix it.

If you prefer to discuss the bit level details, it looks good to me to go to that details.

Quoted 2 lines

--
MST

#64 lore raw
Quoted 6 lines · Parav Pandit
Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 13, 2024 1:22 PM

Removing intel email id again to avoid constant bouncing messages.

Thanks.

Quoted 4 lines
Quoted 4 lines · Parav Pandit
Quoted 1 line

a. suspending a device is non frequent operation (in order of N

operations/sec, where N is roughly in range of 10 or 100) per device?

Quoted 1 line

b. A software-based device may not always want to force VM_EXIT on read

Quoted 8 lines
Quoted 6 lines

and write on the device_status register?

I just see where this is going. Parav, if you want a device with as little as possible memory on it, using DMA as much as possible, what you want is develop an alternative transport that will let you control status without using memory accesses.

No, there is no need of DMA here.
This is not about less memory either.

OK, sorry. I misunderstood.

Quoted 1 line

As I explained, the challenge is implementing suspend bits to operate at sub-microsecond level granularity.

Yes, I think the proposal is too vague on how all this happens. Replied separately.

Quoted 11 lines
Quoted 5 lines

This will also, with time, allow doing this for existing config space. We aren't blocking all spec development until you do, and the tradeoff of easy to debug, robust MMIO control versus complex but maybe cheaper for some hardware vendors DMA based access is up to the device.

No one is blocking the spec etc. Not sure why you say that.
The idea is to have suspend as command register, and its status to reflect in the existing device_status.

Last time you recommended to only make design suggestion and not go down the exact details of the bit and let author to fix it.

If you prefer to discuss the bit level details, it looks good to me to go to that details.

Sorry, reacted too early.

Quoted 2 lines
Quoted 2 lines

--
MST

#65 lore raw
In reply to Zhu Lingshan

Review

Quoted 3 lines · Zhu Lingshan

+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}

(quoted diff) +2 -0
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.

Actually, it has no effect before DRIVER_OK, no? I would forbid that then.

Quoted 3 lines

+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.

This is still vague, I commented on this several times. I think what you mean is that until it reads status with SUSPEND as 1, it does not consider SUSPEND set.

But, what happens if driver clears SUSPEND before it reads it as set? It would seem that should cancel suspend (which is a useful thing to support)? But this creates a problem as it breaks read/modify/write that some hypervisors assumed to be safe. I guess we need an extra SUSPEND_IN_PROGRESS bit then? A little too much for status, at this stage - maybe we need an extra register for this.

Quoted 19 lines

+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize}

content.tex +13 -0
+
+\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.
 
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the Configuration Space.
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in
+its Configuration Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit is set,

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675

Quoted 10 lines
content.tex +10 -0
+the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification after the SUSPEND bit has been cleared.
+
+When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend.
+
+If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,
+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET.
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}:

what does "before presenting" mean? does it return SUSPEND as 0 after driver wrote 1 there and before it completed these actions?

Quoted 23 lines
content.tex +11 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues
+\item Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split
@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.45.2
#66 lore raw
Quoted 6 lines · Michael S. Tsirkin

Review

Quoted 3 lines · Zhu Lingshan

+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}

(quoted diff) +2 -0
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.

Actually, it has no effect before DRIVER_OK, no? I would forbid that then.

There can be two cases:
1) debugging the device before DRIVER_OK
2) VM migration right after boot, and before DRIVER_OK. So it needs to migrate the device before DRIVER_OK

Quoted 10 lines
Quoted 4 lines

+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.

This is still vague, I commented on this several times. I think what you mean is that until it reads status with SUSPEND as 1, it does not consider SUSPEND set.

Quoted 6 lines

But, what happens if driver clears SUSPEND before it reads it as set? It would seem that should cancel suspend (which is a useful thing to support)? But this creates a problem as it breaks read/modify/write that some hypervisors assumed to be safe. I guess we need an extra SUSPEND_IN_PROGRESS bit then? A little too much for status, at this stage - maybe we need an extra register for this.

if the driver reads SUSPEND bit == 0, with the knowledge of its own operations, it knows the device is either in a) normal operation 2). suspending in progress

Cancelling a SUSPEND right after setting SUSPEND is actually a RESUME. The ideal case is the driver waits until SUSPEND == 1, then clear it.

But there are various drivers, not all drivers follow our expectation, so there can be a corner case that how the driver know whether the device is resumed after clearing the SUSPEND bit or not finish suspending.

I think adding a new resister can surely fix this issue, however as you said, add a new status bit is overkill and where to place this new register is a new question.

So, to make things easier, how about suspend & resume the device sequentially:

If the driver has suspended the device by setting the SUSPEND bit to 1, it MUST NOT clear the SUSPEND bit before the device presenting the SUSPEND bit set as 1.

Quoted 24 lines
Quoted 24 lines

+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize}

content.tex +18 -0
+
+\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the Configuration Space.
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in
+its Configuration Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit is set,
+the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification after the SUSPEND bit has been cleared.
+
+When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend.
+
Quoted 8 lines

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675

Quoted 5 lines
content.tex +5 -0
+If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,
+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET.
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}:

what does "before presenting" mean? does it return SUSPEND as 0 after driver wrote 1 there and before it completed these actions?

Yes, it should return 0 if the suspending  operation is still in progress,
It should not change SUSPEND bit value before finish.
Do you suggest we add:
The device MUST present SUSPEND bit set to 1 in \field{device status} once it has been suspended

It says: presenting that the SUSPEND bit is set to 1. Not a native speaker, but this looks clear to me.

Quoted 24 lines
Quoted 24 lines
content.tex +11 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues
+\item Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split
@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.45.2
#67 lore raw
Quoted 4 lines · Zhu Lingshan
Quoted 1 line · Michael S. Tsirkin

On Thu, Aug 01, 2024 at 07:35:16PM +0800, Zhu Lingshan wrote:

Quoted 8 lines
Quoted 5 lines

Review

Quoted 3 lines

+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}

(quoted diff) +2 -0
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.

Actually, it has no effect before DRIVER_OK, no? I would forbid that then.

There can be two cases:
1) debugging the device before DRIVER_OK
2) VM migration right after boot, and before DRIVER_OK. So it needs to migrate the device before DRIVER_OK

Sorry I don't understand. The only effect of SUSPEND as you specified it is not consuming buffers. Buffers are not consumed before DRIVER_OK, thus there is no point to set SUSPEND before DRIVER_OK.

Quoted 25 lines
Quoted 10 lines
Quoted 4 lines

+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.

This is still vague, I commented on this several times. I think what you mean is that until it reads status with SUSPEND as 1, it does not consider SUSPEND set.

Quoted 6 lines

But, what happens if driver clears SUSPEND before it reads it as set? It would seem that should cancel suspend (which is a useful thing to support)? But this creates a problem as it breaks read/modify/write that some hypervisors assumed to be safe. I guess we need an extra SUSPEND_IN_PROGRESS bit then? A little too much for status, at this stage - maybe we need an extra register for this.

if the driver reads SUSPEND bit == 0, with the knowledge of its own operations, it knows the device is either in a) normal operation 2). suspending in progress

Cancelling a SUSPEND right after setting SUSPEND is actually a RESUME. The ideal case is the driver waits until SUSPEND == 1, then clear it. But there are various drivers, not all drivers follow our expectation, so there can be a corner case that how the driver know whether the device is resumed after clearing the SUSPEND bit or not finish suspending.

Quoted 3 lines

I think adding a new resister can surely fix this issue, however as you said, add a new status bit is overkill and where to place this new register is a new question.

Place it after existing registers.

Quoted 5 lines

So, to make things easier, how about suspend & resume the device sequentially:

If the driver has suspended the device by setting the SUSPEND bit to 1, it MUST NOT clear the SUSPEND bit before the device presenting the SUSPEND bit set as 1.

I'm worried that it's fragile - one can no longer deduce the device state by reading the status.

Quoted 32 lines
Quoted 24 lines
Quoted 24 lines

+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize}

content.tex +18 -0
+
+\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the Configuration Space.
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in
+its Configuration Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit is set,
+the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification after the SUSPEND bit has been cleared.
+
+When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend.
+
Quoted 8 lines

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675

Quoted 5 lines
content.tex +5 -0
+If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,
+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET.
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}:

what does "before presenting" mean? does it return SUSPEND as 0 after driver wrote 1 there and before it completed these actions?

Quoted 7 lines

Yes, it should return 0 if the suspending  operation is still in progress,
It should not change SUSPEND bit value before finish.
Do you suggest we add:
The device MUST present SUSPEND bit set to 1 in \field{device status} once it has been suspended

It says: presenting that the SUSPEND bit is set to 1. Not a native speaker, but this looks clear to me.

I don't think this is sufficient, you need to document all assumptions on both device and driver.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
content.tex +11 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues
+\item Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split
@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.45.2
#68 lore raw
Quoted 9 lines · Michael S. Tsirkin
Quoted 8 lines · Zhu Lingshan
Quoted 6 lines · Michael S. Tsirkin

Review

Quoted 3 lines · Zhu Lingshan

+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}

(quoted diff) +2 -0
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.

Actually, it has no effect before DRIVER_OK, no? I would forbid that then.

Quoted 7 lines
Quoted 3 lines

There can be two cases:
1) debugging the device before DRIVER_OK
2) VM migration right after boot, and before DRIVER_OK. So it needs to migrate the device before DRIVER_OK

Sorry I don't understand. The only effect of SUSPEND as you specified it is not consuming buffers. Buffers are not consumed before DRIVER_OK, thus there is no point to set SUSPEND before DRIVER_OK.

Sorry for the late reply, I am very busy these two weeks.

There can be other operations except buffers before DRIVER_OK, like config interrupts. And the hypervisor may migrate VMs right after creating them, at that moment DRIVER_OK may not be set yet, we should not block this use case.

Quoted 24 lines
Quoted 24 lines
Quoted 9 lines
Quoted 4 lines

+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.

This is still vague, I commented on this several times. I think what you mean is that until it reads status with SUSPEND as 1, it does not consider SUSPEND set.

Quoted 6 lines

But, what happens if driver clears SUSPEND before it reads it as set? It would seem that should cancel suspend (which is a useful thing to support)? But this creates a problem as it breaks read/modify/write that some hypervisors assumed to be safe. I guess we need an extra SUSPEND_IN_PROGRESS bit then? A little too much for status, at this stage - maybe we need an extra register for this.

if the driver reads SUSPEND bit == 0, with the knowledge of its own operations, it knows the device is either in a) normal operation 2). suspending in progress

Cancelling a SUSPEND right after setting SUSPEND is actually a RESUME. The ideal case is the driver waits until SUSPEND == 1, then clear it. But there are various drivers, not all drivers follow our expectation, so there can be a corner case that how the driver know whether the device is resumed after clearing the SUSPEND bit or not finish suspending.

Quoted 4 lines
Quoted 3 lines

I think adding a new resister can surely fix this issue, however as you said, add a new status bit is overkill and where to place this new register is a new question.

Place it after existing registers.

I am still confused, SUSPEND and RESUME are both device state transition, just like RESET, FEATURES_OK and DRIVER_OK. I see current virtio works fine for these existing status without an extra register. And we can not assume SUSPEND / RESUME takes longer time than any other states.

So, do you want a new register for all state transitions? Apply to all device states, not only SUSPEND.

Quoted 9 lines
Quoted 6 lines

So, to make things easier, how about suspend & resume the device sequentially:

If the driver has suspended the device by setting the SUSPEND bit to 1, it MUST NOT clear the SUSPEND bit before the device presenting the SUSPEND bit set as 1.

I'm worried that it's fragile - one can no longer deduce the device state by reading the status.

It is the driver reads / sets device status, it has the knowledge of the device status I think. For example, if it sets SUSPEND and reads SUSPEND == 0, means SUSPEND in progress.

so I even fell this is actually not necessary .

Quoted 37 lines
Quoted 37 lines
Quoted 24 lines
Quoted 24 lines

+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize}

content.tex +18 -0
+
+\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.
+
+The device MUST ignore all access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the Configuration Space.
+
+A device MUST NOT send any notifications for any virtqeuues,
+access any virtqueues, or modify any fields in
+its Configuration Space while suspended.
+
+If changes occur in the Configuration Space while the SUSPEND bit is set,
+the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification after the SUSPEND bit has been cleared.
+
+When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend.
+
Quoted 8 lines

content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675

Quoted 5 lines
content.tex +5 -0
+If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,
+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET.
+
+When the driver sets SUSPEND,
+the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}:

what does "before presenting" mean? does it return SUSPEND as 0 after driver wrote 1 there and before it completed these actions?

Yes, it should return 0 if the suspending  operation is still in progress,
It should not change SUSPEND bit value before finish.
Do you suggest we add:
The device MUST present SUSPEND bit set to 1 in \field{device status} once it has been suspended

Quoted 4 lines
Quoted 2 lines

It says: presenting that the SUSPEND bit is set to 1. Not a native speaker, but this looks clear to me.

I don't think this is sufficient, you need to document all assumptions on both device and driver.

Would you please help me understanding what assumptions should be addressed here? IMHO, the driver knows what it is doing I think.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
content.tex +11 -0
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues
+\item Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split
@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can
+   trigger suspending the device via the SUSPEND flag
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.45.2