Skip to contentSkip to conversation

Implement virtio SUSPEND and RESUME feature

PATCH Superseded Zhu Lingshan 4 participants 17 messages Jun 23, 2025 – Jun 27, 2025

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover Implement virtio SUSPEND and RESUME feature

This series implements SUSPEND and RESUME feature for virtio devices. This feature could be used to perform live migration or take snapshots of the devices.

This series re-orders the device status bits, and documents feature bit 42.

Github issue link:
https://github.com/oasis-tcs/virtio-spec/issues/229

Zhu Lingshan (3):
virtio: re-order device status bits
virtio: document feature bit 42
virtio: introduce SUSPEND and RESUME feature

 content.tex | 71 +++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 63 insertions(+), 8 deletions(-)
Signature · 1 line
2.49.0
#1 lore raw

PATCH v3 1/3 virtio: re-order device status bits

This commit re-orders the device status bits.

---
 content.tex | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
content.tex +7 -7
index d3fc6a4..1efc2a5 100644
@@ -36,19 +36,19 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
     this bit.  For example, under Linux, drivers can be loadable modules.
   \end{note}
 
-\item[FAILED (128)] Indicates that something went wrong in the guest,
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason, or
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
+  drive the device.
 
 \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the
   features it understands, and feature negotiation is complete.
 
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
-  drive the device.
-
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
   an error from which it can't recover.
+
+\item[FAILED (128)] Indicates that something went wrong in the guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason, or
+  even a fatal error during device operation.
 \end{description}
 
 The \field{device status} field starts out as 0, and is reinitialized to 0 by
Signature · 1 line
2.49.0
Reviewed-by on v3 1/3
#2 lore raw

RE: [PATCH v3 1/3] virtio: re-order device status bits

Quoted 5 lines

From: Zhu Lingshan <[email protected]>
Sent: 23 June 2025 02:07 PM

This commit re-orders the device status bits.

A better message would be that defines their ordering,

"Rearranged the bit definitions to list them in ascending order."

Quoted 3 lines

---

Anyways,

Quoted 38 lines
 content.tex | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
content.tex +0 -0
index d3fc6a4..1efc2a5 100644
@@ -36,19 +36,19 @@ \section{\field{Device Status} Field}\label{sec:Basic

Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}

-\item[FAILED (128)] Indicates that something went wrong in the guest,

content.tex +7 -7
-  and it has given up on the device. This could be an internal
-  error, or the driver didn't like the device for some reason, or
-  even a fatal error during device operation.
+\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
+  drive the device.
 
 \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the
   features it understands, and feature negotiation is complete.
 
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
-  drive the device.
-
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
   an error from which it can't recover.
+
+\item[FAILED (128)] Indicates that something went wrong in the guest,
+  and it has given up on the device. This could be an internal
+  error, or the driver didn't like the device for some reason, or
+  even a fatal error during device operation.
 \end{description}
 
 The \field{device status} field starts out as 0, and is reinitialized to 0 by
--

2.49.0

#3 lore raw

PATCH v3 2/3 virtio: document feature bit 42

In reply to Zhu Lingshan

This commit documents feture bit 42
VIRTIO_NET_F_GUEST_RSC6

Fixes 94384142 ("content: Declare virtio-net legacy feature bits 41-42")
---
 content.tex | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
content.tex +2 -2
index 1efc2a5..2e8da46 100644
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}
Signature · 1 line
2.49.0
#4 lore raw

RE: [PATCH v3 2/3] virtio: document feature bit 42

Quoted 18 lines

From: Zhu Lingshan <[email protected]>
Sent: 23 June 2025 02:07 PM

This commit documents feture bit 42
VIRTIO_NET_F_GUEST_RSC6

Fixes 94384142 ("content: Declare virtio-net legacy feature bits 41-42")
---
 content.tex | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
content.tex +0 -0
index 1efc2a5..2e8da46 100644

content.tex \section{Feature Bits}\label{sec:Basic Facilities of a · line 104

Quoted 7 lines
content.tex +1 -1
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}
 

This does not look accurate.
Bits 41 and 42 are used by the device specific type and not generic queue related mechanism. In this case it is used by the net device.

So right documentation should be,

0 to 23, 41, 42 and 50 to 127 Feature bits for the specific device type

24 to 40 Feature bits reserved for extensions to the queue and feature negotiation mechanisms

43 to 49, and 128 and above Feature bits reserved for future extensions.

Quoted 10 lines
content.tex +0 -1
-\item[42 to 49, and 128 and above] Feature bits reserved for future

extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}

\begin{note}
--
2.49.0

#5 lore raw

Re: [PATCH v3 2/3] virtio: document feature bit 42

Quoted 28 lines · Parav Pandit
Quoted 17 lines

From: Zhu Lingshan <[email protected]>
Sent: 23 June 2025 02:07 PM

This commit documents feture bit 42
VIRTIO_NET_F_GUEST_RSC6

Fixes 94384142 ("content: Declare virtio-net legacy feature bits 41-42")
---
  content.tex | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
content.tex +0 -0
index 1efc2a5..2e8da46 100644

content.tex \section{Feature Bits}\label{sec:Basic Facilities of a · line 104

Quoted 7 lines
content.tex +1 -1
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
    feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}
 

This does not look accurate.
Bits 41 and 42 are used by the device specific type and not generic queue related mechanism. In this case it is used by the net device.

Quoted 7 lines

So right documentation should be,

0 to 23, 41, 42 and 50 to 127 Feature bits for the specific device type

24 to 40 Feature bits reserved for extensions to the queue and feature negotiation mechanisms

43 to 49, and 128 and above Feature bits reserved for future extensions.

OK, this looks better

Quoted 11 lines
Quoted 11 lines
content.tex +0 -1
-\item[42 to 49, and 128 and above] Feature bits reserved for future

extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}

\begin{note}
--
2.49.0

#6 lore raw

PATCH v3 3/3 virtio: introduce SUSPEND and RESUME feature

In reply to Zhu Lingshan

This commit allows the driver to suspend the device through a new device status bit SUSPEND and resume the device running by re-setting DRIVER_OK bit in device status.

---
 content.tex | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 57 insertions(+), 2 deletions(-)
content.tex +57 -2
index 2e8da46..c987334c 100644
@@ -42,6 +42,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
 \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the
   features it understands, and feature negotiation is complete.
 
+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the
+  device has been suspended by the driver.
+
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
   an error from which it can't recover.
 
@@ -99,10 +102,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 42] Feature bits reserved for extensions to the queue and
+\item[24 to 43] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}
 
-\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[44 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}
@@ -629,6 +632,54 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
 
 Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
 
+\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend}
+
+If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the device by setting the SUSPEND bit in \field{device status} to 1, and the device SHOULD set the DRIVER_OK bit to 0 once it has been suspended.
+
+If the device has been suspended, the driver can resume the device running by setting the DRIVER_OK bit in \field{device status} to 1, and the device should set the SUSPEND bit to 0 once it resumes running.
+
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
+
+The driver SHOULD NOT set SUSPEND bit if DRIVER_OK is not set or VIRTIO_F_SUSPEND is not negotiated.
+
+Once the driver sets SUSPEND bit in \field{device status} to 1:
+\begin{itemize}
+\item The driver MUST verify whether the device has been suspended by re-reading \field{device status}, examining whether the SUSPEND bit is set to 1 and the DRIVER_OK bit is set to 0.
+\item The driver MUST NOT make any more buffers available to the device.
+\item The driver MUST NOT send notifications for any virtqueues.
+\item The driver MUST NOT make any changes to Device Configuration Space except for \field{device status} if it is part of the Configuration Space.
+\end{itemize}
+
+\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore any operations on the SUSPEND bit from the driver if the device has not been completely initialized by the procedures in \ref{sec:General Initialization And Device Operation / Device Initialization}
+
+The device SHOULD ignore any write access to its Configuration Space while
+suspended, except for \field{device status} if it is part of the Configuration Space.
+
+A device MUST NOT send any notifications for any virtqueues,
+access any virtqueues, or modify any fields in
+its Configuration Space while suspended.
+
+If changes occur in the Configuration Space during suspended period,
+the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification when it resumes running.
+
+If the driver sets the SUSPEND bit in \field{device status} to 1,
+the device MUST either suspend itself or set the DEVICE_NEEDS_RESET bit in \field{device status} to 1 when it fails to suspend.
+
+If the device has been suspended and the driver resumes the device running by setting the DRIVER_OK bit in \field{device status} to 1,
+the device MUST either resume normal operation or set the DEVICE_NEEDS_RESET bit in \field{device status} to 1 when it fails to resume.
+
+When the driver sets the SUSPEND bit to 1,
+the device SHOULD perform the following actions before presenting the SUSPEND bit as 1 and DRIVER_OK bit as 0 in the \field{device status}:
+
+\begin{itemize}
+\item Stop consuming more buffers of any virtqueues.
+\item Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split
@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(43)] This feature indicates that the driver can
+   suspend the device by set the SUSPEND bit to 1.
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.49.0
#7 lore raw

RE: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

Quoted 41 lines

From: Zhu Lingshan <[email protected]>
Sent: 23 June 2025 02:07 PM

This commit allows the driver to suspend the device through a new device status bit SUSPEND and resume the device running by re-setting DRIVER_OK bit in device status.

Fixes:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
ub.com%2Foasis-tcs%2Fvirtio- spec%2Fissues%2F229&data=05%7C02%7Cparav%40nvidia.com%7C3085ab 378f264b5c845808ddb2313478%7C43083d15727340c1b7db39efd9ccc17 a%7C0%7C0%7C638862646595099882%7CUnknown%7CTWFpbGZsb3d8e yJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pf2ub%2FwVzdmf A1JjScOJ7D%2B%2BNea9XgVFYdvvDQxym40%3D&reserved=0

---
 content.tex | 59
+++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 57 insertions(+), 2 deletions(-)
content.tex +0 -0
index 2e8da46..c987334c 100644
@@ -42,6 +42,9 @@ \section{\field{Device Status} Field}\label{sec:Basic

Facilities of a Virtio Dev \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the
features it understands, and feature negotiation is complete.

+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that

content.tex +3 -0
+the
+  device has been suspended by the driver.
+
 \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
   an error from which it can't recover.

content.tex \section{Feature Bits}\label{sec:Basic Facilities of a · line 107

Quoted 7 lines
content.tex +1 -1
@@ -99,10 +102,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 42] Feature bits reserved for extensions to the queue and
+\item[24 to 43] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}
 

Ignoring this delta as the fix of patch-2 will change this.

Quoted 17 lines
content.tex +0 -1
-\item[43 to 49, and 128 and above] Feature bits reserved for future

extensions.
+\item[44 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +632,54 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset) before
removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And Device

content.tex +3 -0
+Operation / Device Suspend}
+
+If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the
Quoted 3 lines

device by setting the SUSPEND bit in \field{device status} to 1, and the device SHOULD set the DRIVER_OK bit to 0 once it has been suspended.
+

You ignored the inputs.
I do not agree to add the "SHOULD" normative wording in the general description area. The reasoning is explained already.
Please adapt to the existing style of this spec to keep the normative in requirements section.

If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the device by setting the SUSPEND bit in \field{device status} to 1, and the device sets the DRIVER_OK bit to 0 once it has been suspended.

Is this really that hard to write above way?

Quoted 4 lines

+If the device has been suspended, the driver can resume the device running by setting the DRIVER_OK bit in \field{device status} to 1, and the device should set the SUSPEND bit to 0 once it resumes running.
+

Similarly,

If the device has been suspended, the driver can resume the device running by setting the DRIVER_OK bit in \field{device status} to 1, and the device sets the SUSPEND bit to 0 once it resumes running.

Quoted 72 lines

+\drivernormative{\subsection}{Device Suspend}{General Initialization

content.tex +3 -0
+And Device Operation / Device Suspend}
+
+The driver SHOULD NOT set SUSPEND bit if DRIVER_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+Once the driver sets SUSPEND bit in \field{device status} to 1:
+\begin{itemize}
+\item The driver MUST verify whether the device has been suspended by re-

reading \field{device status}, examining whether the SUSPEND bit is set to 1 and the DRIVER_OK bit is set to 0.
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT send notifications for any virtqueues. +\item The driver MUST NOT make any changes to Device Configuration Space except for \field{device status} if it is part of the Configuration Space. +\end{itemize}

content.tex +11 -0
+
+\devicenormative{\subsection}{Device Suspend}{General Initialization
+And Device Operation / Device Suspend}
+
+The device MUST ignore any operations on the SUSPEND bit from the
+driver if the device has not been completely initialized by the
+procedures in \ref{sec:General Initialization And Device Operation /
+Device Initialization}
+
+The device SHOULD ignore any write access to its Configuration Space
+while suspended, except for \field{device status} if it is part of the

Configuration Space.

content.tex +11 -0
+
+A device MUST NOT send any notifications for any virtqueues, access any
+virtqueues, or modify any fields in its Configuration Space while
+suspended.
+
+If changes occur in the Configuration Space during suspended period,
+the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification when it resumes running.
+
+If the driver sets the SUSPEND bit in \field{device status} to 1, the
+device MUST either suspend itself or set the DEVICE_NEEDS_RESET bit in

\field{device status} to 1 when it fails to suspend.

content.tex +3 -0
+
+If the device has been suspended and the driver resumes the device
+running by setting the DRIVER_OK bit in \field{device status} to 1, the device

MUST either resume normal operation or set the DEVICE_NEEDS_RESET bit in \field{device status} to 1 when it fails to resume.

content.tex +3 -0
+
+When the driver sets the SUSPEND bit to 1, the device SHOULD perform
+the following actions before presenting the SUSPEND bit as 1 and DRIVER_OK

bit as 0 in the \field{device status}:

content.tex +8 -0
+
+\begin{itemize}
+\item Stop consuming more buffers of any virtqueues.
+\item Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
 \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
 Virtio can use various different buses, thus the standard is split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}

for

content.tex +4 -1
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_SUSPEND(43)] This feature indicates that the driver can
+   suspend the device by set the SUSPEND bit to 1.
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
--
Quoted 2 lines

2.49.0

Rest of the requirements and description looks good to me.

Can you please fix these 2 small things in the general description and patch-2?

#8 lore raw

Re: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

Quoted 74 lines · Parav Pandit
Quoted 40 lines

From: Zhu Lingshan<[email protected]>
Sent: 23 June 2025 02:07 PM

This commit allows the driver to suspend the device through a new device status bit SUSPEND and resume the device running by re-setting DRIVER_OK bit in device status.

Fixes:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
ub.com%2Foasis-tcs%2Fvirtio- spec%2Fissues%2F229&data=05%7C02%7Cparav%40nvidia.com%7C3085ab 378f264b5c845808ddb2313478%7C43083d15727340c1b7db39efd9ccc17 a%7C0%7C0%7C638862646595099882%7CUnknown%7CTWFpbGZsb3d8e yJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pf2ub%2FwVzdmf A1JjScOJ7D%2B%2BNea9XgVFYdvvDQxym40%3D&reserved=0

---
  content.tex | 59
+++++++++++++++++++++++++++++++++++++++++++++++++++--
  1 file changed, 57 insertions(+), 2 deletions(-)
content.tex +0 -0
index 2e8da46..c987334c 100644
@@ -42,6 +42,9 @@ \section{\field{Device Status} Field}\label{sec:Basic

Facilities of a Virtio Dev \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the
features it understands, and feature negotiation is complete.

+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that

content.tex +3 -0
+the
+  device has been suspended by the driver.
+
  \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
    an error from which it can't recover.

content.tex \section{Feature Bits}\label{sec:Basic Facilities of a · line 107

Quoted 7 lines
content.tex +1 -1
@@ -99,10 +102,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 42] Feature bits reserved for extensions to the queue and
+\item[24 to 43] Feature bits reserved for extensions to the queue and
    feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}
 

Ignoring this delta as the fix of patch-2 will change this.

Quoted 17 lines
content.tex +0 -1
-\item[43 to 49, and 128 and above] Feature bits reserved for future

extensions.
+\item[44 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -629,6 +632,54 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset) before
removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And Device

content.tex +3 -0
+Operation / Device Suspend}
+
+If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the
Quoted 3 lines

device by setting the SUSPEND bit in \field{device status} to 1, and the device SHOULD set the DRIVER_OK bit to 0 once it has been suspended.
+

You ignored the inputs.
I do not agree to add the "SHOULD" normative wording in the general description area. The reasoning is explained already.
Please adapt to the existing style of this spec to keep the normative in requirements section.

Quoted 5 lines

If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the device by setting the SUSPEND bit in \field{device status} to 1, and the device sets the DRIVER_OK bit to 0 once it has been suspended.

Is this really that hard to write above way?

I believe you totally ignored my replies in the last thread. There are no rules forbid using "SHOULD" in any non-normative sections. Now here I copy the reply here again:

In the spec section 1.3 Terminology, it says:

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC2119] and [RFC8174] when, and only when, they appear in all capitals, as shown here.

RFC 2119 says:

SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

Here this "SHOULD" exactly conforms to the definition.

"SHOULD" has already been used in non-normative sections, for example:
2.5.4 Legacy Interfaces: when using the legacy interface, drivers SHOULD read these fields multiple times until two reads generate a consistent result.

The spec does not say: “SHOULD” can only be used in driver or device requirements section.

@MST, we need your input

Quoted 10 lines
Quoted 5 lines

+If the device has been suspended, the driver can resume the device running by setting the DRIVER_OK bit in \field{device status} to 1, and the device should set the SUSPEND bit to 0 once it resumes running.
+

Similarly,

If the device has been suspended, the driver can resume the device running by setting the DRIVER_OK bit in \field{device status} to 1, and the device sets the SUSPEND bit to 0 once it resumes running.

same as above

Quoted 72 lines
Quoted 72 lines

+\drivernormative{\subsection}{Device Suspend}{General Initialization

content.tex +3 -0
+And Device Operation / Device Suspend}
+
+The driver SHOULD NOT set SUSPEND bit if DRIVER_OK is not set or

VIRTIO_F_SUSPEND is not negotiated.

content.tex +4 -0
+
+Once the driver sets SUSPEND bit in \field{device status} to 1:
+\begin{itemize}
+\item The driver MUST verify whether the device has been suspended by re-

reading \field{device status}, examining whether the SUSPEND bit is set to 1 and the DRIVER_OK bit is set to 0.
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT send notifications for any virtqueues. +\item The driver MUST NOT make any changes to Device Configuration Space except for \field{device status} if it is part of the Configuration Space. +\end{itemize}

content.tex +11 -0
+
+\devicenormative{\subsection}{Device Suspend}{General Initialization
+And Device Operation / Device Suspend}
+
+The device MUST ignore any operations on the SUSPEND bit from the
+driver if the device has not been completely initialized by the
+procedures in \ref{sec:General Initialization And Device Operation /
+Device Initialization}
+
+The device SHOULD ignore any write access to its Configuration Space
+while suspended, except for \field{device status} if it is part of the

Configuration Space.

content.tex +11 -0
+
+A device MUST NOT send any notifications for any virtqueues, access any
+virtqueues, or modify any fields in its Configuration Space while
+suspended.
+
+If changes occur in the Configuration Space during suspended period,
+the device MUST NOT send any configuration change notifications.
+Instead, the device MUST send the notification when it resumes running.
+
+If the driver sets the SUSPEND bit in \field{device status} to 1, the
+device MUST either suspend itself or set the DEVICE_NEEDS_RESET bit in

\field{device status} to 1 when it fails to suspend.

content.tex +3 -0
+
+If the device has been suspended and the driver resumes the device
+running by setting the DRIVER_OK bit in \field{device status} to 1, the device

MUST either resume normal operation or set the DEVICE_NEEDS_RESET bit in \field{device status} to 1 when it fails to resume.

content.tex +3 -0
+
+When the driver sets the SUSPEND bit to 1, the device SHOULD perform
+the following actions before presenting the SUSPEND bit as 1 and DRIVER_OK

bit as 0 in the \field{device status}:

content.tex +8 -0
+
+\begin{itemize}
+\item Stop consuming more buffers of any virtqueues.
+\item Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}
+
  \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
 
  Virtio can use various different buses, thus the standard is split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
  	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}

for
handling features reserved for future use.

content.tex +4 -0
+  \item[VIRTIO_F_SUSPEND(43)] This feature indicates that the driver can
+   suspend the device by set the SUSPEND bit to 1.
+   See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
+
  \end{description}
Quoted 7 lines
Quoted 4 lines

\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
--
2.49.0

Rest of the requirements and description looks good to me.

Can you please fix these 2 small things in the general description and patch-2?

patch-2 can be fixed for sure

#9 lore raw

RE: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

Quoted 2 lines

From: Zhu, Lingshan <[email protected]>
Sent: 26 June 2025 06:56 PM

[....]

I wont be able to reply to this non text format inline. So please find the reply here.

I will not be able to take RFC 2119 keywords sprinkled in general description into the spec.

Even worse, they are sprinkled non-consistently sometimes "SHOULD" and sometimes "should" in your patch.

Virtio spec restricts usage of those keywords only in the requirements section. Existing spec is an example of it in all the sections.

Please refer to the rest of the spec for such nomenclature. The reasons are also explained in [1]. Please refer [1].

You didn't answer 'why is it so hard for your patch to follow the current spec nomenclature.' i.e. to change 4 words...
If there is no reason, I do not want to debate on this anymore.
Both of us have more important things to do in virtio and otherwise. :)

And I didn't ignore your comments, I replied with [2], which the last email in the thread.

[1] https://lore.kernel.org/virtio-comment/CY8PR12MB71957FB8A0337FB62A1CDB41DC6AA@CY8PR12MB7195.namprd12.prod.outlook.com/T/#m3ea175d50e7ffc9816a322384e6766492215469f
[2] https://lore.kernel.org/virtio-comment/CY8PR12MB71957FB8A0337FB62A1CDB41DC6AA@CY8PR12MB7195.namprd12.prod.outlook.com/T/#m5d09faabdaf91a97b5844b040a19f5700c377870

#10 lore raw

Re: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

Quoted 6 lines · Parav Pandit
Quoted 1 line

From: Zhu, Lingshan<[email protected]> Sent: 26 June 2025 06:56 PM

[....]

I wont be able to reply to this non text format inline. So please find the reply here.

Quoted 6 lines

I will not be able to take RFC 2119 keywords sprinkled in general description into the spec.

Even worse, they are sprinkled non-consistently sometimes "SHOULD" and sometimes "should" in your patch.

Virtio spec restricts usage of those keywords only in the requirements section. Existing spec is an example of it in all the sections.

do you find any sections documenting this restriction? If we want this rule, we should document it first in the spec.

Quoted 13 lines

Please refer to the rest of the spec for such nomenclature. The reasons are also explained in [1]. Please refer [1].

You didn't answer 'why is it so hard for your patch to follow the current spec nomenclature.' i.e. to change 4 words...
If there is no reason, I do not want to debate on this anymore.
Both of us have more important things to do in virtio and otherwise. :)

And I didn't ignore your comments, I replied with [2], which the last email in the thread.

[1]https://lore.kernel.org/virtio-comment/CY8PR12MB71957FB8A0337FB62A1CDB41DC6AA@CY8PR12MB7195.namprd12.prod.outlook.com/T/#m3ea175d50e7ffc9816a322384e6766492215469f [2]https://lore.kernel.org/virtio-comment/CY8PR12MB71957FB8A0337FB62A1CDB41DC6AA@CY8PR12MB7195.namprd12.prod.outlook.com/T/#m5d09faabdaf91a97b5844b040a19f5700c377870

#11 lore raw

Re: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

In reply to Zhu, Lingshan
Quoted 44 lines · Lingshan
Quoted 20 lines · Parav Pandit
Quoted 12 lines

From: Zhu Lingshan<[email protected]>
Sent: 23 June 2025 02:07 PM

(quoted diff) +0 -0
@@ -629,6 +632,54 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset) before
removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And Device

(quoted diff) +3 -0
+Operation / Device Suspend}
+
+If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the
Quoted 3 lines

device by setting the SUSPEND bit in \field{device status} to 1, and the device SHOULD set the DRIVER_OK bit to 0 once it has been suspended.
+

You ignored the inputs.
I do not agree to add the "SHOULD" normative wording in the general description area. The reasoning is explained already.
Please adapt to the existing style of this spec to keep the normative in requirements section.

Quoted 5 lines

If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the device by setting the SUSPEND bit in \field{device status} to 1, and the device sets the DRIVER_OK bit to 0 once it has been suspended.

Is this really that hard to write above way?

I believe you totally ignored my replies in the last thread. There are no rules forbid using "SHOULD" in any non-normative sections. Now here I copy the reply here again:

In the spec section 1.3 Terminology, it says:

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC2119] and [RFC8174] when, and only when, they appear in all capitals, as shown here.

RFC 2119 says:

SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

Quoted 9 lines

Here this "SHOULD" exactly conforms to the definition.

"SHOULD" has already been used in non-normative sections, for example:
2.5.4 Legacy Interfaces: when using the legacy interface, drivers SHOULD read these fields multiple times until two reads generate a consistent result.

The spec does not say: “SHOULD” can only be used in driver or device requirements section.

@MST, we need your input

Not MST, but you can have my input.

We refer to the RFC for the definitions of SHOULD and friends, it does not say where to use them.

The intention is to use them in normative statements only, so that implementers have clear guidelines on the requirements. The legacy sections are arguably a special case (for implementers who want to support legacy implementations.) I don't think we want the all-caps key words leaking into other sections (any more than what might have happened already.)

#12 lore raw

RE: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

Quoted 72 lines

From: Cornelia Huck <[email protected]>
Sent: 26 June 2025 09:13 PM

Quoted 10 lines · Lingshan
Quoted 8 lines · Parav Pandit
Quoted 8 lines

From: Zhu Lingshan<[email protected]>
Sent: 23 June 2025 02:07 PM

(quoted diff) +0 -0
@@ -629,6 +632,54 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset)
before removing exposed buffers.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+\section{Device Suspend}\label{sec:General Initialization And

(quoted diff) +4 -0
+Device Operation / Device Suspend}
+
+If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to
+suspend the

device by setting the SUSPEND bit in \field{device status} to 1, and the device SHOULD set the DRIVER_OK bit to 0 once it has been

suspended.

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

+

You ignored the inputs.
I do not agree to add the "SHOULD" normative wording in the general

description area.

Quoted 2 lines
Quoted 2 lines

The reasoning is explained already.
Please adapt to the existing style of this spec to keep the normative in

requirements section.

Quoted 7 lines
Quoted 6 lines

If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the device by setting the SUSPEND bit in \field{device status} to 1, and the device sets the DRIVER_OK bit to 0 once it has been suspended.

Is this really that hard to write above way?

Quoted 7 lines

I believe you totally ignored my replies in the last thread. There are no rules forbid using "SHOULD" in any non-normative sections. Now here I copy the reply here again:

In the spec section 1.3 Terminology, it says:

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,

“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”,
“MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC2119] and [RFC8174] when, and only when, they appear in all capitals, as shown here.

Quoted 9 lines

RFC 2119 says:

SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

Quoted 4 lines

Here this "SHOULD" exactly conforms to the definition.

"SHOULD" has already been used in non-normative sections, for example:
2.5.4 Legacy Interfaces: when using the legacy interface, drivers SHOULD

read these fields multiple times until two reads generate a consistent result.

Quoted 2 lines

The spec does not say: “SHOULD” can only be used in driver or device

requirements section.

Quoted 3 lines

@MST, we need your input

Not MST, but you can have my input.

Quoted 8 lines

We refer to the RFC for the definitions of SHOULD and friends, it does not say where to use them.

The intention is to use them in normative statements only, so that implementers have clear guidelines on the requirements. The legacy sections are arguably a special case (for implementers who want to support legacy implementations.) I don't think we want the all-caps key words leaking into other sections (any more than what might have happened already.)

+1.
Let me send a short patch to update our "terminology" section to provide this guidance. It will be useful in general for everyone.
Sending soon.

#13 lore raw

Re: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

In reply to Cornelia Huck
Quoted 50 lines · Cornelia Huck
Quoted 48 lines · Lingshan
Quoted 20 lines · Parav Pandit
Quoted 12 lines

From: Zhu Lingshan<[email protected]>
Sent: 23 June 2025 02:07 PM

(quoted diff) +0 -0
@@ -629,6 +632,54 @@ \section{Device Cleanup}\label{sec:General

Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset) before
removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And Device

(quoted diff) +3 -0
+Operation / Device Suspend}
+
+If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the
Quoted 3 lines

device by setting the SUSPEND bit in \field{device status} to 1, and the device SHOULD set the DRIVER_OK bit to 0 once it has been suspended.
+

You ignored the inputs.
I do not agree to add the "SHOULD" normative wording in the general description area. The reasoning is explained already.
Please adapt to the existing style of this spec to keep the normative in requirements section.

Quoted 5 lines

If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the device by setting the SUSPEND bit in \field{device status} to 1, and the device sets the DRIVER_OK bit to 0 once it has been suspended.

Is this really that hard to write above way?

I believe you totally ignored my replies in the last thread. There are no rules forbid using "SHOULD" in any non-normative sections. Now here I copy the reply here again:

In the spec section 1.3 Terminology, it says:

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC2119] and [RFC8174] when, and only when, they appear in all capitals, as shown here.

RFC 2119 says:

SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

Here this "SHOULD" exactly conforms to the definition.

"SHOULD" has already been used in non-normative sections, for example:
2.5.4 Legacy Interfaces: when using the legacy interface, drivers SHOULD read these fields multiple times until two reads generate a consistent result.

Quoted 8 lines
Quoted 4 lines

The spec does not say: “SHOULD” can only be used in driver or device requirements section.

@MST, we need your input

Not MST, but you can have my input.

We refer to the RFC for the definitions of SHOULD and friends, it does not say where to use them.

exactly

Quoted 7 lines

The intention is to use them in normative statements only, so that implementers have clear guidelines on the requirements. The legacy sections are arguably a special case (for implementers who want to support legacy implementations.) I don't think we want the all-caps key words leaking into other sections (any more than what might have happened already.)

so this is a new requirement, we need to document these guidance

Quoted 1 line
#14 lore raw

Re: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

Quoted 10 lines · Lingshan
Quoted 6 lines · Cornelia Huck

The intention is to use them in normative statements only, so that implementers have clear guidelines on the requirements. The legacy sections are arguably a special case (for implementers who want to support legacy implementations.) I don't think we want the all-caps key words leaking into other sections (any more than what might have happened already.)

so this is a new requirement, we need to document these guidance

Not new, but not documented, either. I see that Parav is already tackling the latter part (thanks!)

#15 lore raw

Re: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

In reply to Zhu, Lingshan
Quoted 79 lines · Lingshan

On 6/26/2025 7:59 PM, Parav Pandit wrote:

From: Zhu Lingshan <[email protected]>
Sent: 23 June 2025 02:07 PM

This commit allows the driver to suspend the device through a new device status bit SUSPEND and resume the device running by re-setting DRIVER_OK bit in device status.

Fixes:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
ub.com%2Foasis-tcs%2Fvirtio- spec%2Fissues%2F229&data=05%7C02%7Cparav%40nvidia.com%7C3085ab 378f264b5c845808ddb2313478%7C43083d15727340c1b7db39efd9ccc17 a%7C0%7C0%7C638862646595099882%7CUnknown%7CTWFpbGZsb3d8e yJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pf2ub%2FwVzdmf A1JjScOJ7D%2B%2BNea9XgVFYdvvDQxym40%3D&reserved=0
---
content.tex | 59
+++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 57 insertions(+), 2 deletions(-)

diff --git a/content.tex b/content.tex
index 2e8da46..c987334c 100644
--- a/content.tex
+++ b/content.tex
@@ -42,6 +42,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the
features it understands, and feature negotiation is complete.

+\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that +the
+ device has been suspended by the driver.
+
\item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
an error from which it can't recover.

@@ -99,10 +102,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device type

-\item[24 to 42] Feature bits reserved for extensions to the queue and +\item[24 to 43] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}

Ignoring this delta as the fix of patch-2 will change this.

-\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[44 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}

\begin{note}
@@ -629,6 +632,54 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation /

Thus a driver MUST ensure a virtqueue isn't live (by device reset) before
removing exposed buffers.

+\section{Device Suspend}\label{sec:General Initialization And Device +Operation / Device Suspend}
+
+If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the device by setting the SUSPEND bit in \field{device status} to 1, and the device SHOULD set the DRIVER_OK bit to 0 once it has been suspended.
+

You ignored the inputs.
I do not agree to add the "SHOULD" normative wording in the general description area. The reasoning is explained already.
Please adapt to the existing style of this spec to keep the normative in requirements section.

Quoted 8 lines

If VIRTIO_F_SUSPEND is negotiated, the driver is eligible to suspend the device by setting the SUSPEND bit in \field{device status} to 1, and the device sets the DRIVER_OK bit to 0 once it has been suspended.

Is this really that hard to write above way?

I believe you totally ignored my replies in the last thread. There are no rules forbid using "SHOULD" in any non-normative sections.

While true, it is confusing to the reader, and tends to make writers sloopy about adding confirmance statements.

Our recommendation (which we do not always remembered to follow, sadly) is to do it like this:

In the non confirmance sections, avoid these keywords, describing the functionality in a detailed way but without putting too much emphasis on who does what and the specific level or requirement and without using keywords.

Additionally, repeat the requirement in a short but formal way in the confirmance section. For details of the operation, reader has the non-confirmance sections.

Hope that helps.

#16 lore raw

RE: [PATCH v3 3/3] virtio: introduce SUSPEND and RESUME feature

In reply to Parav Pandit

Hi Lingshan,

Quoted 2 lines

From: Parav Pandit <[email protected]>
Sent: 26 June 2025 05:30 PM

[..]

Quoted 3 lines

Rest of the requirements and description looks good to me.

Can you please fix these 2 small things in the general description and patch-2?

This series in my view is reaching nearly good state, when you post v4 with these small changes, Can you please add below line to the commit log?

This helps to connect github issue, patches and subsequently voting ballot.