virtio: introduce SUSPEND bit in device status
PATCH v5 1/1 virtio: introduce SUSPEND bit in device status
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.
This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.
--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)
content.tex +59 -10
@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by
@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a
@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}
@@ -629,6 +633,47 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers. +\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set. +If not, the device does not support the SUSPEND feature. +\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize} + +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}. + +A device MUST NOT send any notifications, access any virtqueues, or modify +any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND, +the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting SUSPEND bit in the \field{device status}: + +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split
@@ -872,6 +917,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + set SUSPEND to the device. + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.45.1
Quoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.
Quoted 7 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)
Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 649
Quoted 8 lines
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
content.tex +7
-0+ +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set. +If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer VIRTIO_F_SUSPEND in the first place in that case... could this rather happen if the device is not able to handle the request at a specific point in time?
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 8 lines
+\end{itemize}
content.tex +7
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}.
...if it is part of the configuration space.
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 665
Quoted 5 lines
content.tex +5
-0+ +A device MUST NOT send any notifications, access any virtqueues, or modify +any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,
"subsequently clears SUSPEND"?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 10 lines
content.tex +10
-0+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting SUSPEND bit in the \field{device status}: + +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}
So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 921
Quoted 10 lines
content.tex +3
-0+ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split@@ -872,6 +917,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + set SUSPEND to the device.
"that the driver can trigger suspending the device via the SUSPEND flag"?
Quoted 5 lines
content.tex +2
-0+ See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Hi Lingshan, David,
Quoted 79 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General Initialization
(quoted diff) +3
-0+And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 4 lines
content.tex +4
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point in time?Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization +And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 3 lines
content.tex +3
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}....if it is part of the configuration space.
Review
Quoted 6 lines
(quoted diff) +6
-0+ +A device MUST NOT send any notifications, access any virtqueues, or +modify any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 3 lines
content.tex +3
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presentingSUSPEND bit in the \field{device status}:
Quoted 11 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 6 lines
content.tex +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait until +all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Additionally, a suspend operation usually involves saving things to a slow memory (or media).
This is because the device implementation wouldn’t know when exactly the device will be resumed.
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMU needs to store this into a (for example) rotating hard disk as 1msec IO latency.
b. a NIC may need to store its RSS, queues, flow filters configuration for several tens of KBs to some slow memory.
c. A block device may prefer to complete some IOs to a threshold level instead of maintaining large list of outstanding IOs in some suspended memory.
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data path hardware engines to suspend DMA operations (read and write both) which are inflight. (without causing DMA errors). Same would apply to the sw backend implementations too.
Therefore, the suspend operation that is initiated by the driver, should get the acknowledgement back from the device that it has been suspend.
Some of the good examples if you prefer to follow, a driver<->device interface needs a suspend register which should behave like below queue reset register.
Spec snippet:
"The device MUST reset the queue when 1 is written to queue_reset. The device MUST continue to present 1 in queue_reset as long as the queue reset is ongoing. The device MUST present 0 in both queue_reset and queue_enable when queue reset has completed."
At minimum, we need, something implementable like,
The device MUST suspend the device when 1 is written to the device_suspend_ctlr. The device MUST continue to present 1 in device_suspend_ctrl as the suspend operation is ongoing in the device. The device MUST present 0 in the device_suspend_ctlr register when device has completely suspended the device.
The device MUST resume the device when 2 is written to the device_suspend_ctrl. The device MUST continue to present 2 in the device_suspend_ctrl as the resume operation may not have yet completed. The device MUST present 0 in the device_suspend_ctrl register when the device has completely resumed the device.
At this point, the driver may resume notifying the device and accessing the configuration space.
Can you please enhance this part?
Quoted 23 lines
Quoted 6 lines
+
\chapter{Virtio Transport Options}\label{sec:Virtio Transport
Options}Virtio can use various different buses, thus the standard is split @@
-872,6 +917,10 @@ \chapter{Reserved Feature Bits}\label{sec:ReservedFeature Bits}
Quoted 1 line
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}
for
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 921
Quoted 4 lines
content.tex +2
-0handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + set SUSPEND to the device."that the driver can trigger suspending the device via the SUSPEND flag"?
Quoted 6 lines
content.tex +2
-0+ See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved FeatureBits}
Quoted 111 lines · Parav Pandit
Hi Lingshan, David,
Quoted 74 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General Initialization
(quoted diff) +3
-0+And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 4 lines
content.tex +4
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point in time?Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization +And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 3 lines
content.tex +3
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}....if it is part of the configuration space.
Review
Quoted 6 lines
(quoted diff) +6
-0+ +A device MUST NOT send any notifications, access any virtqueues, or +modify any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 3 lines
content.tex +3
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presentingSUSPEND bit in the \field{device status}:
Quoted 10 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 6 lines
content.tex +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait until +all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Additionally, a suspend operation usually involves saving things to a slow memory (or media).
This is because the device implementation wouldn’t know when exactly the device will be resumed.
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMU needs to store this into a (for example) rotating hard disk as 1msec IO latency.b. a NIC may need to store its RSS, queues, flow filters configuration for several tens of KBs to some slow memory.
c. A block device may prefer to complete some IOs to a threshold level instead of maintaining large list of outstanding IOs in some suspended memory.
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data path hardware engines to suspend DMA operations (read and write both) which are inflight. (without causing DMA errors). Same would apply to the sw backend implementations too.
Therefore, the suspend operation that is initiated by the driver, should get the acknowledgement back from the device that it has been suspend.
Some of the good examples if you prefer to follow, a driver<->device interface needs a suspend register which should behave like below queue reset register.
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present 1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue reset has completed."At minimum, we need, something implementable like,
Quoted 8 lines
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device. The device MUST present 0 in thedevice_suspend_ctlrregister when device has completely suspended the device.The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yet completed. The device MUST present 0 in thedevice_suspend_ctrlregister when the device has completely resumed the device.
At this point, the driver may resume notifying the device and accessing the configuration space.Can you please enhance this part?
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
This patch focus on the basic facility virito suspending, not transport or device types, this is to make this chageset focused and small.
Thanks,
Zhu Linghsan
Quoted 23 lines
Quoted 23 lines
Quoted 8 lines
+
\chapter{Virtio Transport Options}\label{sec:Virtio Transport
Options}Virtio can use various different buses, thus the standard is split @@
-872,6 +917,10 @@ \chapter{Reserved Feature Bits}\label{sec:ReservedFeature Bits}
Quoted 1 line
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}
for
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 921
Quoted 4 lines
content.tex +2
-0handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + set SUSPEND to the device."that the driver can trigger suspending the device via the SUSPEND flag"?
Quoted 6 lines
content.tex +2
-0+ See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved FeatureBits}
Quoted 146 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 41 lines · Parav Pandit
Hi Lingshan, David,
Quoted 39 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General
(quoted diff) +3
-0+Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 5 lines
content.tex +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to the
device.
Quoted 45 lines
Quoted 35 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any virtqueues or +notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 3 lines
content.tex +3
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}....if it is part of the configuration space.
Review
Quoted 6 lines
(quoted diff) +6
-0+ +A device MUST NOT send any notifications, access any virtqueues, or +modify any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 3 lines
content.tex +3
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presentingSUSPEND bit in the \field{device status}:
Quoted 10 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 6 lines
content.tex +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 1 line
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 1 line
This is because the device implementation wouldn’t know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMU needsto store this into a (for example) rotating hard disk as 1msec IO latency.
Quoted 2 lines
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 2 lines
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 4 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 1 line
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 2 lines
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been suspend.
Quoted 2 lines
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 6 lines
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 4 lines
At minimum, we need, something implementable like,
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device.Quoted 1 line
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 2 lines
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yet completed.Quoted 1 line
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 9 lines
Quoted 1 line
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 2 lines
Can you please enhance this part?
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the device after driver sets it (until the device is suspended). This is ambiguous behavior of the register. Queue_reset register was like that before it was fixed in 1.2.
Quoted 3 lines
This patch focus on the basic facility virito suspending, not transport or device types, this is to make this chageset focused and small.
If suspend bit is part of the device_status, it is already covered by the PCI transport. So it is fine, but its polarity is an issue, that we need to fix.
i.e.
One driver writes 1, it needs to stay 1, while the suspend is ongoing. When suspension is completed in the device, the device to turn it to zero.
If the device_status register does not fullfil this, a new register is worthwhile to add in this patch.
Quoted 139 lines · Parav Pandit
Quoted 139 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 41 lines · Parav Pandit
Hi Lingshan, David,
Quoted 39 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General
(quoted diff) +3
-0+Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 5 lines
content.tex +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 1 line
Quoted 1 line
Quoted 1 line
+\item The driver MUST NOT make any more buffers available to the
device.
Quoted 45 lines
Quoted 35 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any virtqueues or +notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 3 lines
content.tex +3
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}....if it is part of the configuration space.
Review
Quoted 6 lines
(quoted diff) +6
-0+ +A device MUST NOT send any notifications, access any virtqueues, or +modify any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 3 lines
content.tex +3
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presentingSUSPEND bit in the \field{device status}:
Quoted 10 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 6 lines
content.tex +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 1 line
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 1 line
This is because the device implementation wouldn’t know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMU needsto store this into a (for example) rotating hard disk as 1msec IO latency.
Quoted 1 line
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 1 line
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 3 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 1 line
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 1 line
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been suspend.
Quoted 1 line
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 5 lines
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 3 lines
At minimum, we need, something implementable like,
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device.Quoted 1 line
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 1 line
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yet completed.Quoted 1 line
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 1 line
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 1 line
Can you please enhance this part?
Quoted 8 lines
Quoted 5 lines
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the device after driver sets it (until the device is suspended). This is ambiguous behavior of the register.Queue_resetregister was like that before it was fixed in 1.2.
once suspended, the driver should not access the configuration space except for the device status, so it should not reset any vqs through configuration space.
Quoted 6 lines
Quoted 4 lines
This patch focus on the basic facility virito suspending, not transport or device types, this is to make this chageset focused and small.
If suspend bit is part of the
device_status, it is already covered by the PCI transport. So it is fine, but its polarity is an issue, that we need to fix.
Oh, I see your confusion.
All transport have their own implementation of configuration space, this is not PCI specific. PCI config space != virtio configuration space. And this patch focuses on virtio layer.
Quoted 6 lines
i.e.
One driver writes 1, it needs to stay 1, while the suspend is ongoing. When suspension is completed in the device, the device to turn it to zero.If the
device_statusregister does not fullfil this, a new register is worthwhile to add in this patch.
The driver writes SUSPEND, once the device is suspened, it should present SUSPEND in its device status.
Thanks
Zhu Lingshan
Quoted 1 line
Quoted 158 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:03 PMQuoted 153 lines · Parav Pandit
Quoted 153 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 42 lines · Parav Pandit
Hi Lingshan, David,
Quoted 40 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General
(quoted diff) +3
-0+Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 5 lines
content.tex +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 1 line
Quoted 1 line
Quoted 1 line
+\item The driver MUST NOT make any more buffers available to the
device.
Quoted 45 lines
Quoted 35 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any virtqueues or +notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 3 lines
(quoted diff) +3
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status}....if it is part of the configuration space.
Review
Quoted 6 lines
(quoted diff) +6
-0+ +A device MUST NOT send any notifications, access any virtqueues, +or modify any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 3 lines
content.tex +3
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presentingSUSPEND bit in the \field{device status}:
Quoted 10 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 6 lines
content.tex +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 2 lines
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 2 lines
This is because the device implementation wouldn’t know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMU needsto store this into a (for example) rotating hard disk as 1msec IO latency.
Quoted 2 lines
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 2 lines
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 4 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 1 line
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 2 lines
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been suspend.
Quoted 1 line
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 5 lines
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 3 lines
At minimum, we need, something implementable like,
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device.Quoted 1 line
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 1 line
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yet completed.Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 2 lines
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 1 line
Can you please enhance this part?
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
Quoted 8 lines
Quoted 2 lines
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the deviceafter driver sets it (until the device is suspended).
Quoted 1 line
This is ambiguous behavior of the register.
Queue_resetregister was like thatbefore it was fixed in 1.2.
once suspended, the driver should not access the configuration space except for the device status, so it should not reset any vqs through configuration space.
I am not talking about VQ reset at all.
I gave an example of how the VQ reset register polarity works (with was broken like how the proposed suspend bit is broken).
Suspend register needs to work the way the reset register works. Can you please go through my exact text I replied in the previous email?
Basically, the bug is:
When driver has initiated the suspend (writing 1), the device continue to respond 0, while suspend is going on.
At this point, just by looking at the device_status register one cannot figure out what is going on in the device. Is driver initiated suspend is ongoing, or it was never started.
It is ambiguous.
Hence, it needs to work like how the queue_reset works.
i.e. if suspend is ongoing, one should be able to read device_status without involving driver and understand that it is ongoing.
Quoted 13 lines
Quoted 5 lines
Quoted 4 lines
This patch focus on the basic facility virito suspending, not transport or device types, this is to make this chageset focused and small.
If suspend bit is part of the
device_status, it is already covered by the PCItransport.
Quoted 1 line
So it is fine, but its polarity is an issue, that we need to fix.
Oh, I see your confusion.
All transport have their own implementation of configuration space, this is not PCI specific.
PCI config space != virtio configuration space. And this patch focuses on virtio layer.Quoted 1 line
Quoted 8 lines
Quoted 5 lines
i.e.
One driver writes 1, it needs to stay 1, while the suspend is ongoing. When suspension is completed in the device, the device to turn it to zero.If the
device_statusregister does not fullfil this, a new register is worthwhileto add in this patch.
The driver writes SUSPEND, once the device is suspened, it should present SUSPEND in its device status.
Yes, I understood what you are saying.
It is ambiguous like how I explained above.
The suspend bit needs to work like queue_reset register. Please go through the queue reset specification text.
I explained this in context of VQ reset, at [1].
Suspend bit needs to avoid similar ambiguity.
Quoted 170 lines · Parav Pandit
Quoted 156 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:03 PMQuoted 152 lines · Parav Pandit
Quoted 152 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 42 lines · Parav Pandit
Hi Lingshan, David,
Quoted 40 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General
(quoted diff) +3
-0+Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 5 lines
(quoted diff) +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 1 line
Quoted 1 line
Quoted 1 line
+\item The driver MUST NOT make any more buffers available to the
device.
Quoted 45 lines
Quoted 35 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any virtqueues or +notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 3 lines
(quoted diff) +3
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status}....if it is part of the configuration space.
Review
Quoted 6 lines
(quoted diff) +6
-0+ +A device MUST NOT send any notifications, access any virtqueues, +or modify any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.Review
Quoted 3 lines
(quoted diff) +3
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presentingSUSPEND bit in the \field{device status}:
Quoted 10 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 2 lines
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 2 lines
This is because the device implementation wouldn’t know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMU needsto store this into a (for example) rotating hard disk as 1msec IO latency.
Quoted 2 lines
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 2 lines
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 4 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 1 line
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 2 lines
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been suspend.
Quoted 1 line
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 5 lines
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 3 lines
At minimum, we need, something implementable like,
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device.Quoted 1 line
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 1 line
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yet completed.Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 2 lines
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 1 line
Can you please enhance this part?
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
Quoted 8 lines
Quoted 2 lines
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the deviceafter driver sets it (until the device is suspended).
Quoted 1 line
This is ambiguous behavior of the register.
Queue_resetregister was like thatbefore it was fixed in 1.2.
once suspended, the driver should not access the configuration space except for the device status, so it should not reset any vqs through configuration space.I am not talking about VQ reset at all.
I gave an example of how the VQ reset register polarity works (with was broken like how the proposed suspend bit is broken).Suspend register needs to work the way the reset register works. Can you please go through my exact text I replied in the previous email?
Quoted 6 lines
Basically, the bug is:
When driver has initiated the suspend (writing 1), the device continue to respond 0, while suspend is going on.At this point, just by looking at the
device_statusregister one cannot figure out what is going on in the device. Is driver initiated suspend is ongoing, or it was never started.
It is ambiguous.
From the device perspective, there is only one user: the driver, and the driver sets or clears SUSPEND, it knows the status of the device. Differs from vq reset, in this patch there is a section describing how the device should behave suspending. So once the driver sets SUSPEND, the device should perform the required actions and suspend itself, during this process, SUSPEND bit is still 0, and the drivers know that, like how we handle FEATURES_OK. Once the device suspended itself, it present SUSPEND == 1 in the device status.
1 bit controls whether SUSPEND-ed.
Quoted 3 lines
Hence, it needs to work like how the
queue_resetworks.
i.e. if suspend is ongoing, one should be able to readdevice_statuswithout involving driver and understand that it is ongoing.
there are other users than the driver?
Quoted 23 lines
Quoted 12 lines
Quoted 5 lines
Quoted 4 lines
This patch focus on the basic facility virito suspending, not transport or device types, this is to make this chageset focused and small.
If suspend bit is part of the
device_status, it is already covered by the PCItransport.
Quoted 1 line
So it is fine, but its polarity is an issue, that we need to fix.
Oh, I see your confusion.
All transport have their own implementation of configuration space, this is not PCI specific.
PCI config space != virtio configuration space. And this patch focuses on virtio layer.Quoted 8 lines
Quoted 5 lines
i.e.
One driver writes 1, it needs to stay 1, while the suspend is ongoing. When suspension is completed in the device, the device to turn it to zero.If the
device_statusregister does not fullfil this, a new register is worthwhileto add in this patch.
The driver writes SUSPEND, once the device is suspened, it should present SUSPEND in its device status.Yes, I understood what you are saying.
It is ambiguous like how I explained above.
Quoted 6 lines
The suspend bit needs to work like
queue_resetregister. Please go through the queue reset specification text.I explained this in context of VQ reset, at [1].
Suspend bit needs to avoid similar ambiguity.
replied above
Thanks
Zhu Lingshan
Quoted 187 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:42 PMQuoted 70 lines · Parav Pandit
Quoted 70 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:03 PMQuoted 66 lines · Parav Pandit
Quoted 66 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 42 lines · Parav Pandit
Hi Lingshan, David,
Quoted 40 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General
(quoted diff) +3
-0+Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 5 lines
(quoted diff) +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to +the
device.
Quoted 16 lines
Quoted 16 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any virtqueues +or notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status}....if it is part of the configuration space.
Review
Quoted 3 lines
(quoted diff) +3
-0+ +A device MUST NOT send any notifications, access any +virtqueues, or modify any fields in its configuration space whilesuspended.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.Review
Quoted 4 lines
(quoted diff) +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before +presentingSUSPEND bit in the \field{device status}:
Review
Quoted 6 lines
(quoted diff) +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 2 lines
This is because the device implementation wouldn't know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMUneeds
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
to store this into a (for example) rotating hard disk as 1msec IO latency.
Quoted 2 lines
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 2 lines
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 4 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 1 line
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 2 lines
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been suspend.
Quoted 2 lines
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 6 lines
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 2 lines
At minimum, we need, something implementable like,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device.Quoted 1 line
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 1 line
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yetcompleted.
Quoted 16 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 2 lines
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 1 line
Can you please enhance this part?
Quoted 8 lines
Quoted 5 lines
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the deviceafter driver sets it (until the device is suspended).
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
This is ambiguous behavior of the register.
Queue_resetregister was like thatbefore it was fixed in 1.2.
once suspended, the driver should not access the configuration space except for the device status, so it should not reset any vqs through configuration space.I am not talking about VQ reset at all.
I gave an example of how the VQ reset register polarity works (with wasbroken like how the proposed suspend bit is broken).
Quoted 6 lines
Suspend register needs to work the way the reset register works. Can you please go through my exact text I replied in the previous email?
Basically, the bug is:
When driver has initiated the suspend (writing 1), the device continue to
Quoted 8 lines
respond 0, while suspend is going on.
Quoted 2 lines
At this point, just by looking at the
device_statusregister one cannot figureout what is going on in the device.
Quoted 2 lines
Is driver initiated suspend is ongoing, or it was never started. It is ambiguous.
From the device perspective, there is only one user: the driver, and the driver sets or clears SUSPEND, it knows the status of the device.
It does not work elegantly when the hypervisor or other system is looking dealing with this register.
In current proposal, anyone else than the driver looking at the device_status, it is ambiguous.
For example,suspend_bit: 0x1 suspended.
0x0 not suspended or suspend is ongoing. (ambiguous for hypervisor debugging this device)
For device migration, it requires yet another side bit in the device parts.
An elegant and non-ambiguous interface can be,
ctrl_register = 0x0 (a usual default)
0x1, suspend the device 0x2, resume the device. status_register = 0x0, (still) running. 0x1, suspended.
With this driver and hypervisor (both) has very clear view of what is going on and what is the status. No more ambiguity for anyone.
Please consider pros and cons of both approaches.
Quoted 210 lines · Parav Pandit
Quoted 185 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:42 PMQuoted 70 lines · Parav Pandit
Quoted 70 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:03 PMQuoted 66 lines · Parav Pandit
Quoted 66 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 42 lines · Parav Pandit
Hi Lingshan, David,
Quoted 40 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General
(quoted diff) +3
-0+Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 5 lines
(quoted diff) +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to +the
device.
Quoted 16 lines
Quoted 16 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any virtqueues +or notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status}....if it is part of the configuration space.
Review
Quoted 3 lines
(quoted diff) +3
-0+ +A device MUST NOT send any notifications, access any +virtqueues, or modify any fields in its configuration space whilesuspended.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.Review
Quoted 4 lines
(quoted diff) +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before +presentingSUSPEND bit in the \field{device status}:
Review
Quoted 6 lines
(quoted diff) +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 2 lines
This is because the device implementation wouldn't know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMUneeds
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
to store this into a (for example) rotating hard disk as 1msec IO latency.
Quoted 2 lines
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 2 lines
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 4 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 1 line
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 2 lines
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been suspend.
Quoted 2 lines
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 6 lines
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 2 lines
At minimum, we need, something implementable like,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device.Quoted 1 line
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 1 line
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yetcompleted.
Quoted 16 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 2 lines
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 1 line
Can you please enhance this part?
Quoted 8 lines
Quoted 5 lines
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the deviceafter driver sets it (until the device is suspended).
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
This is ambiguous behavior of the register.
Queue_resetregister was like thatbefore it was fixed in 1.2.
once suspended, the driver should not access the configuration space except for the device status, so it should not reset any vqs through configuration space.I am not talking about VQ reset at all.
I gave an example of how the VQ reset register polarity works (with wasbroken like how the proposed suspend bit is broken).
Quoted 5 lines
Suspend register needs to work the way the reset register works. Can you please go through my exact text I replied in the previous email?
Basically, the bug is:
When driver has initiated the suspend (writing 1), the device continue toQuoted 7 lines
respond 0, while suspend is going on.
Quoted 1 line
At this point, just by looking at the
device_statusregister one cannot figureout what is going on in the device.
Quoted 2 lines
Is driver initiated suspend is ongoing, or it was never started. It is ambiguous.
From the device perspective, there is only one user: the driver, and the driver sets or clears SUSPEND, it knows the status of the device.
It does not work elegantly when the hypervisor or other system is looking dealing with this register.
In current proposal, anyone else than the driver looking at the
device_status, it is ambiguous.For example,
suspend_bit:0x1suspended.0x0 not suspended or suspend is ongoing. (ambiguous for hypervisor debugging this device)For device migration, it requires yet another side bit in the device parts.
An elegant and non-ambiguous interface can be,
ctrl_register=0x0(a usual default)0x1, suspend the device 0x2, resume the device. status_register = 0x0, (still) running. 0x1, suspended.
Quoted 4 lines
With this driver and hypervisor (both) has very clear view of what is going on and what is the status. No more ambiguity for anyone.
Please consider pros and cons of both approaches.
The driver "owns" the device, other components access the device configurations through the driver, I am not sure there are other "owners" than the driver.
Thanks
Zhu Lingshan
Quoted 224 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, June 12, 2024 2:50 PMQuoted 33 lines · Parav Pandit
Quoted 33 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:42 PMQuoted 29 lines · Parav Pandit
Quoted 29 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:03 PMQuoted 25 lines · Parav Pandit
Quoted 25 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 21 lines · Parav Pandit
Hi Lingshan, David,
Quoted 19 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 14 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing
lists...
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General
(quoted diff) +3
-0+Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 2 lines
(quoted diff) +2
-0+ +Once the driver sets SUSPEND to \field{device status} of thedevice:
Quoted 144 lines
Quoted 144 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to +the
device.
Quoted 21 lines
Quoted 21 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any virtqueues +or notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 3 lines
(quoted diff) +3
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status}....if it is part of the configuration space.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ +A device MUST NOT send any notifications, access any +virtqueues, or modify any fields in its configuration space +whilesuspended.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.Review
Quoted 3 lines
(quoted diff) +3
-0+ +When the driver sets SUSPEND, the device SHOULD perform the +following actions before presentingSUSPEND bit in the \field{device status}:
Review
Quoted 6 lines
(quoted diff) +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item +Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 2 lines
This is because the device implementation wouldn't know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMUneeds
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
to store this into a (for example) rotating hard disk as 1msec IO latency.
Quoted 2 lines
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 2 lines
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 4 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 1 line
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 2 lines
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been suspend.
Quoted 2 lines
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 6 lines
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 3 lines
At minimum, we need, something implementable like,
The device MUST suspend the device when 1 is written to the
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device.Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 1 line
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yetcompleted.
Quoted 18 lines
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 2 lines
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 1 line
Can you please enhance this part?
Quoted 8 lines
Quoted 5 lines
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the deviceafter driver sets it (until the device is suspended).
Quoted 2 lines
This is ambiguous behavior of the register.
Queue_resetregister was like thatQuoted 7 lines
Quoted 4 lines
before it was fixed in 1.2.
once suspended, the driver should not access the configuration space except for the device status, so it should not reset any vqs through configuration space.I am not talking about VQ reset at all.
I gave an example of how the VQ reset register polarity works (with wasbroken like how the proposed suspend bit is broken).
Quoted 6 lines
Suspend register needs to work the way the reset register works. Can you please go through my exact text I replied in the previous email?
Basically, the bug is:
When driver has initiated the suspend (writing 1), the device continue torespond 0, while suspend is going on.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
At this point, just by looking at the
device_statusregister one cannot figureout what is going on in the device.
Quoted 2 lines
Is driver initiated suspend is ongoing, or it was never started. It is ambiguous.
From the device perspective, there is only one user: the driver, and the driver sets or clears SUSPEND, it knows the status of the device.
It does not work elegantly when the hypervisor or other system is looking
dealing with this register.
Quoted 2 lines
In current proposal, anyone else than the driver looking at the
device_status,it is ambiguous.
Quoted 16 lines
For example,
suspend_bit:0x1suspended.0x0 not suspended or suspend is ongoing. (ambiguous forhypervisor debugging this device)
For device migration, it requires yet another side bit in the device parts.
An elegant and non-ambiguous interface can be,
ctrl_register=0x0(a usual default)0x1, suspend the device 0x2, resume the device. status_register = 0x0, (still) running. 0x1, suspended.
Quoted 8 lines
Quoted 1 line
With this driver and hypervisor (both) has very clear view of what is going on
and what is the status.
Quoted 3 lines
No more ambiguity for anyone.
Please consider pros and cons of both approaches.
The driver "owns" the device, other components access the device configurations through the driver, I am not sure there are other "owners" than the driver.
Device migration and debug channel [1] plans to access the device from the hypervisor. I don’t see any disadvantage of separate control and status register.
Do you see any?
Quoted 225 lines · Parav Pandit
Quoted 225 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, June 12, 2024 2:50 PMQuoted 33 lines · Parav Pandit
Quoted 33 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:42 PMQuoted 29 lines · Parav Pandit
Quoted 29 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:03 PMQuoted 25 lines · Parav Pandit
Quoted 25 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 21 lines · Parav Pandit
Hi Lingshan, David,
Quoted 19 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 14 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing
lists...
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General
(quoted diff) +3
-0+Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 2 lines
(quoted diff) +2
-0+ +Once the driver sets SUSPEND to \field{device status} of thedevice:
Quoted 144 lines
Quoted 144 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to +the
device.
Quoted 21 lines
Quoted 21 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any virtqueues +or notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 3 lines
(quoted diff) +3
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status}....if it is part of the configuration space.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ +A device MUST NOT send any notifications, access any +virtqueues, or modify any fields in its configuration space +whilesuspended.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.Review
Quoted 3 lines
(quoted diff) +3
-0+ +When the driver sets SUSPEND, the device SHOULD perform the +following actions before presentingSUSPEND bit in the \field{device status}:
Review
Quoted 6 lines
(quoted diff) +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item +Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 2 lines
This is because the device implementation wouldn't know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMUneeds
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
to store this into a (for example) rotating hard disk as 1msec IO latency.
Quoted 2 lines
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 2 lines
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 4 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 1 line
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 2 lines
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been suspend.
Quoted 2 lines
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 6 lines
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 3 lines
At minimum, we need, something implementable like,
The device MUST suspend the device when 1 is written to the
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device.Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 1 line
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yetcompleted.
Quoted 18 lines
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 2 lines
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 1 line
Can you please enhance this part?
Quoted 8 lines
Quoted 5 lines
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the deviceafter driver sets it (until the device is suspended).
Quoted 2 lines
This is ambiguous behavior of the register.
Queue_resetregister was like thatQuoted 7 lines
Quoted 4 lines
before it was fixed in 1.2.
once suspended, the driver should not access the configuration space except for the device status, so it should not reset any vqs through configuration space.I am not talking about VQ reset at all.
I gave an example of how the VQ reset register polarity works (with wasbroken like how the proposed suspend bit is broken).
Quoted 6 lines
Suspend register needs to work the way the reset register works. Can you please go through my exact text I replied in the previous email?
Basically, the bug is:
When driver has initiated the suspend (writing 1), the device continue torespond 0, while suspend is going on.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
At this point, just by looking at the
device_statusregister one cannot figureout what is going on in the device.
Quoted 2 lines
Is driver initiated suspend is ongoing, or it was never started. It is ambiguous.
From the device perspective, there is only one user: the driver, and the driver sets or clears SUSPEND, it knows the status of the device.
It does not work elegantly when the hypervisor or other system is looking
dealing with this register.
Quoted 1 line
In current proposal, anyone else than the driver looking at the
device_status,it is ambiguous.
Quoted 9 lines
For example,
suspend_bit:0x1suspended.0x0 not suspended or suspend is ongoing. (ambiguous forhypervisor debugging this device)
For device migration, it requires yet another side bit in the device parts.
An elegant and non-ambiguous interface can be,
Quoted 7 lines
ctrl_register=0x0(a usual default)0x1, suspend the device 0x2, resume the device. status_register = 0x0, (still) running. 0x1, suspended.With this driver and hypervisor (both) has very clear view of what is going on
and what is the status.
Quoted 3 lines
No more ambiguity for anyone.
Please consider pros and cons of both approaches.
Quoted 8 lines
Quoted 3 lines
The driver "owns" the device, other components access the device configurations through the driver, I am not sure there are other "owners" than the driver.
Device migration and debug channel [1] plans to access the device from the hypervisor. I don’t see any disadvantage of separate control and status register.
Do you see any?
is the admin vq governed by the driver? There should be only one owner of the device, or you need locks.
I am not sure we want to repeat the discussion on the admin vq for live migration last year, you know it breaks nested.
Quoted 271 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, June 12, 2024 4:00 PMQuoted 22 lines · Parav Pandit
Quoted 22 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, June 12, 2024 2:50 PMQuoted 18 lines · Parav Pandit
Quoted 18 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:42 PMQuoted 14 lines · Parav Pandit
Quoted 14 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:03 PMQuoted 10 lines · Parav Pandit
Quoted 10 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 6 lines · Parav Pandit
Hi Lingshan, David,
Quoted 4 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMOn Fri, Jun 07 2024, Zhu Lingshan <[email protected]>
Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device supportSUSPEND.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69+++++++++++++++++++++++++++++++++++++++++++++------
--
Quoted 67 lines
Quoted 67 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing
lists...
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
(...)
Review
Quoted 5 lines
+\drivernormative{\subsection}{Device Suspend}{General
(quoted diff) +4
-0+Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set +or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 3 lines
(quoted diff) +3
-0+ +Once the driver sets SUSPEND to \field{device status} of +thedevice:
Quoted 50 lines
Quoted 50 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to +verify whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available +to the
device.
Quoted 21 lines
Quoted 21 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any +virtqueues or notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Review
Quoted 3 lines
(quoted diff) +3
-0+ +The device MUST ignore all access to its Configuration +Space while suspended, except for \field{device status}....if it is part of the configuration space.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ +A device MUST NOT send any notifications, access any +virtqueues, or modify any fields in its configuration space +whilesuspended.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.Review
Quoted 3 lines
(quoted diff) +3
-0+ +When the driver sets SUSPEND, the device SHOULD perform the +following actions before presentingSUSPEND bit in the \field{device status}:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item +Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting
SUSPEND?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it
again...
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 2 lines
This is because the device implementation wouldn't know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended,QEMU
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
needs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to store this into a (for example) rotating hard disk as 1msec IO
latency.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 2 lines
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 4 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 2 lines
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 2 lines
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been
suspend.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 2 lines
Spec snippet:
"The device MUST reset the queue when 1 is written to
queue_reset.Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 3 lines
At minimum, we need, something implementable like,
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in thedevice.
Quoted 71 lines
Quoted 71 lines
Quoted 43 lines
Quoted 43 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 1 line
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yetcompleted.
Quoted 18 lines
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 2 lines
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 1 line
Can you please enhance this part?
Quoted 8 lines
Quoted 5 lines
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the deviceafter driver sets it (until the device is suspended).
Quoted 2 lines
This is ambiguous behavior of the register.
Queue_resetregister was like thatQuoted 7 lines
Quoted 4 lines
before it was fixed in 1.2.
once suspended, the driver should not access the configuration space except for the device status, so it should not reset any vqs through configuration space.I am not talking about VQ reset at all.
I gave an example of how the VQ reset register polarity works (with wasbroken like how the proposed suspend bit is broken).
Quoted 6 lines
Suspend register needs to work the way the reset register works. Can you please go through my exact text I replied in the previous email?
Basically, the bug is:
When driver has initiated the suspend (writing 1), the device continue torespond 0, while suspend is going on.
Quoted 2 lines
At this point, just by looking at the
device_statusregister one cannot figureQuoted 7 lines
Quoted 5 lines
out what is going on in the device.
Quoted 2 lines
Is driver initiated suspend is ongoing, or it was never started. It is ambiguous.
From the device perspective, there is only one user: the driver, and the driver sets or clears SUSPEND, it knows the status of the device.
It does not work elegantly when the hypervisor or other system is looking
dealing with this register.
Quoted 2 lines
In current proposal, anyone else than the driver looking at the
device_status,it is ambiguous.
Quoted 17 lines
For example,
suspend_bit:0x1suspended.0x0 not suspended or suspend is ongoing. (ambiguous forhypervisor debugging this device)
For device migration, it requires yet another side bit in the device parts.
An elegant and non-ambiguous interface can be,
ctrl_register=0x0(a usual default)0x1, suspend the device 0x2, resume the device. status_register = 0x0, (still) running. 0x1, suspended.With this driver and hypervisor (both) has very clear view of what is going on
Quoted 8 lines
Quoted 7 lines
and what is the status.
Quoted 3 lines
No more ambiguity for anyone.
Please consider pros and cons of both approaches.
The driver "owns" the device, other components access the device configurations through the driver, I am not sure there are other "owners" than the driver.
Device migration and debug channel [1] plans to access the device from the
hypervisor.
Quoted 5 lines
I don't see any disadvantage of separate control and status register. Do you see any?
[1]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore
Quoted 1 line
.kernel.org%2Fvirtio-comment%2F20240601145042.2074739-1-
parav%40nvidia
Quoted 1 line
.com%2FT%2F%23m14b80fd996e8d2522723967d8af844b70e131a90&dat a=05%7C02%7
Quoted 1 line
Cparav%40nvidia.com%7C06c09157b2d74af8b7f608dc8acaab0a%7C43083
d1572734Quoted 1 line
0c1b7db39efd9ccc17a%7C0%7C0%7C638537850271999791%7CUnknown %7CTWFpbGZsb
Quoted 1 line
Quoted 9 lines
3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn 0%3D%
Quoted 1 line
7C0%7C%7C%7C&sdata=e%2BBDJMpuFzmj47CKqnWU1jXcRZUXATmqhhMZ 7WC8zj8%3D&re
Quoted 1 line
served=0
is the admin vq governed by the driver? There should be only one owner of the device, or you need locks.
No locks are needed. Hypervisor is accessing the admin vq on the PF.
VF is accessing the suspend ctrl and status register of its own from its own MMIO space.
If for some reason, this registers is trapped and accessed in the hypervisor too, it is upto the hypervisor on lock/no-lock etc.
Quoted 2 lines
I am not sure we want to repeat the discussion on the admin vq for live migration last year, you know it breaks nested.
I am showing you the use case where ctrl+status is useful without any negatives. You are diverging the discussion to a point of no return.
I wish you can be more cooperative by seeing larger picture. :)
You didn't answer my question, do you see any disadvantage of ctrl+status register?
If not, lets converge and progress as it enables suspend functionality + device migration both using non-ambiguous register pair.
Quoted 1 line · Parav Pandit
No locks are needed. Hypervisor is accessing the admin vq on the PF.
I'm not even going to try and get through a jumble of this thread heavily corrupted by outlook and whatnot, but from the glimpses I got this discussion looks like just two people talking past each other. Lingshan, Parav's not the only one bothered by how the device seemingly has to block a transaction until it completes suspend. Parav, I feel how to address this is up to the author to resolve really, leave it at that.
Thanks,
Signature · 1 line
MST
Quoted 10 lines · Michael S. Tsirkin
Quoted 1 line · Parav Pandit
No locks are needed. Hypervisor is accessing the admin vq on the PF.
I'm not even going to try and get through a jumble of this thread heavily corrupted by outlook and whatnot, but from the glimpses I got this discussion looks like just two people talking past each other. Lingshan, Parav's not the only one bothered by how the device seemingly has to block a transaction until it completes suspend. Parav, I feel how to address this is up to the author to resolve really, leave it at that.
IIUC, Prava's concern here is that with the current proposal, the device's state can't be determined unambiguously by reading the device status field. If the SUSPEND bit is 0, it's ambiguous as to whether the device is running normally or if it's in the process of suspending, and there is a corresponding issue for suspended vs resuming. The device and driver can know based on their internal information what state the device is in. However, some hypothetical admin tool in guest userspace wouldn't be able to tell simply by reading the values in the PCI BAR.
Assuming that's actually a problem that we want to solve, then I think the most straightforward way to address this would be to use another bit in the status field as a SUSPEND_TRANSITION bit, with something like:
"After the driver writes or clears the SUSPEND bit, the device MUST present the SUSPEND_TRANSITION bit until it completes the requested suspend or resume transition. If the device is unable to successfully complete the transition, it MUST present DEVICE_NEEDS_RESET."
Doing that would give distinct values for each of the running/suspending/suspended/resuming states. That said, although the device status field doesn't have a defined width, both the PCI and channel transports assume that it's a single byte, so we'd be using the last remaining bits those transports have available.
-David
Quoted 29 lines · David Stevens
Quoted 8 lines · Michael S. Tsirkin
Quoted 1 line · Parav Pandit
No locks are needed. Hypervisor is accessing the admin vq on the PF.
I'm not even going to try and get through a jumble of this thread heavily corrupted by outlook and whatnot, but from the glimpses I got this discussion looks like just two people talking past each other. Lingshan, Parav's not the only one bothered by how the device seemingly has to block a transaction until it completes suspend. Parav, I feel how to address this is up to the author to resolve really, leave it at that.
IIUC, Prava's concern here is that with the current proposal, the device's state can't be determined unambiguously by reading the device status field. If the SUSPEND bit is 0, it's ambiguous as to whether the device is running normally or if it's in the process of suspending, and there is a corresponding issue for suspended vs resuming. The device and driver can know based on their internal information what state the device is in. However, some hypothetical admin tool in guest userspace wouldn't be able to tell simply by reading the values in the PCI BAR.
Assuming that's actually a problem that we want to solve, then I think the most straightforward way to address this would be to use another bit in the status field as a
SUSPEND_TRANSITIONbit, with something like:"After the driver writes or clears the SUSPEND bit, the device MUST present the
SUSPEND_TRANSITIONbit until it completes the requested suspend or resume transition. If the device is unable to successfully complete the transition, it MUST presentDEVICE_NEEDS_RESET."
Quoted 5 lines
Doing that would give distinct values for each of the running/suspending/suspended/resuming states. That said, although the device status field doesn't have a defined width, both the PCI and channel transports assume that it's a single byte, so we'd be using the last remaining bits those transports have available.
Current spec does not handle status transitions well, not only for SUSPEND and resuming. other status transitions may also take longer time than expect depends on how the device is implemented, it can be sluggish if implemented by SW during high traffic, even when handling FEATURES_OK or present NEEDS_RESET.
This patch wants to copy the mechanism of handling FEATURES_OK, but if we want to resolve the problem of device status transition, I agree to add a new status bit: TRANSITION.
The device should present TRANSITION until it complete the requested status transition. Combining the knowledge of the device, the driver can tell the exact device status.
Any thoughts?
Thanks
Zhu Lingshan
Quoted 2 lines
-David
Quoted 47 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, June 13, 2024 3:29 PMQuoted 1 line · David Stevens
On Wed, Jun 12, 2024 at 9:20 PM Michael S. Tsirkin <[email protected]>
Quoted 8 lines · wrote:
Quoted 8 lines · Parav Pandit
Quoted 1 line
No locks are needed. Hypervisor is accessing the admin vq on the PF.
I'm not even going to try and get through a jumble of this thread heavily corrupted by outlook and whatnot, but from the glimpses I got this discussion looks like just two people talking past each other. Lingshan, Parav's not the only one bothered by how the device seemingly has to block a transaction until it completes suspend. Parav, I feel how to address this is up to the author to resolve really, leave
it at that.
Quoted 20 lines
IIUC, Prava's concern here is that with the current proposal, the device's state can't be determined unambiguously by reading the device status field. If the SUSPEND bit is 0, it's ambiguous as to whether the device is running normally or if it's in the process of suspending, and there is a corresponding issue for suspended vs resuming. The device and driver can know based on their internal information what state the device is in. However, some hypothetical admin tool in guest userspace wouldn't be able to tell simply by reading the values in the PCI BAR.
Assuming that's actually a problem that we want to solve, then I think the most straightforward way to address this would be to use another bit in the status field as a
SUSPEND_TRANSITIONbit, with something like:"After the driver writes or clears the SUSPEND bit, the device MUST present the
SUSPEND_TRANSITIONbit until it completes the requested suspend or resume transition. If the device is unable to successfully complete the transition, it MUST presentDEVICE_NEEDS_RESET."Quoted 5 lines
Doing that would give distinct values for each of the running/suspending/suspended/resuming states. That said, although the device status field doesn't have a defined width, both the PCI and channel transports assume that it's a single byte, so we'd be using the last remaining bits those transports have available.
Current spec does not handle status transitions well, not only for SUSPEND and resuming.
other status transitions may also take longer time than expect depends on how the device is implemented, it can be sluggish if implemented by SW during high traffic, even when handlingFEATURES_OKor presentNEEDS_RESET.
Quoted 9 lines
This patch wants to copy the mechanism of handling
FEATURES_OK, but if we want to resolve the problem of device status transition, I agree to add a new status bit: TRANSITION.The device should present TRANSITION until it complete the requested status transition.
Combining the knowledge of the device, the driver can tell the exact device status.
Yes, this will be good idea.
The TRANSITION bit which must turn very quickly to 1 on an MMIO write requires very special circuitry. It needs to turn 1 just before the next read arrives from the guest driver.
This is roughly 10nsec of time (write followed by a read).
To simplify this aspect, a better approach can be,
1. device_status to have SUSPENDED bit, which starts with 0.
2. a new ctrl_register is added. When cmd is written there, its status reflects in device_status.
3. So even if the device takes 30nsec or 30usec to respond, things don’t break.
For example,
a. driver writes device_ctrl = 0x1 (cmd=suspend)
b. driver reads device_status = (suspended=0) -> not yet suspended c. driver loops on #b few more nsecs.
d. finally device turns it suspended. So #b returns supended=1.
And we can extend it more to better restructure the other operations too such as features_ok, driver_ok etc. And can also make the current "reset" from SHOULD to MUST.
Its one time price to pay for the extra register but it is far easier to implement for the devices (and works fine for sw backend too).
Quoted 66 lines · Parav Pandit
Quoted 48 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, June 13, 2024 3:29 PMQuoted 1 line · David Stevens
On Wed, Jun 12, 2024 at 9:20 PM Michael S. Tsirkin <[email protected]>
Quoted 8 lines · wrote:
Quoted 8 lines · Parav Pandit
Quoted 1 line
No locks are needed. Hypervisor is accessing the admin vq on the PF.
I'm not even going to try and get through a jumble of this thread heavily corrupted by outlook and whatnot, but from the glimpses I got this discussion looks like just two people talking past each other. Lingshan, Parav's not the only one bothered by how the device seemingly has to block a transaction until it completes suspend. Parav, I feel how to address this is up to the author to resolve really, leave
it at that.
Quoted 20 lines
IIUC, Prava's concern here is that with the current proposal, the device's state can't be determined unambiguously by reading the device status field. If the SUSPEND bit is 0, it's ambiguous as to whether the device is running normally or if it's in the process of suspending, and there is a corresponding issue for suspended vs resuming. The device and driver can know based on their internal information what state the device is in. However, some hypothetical admin tool in guest userspace wouldn't be able to tell simply by reading the values in the PCI BAR.
Assuming that's actually a problem that we want to solve, then I think the most straightforward way to address this would be to use another bit in the status field as a
SUSPEND_TRANSITIONbit, with something like:"After the driver writes or clears the SUSPEND bit, the device MUST present the
SUSPEND_TRANSITIONbit until it completes the requested suspend or resume transition. If the device is unable to successfully complete the transition, it MUST presentDEVICE_NEEDS_RESET."Quoted 5 lines
Doing that would give distinct values for each of the running/suspending/suspended/resuming states. That said, although the device status field doesn't have a defined width, both the PCI and channel transports assume that it's a single byte, so we'd be using the last remaining bits those transports have available.
Current spec does not handle status transitions well, not only for SUSPEND and resuming.
other status transitions may also take longer time than expect depends on how the device is implemented, it can be sluggish if implemented by SW during high traffic, even when handlingFEATURES_OKor presentNEEDS_RESET.Quoted 9 lines
This patch wants to copy the mechanism of handling
FEATURES_OK, but if we want to resolve the problem of device status transition, I agree to add a new status bit: TRANSITION.The device should present TRANSITION until it complete the requested status transition.
Combining the knowledge of the device, the driver can tell the exact device status.Yes, this will be good idea.
The TRANSITION bit which must turn very quickly to 1 on an MMIO write requires very special circuitry. It needs to turn 1 just before the next read arrives from the guest driver.
This is roughly 10nsec of time (write followed by a read).To simplify this aspect, a better approach can be,
1.device_statusto have SUSPENDED bit, which starts with 0.
2. a newctrl_registeris added. When cmd is written there, its status reflects indevice_status.
Quoted 7 lines
3. So even if the device takes 30nsec or 30usec to respond, things don’t break.
For example,
a. driver writesdevice_ctrl=0x1(cmd=suspend)
b. driver readsdevice_status= (suspended=0) -> not yet suspended c. driver loops on #b few more nsecs.
d. finally device turns it suspended. So #b returns supended=1.
Is there any real difference? Currently, transitions could be spotted by the driver easily: read after write doesn't match.
Quoted 5 lines
And we can extend it more to better restructure the other operations too such as
features_ok,driver_oketc. And can also make the current "reset" from SHOULD to MUST.Its one time price to pay for the extra register but it is far easier to implement for the devices (and works fine for sw backend too).
Well, I don't know why this needs to be addressed in this series as it's not specific to suspend.
Thanks
Quoted 4 lines
Quoted 80 lines
From: Jason Wang <[email protected]>
Sent: Monday, June 17, 2024 7:52 AMQuoted 34 lines · Parav Pandit
Quoted 34 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, June 13, 2024 3:29 PMQuoted 2 lines · David Stevens
On Wed, Jun 12, 2024 at 9:20 PM Michael S. Tsirkin <[email protected]>
Quoted 9 lines · wrote:
Quoted 9 lines · Parav Pandit
Quoted 1 line
No locks are needed. Hypervisor is accessing the admin vq on the PF.
I'm not even going to try and get through a jumble of this thread heavily corrupted by outlook and whatnot, but from the glimpses I got this discussion looks like just two people talking past each other. Lingshan, Parav's not the only one bothered by how the device seemingly has to block a transaction until it completes suspend. Parav, I feel how to address this is up to the author to resolve really, leave
it at that.
Quoted 16 lines
IIUC, Prava's concern here is that with the current proposal, the device's state can't be determined unambiguously by reading the device status field. If the SUSPEND bit is 0, it's ambiguous as to whether the device is running normally or if it's in the process of suspending, and there is a corresponding issue for suspended vs resuming. The device and driver can know based on their internal information what state the device is in. However, some hypothetical admin tool in guest userspace wouldn't be able to tell simply by reading the values in the PCI BAR.
Assuming that's actually a problem that we want to solve, then I think the most straightforward way to address this would be to use another bit in the status field as a
SUSPEND_TRANSITIONbit, with something
like:Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
"After the driver writes or clears the SUSPEND bit, the device MUST present the
SUSPEND_TRANSITIONbit until it completes the requested suspend or resume transition. If the device is unable to successfully complete the transition, it MUST present
DEVICE_NEEDS_RESET."Quoted 17 lines
Quoted 17 lines
Quoted 6 lines
Doing that would give distinct values for each of the running/suspending/suspended/resuming states. That said, although the device status field doesn't have a defined width, both the PCI and channel transports assume that it's a single byte, so we'd be using the last remaining bits those transports have available.
Current spec does not handle status transitions well, not only for SUSPEND and resuming.
other status transitions may also take longer time than expect depends on how the device is implemented, it can be sluggish if implemented by SW during high traffic, even when handlingFEATURES_OKor presentNEEDS_RESET.This patch wants to copy the mechanism of handling
FEATURES_OK, but if we want to resolve the problem of device status transition, I agree to add a new status bit: TRANSITION.Quoted 7 lines
Quoted 5 lines
The device should present TRANSITION until it complete the requested status transition.
Combining the knowledge of the device, the driver can tell the exact device status.Yes, this will be good idea.
The TRANSITION bit which must turn very quickly to 1 on an MMIO writerequires very special circuitry.
Quoted 6 lines
It needs to turn 1 just before the next read arrives from the guest driver. This is roughly 10nsec of time (write followed by a read).
To simplify this aspect, a better approach can be, 1.
device_statusto have SUSPENDED bit, which starts with 0.
2. a newctrl_registeris added. When cmd is written there, its statusreflects in
device_status.Quoted 2 lines
3. So even if the device takes 30nsec or 30usec to respond, things don’t
break.
Quoted 1 line
Quoted 9 lines
Quoted 5 lines
For example,
a. driver writesdevice_ctrl=0x1(cmd=suspend) b. driver readsdevice_status= (suspended=0) -> not yet suspended c. driver loops on #b few more nsecs.
d. finally device turns it suspended. So #b returns supended=1.Is there any real difference? Currently, transitions could be spotted by the driver easily: read after write doesn't match.
Yes, the real difference is: simplified device implementation.
The device does not have the time constraint to reply (transition=1) to the incoming read in next 10nsec.
Quoted 267 lines · Parav Pandit
Quoted 267 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, June 12, 2024 4:00 PMQuoted 22 lines · Parav Pandit
Quoted 22 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, June 12, 2024 2:50 PMQuoted 18 lines · Parav Pandit
Quoted 18 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:42 PMQuoted 14 lines · Parav Pandit
Quoted 14 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 3:03 PMQuoted 10 lines · Parav Pandit
Quoted 10 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, June 11, 2024 1:57 PMQuoted 6 lines · Parav Pandit
Hi Lingshan, David,
Quoted 4 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMOn Fri, Jun 07 2024, Zhu Lingshan <[email protected]>
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This commit allows the driver to suspend the device by
introducing a new status bit SUSPEND in device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPEND which indicating whether the device supportSUSPEND.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Signed-off-by: Zhu Lingshan <[email protected]>
Signed-off-by: Zhu Lingshan <[email protected]>
Signed-off-by: Eugenio Pérez <[email protected]>
Signed-off-by: David Stevens <[email protected]>
---
content.tex | 69
+++++++++++++++++++++++++++++++++++++++++++++--------
Quoted 67 lines
Quoted 67 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
1 file changed, 59 insertions(+), 10 deletions(-)
Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing
lists...
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
(...)
Quoted 5 lines
+\drivernormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The driver MUST NOT set SUSPEND if FEATURES_OK is not set
+or
VIRTIO_F_SUSPENDis not negotiated.Quoted 3 lines
+
+Once the driver sets SUSPEND to \field{device status} of
+thedevice:
Quoted 50 lines
Quoted 50 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to
+verify whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point intime?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available
+to thedevice.
Quoted 21 lines
Quoted 21 lines
Quoted 2 lines
+\item The driver MUST NOT access any fields of any
+virtqueues or notifyany virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Quoted 6 lines
+\end{itemize}
+
+\devicenormative{\subsection}{Device Suspend}{General
+Initialization And Device Operation / Device Suspend}
+
+The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.Quoted 3 lines
+
+The device MUST ignore all access to its Configuration
+Space while suspended, except for \field{device status}....if it is part of the configuration space.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
+
+A device MUST NOT send any notifications, access any
+virtqueues, or modify any fields in its configuration space
+whilesuspended.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
+
+If SUSPEND is set in \field{device status}, when the driver
+clears SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.Quoted 3 lines
+
+When the driver sets SUSPEND, the device SHOULD perform the
+following actions before presentingSUSPEND bit in the \field{device status}:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
+
+\begin{itemize}
+\item Stop processing more buffers of any virtqueues \item
+Wait until all buffers that are being processed have been used.
+\item Send used buffer notifications to the driver.
+\end{itemize}So, is there any opportunity for the device to fail setting
SUSPEND?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it
again...
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Additionally, a suspend operation usually involves saving things to a slow
memory (or media).
Quoted 2 lines
This is because the device implementation wouldn't know when exactly the
device will be resumed.
Quoted 2 lines
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended,QEMU
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
needs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to store this into a (for example) rotating hard disk as 1msec IO
latency.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
b. a NIC may need to store its RSS, queues, flow filters configuration for
several tens of KBs to some slow memory.
Quoted 2 lines
c. A block device may prefer to complete some IOs to a threshold level
instead of maintaining large list of outstanding IOs in some suspended memory.
Quoted 4 lines
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
path hardware engines to suspend DMA operations (read and write both) which are inflight.
Quoted 2 lines
(without causing DMA errors). Same would apply to the sw backend
implementations too.
Quoted 2 lines
Therefore, the suspend operation that is initiated by the driver, should get
the acknowledgement back from the device that it has been
suspend.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Some of the good examples if you prefer to follow, a driver<->device
interface needs a suspend register which should behave like below queue reset register.
Quoted 2 lines
Spec snippet:
"The device MUST reset the queue when 1 is written to
queue_reset.Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
The device MUST continue to present
1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue resethas completed."
Quoted 3 lines
At minimum, we need, something implementable like,
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in thedevice.
Quoted 71 lines
Quoted 71 lines
Quoted 43 lines
Quoted 43 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctlrregister whendevice has completely suspended the device.
Quoted 1 line
The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yetcompleted.
Quoted 18 lines
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
The device MUST present 0 in the
device_suspend_ctrlregister when thedevice has completely resumed the device.
Quoted 2 lines
At this point, the driver may resume notifying the device and accessing the
configuration space.
Quoted 1 line
Can you please enhance this part?
Quoted 8 lines
Quoted 5 lines
Hi Parav
Yes they are necessary contents and we will address them in the following patches.
The register polarity is critical.
In your reply to Cornelia, you descried that suspend bit cleared in the deviceafter driver sets it (until the device is suspended).
Quoted 2 lines
This is ambiguous behavior of the register.
Queue_resetregister was like thatQuoted 7 lines
Quoted 4 lines
before it was fixed in 1.2.
once suspended, the driver should not access the configuration space except for the device status, so it should not reset any vqs through configuration space.I am not talking about VQ reset at all.
I gave an example of how the VQ reset register polarity works (with wasbroken like how the proposed suspend bit is broken).
Quoted 6 lines
Suspend register needs to work the way the reset register works. Can you please go through my exact text I replied in the previous email?
Basically, the bug is:
When driver has initiated the suspend (writing 1), the device continue torespond 0, while suspend is going on.
Quoted 2 lines
At this point, just by looking at the
device_statusregister one cannot figureQuoted 7 lines
Quoted 5 lines
out what is going on in the device.
Quoted 2 lines
Is driver initiated suspend is ongoing, or it was never started. It is ambiguous.
From the device perspective, there is only one user: the driver, and the driver sets or clears SUSPEND, it knows the status of the device.
It does not work elegantly when the hypervisor or other system is looking
dealing with this register.
Quoted 2 lines
In current proposal, anyone else than the driver looking at the
device_status,it is ambiguous.
Quoted 17 lines
For example,
suspend_bit:0x1suspended.0x0 not suspended or suspend is ongoing. (ambiguous forhypervisor debugging this device)
For device migration, it requires yet another side bit in the device parts.
An elegant and non-ambiguous interface can be,
ctrl_register=0x0(a usual default)0x1, suspend the device 0x2, resume the device. status_register = 0x0, (still) running. 0x1, suspended.With this driver and hypervisor (both) has very clear view of what is going on
Quoted 8 lines
Quoted 7 lines
and what is the status.
Quoted 3 lines
No more ambiguity for anyone.
Please consider pros and cons of both approaches.
The driver "owns" the device, other components access the device configurations through the driver, I am not sure there are other "owners" than the driver.
Device migration and debug channel [1] plans to access the device from the
hypervisor.
Quoted 5 lines
I don't see any disadvantage of separate control and status register. Do you see any?
[1]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore
Quoted 1 line
.kernel.org%2Fvirtio-comment%2F20240601145042.2074739-1-
parav%40nvidia .com%2FT%2F%23m14b80fd996e8d2522723967d8af844b70e131a90&dat a=05%7C02%7 Cparav%40nvidia.com%7C06c09157b2d74af8b7f608dc8acaab0a%7C43083
d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638537850271999791%7CUnknown %7CTWFpbGZsb 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn 0%3D% 7C0%7C%7C%7C&sdata=e%2BBDJMpuFzmj47CKqnWU1jXcRZUXATmqhhMZ 7WC8zj8%3D&re
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
served=0
is the admin vq governed by the driver? There should be only one owner of the device, or you need locks.
No locks are needed. Hypervisor is accessing the admin vq on the PF.
VF is accessing the suspend ctrl and status register of its own from its own MMIO space.
If for some reason, this registers is trapped and accessed in the hypervisor too, it is upto the hypervisor on lock/no-lock etc.
Even access device status from PF through admin vq, it is still trying to RW the unique vf->dev_status, so there are too individuals accessing the same copy of resource, there need a lock.
Quoted 9 lines
Quoted 3 lines
I am not sure we want to repeat the discussion on the admin vq for live migration last year, you know it breaks nested.
I am showing you the use case where ctrl+status is useful without any negatives. You are diverging the discussion to a point of no return.
I wish you can be more cooperative by seeing larger picture. :)You didn't answer my question, do you see any disadvantage of ctrl+status register?
If not, lets converge and progress as it enables suspend functionality + device migration both using non-ambiguous register pair.
This is not about SUSPEND or ctrl + status. If we take the transition period into consideration, we need a total solution for all status transitions. For example DRIVER_OK and FEATURES_OK, or the device setting NEEDS_RESET,
they can also take longer time than expect to take effects, depends on the device implementation.
Maybe we need a new bit in the device stauts: TRANSITION.
Quoted 117 lines · Parav Pandit
Hi Lingshan, David,
Quoted 79 lines
From: Cornelia Huck <[email protected]>
Sent: Monday, June 10, 2024 9:34 PMQuoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 8 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
Review
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General Initialization
(quoted diff) +3
-0+And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 4 lines
content.tex +4
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether theSUSPEND bit is set.
Quoted 1 line
+If not, the device does not support the SUSPEND feature.
That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point in time?Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify
any virtqueues.
"send notifications for any virtqueues"?
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Review
Quoted 6 lines
+\end{itemize}
(quoted diff) +5
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization +And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 3 lines
content.tex +3
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}....if it is part of the configuration space.
Review
Quoted 6 lines
(quoted diff) +6
-0+ +A device MUST NOT send any notifications, access any virtqueues, or +modify any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND,"subsequently clears SUSPEND"?
Quoted 1 line
+the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation /
Quoted 3 lines
content.tex +3
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presentingSUSPEND bit in the \field{device status}:
Quoted 11 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 6 lines
content.tex +6
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait until +all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Additionally, a suspend operation usually involves saving things to a slow memory (or media).
This is because the device implementation wouldn’t know when exactly the device will be resumed.
Few examples, are:
a. A gpu device with 128MB of video RAM when suspended, QEMU needs to store this into a (for example) rotating hard disk as 1msec IO latency.b. a NIC may need to store its RSS, queues, flow filters configuration for several tens of KBs to some slow memory.
c. A block device may prefer to complete some IOs to a threshold level instead of maintaining large list of outstanding IOs in some suspended memory.
d. May be more in future.
Additionally, suspend operation needs to be synchronized with certain data path hardware engines to suspend DMA operations (read and write both) which are inflight. (without causing DMA errors). Same would apply to the sw backend implementations too.
Therefore, the suspend operation that is initiated by the driver, should get the acknowledgement back from the device that it has been suspend.
Some of the good examples if you prefer to follow, a driver<->device interface needs a suspend register which should behave like below queue reset register.
Spec snippet:
"The device MUST reset the queue when 1 is written toqueue_reset. The device MUST continue to present 1 inqueue_resetas long as the queue reset is ongoing. The device MUST present 0 in bothqueue_resetandqueue_enablewhen queue reset has completed."At minimum, we need, something implementable like,
Quoted 8 lines
The device MUST suspend the device when 1 is written to the
device_suspend_ctlr. The device MUST continue to present 1 indevice_suspend_ctrlas the suspend operation is ongoing in the device. The device MUST present 0 in thedevice_suspend_ctlrregister when device has completely suspended the device.The device MUST resume the device when 2 is written to the
device_suspend_ctrl. The device MUST continue to present 2 in thedevice_suspend_ctrlas the resume operation may not have yet completed. The device MUST present 0 in thedevice_suspend_ctrlregister when the device has completely resumed the device.
At this point, the driver may resume notifying the device and accessing the configuration space.Can you please enhance this part?
Yea, this gives me pause too.
Quoted 25 lines
Quoted 24 lines
Quoted 7 lines
+
\chapter{Virtio Transport Options}\label{sec:Virtio Transport
Options}Virtio can use various different buses, thus the standard is split @@
-872,6 +917,10 @@ \chapter{Reserved Feature Bits}\label{sec:ReservedFeature Bits}
Quoted 1 line
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}
for
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 921
Quoted 4 lines
content.tex +2
-0handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + set SUSPEND to the device."that the driver can trigger suspending the device via the SUSPEND flag"?
Quoted 6 lines
content.tex +2
-0+ See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved FeatureBits}
Quoted 15 lines · Cornelia Huck
Quoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 7 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)
Quoted 4 lines
Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
will add change log in next version.
Quoted 13 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 649
Quoted 9 lines
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
content.tex +7
-0+ +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set. +If not, the device does not support the SUSPEND feature.That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point in time?
Yes, this can be improved
How about:
If not, this indicating the device is not able to handle SUSPEND at the moment.
Quoted 4 lines
Quoted 3 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify any virtqueues.
"send notifications for any virtqueues"?
OK
Quoted 3 lines
Quoted 2 lines
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Yes, that is the original design, then I am convinced that the status fields is not a part of the config space because it has a dedicated section in the spec.
I will add this sentence in the next version.
Quoted 10 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 9 lines
+\end{itemize}
content.tex +7
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}....if it is part of the configuration space.
Yes
Quoted 7 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 665
Quoted 6 lines
content.tex +5
-0+ +A device MUST NOT send any notifications, access any virtqueues, or modify +any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,"subsequently clears SUSPEND"?
I am a native speaker, but "subsequently" sounds like clearing SUSPEND right after "setting SUSPEND", I guess there can be some operations during SUSPEND.
Quoted 14 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 11 lines
content.tex +10
-0+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting SUSPEND bit in the \field{device status}: + +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
The device should not present SUSPEND bit in the device status until it is suspended, so I am afraid there is not a SUSPEND bit to clear if the device fails to suspend itself.
I think there can be two indications of a failed suspend:
1) The driver sets SUSPEND, but the device does not present SUSPEND after a while, depends on the driver tolerance, the driver may assume it is a failed SUSPEND, like how we handle FEATURES_OK 2)The device runs into errors, presenting NEEDS_RESET.
Quoted 4 lines
Quoted 4 lines
+
\chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
Quoted 8 lines
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 921
Quoted 7 lines
Virtio can use various different buses, thus the standard is split
content.tex +2
-0@@ -872,6 +917,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + set SUSPEND to the device."that the driver can trigger suspending the device via the SUSPEND flag"?
Yes
Quoted 6 lines
Quoted 6 lines
content.tex +2
-0+ See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Quoted 36 lines · Zhu Lingshan
Quoted 15 lines · Cornelia Huck
Quoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 7 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Quoted 4 lines
Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
will add change log in next version.
Quoted 13 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 649
Quoted 9 lines
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
content.tex +7
-0+ +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set. +If not, the device does not support the SUSPEND feature.That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point in time?
Quoted 4 lines
Yes, this can be improved
How about:
If not, this indicating the device is not able to handle SUSPEND at the moment.
I don't get what this means, agrammatical and vague. Please make it explicit. Do you mean if driver reads status and SUSPEND is clear then devide failed to enter suspend state? Might be problematic: time to suspend might be quite high.
Quoted 46 lines
Quoted 3 lines
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify any virtqueues.
"send notifications for any virtqueues"?
OK
Quoted 3 lines
Quoted 2 lines
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Yes, that is the original design, then I am convinced that the status fields is not a part of the config space because it has a dedicated section in the spec.
I will add this sentence in the next version.
Quoted 10 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 9 lines
+\end{itemize}
content.tex +7
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}....if it is part of the configuration space.
Yes
Quoted 7 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 665
Quoted 6 lines
content.tex +5
-0+ +A device MUST NOT send any notifications, access any virtqueues, or modify +any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,"subsequently clears SUSPEND"?
I am a native speaker, but "subsequently" sounds like clearing SUSPEND right after "setting SUSPEND", I guess there can be some operations during SUSPEND.
Quoted 14 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 11 lines
content.tex +10
-0+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting SUSPEND bit in the \field{device status}: + +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 6 lines
The device should not present SUSPEND bit in the device status until it is suspended, so I am afraid there is not a SUSPEND bit to clear if the device fails to suspend itself.
I think there can be two indications of a failed suspend:
1) The driver sets SUSPEND, but the device does not present SUSPEND after a while, depends on the driver tolerance, the driver may assume it is a failed SUSPEND, like how we handleFEATURES_OK
This is not how we handle FEATURES_OK.
Quoted 2 lines
2)The device runs into errors, presenting
NEEDS_RESET.
More reasonable.
Quoted 19 lines
Quoted 4 lines
Quoted 4 lines
+
\chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}Quoted 8 lines
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 921
Quoted 7 lines
Virtio can use various different buses, thus the standard is split
content.tex +2
-0@@ -872,6 +917,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + set SUSPEND to the device."that the driver can trigger suspending the device via the SUSPEND flag"?
Yes
Quoted 6 lines
Quoted 6 lines
content.tex +2
-0+ See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Quoted 37 lines · Michael S. Tsirkin
Quoted 36 lines · Zhu Lingshan
Quoted 15 lines · Cornelia Huck
Quoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 7 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Quoted 4 lines
Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
will add change log in next version.
Quoted 4 lines
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
content.tex +3
-0+ +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. +Quoted 8 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 649
Quoted 4 lines
content.tex +4
-0+Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set. +If not, the device does not support the SUSPEND feature.That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point in time?Yes, this can be improved
Quoted 8 lines
Quoted 2 lines
How about:
If not, this indicating the device is not able to handle SUSPEND at the moment.I don't get what this means, agrammatical and vague. Please make it explicit. Do you mean if driver reads status and SUSPEND is clear then devide failed to enter suspend state? Might be problematic: time to suspend might be quite high.
I think this is a common problem how the driver handle changes of the device status.
Even without this SUSPEND feature, spec section 3.1.1 Driver Requirements: Device Initialization says:
“Re-read device status to ensure the FEATURES_OK bit is still set: otherwise, the device does not support our subset of features and the device is unusable” Other status changing may also be costly, so how long should the driver wait for the device responding to status changes?
This currently depends on the driver's tolerance.
What should the driver do when overdue? Shall it reset the device or set FAILED?
I think the device should either SUSPEND successfully or set NEEDS_RESET, but maybe not a good idea to set a time-out in the spec, the driver can set FAILED or just reset the device if the operation exceeds the tolerance in pulling.
Any suggestions?
Quoted 44 lines
Quoted 44 lines
Quoted 5 lines
Quoted 4 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify any virtqueues.
"send notifications for any virtqueues"?
OK
Quoted 2 lines
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Yes, that is the original design, then I am convinced that the status fields is not a part of the config space because it has a dedicated section in the spec.
I will add this sentence in the next version.
Quoted 9 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 8 lines
+\end{itemize}
content.tex +7
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}....if it is part of the configuration space.
Yes
Quoted 6 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 665
Quoted 5 lines
content.tex +5
-0+ +A device MUST NOT send any notifications, access any virtqueues, or modify +any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,"subsequently clears SUSPEND"?
I am a native speaker, but "subsequently" sounds like clearing SUSPEND right after "setting SUSPEND", I guess there can be some operations during SUSPEND.
Quoted 13 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 10 lines
content.tex +10
-0+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting SUSPEND bit in the \field{device status}: + +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 8 lines
Quoted 6 lines
The device should not present SUSPEND bit in the device status until it is suspended, so I am afraid there is not a SUSPEND bit to clear if the device fails to suspend itself.
I think there can be two indications of a failed suspend:
1) The driver sets SUSPEND, but the device does not present SUSPEND after a while, depends on the driver tolerance, the driver may assume it is a failed SUSPEND, like how we handleFEATURES_OKThis is not how we handle
FEATURES_OK.
setting FEATURES_OK may not always be successful and can take longer time than expect. What should the driver do when the pulling re-reads does not return FEATURES_OK?
Quoted 4 lines
Quoted 3 lines
2)The device runs into errors, presenting
NEEDS_RESET.More reasonable.
OK, so the driver should re-read the status, it should see either SUSPEND or NEEDS_RESET, this works for me.
Thanks
Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
+
\chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}Quoted 8 lines
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 921
Quoted 7 lines
Virtio can use various different buses, thus the standard is split
content.tex +2
-0@@ -872,6 +917,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + set SUSPEND to the device."that the driver can trigger suspending the device via the SUSPEND flag"?
Yes
Quoted 5 lines
Quoted 5 lines
content.tex +2
-0+ See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Quoted 48 lines · Zhu Lingshan
Quoted 37 lines · Michael S. Tsirkin
Quoted 36 lines · Zhu Lingshan
Quoted 15 lines · Cornelia Huck
Quoted 6 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 7 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 69 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-)Quoted 4 lines
Can you please add a changelog? Especially as some of the previous discussion has been lost due to the broken old mailing lists...
(...)
will add change log in next version.
Quoted 4 lines
Quoted 4 lines
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
content.tex +3
-0+ +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. +Quoted 8 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 649
Quoted 4 lines
content.tex +4
-0+Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set. +If not, the device does not support the SUSPEND feature.That sentence is a bit weird: I'd expect the device to not offer
VIRTIO_F_SUSPENDin the first place in that case... could this rather happen if the device is not able to handle the request at a specific point in time?Yes, this can be improved
Quoted 8 lines
Quoted 2 lines
How about:
If not, this indicating the device is not able to handle SUSPEND at the moment.I don't get what this means, agrammatical and vague. Please make it explicit. Do you mean if driver reads status and SUSPEND is clear then devide failed to enter suspend state? Might be problematic: time to suspend might be quite high.
Quoted 4 lines
I think this is a common problem how the driver handle changes of the device status.
Even without this SUSPEND feature, spec section 3.1.1 Driver Requirements: Device Initialization says:
“Re-read device status to ensure theFEATURES_OKbit is still set: otherwise, the device does not support our subset of features and the device is unusable” Other status changing may also be costly, so how long should the driver wait for the device responding to status changes?
The driver does not wait with FEATURES_OK. It does a single write followed by a single read. If it sees 0 there, then these features do not work.
Quoted 2 lines
This currently depends on the driver's tolerance.
What should the driver do when overdue? Shall it reset the device or set FAILED?
Actually drivers in the field can and do change features and try again, that is also an option.
Quoted 4 lines
I think the device should either SUSPEND successfully or set
NEEDS_RESET, but maybe not a good idea to set a time-out in the spec, the driver can set FAILED or just reset the device if the operation exceeds the tolerance in pulling.Any suggestions?
Seems ok to me.
Quoted 55 lines
Quoted 44 lines
Quoted 44 lines
Quoted 5 lines
Quoted 4 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify any virtqueues.
"send notifications for any virtqueues"?
OK
Quoted 2 lines
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Yes, that is the original design, then I am convinced that the status fields is not a part of the config space because it has a dedicated section in the spec.
I will add this sentence in the next version.
Quoted 9 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 8 lines
+\end{itemize}
content.tex +7
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status}....if it is part of the configuration space.
Yes
Quoted 6 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 665
Quoted 5 lines
content.tex +5
-0+ +A device MUST NOT send any notifications, access any virtqueues, or modify +any fields in its configuration space while suspended. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND,"subsequently clears SUSPEND"?
I am a native speaker, but "subsequently" sounds like clearing SUSPEND right after "setting SUSPEND", I guess there can be some operations during SUSPEND.
Quoted 13 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 10 lines
content.tex +10
-0+the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting SUSPEND bit in the \field{device status}: + +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}So, is there any opportunity for the device to fail setting SUSPEND? I mean, if the driver is supposed to look whether it sticks, there should be conditions for when the device might clear it again...
Quoted 8 lines
Quoted 6 lines
The device should not present SUSPEND bit in the device status until it is suspended, so I am afraid there is not a SUSPEND bit to clear if the device fails to suspend itself.
I think there can be two indications of a failed suspend:
1) The driver sets SUSPEND, but the device does not present SUSPEND after a while, depends on the driver tolerance, the driver may assume it is a failed SUSPEND, like how we handleFEATURES_OKThis is not how we handle
FEATURES_OK.setting
FEATURES_OKmay not always be successful and can take longer time than expect. What should the driver do when the pulling re-reads does not returnFEATURES_OK?Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
2)The device runs into errors, presenting
NEEDS_RESET.More reasonable.
OK, so the driver should re-read the status, it should see either SUSPEND or
NEEDS_RESET, this works for me.Thanks
Or neither to mean "suspend in progress"?
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
+
\chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}Quoted 8 lines
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 921
Quoted 7 lines
Virtio can use various different buses, thus the standard is split
content.tex +2
-0@@ -872,6 +917,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + set SUSPEND to the device."that the driver can trigger suspending the device via the SUSPEND flag"?
Yes
Quoted 5 lines
Quoted 5 lines
content.tex +2
-0+ See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Quoted 11 lines
Quoted 3 lines
Quoted 2 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or notify any virtqueues.
"send notifications for any virtqueues"?
OK
Quoted 3 lines
Quoted 2 lines
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
Yes, that is the original design, then I am convinced that the status fields is not a part of the config space because it has a dedicated section in the spec.
I will add this sentence in the next version.
In the discussion on v3 (unarchived due to the migration), Michael stated:
Quoted 2 lines
by the way device status is not in device configuration space at all so this sentence does not make sense either.
Doesn't that assertion directly contradict Cornelia's feedback on this series, which seems to be saying that the device status field can be part of the config space? Can we get some clarity here on what exactly the existing spec is?
-David