virtio: introduce SUSPEND bit in device status
PATCH v7 1/1 virtio: introduce SUSPEND bit in device status
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.
This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.
--- content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 65 insertions(+), 10 deletions(-)
Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.
Changes from V5:
- the device should present NEEDS_RESET if failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvements
Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqs
Changes from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.
content.tex +65 -10
@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by
@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a
@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}
@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers. +\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set. +\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize} + +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the Configuration Space. + +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in +its Configuration Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND bit is set, +the device MUST NOT send any configuration change notifications. +Instead, the device MUST send the notification after the SUSPEND bit has been cleared. + +When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND, +the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}: + +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split
@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.45.2
Hi Lingshan, David,
Quoted 122 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++----- -- 1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:BasicFacilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +10
-5- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that +the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by @@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}.
content.tex +5
-2-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than +SUSPEND except when setting \field{device status} to 0 as a +transport-specific way to initiate a reset. If the driver sets the +FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a @@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device type-\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:GeneralInitialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device reset) before
removing exposed buffers.+\section{Device Suspend}\label{sec:General Initialization And Device
content.tex +10
-0+Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the SUSPEND bit +in \field{device status} to suspend a device, and can clear the SUSPEND +bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization +And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the
Quoted 7 lines
SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+
Do we agree that
a. suspending a device is non frequent operation (in order of N operations/sec, where N is roughly in range of 10 or 100) per device? b. A software-based device may not always want to force VM_EXIT on read and write on the device_status register?
Quoted 55 lines
+\devicenormative{\subsection}{Device Suspend}{General Initialization
content.tex +3
-0+And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +3
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the ConfigurationSpace.
content.tex +8
-0+ +A device MUST NOT send any notifications for any virtqeuues, access any +virtqueues, or modify any fields in its Configuration Space while +suspended. + +If changes occur in the Configuration Space while the SUSPEND bit is +set, the device MUST NOT send any configuration change notifications. +Instead, the device MUST send the notification after the SUSPEND bit hasbeen cleared.
content.tex +2
-0+ +When the driver sets SUSPEND, the device MUST either suspend itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND, the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex +3
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +8
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait until +all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split @@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved FeatureBits}
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}
forcontent.tex +4
-1handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} --2.45.2
Quoted 123 lines · Parav Pandit
Hi Lingshan, David,
Quoted 121 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++----- -- 1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:BasicFacilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +10
-5- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that +the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by @@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}.
content.tex +5
-2-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than +SUSPEND except when setting \field{device status} to 0 as a +transport-specific way to initiate a reset. If the driver sets the +FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a @@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device type-\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:GeneralInitialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device reset) before
removing exposed buffers.+\section{Device Suspend}\label{sec:General Initialization And Device
content.tex +10
-0+Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the SUSPEND bit +in \field{device status} to suspend a device, and can clear the SUSPEND +bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization +And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +3
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize}
Quoted 8 lines
Quoted 8 lines
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+
Hi Parva
Quoted 2 lines
Do we agree that
a. suspending a device is non frequent operation (in order of N operations/sec, where N is roughly in range of 10 or 100) per device?
Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.
Quoted 1 line
b. A software-based device may not always want to force
VM_EXITon read and write on thedevice_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.
Thanks
Zhu Lingshan
Quoted 57 lines
Quoted 57 lines
+\devicenormative{\subsection}{Device Suspend}{General Initialization
content.tex +3
-0+And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +3
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the ConfigurationSpace.
content.tex +8
-0+ +A device MUST NOT send any notifications for any virtqeuues, access any +virtqueues, or modify any fields in its Configuration Space while +suspended. + +If changes occur in the Configuration Space while the SUSPEND bit is +set, the device MUST NOT send any configuration change notifications. +Instead, the device MUST send the notification after the SUSPEND bit hasbeen cleared.
content.tex +2
-0+ +When the driver sets SUSPEND, the device MUST either suspend itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND, the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex +3
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +8
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait until +all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split @@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved FeatureBits}
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}
forcontent.tex +4
-1handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} --2.45.2
Quoted 155 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 17 lines · Parav Pandit
Hi Lingshan, David,
Quoted 15 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++-----
Quoted 41 lines
Quoted 41 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +3
-3- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready +to + drive the device.Quoted 1 line
Quoted 1 line
\item[FEATURES_OK (8)] Indicates that the driver has acknowledged all
the
Quoted 10 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
Quoted 10 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
content.tex +3
-1- drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates +that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 9 lines
Quoted 9 lines
an error from which it can't recover.
content.tex +6
-0+ +\item[FAILED (128)] Indicates that something went wrong in the +guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description}Quoted 5 lines
Quoted 5 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 46 lines
Quoted 46 lines
content.tex +6
-3-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than +SUSPEND except when setting \field{device status} to 0 as a +transport-specific way to initiate a reset. If the driver sets the +FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a @@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device
type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue +and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:GeneralInitialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device reset)
before removing exposed buffers.+\section{Device Suspend}\label{sec:General Initialization And Device
content.tex +10
-0+Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the SUSPEND +bit in \field{device status} to suspend a device, and can clear the +SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization +And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set orQuoted 8 lines
Quoted 8 lines
VIRTIO_F_SUSPENDis not negotiated.content.tex +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 6 lines
Quoted 2 lines
Do we agree that
a. suspending a device is non frequent operation (in order of Noperations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
Quoted 5 lines
Quoted 1 line
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.
Sure, I didn’t suggest to put such things in the spec.
My question is, whether to trap and emulate or not is a choice of the software. Do you agree?
Quoted 66 lines
Thanks
Zhu LingshanQuoted 64 lines
Quoted 64 lines
+\devicenormative{\subsection}{Device Suspend}{General Initialization
content.tex +3
-0+And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the +ConfigurationSpace.
content.tex +9
-0+ +A device MUST NOT send any notifications for any virtqeuues, access +any virtqueues, or modify any fields in its Configuration Space +while suspended. + +If changes occur in the Configuration Space while the SUSPEND bit is +set, the device MUST NOT send any configuration change notifications. +Instead, the device MUST send the notification after the SUSPEND bit +hasbeen cleared.
content.tex +3
-0+ +When the driver sets SUSPEND, the device MUST either suspend itself +or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND, the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting +thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio TransportOptions}
Virtio can use various different buses, thus the standard is split
@@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature
Bits} forcontent.tex +4
-0handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved FeatureBits}
--
2.45.2
Quoted 154 lines · Parav Pandit
Quoted 154 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 17 lines · Parav Pandit
Hi Lingshan, David,
Quoted 15 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75 ++++++++++++++++++++++++++++++++++++++++++++++-----
Quoted 41 lines
Quoted 41 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +3
-3- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready +to + drive the device.Quoted 1 line
Quoted 1 line
\item[FEATURES_OK (8)] Indicates that the driver has acknowledged all
the
Quoted 10 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
Quoted 10 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
content.tex +3
-1- drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates +that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 9 lines
Quoted 9 lines
an error from which it can't recover.
content.tex +6
-0+ +\item[FAILED (128)] Indicates that something went wrong in the +guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description}Quoted 5 lines
Quoted 5 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 46 lines
Quoted 46 lines
content.tex +6
-3-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than +SUSPEND except when setting \field{device status} to 0 as a +transport-specific way to initiate a reset. If the driver sets the +FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a @@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device
type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue +and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
\end{description}\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:GeneralInitialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device reset)
before removing exposed buffers.+\section{Device Suspend}\label{sec:General Initialization And Device
content.tex +10
-0+Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the SUSPEND +bit in \field{device status} to suspend a device, and can clear the +SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization +And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set orQuoted 8 lines
Quoted 8 lines
VIRTIO_F_SUSPENDis not negotiated.content.tex +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Do we agree that
a. suspending a device is non frequent operation (in order of Noperations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
I think we don't have an intention of the frequency in the spec.
The spec only provides generic mechanisms and interfaces.
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.Sure, I didn’t suggest to put such things in the spec.
My question is, whether to trap and emulate or not is a choice of the software. Do you agree?
The device emulator does not know anything about whether trapped or not. Trapping and Emulation is a hypervisor thing.
If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of VM_EXIT & VM_ENTRY.
this is out of the spec anyway.
Thanks
Zhu Lingshan
Quoted 67 lines
Quoted 67 lines
Thanks
Zhu LingshanQuoted 63 lines
Quoted 63 lines
+\devicenormative{\subsection}{Device Suspend}{General Initialization
content.tex +3
-0+And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the +ConfigurationSpace.
content.tex +9
-0+ +A device MUST NOT send any notifications for any virtqeuues, access +any virtqueues, or modify any fields in its Configuration Space +while suspended. + +If changes occur in the Configuration Space while the SUSPEND bit is +set, the device MUST NOT send any configuration change notifications. +Instead, the device MUST send the notification after the SUSPEND bit +hasbeen cleared.
content.tex +3
-0+ +When the driver sets SUSPEND, the device MUST either suspend itself +or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND, the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting +thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio TransportOptions}
Virtio can use various different buses, thus the standard is split
@@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device / Feature
Bits} forcontent.tex +4
-0handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved FeatureBits}
--
2.45.2
Quoted 5 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AM
I removed the unreachable intel email id as every single email is bouncing from it. Please consider dropping that email from v8 as it will cause all reviewers email to bounce.
Quoted 163 lines · Parav Pandit
Quoted 16 lines
Quoted 16 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 11 lines · Parav Pandit
Hi Lingshan, David,
Quoted 9 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75++++++++++++++++++++++++++++++++++++++++++++++-
Quoted 46 lines
Quoted 46 lines
----
Quoted 42 lines
Quoted 42 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.content.tex index 0a62dce..2d1bee8 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +3
-3- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready +to + drive the device.Quoted 2 lines
Quoted 2 lines
\item[FEATURES_OK (8)] Indicates that the driver has acknowledged
allthe
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] WhenVIRTIO_F_SUSPENDis negotiated,indicates
Quoted 20 lines
Quoted 20 lines
Quoted 4 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
Quoted 4 lines
content.tex +2
-0+that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 9 lines
Quoted 9 lines
an error from which it can't recover.
content.tex +6
-0+ +\item[FAILED (128)] Indicates that something went wrong in the +guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description}Quoted 5 lines
Quoted 5 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
content.tex
Quoted 7 lines
content.tex +4
-2-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than +SUSPEND except when setting \field{device status} to 0 as a +transport-specific way to initiate a reset. If the driver sets the +FAILED bit, the driver MUST later reset the device before attempting to re-initialize.
Quoted 58 lines
Quoted 58 lines
Quoted 40 lines
Quoted 40 lines
The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device
type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue +and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:GeneralInitialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device reset)
before removing exposed buffers.+\section{Device Suspend}\label{sec:General Initialization And
content.tex +10
-0+Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set orQuoted 8 lines
Quoted 8 lines
VIRTIO_F_SUSPENDis not negotiated.content.tex +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 2 lines
Do we agree that
a. suspending a device is non frequent operation (in order of N
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
I think we don't have an intention of the frequency in the spec.
Sure.
Quoted 1 line
The spec only provides generic mechanisms and interfaces.
Sure.
Quoted 2 lines
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.
As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is < 50nsec of time.
The suspend operation for a net device to store hundreds of queues, RSS table, flow filters, takes plenty of time (at least more than 50nsec :) ). Similarly for the GPU to store some MBs of memory takes more than 50nsec of time, for example to store in a file for a software-based GPU device. So a device cannot respond back suspend=true in next 50nsec time.
More below.
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.Sure, I didn’t suggest to put such things in the spec.
My question is, whether to trap and emulate or not is a choice of the
Quoted 9 lines
software.
Quoted 1 line
Do you agree?
The device emulator does not know anything about whether trapped or not. Trapping and Emulation is a hypervisor thing.
If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of
VM_EXIT&VM_ENTRY.
Right. So a software wants to implement device_status as pure MMIO writes. (and not VM_EXIT). And prefer to returning SUSPEND=true at slow pace.
This means, the device implementation cannot immediately return suspend=true right after it was written. A MMIO read will read it back, as suspend=true.
An alternative would be, to forward CPU loads and CPU stores to different address. However, this does not work for the hw based devices.
That means, PCI HW needs to return suspend=0, until the device is not suspended.
In this example, the device cannot build special circuitry to answer suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver reads suspend_status, and receives not_completed=(false). This is the default value.
3. When the device completes suspend, it changes the polarity of suspend_status=true.
This has two main benefits:
[A] This will enable software-based devices to write data to slow files and does not have to force VM_EXITs.
[B] It also enables hw based devices to not build special circuitry to answer within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
Quoted 74 lines
this is out of the spec anyway.
Thanks
Zhu LingshanQuoted 14 lines
Quoted 14 lines
Thanks
Zhu LingshanQuoted 10 lines
Quoted 10 lines
+\devicenormative{\subsection}{Device Suspend}{General
content.tex +3
-0+Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the +ConfigurationQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
Space.
content.tex +7
-0+ +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in its Configuration +Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND bit +is set, the device MUST NOT send any configuration changenotifications.
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
content.tex +2
-0+Instead, the device MUST send the notification after the SUSPEND +bit hasbeen cleared.
content.tex +3
-0+ +When the driver sets SUSPEND, the device MUST either suspend +itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND, the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting +thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio TransportOptions}
Virtio can use various different buses, thus the standard is split
@@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver
can
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex +3
-0+ trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved FeatureBits}
--
2.45.2
Quoted 180 lines · Parav Pandit
Quoted 4 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AMI removed the unreachable intel email id as every single email is bouncing from it. Please consider dropping that email from v8 as it will cause all reviewers email to bounce.
Quoted 161 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 11 lines · Parav Pandit
Hi Lingshan, David,
Quoted 9 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75++++++++++++++++++++++++++++++++++++++++++++++-
Quoted 46 lines
Quoted 46 lines
----
Quoted 42 lines
Quoted 42 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.content.tex index 0a62dce..2d1bee8 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +3
-3- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready +to + drive the device.Quoted 2 lines
Quoted 2 lines
\item[FEATURES_OK (8)] Indicates that the driver has acknowledged
allthe
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] WhenVIRTIO_F_SUSPENDis negotiated,indicates
Quoted 20 lines
Quoted 20 lines
Quoted 4 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev
Quoted 4 lines
content.tex +2
-0+that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 9 lines
Quoted 9 lines
an error from which it can't recover.
content.tex +6
-0+ +\item[FAILED (128)] Indicates that something went wrong in the +guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description}Quoted 5 lines
Quoted 5 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
content.tex
Quoted 7 lines
content.tex +4
-2-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than +SUSPEND except when setting \field{device status} to 0 as a +transport-specific way to initiate a reset. If the driver sets the +FAILED bit, the driver MUST later reset the device before attempting to re-initialize.
Quoted 57 lines
Quoted 57 lines
Quoted 39 lines
Quoted 39 lines
The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific device
type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue +and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:GeneralInitialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device reset)
before removing exposed buffers.+\section{Device Suspend}\label{sec:General Initialization And
content.tex +10
-0+Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set orQuoted 8 lines
Quoted 8 lines
VIRTIO_F_SUSPENDis not negotiated.content.tex +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 2 lines
Do we agree that
a. suspending a device is non frequent operation (in order of NQuoted 7 lines
Quoted 6 lines
Quoted 4 lines
operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
I think we don't have an intention of the frequency in the spec.
Sure.
Quoted 1 line
The spec only provides generic mechanisms and interfaces.
Sure.
Quoted 7 lines
Quoted 2 lines
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.
As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is < 50nsec of time.
The suspend operation for a net device to store hundreds of queues, RSS table, flow filters, takes plenty of time (at least more than 50nsec :) ). Similarly for the GPU to store some MBs of memory takes more than 50nsec of time, for example to store in a file for a software-based GPU device. So a device cannot respond back suspend=true in next 50nsec time.
It's OK for the device to take longer time to respond, the driver simply re-reads device status.
Quoted 13 lines
More below.
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.Sure, I didn’t suggest to put such things in the spec.
My question is, whether to trap and emulate or not is a choice of thesoftware.
Quoted 1 line
Do you agree?
Quoted 8 lines
Quoted 7 lines
The device emulator does not know anything about whether trapped or not. Trapping and Emulation is a hypervisor thing.
If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of
VM_EXIT&VM_ENTRY.Right. So a software wants to implement
device_statusas pure MMIO writes. (and notVM_EXIT).
This is not always true, there can be VM_EXIT of pure emulated devices. The HW registers are sensitive resource and any access to them need to be trapped and emulated.
Quoted 9 lines
And prefer to returning SUSPEND=true at slow pace.
This means, the device implementation cannot immediately return suspend=true right after it was written. A MMIO read will read it back, as suspend=true.An alternative would be, to forward CPU loads and CPU stores to different address. However, this does not work for the hw based devices.
That means, PCI HW needs to return suspend=0, until the device is not suspended.
In this example, the device cannot build special circuitry to answer suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.
Quoted 5 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
Quoted 8 lines
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This is the default value.
3. When the device completes suspend, it changes the polarity of suspend_status=true.This has two main benefits:
[A] This will enable software-based devices to write data to slow files and does not have to forceVM_EXITs.[B] It also enables hw based devices to not build special circuitry to answer within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful comments
Quoted 73 lines
Quoted 73 lines
this is out of the spec anyway.
Thanks
Zhu LingshanQuoted 12 lines
Quoted 12 lines
Thanks
Zhu LingshanQuoted 9 lines
Quoted 9 lines
+\devicenormative{\subsection}{Device Suspend}{General
content.tex +3
-0+Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the +ConfigurationQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
Space.
content.tex +7
-0+ +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in its Configuration +Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND bit +is set, the device MUST NOT send any configuration changenotifications.
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
content.tex +2
-0+Instead, the device MUST send the notification after the SUSPEND +bit hasbeen cleared.
content.tex +3
-0+ +When the driver sets SUSPEND, the device MUST either suspend +itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver clears +SUSPEND, the device MUST either resume normal operation or set
DEVICE_NEEDS_RESET.content.tex +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting +thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio TransportOptions}
Virtio can use various different buses, thus the standard is split
@@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver
can
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex +3
-0+ trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved FeatureBits}
--
2.45.2
Quoted 193 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PMQuoted 5 lines · Parav Pandit
Quoted 4 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AMI removed the unreachable intel email id as every single email is bouncing
from it.
Quoted 1 line
Please consider dropping that email from v8 as it will cause all reviewers
email to bounce.
Quoted 174 lines
Quoted 165 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 11 lines · Parav Pandit
Hi Lingshan, David,
Quoted 9 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75++++++++++++++++++++++++++++++++++++++++++++++-
Quoted 47 lines
Quoted 47 lines
----
Quoted 43 lines
Quoted 43 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.content.tex index 0a62dce..2d1bee8 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +0
-2- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason,or
content.tex +3
-1- even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and +ready to + drive the device.Quoted 2 lines
Quoted 2 lines
\item[FEATURES_OK (8)] Indicates that the driver has
acknowledged allthe
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] WhenVIRTIO_F_SUSPENDis negotiated,indicates
Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
content.tex
Quoted 4 lines
content.tex +2
-0+that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 10 lines
Quoted 10 lines
an error from which it can't recover.
content.tex +7
-0+ +\item[FAILED (128)] Indicates that something went wrong in the +guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, +or + even a fatal error during device operation. \end{description}Quoted 6 lines
Quoted 6 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
content.tex
Quoted 7 lines
content.tex +4
-2-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than +SUSPEND except when setting \field{device status} to 0 as a +transport-specific way to initiate a reset. If the driver sets +the FAILED bit, the driver MUST later reset the device before attempting to re-initialize.
Quoted 57 lines
Quoted 57 lines
Quoted 39 lines
Quoted 39 lines
The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:GeneralInitialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.+\section{Device Suspend}\label{sec:General Initialization And
content.tex +10
-0+Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and +can clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set orQuoted 8 lines
Quoted 8 lines
VIRTIO_F_SUSPENDis not negotiated.content.tex +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 2 lines
Do we agree that
a. suspending a device is non frequent operation (in order of NQuoted 7 lines
Quoted 6 lines
Quoted 4 lines
operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
I think we don't have an intention of the frequency in the spec.
Sure.
Quoted 7 lines
Quoted 1 line
The spec only provides generic mechanisms and interfaces.
Sure.
Quoted 2 lines
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.
As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <
Quoted 8 lines
50nsec of time.
Quoted 1 line
The suspend operation for a net device to store hundreds of queues, RSS
table, flow filters, takes plenty of time (at least more than 50nsec :) ).
Quoted 1 line
Similarly for the GPU to store some MBs of memory takes more than 50nsec
of time, for example to store in a file for a software-based GPU device.
Quoted 1 line
So a device cannot respond back suspend=true in next 50nsec time.
It's OK for the device to take longer time to respond, the driver simply re-reads device status.
Sure, the issue is, when the driver re-reads, the device must present suspend=false within 50nsec. (because device didn't suspend it yet).
As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly differently.
Quoted 18 lines
Quoted 15 lines
More below.
Quoted 12 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the
software.
Quoted 1 line
Do you agree?
The device emulator does not know anything about whether trapped or
not.
Quoted 2 lines
Quoted 2 lines
Trapping and Emulation is a hypervisor thing.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of
VM_EXIT&VM_ENTRY.Right. So a software wants to implement
device_statusas pure MMIOwrites. (and not
VM_EXIT).
This is not always true, there can beVM_EXITof pure emulated devices.
I am not denying that VM_EXIT can/cannot be there.
I am saying, the proposal forces VM_EXIT based approach in my understanding of this patch. If that is not true, may be can you please explain how this can be implemented without VM_EXIT?
We should have an interface that can be done with and without VM_EXIT method at least for any new additions.
Quoted 3 lines
The
HW registers are sensitive resource and any access to them need to be trapped and emulated.
This does not apply to PCI PFs and VFs which are HW devices (mainly PFs). so this trap + emulation is narrow view that we better avoid.
If you think this is the way forward, you should put forward in patch as MUST requirement. and that does not look right to me.
I hope you also don't mean to force this method to device implementations. Right?
Quoted 9 lines
Quoted 2 lines
And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return
suspend=true right after it was written.
Quoted 3 lines
A MMIO read will read it back, as suspend=true.
An alternative would be, to forward CPU loads and CPU stores to different
address.
Quoted 2 lines
However, this does not work for the hw based devices.
Quoted 7 lines
Quoted 1 line
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 1 line
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.
When driver wrote, it wrote suspend=true,
And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec or less to answer suspend=false.
And even worst, it needs to suspend=true when unsuspending within 50nsec when resuming is ongoing.
Quoted 13 lines
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files and
Quoted 8 lines
does not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful comments
Unfortunately, not. His comment was that it is not specific to suspend.
But here we are introducing a new interface and functionality that does not need to suffer or follow anything that may not be efficient.
Quoted 77 lines
Quoted 75 lines
Quoted 75 lines
this is out of the spec anyway.
Thanks
Zhu LingshanQuoted 12 lines
Quoted 12 lines
Thanks
Zhu LingshanQuoted 9 lines
Quoted 9 lines
+\devicenormative{\subsection}{Device Suspend}{General
content.tex +3
-0+Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status} if it is part +of the ConfigurationQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
Space.
content.tex +7
-0+ +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in its Configuration +Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND +bit is set, the device MUST NOT send any configuration changenotifications.
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
content.tex +2
-0+Instead, the device MUST send the notification after the SUSPEND +bit hasbeen cleared.
content.tex +3
-0+ +When the driver sets SUSPEND, the device MUST either suspend +itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +4
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND, the device MUST either resume normal operation +or set
DEVICE_NEEDS_RESET.content.tex +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before +presenting thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio TransportOptions}
Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the
+ drivercan
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex +3
-0+ trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{ReservedFeature Bits}
--
2.45.2
Quoted 196 lines · Parav Pandit
Quoted 196 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PMQuoted 4 lines · Parav Pandit
Quoted 3 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AMI removed the unreachable intel email id as every single email is bouncing
from it.
Quoted 1 line
Please consider dropping that email from v8 as it will cause all reviewers
email to bounce.
Quoted 173 lines
Quoted 164 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 11 lines · Parav Pandit
Hi Lingshan, David,
Quoted 9 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75++++++++++++++++++++++++++++++++++++++++++++++-
Quoted 47 lines
Quoted 47 lines
----
Quoted 43 lines
Quoted 43 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.content.tex index 0a62dce..2d1bee8 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadable modules.
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +0
-2- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason,or
content.tex +3
-1- even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and +ready to + drive the device.Quoted 2 lines
Quoted 2 lines
\item[FEATURES_OK (8)] Indicates that the driver has
acknowledged allthe
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] WhenVIRTIO_F_SUSPENDis negotiated,indicates
Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
content.tex
Quoted 4 lines
content.tex +2
-0+that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 10 lines
Quoted 10 lines
an error from which it can't recover.
content.tex +7
-0+ +\item[FAILED (128)] Indicates that something went wrong in the +guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, +or + even a fatal error during device operation. \end{description}Quoted 6 lines
Quoted 6 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
content.tex
Quoted 7 lines
content.tex +4
-2-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than +SUSPEND except when setting \field{device status} to 0 as a +transport-specific way to initiate a reset. If the driver sets +the FAILED bit, the driver MUST later reset the device before attempting to re-initialize.
Quoted 57 lines
Quoted 57 lines
Quoted 39 lines
Quoted 39 lines
The driver SHOULD NOT rely on completion of operations of a @@
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a +Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{Device Cleanup}\label{sec:GeneralInitialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.+\section{Device Suspend}\label{sec:General Initialization And
content.tex +10
-0+Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and +can clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set orQuoted 8 lines
Quoted 8 lines
VIRTIO_F_SUSPENDis not negotiated.content.tex +5
-0+ +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify +whether theSUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 2 lines
Do we agree that
a. suspending a device is non frequent operation (in order of NQuoted 7 lines
Quoted 6 lines
Quoted 4 lines
operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
I think we don't have an intention of the frequency in the spec.
Sure.
Quoted 7 lines
Quoted 1 line
The spec only provides generic mechanisms and interfaces.
Sure.
Quoted 2 lines
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.
As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <50nsec of time.
Quoted 1 line
The suspend operation for a net device to store hundreds of queues, RSS
table, flow filters, takes plenty of time (at least more than 50nsec :) ).
Quoted 1 line
Similarly for the GPU to store some MBs of memory takes more than 50nsec
of time, for example to store in a file for a software-based GPU device.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
So a device cannot respond back suspend=true in next 50nsec time.
It's OK for the device to take longer time to respond, the driver simply re-reads device status.
Sure, the issue is, when the driver re-reads, the device must present suspend=false within 50nsec. (because device didn't suspend it yet).
As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly differently.
why there is a constraint condition of the time? Are there any similar constraining for other states like RESET or DRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.
Quoted 22 lines
Quoted 22 lines
Quoted 15 lines
More below.
Quoted 12 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the
software.
Quoted 1 line
Do you agree?
The device emulator does not know anything about whether trapped or
not.
Quoted 6 lines
Quoted 6 lines
Trapping and Emulation is a hypervisor thing.
If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of
VM_EXIT&VM_ENTRY.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Right. So a software wants to implement
device_statusas pure MMIOwrites. (and not
VM_EXIT).
This is not always true, there can beVM_EXITof pure emulated devices.I am not denying that
VM_EXITcan/cannot be there.
I am saying, the proposal forcesVM_EXITbased approach in my understanding of this patch. If that is not true, may be can you please explain how this can be implemented withoutVM_EXIT?We should have an interface that can be done with and without
VM_EXITmethod at least for any new additions.
VM_EXIT is out of spec, it is a hypervisor and the processor thing.
In non-pass-through case, any register access are sensitive and will trigger VM_EXIT. Like RESET or DRIVER_OK needs to access device_status, nothing different.
Quoted 6 lines
Quoted 4 lines
The
HW registers are sensitive resource and any access to them need to be trapped and emulated.This does not apply to PCI PFs and VFs which are HW devices (mainly PFs). so this trap + emulation is narrow view that we better avoid.
This is how *basic* virtualization work, once access sensitive resource, trap it.
Quoted 5 lines
If you think this is the way forward, you should put forward in patch as MUST requirement. and that does not look right to me.
I hope you also don't mean to force this method to device implementations. Right?
Again, VM_EXIT is a hypervisor thing, out of spec. Whether there is a VM_EXIT when setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different from DRIVER_OK.
Means, if your virtualization needs to trap SUSPEND, it also needs to trap DRIVER_OK, and don't assume DRIVER_OK is faster than SUSPEND.
Quoted 17 lines
Quoted 10 lines
Quoted 3 lines
And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return
suspend=true right after it was written.
Quoted 3 lines
A MMIO read will read it back, as suspend=true.
An alternative would be, to forward CPU loads and CPU stores to different
address.
Quoted 2 lines
However, this does not work for the hw based devices.
Quoted 7 lines
Quoted 1 line
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 1 line
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.
Quoted 5 lines
When driver wrote, it wrote suspend=true,
And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec or less to answer suspend=false.And even worst, it needs to suspend=true when unsuspending within 50nsec when resuming is ongoing.
again, there is not a 50nsec constraining and please take a reference of how DRIVER_OK work with virtualization.
Quoted 96 lines
Quoted 10 lines
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 9 lines
Quoted 2 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 1 line
[B] It also enables hw based devices to not build special circuitry to answer
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful commentsUnfortunately, not. His comment was that it is not specific to suspend.
But here we are introducing a new interface and functionality that does not need to suffer or follow anything that may not be efficient.Quoted 74 lines
Quoted 73 lines
Quoted 73 lines
this is out of the spec anyway.
Thanks
Zhu LingshanQuoted 11 lines
Quoted 11 lines
Thanks
Zhu LingshanQuoted 9 lines
Quoted 9 lines
+\devicenormative{\subsection}{Device Suspend}{General
content.tex +3
-0+Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status} if it is part +of the ConfigurationQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
Space.
content.tex +7
-0+ +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in its Configuration +Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND +bit is set, the device MUST NOT send any configuration changenotifications.
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
content.tex +2
-0+Instead, the device MUST send the notification after the SUSPEND +bit hasbeen cleared.
content.tex +3
-0+ +When the driver sets SUSPEND, the device MUST either suspend +itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +4
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND, the device MUST either resume normal operation +or set
DEVICE_NEEDS_RESET.content.tex +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before +presenting thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item Wait +until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio TransportOptions}
Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the
+ drivercan
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex +3
-0+ trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{ReservedFeature Bits}
--
2.45.2
Quoted 216 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:02 AMQuoted 61 lines · Parav Pandit
Quoted 61 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PMQuoted 5 lines · Parav Pandit
Quoted 3 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AMI removed the unreachable intel email id as every single email is bouncing
from it.
Quoted 2 lines
Please consider dropping that email from v8 as it will cause all reviewers
email to bounce.
Quoted 48 lines
Quoted 48 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 11 lines · Parav Pandit
Hi Lingshan, David,
Quoted 9 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75++++++++++++++++++++++++++++++++++++++++++++++-
Quoted 25 lines
Quoted 25 lines
----
Quoted 24 lines
Quoted 24 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
content.tex index 0a62dce..2d1bee8 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadablemodules.
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +0
-2- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason,or
content.tex +3
-1- even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and +ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver hasacknowledged all
Quoted 7 lines
Quoted 7 lines
the
Quoted 6 lines
Quoted 6 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] WhenVIRTIO_F_SUSPENDis negotiated,indicates
Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
content.tex
Quoted 4 lines
content.tex +2
-0+that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 10 lines
Quoted 10 lines
an error from which it can't recover.
content.tex +7
-0+ +\item[FAILED (128)] Indicates that something went wrong in the +guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, +or + even a fatal error during device operation. \end{description}Quoted 6 lines
Quoted 6 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex +4
-2-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other +than SUSPEND except when setting \field{device status} to 0 as +a transport-specific way to initiate a reset. If the driver +sets the FAILED bit, the driver MUST later reset the device before attempting tore-
initialize.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The driver SHOULD NOT rely on completion of operations of a
@@
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of +a Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
content.tex \section{Device
Quoted 2 lines
\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{DeviceCleanup}\label{sec:General
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Initialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
+\section{Device Suspend}\label{sec:General Initialization And
content.tex +5
-0+Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and +can clear the SUSPEND bit to resume a suspended device.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
content.tex +5
-0+ +\drivernormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +2
-0+ +Once the driver sets SUSPEND to \field{device status} of thedevice:
Quoted 43 lines
Quoted 43 lines
Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify +whether the
SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to +thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 2 lines
Do we agree that
a. suspending a device is non frequent operation (in order of NQuoted 7 lines
Quoted 6 lines
Quoted 4 lines
operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
I think we don't have an intention of the frequency in the spec.
Sure.
Quoted 8 lines
Quoted 1 line
The spec only provides generic mechanisms and interfaces.
Sure.
Quoted 2 lines
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.
As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <50nsec of time.
Quoted 2 lines
The suspend operation for a net device to store hundreds of queues, RSS
table, flow filters, takes plenty of time (at least more than 50nsec :) ).
Quoted 2 lines
Similarly for the GPU to store some MBs of memory takes more than 50nsec
of time, for example to store in a file for a software-based GPU device.
Quoted 1 line
So a device cannot respond back suspend=true in next 50nsec time.
It's OK for the device to take longer time to respond, the driver simply re-reads device status.
Quoted 8 lines
Quoted 1 line
Sure, the issue is, when the driver re-reads, the device must present
suspend=false within 50nsec.
Quoted 4 lines
(because device didn't suspend it yet).
As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly
differently.
why there is a constraint condition of the time?
Because this is what driver does as you explained in [1] indicating "we must be able to handle ..."
Quoted 3 lines
Are there any similar
constraining for other states like RESET orDRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.
DRIVER_OK does not suffer from it because it is async notification. A device may start slow after DRIVER_OK.
SUSPEND operation cannot rely on such async behavior.
RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.
Here a new functionality is being proposed and it has a chance for efficient device implementation. Therefore the request is to improve it.
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
More below.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the
software.
Quoted 1 line
Do you agree?
The device emulator does not know anything about whether trapped
or
Quoted 7 lines
Quoted 7 lines
not.
Quoted 6 lines
Quoted 6 lines
Trapping and Emulation is a hypervisor thing.
If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of
VM_EXIT&VM_ENTRY.
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Right. So a software wants to implement
device_statusas pure MMIOwrites. (and not
VM_EXIT).
This is not always true, there can beVM_EXITof pure emulated devices.I am not denying that
VM_EXITcan/cannot be there. I am saying, the proposal forcesVM_EXITbased approach in myunderstanding of this patch.
[MARKER_1]
Quoted 6 lines
Quoted 1 line
If that is not true, may be can you please explain how this can be
implemented without
VM_EXIT?Quoted 2 lines
We should have an interface that can be done with and without
VM_EXITmethod at least for any new additions.
VM_EXITis out of spec, it is a hypervisor and the processor thing.
You keep repeating VM_EXIST is out of spec.
I already replied at [MARKER_1], sure it is out of spec, the current approach forces one to do VM_EXIT based approach. And if not, please explain, how can it be achieved?
Quoted 3 lines
In non-pass-through case, any register access are sensitive and will trigger
VM_EXIT. Like RESET orDRIVER_OKneeds to accessdevice_status, nothing different.
I don't think you understood the point.
Let me repeat, The question is, if the device implementation wants to achieve the functionality without VM_EXIT, what is the way?
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
The
HW registers are sensitive resource and any access to them need to be trapped and emulated.This does not apply to PCI PFs and VFs which are HW devices (mainly PFs). so this trap + emulation is narrow view that we better avoid.
This is how *basic* virtualization work, once access sensitive resource, trap it.
Quoted 2 lines
If you think this is the way forward, you should put forward in patch as
Quoted 9 lines
MUST requirement.
Quoted 2 lines
and that does not look right to me.
I hope you also don't mean to force this method to deviceimplementations.
Quoted 1 line
Right?
Again,
VM_EXITis a hypervisor thing, out of spec. Whether there is aVM_EXITwhen setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different fromDRIVER_OK.
Please avoid repeating the point that VM_EXIT is hypervosor thing.
No one asked to put this in spec. Please re-read [MARKER1]. You probably missed that.
SUSPEND is different than DRIVER_OK. I explained the timing constraints and the required circuitry needed to fulfill the proposal. And with the additional register, such complicated circuitry can be easily avoided.
Quoted 2 lines
Means, if your virtualization needs to trap SUSPEND, it also needs to trap
DRIVER_OK, and don't assumeDRIVER_OKis faster than SUSPEND.
DRIVER_OK is by law of physics is faster than SUSPEND because it does not demand the driver of reading back. There is no driver side loop to check if the device accepted DRIVER_OK or not.
Agree?
Quoted 22 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return
suspend=true right after it was written.
Quoted 4 lines
A MMIO read will read it back, as suspend=true.
An alternative would be, to forward CPU loads and CPU stores to different
address.
Quoted 4 lines
However, this does not work for the hw based devices.
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation. why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.
When driver wrote, it wrote suspend=true, And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec
Quoted 6 lines
or less to answer suspend=false.
Quoted 2 lines
And even worst, it needs to suspend=true when unsuspending within
50nsec when resuming is ongoing.
again, there is not a 50nsec constraining and please take a reference of howDRIVER_OKwork with virtualization.
There is. The device is expected to return back the desired value to indicate driver that suspend is ongoing. It is different than DRIVER_OK.
Quoted 121 lines
Quoted 17 lines
Quoted 17 lines
Quoted 8 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 7 lines
Quoted 5 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful commentsUnfortunately, not. His comment was that it is not specific to suspend. But here we are introducing a new interface and functionality that does not
need to suffer or follow anything that may not be efficient.
Quoted 2 lines
Quoted 2 lines
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
Quoted 1 line
Quoted 1 line
e.kernel.org%2Fvirtio-comment%2F20240612082055-mutt-send-email-
mst%40
Quoted 1 line
Quoted 1 line
kernel.org%2FT%2F%23mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9&dat a=05
Quoted 1 line
Quoted 1 line
%7C02%7Cparav%40nvidia.com%7C619c82b60b824ca03f8808dcc89beb4a%7 C43083
Quoted 1 line
Quoted 1 line
d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638605819247985989%7CUn known%7C
Quoted 1 line
Quoted 1 line
TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL CJXV
Quoted 1 line
Quoted 1 line
CI6Mn0%3D%7C0%7C%7C%7C&sdata=VqmJmqt3k5tf3x4ihjE5Bd7u59GadOn OaOfJ5lvG
Quoted 25 lines
Quoted 25 lines
1DE%3D&reserved=0
Quoted 24 lines
Quoted 24 lines
this is out of the spec anyway.
Thanks
Zhu LingshanQuoted 11 lines
Quoted 11 lines
Thanks
Zhu LingshanQuoted 9 lines
Quoted 9 lines
+\devicenormative{\subsection}{Device Suspend}{General
content.tex +3
-0+Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status} if it is +part of the ConfigurationQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
Space.
content.tex +7
-0+ +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in its +Configuration Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND +bit is set, the device MUST NOT send any configuration changenotifications.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex
Quoted 10 lines
content.tex +2
-0+Instead, the device MUST send the notification after the +SUSPEND bit hasbeen cleared.
content.tex +3
-0+ +When the driver sets SUSPEND, the device MUST either suspend +itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND, the device MUST either resume normaloperation
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
+or set
DEVICE_NEEDS_RESET.content.tex +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before +presenting thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item +Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio TransportOptions}
Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the
+ drivercan
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
content.tex +2
-0+ trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device StatusField}.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+
\end{description}\drivernormative{\section}{Reserved Feature Bits}{Reserved
Feature Bits}
--
2.45.2
Quoted 219 lines · Parav Pandit
Quoted 219 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:02 AMQuoted 61 lines · Parav Pandit
Quoted 61 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PMQuoted 5 lines · Parav Pandit
Quoted 3 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AMI removed the unreachable intel email id as every single email is bouncing
from it.
Quoted 2 lines
Please consider dropping that email from v8 as it will cause all reviewers
email to bounce.
Quoted 48 lines
Quoted 48 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 11 lines · Parav Pandit
Hi Lingshan, David,
Quoted 9 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75++++++++++++++++++++++++++++++++++++++++++++++-
Quoted 25 lines
Quoted 25 lines
----
Quoted 24 lines
Quoted 24 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsChanges from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume the device.
- disallow access to config space while suspended.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
content.tex index 0a62dce..2d1bee8 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be loadablemodules.
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
content.tex +0
-2- and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason,or
content.tex +3
-1- even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and +ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver hasacknowledged all
Quoted 7 lines
Quoted 7 lines
the
Quoted 6 lines
Quoted 6 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] WhenVIRTIO_F_SUSPENDis negotiated,indicates
Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
content.tex
Quoted 4 lines
content.tex +2
-0+that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 10 lines
Quoted 10 lines
an error from which it can't recover.
content.tex +7
-0+ +\item[FAILED (128)] Indicates that something went wrong in the +guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, +or + even a fatal error during device operation. \end{description}Quoted 6 lines
Quoted 6 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex +4
-2-The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other +than SUSPEND except when setting \field{device status} to 0 as +a transport-specific way to initiate a reset. If the driver +sets the FAILED bit, the driver MUST later reset the device before attempting tore-
initialize.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The driver SHOULD NOT rely on completion of operations of a
@@
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of +a Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
content.tex \section{Device
Quoted 2 lines
\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{DeviceCleanup}\label{sec:General
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Initialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
+\section{Device Suspend}\label{sec:General Initialization And
content.tex +5
-0+Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and +can clear the SUSPEND bit to resume a suspended device.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
content.tex +5
-0+ +\drivernormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +2
-0+ +Once the driver sets SUSPEND to \field{device status} of thedevice:
Quoted 36 lines
Quoted 36 lines
Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify +whether the
SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to +thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 2 lines
Do we agree that
a. suspending a device is non frequent operation (in order of NQuoted 7 lines
Quoted 6 lines
Quoted 4 lines
operations/sec, where N is roughly in range of 10 or 100) per device? Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is often.
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
I think we don't have an intention of the frequency in the spec.
Sure.
Quoted 8 lines
Quoted 1 line
The spec only provides generic mechanisms and interfaces.
Sure.
Quoted 2 lines
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.
As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <50nsec of time.
Quoted 2 lines
The suspend operation for a net device to store hundreds of queues, RSS
Quoted 8 lines
Quoted 7 lines
table, flow filters, takes plenty of time (at least more than 50nsec :) ).
Quoted 2 lines
Similarly for the GPU to store some MBs of memory takes more than 50nsec
of time, for example to store in a file for a software-based GPU device.
Quoted 1 line
So a device cannot respond back suspend=true in next 50nsec time.
It's OK for the device to take longer time to respond, the driver simply re-reads device status.
Sure, the issue is, when the driver re-reads, the device must present
suspend=false within 50nsec.
Quoted 2 lines
(because device didn't suspend it yet).
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly
differently.
why there is a constraint condition of the time?Because this is what driver does as you explained in [1] indicating "we must be able to handle ..."
The sentence is "we must be able to handle the scenario in which SUSPENDING is often", where do you see any constraint conditions of time?
Quoted 7 lines
Quoted 4 lines
Are there any similar
constraining for other states like RESET orDRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.
DRIVER_OKdoes not suffer from it because it is async notification. A device may start slow afterDRIVER_OK.
SUSPEND operation cannot rely on such async behavior.
The driver can set suspend and re-read & wait. This is a common routine in the driver.
Quoted 6 lines
RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.
Here a new functionality is being proposed and it has a chance for efficient device implementation. Therefore the request is to improve it.
sure, as long as we confirm this new register apply for all device status transitions, not only for SUSPEND.
Quoted 32 lines
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
More below.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the
software.
Quoted 1 line
Do you agree?
The device emulator does not know anything about whether trapped
or
Quoted 7 lines
Quoted 7 lines
not.
Quoted 6 lines
Quoted 6 lines
Trapping and Emulation is a hypervisor thing.
If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of
VM_EXIT&VM_ENTRY.Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Right. So a software wants to implement
device_statusas pure MMIOwrites. (and not
VM_EXIT).
This is not always true, there can beVM_EXITof pure emulated devices.I am not denying that
VM_EXITcan/cannot be there. I am saying, the proposal forcesVM_EXITbased approach in myunderstanding of this patch.
[
MARKER_1]
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
If that is not true, may be can you please explain how this can be
implemented without
VM_EXIT?Quoted 1 line
We should have an interface that can be done with and without
VM_EXITmethod at least for any new additions.
VM_EXITis out of spec, it is a hypervisor and the processor thing.You keep repeating
VM_EXISTis out of spec.
I already replied at [MARKER_1], sure it is out of spec, the current approach forces one to doVM_EXITbased approach. And if not, please explain, how can it be achieved?
It is not the current solution force the hypervisor & guest to perform VM_EXIT.
In basic virtualization, any access to hypervisor emulated HW registers would be Traped and Emulated, means a VM_EXIT and a VM_ENTRY.
Quoted 6 lines
Quoted 4 lines
In non-pass-through case, any register access are sensitive and will trigger
VM_EXIT. Like RESET orDRIVER_OKneeds to accessdevice_status, nothing different.I don't think you understood the point.
Let me repeat, The question is, if the device implementation wants to achieve the functionality withoutVM_EXIT, what is the way?
Map the register to guest address space, out of the spec.
Quoted 17 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
The
HW registers are sensitive resource and any access to them need to be trapped and emulated.This does not apply to PCI PFs and VFs which are HW devices (mainly PFs). so this trap + emulation is narrow view that we better avoid.
This is how *basic* virtualization work, once access sensitive resource, trap it.
Quoted 1 line
If you think this is the way forward, you should put forward in patch as
Quoted 9 lines
MUST requirement.
Quoted 2 lines
and that does not look right to me.
I hope you also don't mean to force this method to deviceimplementations.
Quoted 1 line
Right?
Again,
VM_EXITis a hypervisor thing, out of spec. Whether there is aVM_EXITwhen setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different fromDRIVER_OK.
Quoted 5 lines
Please avoid repeating the point that
VM_EXITis hypervosor thing.
No one asked to put this in spec. Please re-read [MARKER1]. You probably missed that.SUSPEND is different than
DRIVER_OK. I explained the timing constraints and the required circuitry needed to fulfill the proposal. And with the additional register, such complicated circuitry can be easily avoided.
Please read QEMU code, that can help you find how the trap - emulate mechanism work for guest when it tries to access sensitive resources.
Quoted 6 lines
Quoted 3 lines
Means, if your virtualization needs to trap SUSPEND, it also needs to trap
DRIVER_OK, and don't assumeDRIVER_OKis faster than SUSPEND.
DRIVER_OKis by law of physics is faster than SUSPEND because it does not demand the driver of reading back. There is no driver side loop to check if the device acceptedDRIVER_OKor not.
Agree?
Why do you assume so? In some vendor implementation, DRIVER_OK can be slow. Can DRIVER_OK implement a deferred initialization even deferred resource allocation? Does the spec say that DRIVER_OK must be faster?
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 3 lines
And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return
suspend=true right after it was written.
Quoted 4 lines
A MMIO read will read it back, as suspend=true.
An alternative would be, to forward CPU loads and CPU stores to different
address.
Quoted 2 lines
However, this does not work for the hw based devices.
Quoted 8 lines
Quoted 2 lines
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 1 line
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation. why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.
When driver wrote, it wrote suspend=true, And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsec
Quoted 7 lines
Quoted 5 lines
or less to answer suspend=false.
Quoted 1 line
And even worst, it needs to suspend=true when unsuspending within
50nsec when resuming is ongoing.
again, there is not a 50nsec constraining and please take a reference of howDRIVER_OKwork with virtualization.There is. The device is expected to return back the desired value to indicate driver that suspend is ongoing. It is different than
DRIVER_OK.
where does the spec or any code say 50nsec?
Quoted 114 lines
Quoted 114 lines
Quoted 17 lines
Quoted 17 lines
Quoted 8 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 7 lines
Quoted 5 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful commentsUnfortunately, not. His comment was that it is not specific to suspend. But here we are introducing a new interface and functionality that does not
need to suffer or follow anything that may not be efficient.
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2FlorQuoted 1 line
Quoted 1 line
e.kernel.org%2Fvirtio-comment%2F20240612082055-mutt-send-email-
mst%40 kernel.org%2FT%2F%23mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9&dat a=05 %7C02%7Cparav%40nvidia.com%7C619c82b60b824ca03f8808dcc89beb4a%7 C43083 d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638605819247985989%7CUn known%7C TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL CJXV CI6Mn0%3D%7C0%7C%7C%7C&sdata=VqmJmqt3k5tf3x4ihjE5Bd7u59GadOn OaOfJ5lvG
Quoted 25 lines
Quoted 25 lines
1DE%3D&reserved=0
Quoted 24 lines
Quoted 24 lines
this is out of the spec anyway.
Thanks
Zhu LingshanQuoted 11 lines
Quoted 11 lines
Thanks
Zhu LingshanQuoted 9 lines
Quoted 9 lines
+\devicenormative{\subsection}{Device Suspend}{General
content.tex +3
-0+Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status} if it is +part of the ConfigurationQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
Space.
content.tex +7
-0+ +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in its +Configuration Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND +bit is set, the device MUST NOT send any configuration changenotifications.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex
Quoted 10 lines
content.tex +2
-0+Instead, the device MUST send the notification after the +SUSPEND bit hasbeen cleared.
content.tex +3
-0+ +When the driver sets SUSPEND, the device MUST either suspend +itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND, the device MUST either resume normaloperation
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
+or set
DEVICE_NEEDS_RESET.content.tex +4
-0+ +When the driver sets SUSPEND, +the device SHOULD perform the following actions before +presenting thatthe SUSPEND bit is set to 1 in the \field{device status}:
content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item +Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio TransportOptions}
Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the
+ drivercan
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
content.tex +2
-0+ trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device StatusField}.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+
\end{description}\drivernormative{\section}{Reserved Feature Bits}{Reserved
Feature Bits}
--
2.45.2
Quoted 7 lines · Zhu Lingshan
Quoted 6 lines
Quoted 3 lines
Are there any similar
constraining for other states like RESET orDRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.
DRIVER_OKdoes not suffer from it because it is async notification. A device may start slow afterDRIVER_OK.
SUSPEND operation cannot rely on such async behavior.The driver can set suspend and re-read & wait. This is a common routine in the driver.
I don't buy all this talk about special machinery, some 10 nor gates, the cost is in the noise - we are talking about cards with several ARM processors on chip.
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.
we could steal 2 bits from status but it seems a bit much.
Quoted 6 lines
Quoted 5 lines
RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.
Here a new functionality is being proposed and it has a chance for efficient device implementation. Therefore the request is to improve it.
sure, as long as we confirm this new register apply for all device status transitions, not only for SUSPEND.
Quoted 12 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, September 3, 2024 3:16 PMQuoted 6 lines · Zhu Lingshan
Quoted 6 lines
Quoted 3 lines
Are there any similar
constraining for other states like RESET orDRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.
DRIVER_OKdoes not suffer from it because it is async notification. A device may start slow afterDRIVER_OK.
SUSPEND operation cannot rely on such async behavior.
Quoted 6 lines
Quoted 1 line
The driver can set suspend and re-read & wait. This is a common routine in
the driver.
I don't buy all this talk about special machinery, some 10 nor gates, the cost is in the noise - we are talking about cards with several ARM processors on chip.
They may not respond back under 50nsec.
How many cores would you like to dedicate and for how many PCI functions? And how they can be easily underutilized...
So, it is not a noise.
Circuity is the device abstraction that we keep instead of digging down on gates.
Also can you please explain how can it work without VM_EXIT? You have better insight to this than me..
Quoted 9 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
Quoted 18 lines · Parav Pandit
Quoted 12 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, September 3, 2024 3:16 PMQuoted 6 lines · Zhu Lingshan
Quoted 6 lines
Quoted 3 lines
Are there any similar
constraining for other states like RESET orDRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.
DRIVER_OKdoes not suffer from it because it is async notification. A device may start slow afterDRIVER_OK.
SUSPEND operation cannot rely on such async behavior.Quoted 6 lines
Quoted 1 line
The driver can set suspend and re-read & wait. This is a common routine in
the driver.
I don't buy all this talk about special machinery, some 10 nor gates, the cost is in the noise - we are talking about cards with several ARM processors on chip.
Quoted 8 lines
They may not respond back under 50nsec.
How many cores would you like to dedicate and for how many PCI functions? And how they can be easily underutilized...
So, it is not a noise.Circuity is the device abstraction that we keep instead of digging down on gates.
Also can you please explain how can it work without
VM_EXIT? You have better insight to this than me..
I don't know which interface we are arguing about here, when there's a specific proposal, I can comment.
Quoted 10 lines
Quoted 9 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Quoted 7 lines · Michael S. Tsirkin
Quoted 7 lines
Quoted 7 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status. If we need to reflect and control device status, we need something else.
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Michael S. Tsirkin
Quoted 7 lines
Quoted 7 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others. Anyhow driver know its own status.
Thanks
Quoted 3 lines
If we need
to reflect and control device status, we need something else.
Quoted 15 lines · Jason Wang
Quoted 14 lines · Michael S. Tsirkin
Quoted 12 lines · Michael S. Tsirkin
Quoted 10 lines
Quoted 9 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.
Quoted 24 lines · Michael S. Tsirkin
Quoted 15 lines · Jason Wang
Quoted 14 lines · Michael S. Tsirkin
Quoted 12 lines · Michael S. Tsirkin
Quoted 10 lines
Quoted 9 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.
We've already had msleep() for vp_reset(), anyhow we can increase the sleep time, if it can overload the pci:
while (vp_modern_get_status(mdev))
msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
Quoted 15 lines
Quoted 5 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.
Thanks
Quoted 35 lines · Jason Wang
Quoted 17 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Thanks
Zhu Lingshan
Quoted 15 lines
Quoted 13 lines
Quoted 5 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 47 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 1 line · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>
Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to
control thing is elegant.
Quoted 26 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
Quoted 7 lines
Quoted 4 lines
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Sounds good to me as long as new functionality does not force the device to react in 50nsec. In the current state it forces the device so please upgrade the proposal as Michael guided. Thanks.
Quoted 49 lines · Parav Pandit
Quoted 49 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 1 line · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>
Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to
control thing is elegant.
Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
Quoted 5 lines
Quoted 3 lines
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Sounds good to me as long as new functionality does not force the device to react in 50nsec. In the current state it forces the device so please upgrade the proposal as Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation. But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time
Quoted 1 line
Thanks.
Quoted 59 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PMQuoted 53 lines · Parav Pandit
Quoted 53 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 1 line · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>
Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to
control thing is elegant.
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 9 lines
Quoted 1 line
Sounds good to me as long as new functionality does not force the device
to react in 50nsec.
Quoted 1 line
In the current state it forces the device so please upgrade the proposal as
Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain timeQuoted 1 line
Thanks.
You are completely ignoring the comments.
Good luck..
Quoted 61 lines · Parav Pandit
Quoted 61 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PMQuoted 46 lines · Parav Pandit
Quoted 46 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 1 line · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>
Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to
control thing is elegant.
Quoted 25 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Sounds good to me as long as new functionality does not force the device
to react in 50nsec.
Quoted 1 line
In the current state it forces the device so please upgrade the proposal as
Quoted 8 lines
Quoted 6 lines
Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain timeQuoted 1 line
Thanks.
You are completely ignoring the comments.
Good luck..
VM_EXIT and a VM_ENTRY work in a pair.
Quoted 1 line
Quoted 78 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:00 PMQuoted 17 lines · Parav Pandit
Quoted 17 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PMQuoted 12 lines · Parav Pandit
Quoted 12 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 2 lines · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for
debuggability.
Quoted 51 lines
Quoted 51 lines
Quoted 38 lines
Quoted 38 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to
control thing is elegant.
Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
Quoted 5 lines
Quoted 3 lines
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Sounds good to me as long as new functionality does not force the device
to react in 50nsec.
Quoted 2 lines
In the current state it forces the device so please upgrade the proposal as
Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Thanks.
You are completely ignoring the comments.
Good luck..
VM_EXITand aVM_ENTRYwork in a pair.Quoted 1 line
For hw pci pf where there is no virtualization used? How?
Quoted 77 lines · Parav Pandit
Quoted 77 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:00 PMQuoted 17 lines · Parav Pandit
Quoted 17 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PMQuoted 12 lines · Parav Pandit
Quoted 12 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 2 lines · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
But I don't like it that looking at the registers, one does
not know the device state. Hidden state is bad fordebuggability.
Quoted 51 lines
Quoted 51 lines
Quoted 38 lines
Quoted 38 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to
control thing is elegant.
Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
Quoted 5 lines
Quoted 3 lines
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Sounds good to me as long as new functionality does not force the device
to react in 50nsec.
Quoted 2 lines
In the current state it forces the device so please upgrade the proposal as
Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Thanks.
You are completely ignoring the comments.
Good luck..
VM_EXITand aVM_ENTRYwork in a pair.For hw pci pf where there is no virtualization used? How?
you bring the VM_EXIT problem to me, if without virtualization, there is no VM_EXIT.
Quoted 1 line
Quoted 78 lines · Zhu Lingshan
Quoted 77 lines · Parav Pandit
Quoted 77 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:00 PMQuoted 17 lines · Parav Pandit
Quoted 17 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PMQuoted 12 lines · Parav Pandit
Quoted 12 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 2 lines · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
But I don't like it that looking at the registers, one does
not know the device state. Hidden state is bad fordebuggability.
Quoted 51 lines
Quoted 51 lines
Quoted 38 lines
Quoted 38 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control
register tocontrol thing is elegant.
Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
Quoted 5 lines
Quoted 3 lines
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Sounds good to me as long as new functionality does not force the device
to react in 50nsec.
Quoted 2 lines
In the current state it forces the device so please upgrade the proposal as
Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Thanks.
You are completely ignoring the comments.
Good luck..
VM_EXITand aVM_ENTRYwork in a pair.For hw pci pf where there is no virtualization used? How?
you bring the
VM_EXITproblem to me, if without virtualization, there is noVM_EXIT.
The problem Parav is raising is that the single bit approach effectively requires that the device handles updates to the status field synchronously. In a virtualization context with VM_EXIT/VM_ENTRY, this synchronous handling is easy. However, it's not necessarily easy with bare metal.
Even in the virtualization context, it's entirely possible to implement a virtualized device where the status field is plain memory that is periodically checked by the device. If the device only checks the status field once per second, then it's pretty clear that the single suspend bit approach won't work at all. I don't know why someone would implement a device like this, but it's not something that the virtio spec should rule out.
-David
Quoted 100 lines
From: David Stevens <[email protected]>
Sent: Thursday, September 5, 2024 2:12 PMQuoted 79 lines · Zhu Lingshan
Quoted 78 lines · Parav Pandit
Quoted 78 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:00 PMQuoted 18 lines · Parav Pandit
Quoted 18 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 12:45 PMQuoted 13 lines · Parav Pandit
Quoted 13 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 2 lines · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
But I don't like it that looking at the registers, one
does not know the device state. Hidden state is bad fordebuggability.
Quoted 51 lines
Quoted 51 lines
Quoted 38 lines
Quoted 38 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and
control register tocontrol thing is elegant.
Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
Quoted 5 lines
Quoted 3 lines
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Sounds good to me as long as new functionality does not force the device
to react in 50nsec.
Quoted 2 lines
In the current state it forces the device so please upgrade the proposal as
Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation.
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain timeQuoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Thanks.
You are completely ignoring the comments.
Good luck..
VM_EXITand aVM_ENTRYwork in a pair.For hw pci pf where there is no virtualization used? How?
you bring the
VM_EXITproblem to me, if without virtualization, there is no
VM_EXIT.The problem Parav is raising is that the single bit approach effectively requires that the device handles updates to the status field synchronously. In a virtualization context with VM_EXIT/VM_ENTRY, this synchronous handling is easy. However, it's not necessarily easy with bare metal.
Quoted 5 lines
Even in the virtualization context, it's entirely possible to implement a virtualized device where the status field is plain memory that is periodically checked by the device. If the device only checks the status field once per second, then it's pretty clear that the single suspend bit approach won't work at all.
Thanks a lot David. You captured the summary precisely.
Quoted 4 lines
I don't know why someone would implement a device like this, but it's not something that the virtio spec should rule out.
-David
Quoted 52 lines · Zhu Lingshan
Quoted 49 lines · Parav Pandit
Quoted 49 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 1 line · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>
Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to
control thing is elegant.
Quoted 29 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Sounds good to me as long as new functionality does not force the device to react in 50nsec. In the current state it forces the device so please upgrade the proposal as Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation. But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time
The spec should not assume devices are virtual, there are baremetal deployments of virtio. In such setups, there is no one to trap or
emulate: driver must do everything itself.
Quoted 1 line
Quoted 1 line
Thanks.
Quoted 55 lines · Michael S. Tsirkin
Quoted 54 lines · Zhu Lingshan
Quoted 52 lines · Parav Pandit
Quoted 52 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 1 line · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Jason Wang
Quoted 1 line
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin <[email protected]>
Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability. We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to
control thing is elegant.
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Sounds good to me as long as new functionality does not force the device to react in 50nsec. In the current state it forces the device so please upgrade the proposal as Michael guided.
Trap and emulate is very often in virtualization, of course the sooner the better for emulation. But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time
The spec should not assume devices are virtual, there are baremetal deployments of virtio. In such setups, there is no one to trap or
emulate: driver must do everything itself.
sure thing, I am also confused why VM_EXIT & VM_ENTRY and is a problem here...
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Thanks.
Quoted 71 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 1:01 PMQuoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Zhu Lingshan
Quoted 12 lines · Parav Pandit
Quoted 12 lines
From: Zhu Lingshan <[email protected]>
Sent: Wednesday, September 4, 2024 12:09 PMQuoted 2 lines · Jason Wang
On Wed, Sep 4, 2024 at 12:03 PM Michael S. Tsirkin <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines · Jason Wang
Quoted 2 lines
On Tue, Sep 3, 2024 at 6:37 PM Michael S. Tsirkin
<[email protected]>Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Sep 03, 2024 at 06:35:59AM -0400, Michael S. Tsirkin
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for
debuggability.
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to
control thing is elegant.
Quoted 25 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling.
The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Sounds good to me as long as new functionality does not force the
device to react in 50nsec.
Quoted 1 line
Quoted 1 line
Quoted 1 line
In the current state it forces the device so please upgrade the proposal
as Michael guided.
Quoted 1 line
Quoted 1 line
Trap and emulate is very often in virtualization, of course the sooner the
Quoted 8 lines
better for emulation.
Quoted 5 lines
Quoted 2 lines
But there are no 50nsec assumption and no one can force the hypervisor to finish any emulations within a certain time
The spec should not assume devices are virtual, there are baremetal deployments of virtio. In such setups, there is no one to trap or
emulate: driver must do everything itself.sure thing, I am also confused why
VM_EXIT&VM_ENTRYand is a problem here...
How does proposed VM_EXIT and VM_ENTRY work for basic PCI PF hw device?
How can one build sw without VM_EXIT and VM_ENTRY restriction if they want to?
Apart from other timing restrictions imposed that makes this very high complexity device for rare event. Please list down the assumptions and limitations so that it becomes clear the value of this bit..
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Thanks.
Quoted 46 lines · Zhu Lingshan
Quoted 35 lines · Jason Wang
Quoted 17 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
Quoted 5 lines
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Thanks
Zhu Lingshan
If you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 15 lines
Quoted 15 lines
Quoted 13 lines
Quoted 5 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 7 lines
Quoted 2 lines
Thanks
Zhu LingshanIf you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
I guess the driver can reset the device even after DRIVER_OK. IMHO the indicator should work for all device_status transitions, not only for RESET or SUSPEND, it should also apply for FEATURE_OK and DRIVER_OK. So we don't need a feature bit, we may just place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!
Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes device_status.
Thanks
Quoted 19 lines
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Thanks
Zhu LingshanIf you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
I guess the driver can reset the device even after
DRIVER_OK.
in fact, unlike suspend - at any point at all.
Quoted 3 lines
IMHO the indicator should work for all
device_statustransitions, not only for RESET or SUSPEND, it should also apply forFEATURE_OKandDRIVER_OK.
I don't know what kind of transition is there for DRIVER/DRIVER_OK. FEATURES_OK brings with it more issues, e.g. it can fail.
Quoted 3 lines
So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!
I have to say, this project is really ballooning out ;)
Quoted 5 lines
Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes
device_status.Thanks
or just a register for suspend transitions, worry about extending it later/
Quoted 19 lines
Quoted 19 lines
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 72 lines · Michael S. Tsirkin
Quoted 49 lines · Zhu Lingshan
Quoted 47 lines · Michael S. Tsirkin
Quoted 46 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Thanks
Zhu LingshanIf you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
I guess the driver can reset the device even after
DRIVER_OK.in fact, unlike suspend - at any point at all.
Quoted 3 lines
IMHO the indicator should work for all
device_statustransitions, not only for RESET or SUSPEND, it should also apply forFEATURE_OKandDRIVER_OK.I don't know what kind of transition is there for DRIVER/DRIVER_OK.
FEATURES_OKbrings with it more issues, e.g. it can fail.Quoted 3 lines
So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!I have to say, this project is really ballooning out ;)
Quoted 7 lines
Quoted 5 lines
Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes
device_status.Thanks
or just a register for suspend transitions, worry about extending it later/
OK, will implement it in V8
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 9 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, September 5, 2024 2:40 PMQuoted 3 lines
Quoted 1 line
Thanks
or just a register for suspend transitions, worry about extending it later/
OK, will implement it in V8
Thanks Zhu Lingshan.
Quoted 78 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Thanks
Zhu LingshanIf you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
I guess the driver can reset the device even after
DRIVER_OK.in fact, unlike suspend - at any point at all.
Quoted 3 lines
IMHO the indicator should work for all
device_statustransitions, not only for RESET or SUSPEND, it should also apply forFEATURE_OKandDRIVER_OK.I don't know what kind of transition is there for DRIVER/DRIVER_OK.
FEATURES_OKbrings with it more issues, e.g. it can fail.Quoted 3 lines
So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!I have to say, this project is really ballooning out ;)
Quoted 8 lines
Quoted 5 lines
Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes
device_status.Thanks
or just a register for suspend transitions, worry about extending it later/
Is this for PCI only? Another question is that, if suspend needs that, reset would also want. Or it doesn't justify itself as reset needs to take longer time than reset.
Thanks
Quoted 21 lines
Quoted 20 lines
Quoted 20 lines
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 77 lines · Jason Wang
Quoted 76 lines · Michael S. Tsirkin
Quoted 49 lines · Zhu Lingshan
Quoted 47 lines · Michael S. Tsirkin
Quoted 46 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Thanks
Zhu LingshanIf you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
I guess the driver can reset the device even after
DRIVER_OK.in fact, unlike suspend - at any point at all.
Quoted 3 lines
IMHO the indicator should work for all
device_statustransitions, not only for RESET or SUSPEND, it should also apply forFEATURE_OKandDRIVER_OK.I don't know what kind of transition is there for DRIVER/DRIVER_OK.
FEATURES_OKbrings with it more issues, e.g. it can fail.Quoted 3 lines
So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!I have to say, this project is really ballooning out ;)
Quoted 4 lines
Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes
device_status.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Thanks
or just a register for suspend transitions, worry about extending it later/
Is this for PCI only? Another question is that, if suspend needs that, reset would also want. Or it doesn't justify itself as reset needs to take longer time than reset.
Ping Michael.
I think Jason's question is critical and we should address it before we proceed v8
Thanks
Quoted 22 lines
Thanks
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 83 lines · Jason Wang
Quoted 82 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device state. Hidden state is bad for debuggability.
We have 4 states:suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Thanks
Zhu LingshanIf you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
I guess the driver can reset the device even after
DRIVER_OK.in fact, unlike suspend - at any point at all.
Quoted 3 lines
IMHO the indicator should work for all
device_statustransitions, not only for RESET or SUSPEND, it should also apply forFEATURE_OKandDRIVER_OK.I don't know what kind of transition is there for DRIVER/DRIVER_OK.
FEATURES_OKbrings with it more issues, e.g. it can fail.Quoted 3 lines
So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!I have to say, this project is really ballooning out ;)
Quoted 4 lines
Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes
device_status.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thanks
or just a register for suspend transitions, worry about extending it later/
Is this for PCI only?
It's best to add it to all transports. Not rocket science at all.
Quoted 2 lines
Another question is that, if suspend needs that,
reset would also want.
I don't know what does "reset would also want" means. Unlike suspend, reset is not a special state, so it does not really require a spcial register to track that state.
Quoted 5 lines
Or it doesn't justify itself as reset needs to
take longer time than reset.Thanks
Don't know what this means, either.
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 20 lines
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 85 lines · Michael S. Tsirkin
Quoted 83 lines · Jason Wang
Quoted 82 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device
state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Thanks
Zhu LingshanIf you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
I guess the driver can reset the device even after
DRIVER_OK.in fact, unlike suspend - at any point at all.
Quoted 3 lines
IMHO the indicator should work for all
device_statustransitions, not only for RESET or SUSPEND, it should also apply forFEATURE_OKandDRIVER_OK.I don't know what kind of transition is there for DRIVER/DRIVER_OK.
FEATURES_OKbrings with it more issues, e.g. it can fail.Quoted 3 lines
So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!I have to say, this project is really ballooning out ;)
Quoted 4 lines
Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes
device_status.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thanks
or just a register for suspend transitions, worry about extending it later/
Is this for PCI only?
It's best to add it to all transports. Not rocket science at all.
I'm asking since I wonder if the "issue" exists only for PCI. If yes, solving the PCI transport (registers) issue at the basic facility level seems like a layer violation. And I still don't see why introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions. As mentioned before, re-read has been used for both FEAUTRES_OK and reset.
Quoted 5 lines
Quoted 3 lines
Another question is that, if suspend needs that,
reset would also want.I don't know what does "reset would also want" means.
I meant we have already had a state transition like reset.
For example, you worries about:
suspending->suspended->resuming->resumed
But we've already had
setting_features -> feature_ok(or not) -> resetting -> reseted
Quoted 1 line
Unlike suspend, reset is not a special state,
It depends but we had a dedicated chapter in the basic facility to describe the reset state.
Quoted 3 lines
so it
does not really require a spcial register to track that state.
This seems to be PCI specific. Note that ccw has a dedicated command for reset:
#define CCW_CMD_VDEV_RESET 0x33
...
#define CCW_CMD_WRITE_STATUS 0x31
Quoted 8 lines
Quoted 6 lines
Or it doesn't justify itself as reset needs to
take longer time than reset.Thanks
Don't know what this means, either.
I meant
1) driver knows its own status and it can read device status
2) reset may take much longer time than suspend
If we really need a new status register (I don't see why so far), we should not only suspend.
Thanks
Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 91 lines · Jason Wang
Quoted 90 lines · Michael S. Tsirkin
Quoted 88 lines · Jason Wang
Quoted 86 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
But I don't like it that looking at the registers, one does not know the device
state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.
Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Thanks
Zhu LingshanIf you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
I guess the driver can reset the device even after
DRIVER_OK.in fact, unlike suspend - at any point at all.
Quoted 3 lines
IMHO the indicator should work for all
device_statustransitions, not only for RESET or SUSPEND, it should also apply forFEATURE_OKandDRIVER_OK.I don't know what kind of transition is there for DRIVER/DRIVER_OK.
FEATURES_OKbrings with it more issues, e.g. it can fail.Quoted 3 lines
So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!I have to say, this project is really ballooning out ;)
Quoted 5 lines
Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes
device_status.Thanks
or just a register for suspend transitions, worry about extending it later/
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Is this for PCI only?
It's best to add it to all transports. Not rocket science at all.
I'm asking since I wonder if the "issue" exists only for PCI. If yes,
solving the PCI transport (registers) issue at the basic facility level seems like a layer violation.
I don't think so. Suspend is fundamentally a slow operation for any transport.
Quoted 3 lines
And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.
reset is not a state.
Quoted 3 lines
As
mentioned before, re-read has been used for bothFEAUTRES_OKand reset.
FEAUTRES_OK is driver state, so can take place immediately
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Another question is that, if suspend needs that,
reset would also want.I don't know what does "reset would also want" means.
I meant we have already had a state transition like reset.
Quoted 7 lines
For example, you worries about:
suspending->suspended->resuming->resumedBut we've already had
setting_features->feature_ok(or not)-> resetting -> reseted
features ok just has to validate the bitmap, so it can take place immediately. no intermediate state.
Quoted 4 lines
Quoted 1 line
Unlike suspend, reset is not a special state,
It depends but we had a dedicated chapter in the basic facility to describe the reset state.
this is two different meanings of the same word. reset is not a special state, device operates normally.
Quoted 9 lines
Quoted 3 lines
so it
does not really require a spcial register to track that state.This seems to be PCI specific. Note that ccw has a dedicated command for reset:
#define
CCW_CMD_VDEV_RESET0x33
...
#defineCCW_CMD_WRITE_STATUS0x31
maybe reusing state 0 to reset was a bad idea.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Or it doesn't justify itself as reset needs to
take longer time than reset.Thanks
Quoted 6 lines
Quoted 1 line
Don't know what this means, either.
I meant
1) driver knows its own status and it can read device status
2) reset may take much longer time than suspend
that is why driver re-reads status to check for reset.
Quoted 5 lines
If we really need a new status register (I don't see why so far), we should not only suspend.
Thanks
it's not a new status register, it's a suspend register used for PM.
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 97 lines · Michael S. Tsirkin
Quoted 95 lines · Jason Wang
Quoted 93 lines · Michael S. Tsirkin
Quoted 83 lines · Jason Wang
Quoted 82 lines · Michael S. Tsirkin
Quoted 51 lines · Zhu Lingshan
Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Zhu Lingshan
Quoted 34 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 8 lines · Jason Wang
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
But I don't like it that looking at the registers, one does not know the device
state. Hidden state is bad for debuggability.
We have 4 states:
suspending->suspended->resuming->resumed
so we need a register with at least 2 bits.Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
we could steal 2 bits from status but it seems a bit much.
This is why letting the status tell the status and control register to control thing is elegant.
No argument here.Or, to be more precise, our status is driver status.
It looks like the device actually otherwise there's no need for re-read or poll for things like reset and others.
Quoted 9 lines
The need is there for complex device state transitions, which can not reasonably block a read response.
Another standard approach with PCI is to specify the time transitions can take. I consider that less elegant - is this what you are advocating? The advantage is that driver does not load the pci bus with constant re-polling. The disadvantage is that it is hard to pick a universal number. A combination of these approaches might work, e.g. a recommended timeout then poll.We've already had
msleep()forvp_reset(), anyhow we can increase the sleep time, if it can overload the pci:while (vp_modern_get_status(mdev)) msleep(1);We can do the same for suspending.
Quoted 8 lines
The main blocker for timeout is that it may break migration and complicate the hardening. Another proposal in the past is to have a notification.
But what I don't understand here is that suspend/resume should be lighter than reset. If we can afford a reset, so did the suspending/resume. If we want to have something new, that's fine but it should be orthogonal to a specific new status bit?
I agree, if we want new status indicator, then the new indicator should not be specific to SUSPEND.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Thanks
Zhu LingshanIf you mean reset, we have a problem. Specifically, reset has to work before feature negotiation. So we can not do as we would with SUSPEND and use a feature bit to expose presence of the indicator register.
I guess the driver can reset the device even after
DRIVER_OK.in fact, unlike suspend - at any point at all.
Quoted 3 lines
IMHO the indicator should work for all
device_statustransitions, not only for RESET or SUSPEND, it should also apply forFEATURE_OKandDRIVER_OK.I don't know what kind of transition is there for DRIVER/DRIVER_OK.
FEATURES_OKbrings with it more issues, e.g. it can fail.Quoted 3 lines
So we don't need a feature bit, we may just
place it in the common config space as you proposed before to define common config space for all transport. We can do this for sure!I have to say, this project is really ballooning out ;)
Quoted 4 lines
Of course another approach is what Jason proposed, the existing msleep and poll, driver knows whether the device is in progress of status transitions because it is the driver writes
device_status.Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thanks
or just a register for suspend transitions, worry about extending it later/
Is this for PCI only?
It's best to add it to all transports. Not rocket science at all.
Quoted 8 lines
Quoted 4 lines
I'm asking since I wonder if the "issue" exists only for PCI. If yes,
solving the PCI transport (registers) issue at the basic facility level seems like a layer violation.I don't think so. Suspend is fundamentally a slow operation for any transport.
Probably, but how such slowness makes a new register a must? And my understanding is, reset should be even slower than suspend.
Quoted 7 lines
Quoted 5 lines
And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.reset is not a state.
Well, I'm not sure I get this. Each value read from status should represent a device state. And 0 is a valid status which means device is resetted:
"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""
Quoted 6 lines
Quoted 4 lines
As
mentioned before, re-read has been used for bothFEAUTRES_OKand reset.
FEAUTRES_OKis driver state, so can take place immediately
I'm not sure how to define immediately, but suspend could be very fast depending on the implementation.
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Quoted 4 lines
Another question is that, if suspend needs that,
reset would also want.I don't know what does "reset would also want" means.
I meant we have already had a state transition like reset.
For example, you worries about:
suspending->suspended->resuming->resumed
Quoted 7 lines
Quoted 3 lines
But we've already had
setting_features->feature_ok(or not)-> resetting -> resetedfeatures ok just has to validate the bitmap, so it can take place immediately. no intermediate state.
True but there's an intermediate state between resetting and reseted. When is the state when the driver has written 0 to status but not read 0 from status.
Quoted 14 lines
Quoted 5 lines
Quoted 2 lines
Unlike suspend, reset is not a special state,
It depends but we had a dedicated chapter in the basic facility to describe the reset state.
this is two different meanings of the same word. reset is not a special state, device operates normally.
Quoted 4 lines
Quoted 3 lines
so it
does not really require a spcial register to track that state.
Quoted 8 lines
Quoted 5 lines
This seems to be PCI specific. Note that ccw has a dedicated command for reset:
#define
CCW_CMD_VDEV_RESET0x33
...
#defineCCW_CMD_WRITE_STATUS0x31maybe reusing state 0 to reset was a bad idea.
Not sure, but it's too late to change that.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Or it doesn't justify itself as reset needs to
take longer time than reset.Thanks
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Don't know what this means, either.
I meant
1) driver knows its own status and it can read device status
2) reset may take much longer time than suspendthat is why driver re-reads status to check for reset.
Yes, but this is exactly how suspend is supposed to work for this patch.
Reset has one intermediate state and drivers need to poll for the completion. Suspending has two intermediate states, drivers need to poll the completion for them.
Quoted 9 lines
Quoted 7 lines
If we really need a new status register (I don't see why so far), we should not only suspend.
Thanks
it's not a new status register, it's a suspend register used for PM.
I think power management should belong to the transport layer, not the virtio layer.
Thanks
Quoted 26 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 23 lines
Regrettably, reset will have to still be supported through clearing status just because this is always there.
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
Quoted 4 lines
Anyhow driver know
its own status.Thanks
indeed, the status register is there to inform the device about the driver status.
Quoted 3 lines
Quoted 3 lines
If we need
to reflect and control device status, we need something else.Thanks
Quoted 9 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines
And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.reset is not a state.
Well, I'm not sure I get this. Each value read from status should represent a device state.
What makes you say so? status represents driver state, not device state. In fact, driver writes there and does not double check with the sole exception of reset.
Quoted 7 lines
And 0 is a valid status which means device
is resetted:"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""
It means all registers have their reset values, but it is not a special state in that device behaves normally in this state, it does not need to be handles in any special way.
Contrast with suspend where device does not respond and must be first resumed.
Signature · 1 line
MST
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines
And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.reset is not a state.
Quoted 7 lines
Quoted 2 lines
Well, I'm not sure I get this. Each value read from status should represent a device state.
What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.
For example, spec said
"""
2.1 Device Status Field
During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.
The device status field provides a simple low-level indication of the completed steps of this sequence. It’s most useful to imagine it hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):
"""
Driver knows its own state, so there's no need for the driver to read it from the device.
Quoted 10 lines
Quoted 8 lines
And 0 is a valid status which means device
is resetted:"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""It means all registers have their reset values,
This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.
Quoted 6 lines
but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.Contrast with suspend where device does not respond and must be first resumed.
I think not. Driver can choose to reset the device when the device is suspended.
Thanks
Quoted 7 lines
--
MST
Quoted 19 lines · Jason Wang
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines
And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.reset is not a state.
Quoted 7 lines
Quoted 2 lines
Well, I'm not sure I get this. Each value read from status should represent a device state.
What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.
Quoted 5 lines
For example, spec said
"""
2.1 Device Status Field
yes, this is confusingly written
Quoted 5 lines
During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.
The device status field provides a simple low-level indication of the completed steps of this sequence.
And note the sequence is driver sequence.
Quoted 5 lines
It’s most useful to imagine it
hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):
Quoted 7 lines
"""
Driver knows its own state, so there's no need for the driver to read it from the device.
That is why it never reads status.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
And 0 is a valid status which means device
is resetted:"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""
Quoted 4 lines
Quoted 1 line
It means all registers have their reset values,
This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.
yes but otherwise, all registers respond normally device is assumed to be fully operational, and it is assumed that no time at all is needed to get from that to DRIVER.
Quoted 10 lines
Quoted 6 lines
but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.Contrast with suspend where device does not respond and must be first resumed.
I think not. Driver can choose to reset the device when the device is suspended.
Thanks
Yes, reset should take device out of suspend.
Quoted 7 lines
Quoted 7 lines
--
MST
Quoted 21 lines · Michael S. Tsirkin
Quoted 19 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 9 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines
And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.reset is not a state.
Well, I'm not sure I get this. Each value read from status should represent a device state.
What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.
Quoted 7 lines
Quoted 5 lines
For example, spec said
"""
2.1 Device Status Field
yes, this is confusingly written
Anything makes you think it is confusing?
Quoted 8 lines
Quoted 6 lines
During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.
The device status field provides a simple low-level indication of the completed steps of this sequence.
And note the sequence is driver sequence.
Right but it means the driver needs to know the status of the device in order to proceed.
Quoted 11 lines
Quoted 7 lines
It’s most useful to imagine it
hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):"""
Quoted 4 lines
Quoted 2 lines
Driver knows its own state, so there's no need for the driver to read it from the device.
That is why it never reads status.
Well, looking at the current Linux drivers there're plenty (in virtio core, there could be other in the transport)
drivers/virtio/virtio.c: return sysfs_emit(buf, "0x%08x\n",dev->config->get_status(dev));drivers/virtio/virtio.c: dev->config->set_status(dev,dev->config->get_status(dev) | status);
drivers/virtio/virtio.c: status = dev->config->get_status(dev); drivers/virtio/virtio.c: if (!(dev->config->get_status(dev) & VIRTIO_CONFIG_S_DRIVER_OK)) drivers/virtio/virtio.c: WARN_ON_ONCE(dev->config->get_status(dev)); drivers/virtio/virtio.c: dev->failed =
dev->config->get_status(dev) & VIRTIO_CONFIG_S_FAILED;drivers/virtio/virtio.c: if (!(dev->config->get_status(dev) &VIRTIO_CONFIG_S_DRIVER_OK))
Quoted 26 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
And 0 is a valid status which means device
is resetted:"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""Quoted 4 lines
Quoted 1 line
It means all registers have their reset values,
This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.
yes but otherwise, all registers respond normally device is assumed to be fully operational, and it is assumed that no time at all is needed to get from that to DRIVER.
Quoted 4 lines
Quoted 4 lines
but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Contrast with suspend where device does not respond and must be first resumed.
I think not. Driver can choose to reset the device when the device is suspended.
Thanks
Yes, reset should take device out of suspend.
Thanks
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
--
MST
Quoted 24 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Jason Wang
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines
And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.reset is not a state.
Quoted 7 lines
Quoted 2 lines
Well, I'm not sure I get this. Each value read from status should represent a device state.
What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.For example, spec said
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
"""
2.1 Device Status Field
yes, this is confusingly written
Anything makes you think it is confusing?
It says Device Status but most of it reflects the driver probe progress. Compare to pci RO status and RW control.
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.
The device status field provides a simple low-level indication of the completed steps of this sequence.
And note the sequence is driver sequence.
Right but it means the driver needs to know the status of the device in order to proceed.
again this is not device status at all.
Quoted 23 lines
Quoted 11 lines
Quoted 7 lines
It’s most useful to imagine it
hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):"""
Quoted 4 lines
Quoted 2 lines
Driver knows its own state, so there's no need for the driver to read it from the device.
That is why it never reads status.
Well, looking at the current Linux drivers there're plenty (in virtio core, there could be other in the transport)
drivers/virtio/virtio.c: return sysfs_emit(buf, "0x%08x\n",dev->config->get_status(dev));drivers/virtio/virtio.c: dev->config->set_status(dev,dev->config->get_status(dev)| status);
Quoted 8 lines
drivers/virtio/virtio.c: status = dev->config->get_status(dev); drivers/virtio/virtio.c: if (!(dev->config->get_status(dev) & VIRTIO_CONFIG_S_DRIVER_OK)) drivers/virtio/virtio.c: WARN_ON_ONCE(dev->config->get_status(dev)); drivers/virtio/virtio.c: dev->failed =
dev->config->get_status(dev)&VIRTIO_CONFIG_S_FAILED;drivers/virtio/virtio.c: if (!(dev->config->get_status(dev) &VIRTIO_CONFIG_S_DRIVER_OK))
It's done like this for robustness and simplicity. But we could easily cache it in the driver on write.
Quoted 46 lines
Quoted 26 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
And 0 is a valid status which means device
is resetted:"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""Quoted 4 lines
Quoted 1 line
It means all registers have their reset values,
This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.
yes but otherwise, all registers respond normally device is assumed to be fully operational, and it is assumed that no time at all is needed to get from that to DRIVER.
Quoted 4 lines
Quoted 4 lines
but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Contrast with suspend where device does not respond and must be first resumed.
I think not. Driver can choose to reset the device when the device is suspended.
Thanks
Yes, reset should take device out of suspend.
Thanks
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
--
MST
Quoted 30 lines · Michael S. Tsirkin
Quoted 28 lines · Jason Wang
Quoted 27 lines · Michael S. Tsirkin
Quoted 24 lines · Jason Wang
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines
And I still don't see why
introducing a single new bit in the status brings any new troubles compared with the existing reset and other state transitions.reset is not a state.
Quoted 7 lines
Quoted 2 lines
Well, I'm not sure I get this. Each value read from status should represent a device state.
What makes you say so?
status represents driver state, not device state.
In fact, driver writes there and does not double check with the sole exception of reset.For example, spec said
"""
2.1 Device Status Field
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
yes, this is confusingly written
Anything makes you think it is confusing?
It says Device Status but most of it reflects the driver probe progress. Compare to pci RO status and RW control.
My understanding is that
1) any value that is read from the device status reflects a device status.
2) any value write from the driver for the device status reflects a device status that driver want to go
Here are some examples:
1) DRIVER_OK:
"""
The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK.
"""
It's a device status that "Device knows driver is OK"
2) FEATURES_OK
"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the FEATURES_OK device status bit when the driver writes it. """
Device tells the driver that it doesn't support some of the features.
3) NEEDS_RESET:
"""
The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.
"""
This is a device status but not a driver.
And usually, device status is a bitmask of several bits of status defined in the spec.
Quoted 49 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
The device status field provides a simple low-level indication of the completed steps of this sequence.
And note the sequence is driver sequence.
Right but it means the driver needs to know the status of the device in order to proceed.
again this is not device status at all.
Quoted 23 lines
Quoted 11 lines
Quoted 7 lines
It’s most useful to imagine it
hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):"""
Quoted 4 lines
Quoted 2 lines
Driver knows its own state, so there's no need for the driver to read it from the device.
That is why it never reads status.
Well, looking at the current Linux drivers there're plenty (in virtio core, there could be other in the transport)
drivers/virtio/virtio.c: return sysfs_emit(buf, "0x%08x\n",dev->config->get_status(dev));drivers/virtio/virtio.c: dev->config->set_status(dev,dev->config->get_status(dev)| status);Quoted 8 lines
drivers/virtio/virtio.c: status = dev->config->get_status(dev); drivers/virtio/virtio.c: if (!(dev->config->get_status(dev) & VIRTIO_CONFIG_S_DRIVER_OK)) drivers/virtio/virtio.c: WARN_ON_ONCE(dev->config->get_status(dev)); drivers/virtio/virtio.c: dev->failed =
dev->config->get_status(dev)&VIRTIO_CONFIG_S_FAILED;drivers/virtio/virtio.c: if (!(dev->config->get_status(dev) &VIRTIO_CONFIG_S_DRIVER_OK))
Quoted 2 lines
It's done like this for robustness and simplicity. But we could easily cache it in the driver on write.
Probably not, one example:
virtio_add_status(dev, VIRTIO_CONFIG_S_FEATURES_OK);
status = dev->config->get_status(dev);
if (!(status & VIRTIO_CONFIG_S_FEATURES_OK)) {
dev_err(&dev->dev, "virtio: device refuses features: %x\n",
status);
return -ENODEV;
}Devices may refuse the features.
Thanks
Quoted 49 lines
Quoted 48 lines
Quoted 28 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
And 0 is a valid status which means device
is resetted:"""
The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.
"""Quoted 4 lines
Quoted 1 line
It means all registers have their reset values,
This is exactly the device status? Drivers need to know that there's a device state that is safe to start to program.
yes but otherwise, all registers respond normally device is assumed to be fully operational, and it is assumed that no time at all is needed to get from that to DRIVER.
Quoted 4 lines
Quoted 4 lines
but it is
not a special state in that device behaves normally in this state, it does not need to be handles in any special way.Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Contrast with suspend where device does not respond and must be first resumed.
I think not. Driver can choose to reset the device when the device is suspended.
Thanks
Yes, reset should take device out of suspend.
Thanks
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
--
MST
Quoted 259 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, September 3, 2024 2:36 PMQuoted 25 lines · Parav Pandit
Quoted 25 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:02 AMQuoted 20 lines · Parav Pandit
Quoted 20 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PMQuoted 5 lines · Parav Pandit
Quoted 3 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AMI removed the unreachable intel email id as every single email is bouncing
from it.
Quoted 2 lines
Please consider dropping that email from v8 as it will cause all reviewers
email to bounce.
Quoted 7 lines
Quoted 7 lines · Parav Pandit
Quoted 6 lines
Quoted 6 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 2 lines · Parav Pandit
Hi Lingshan, David,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDQuoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
which indicating whether the device support SUSPEND.
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75++++++++++++++++++++++++++++++++++++++++++++++-
Quoted 17 lines
Quoted 17 lines
----
Quoted 16 lines
Quoted 16 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume thedevice.
Quoted 97 lines
Quoted 97 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
- disallow access to config space while suspended.
content.tex index 0a62dce..2d1bee8 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be
loadablemodules.
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
- and it has given up on the device. This could be an internal
- error, or the driver didn't like the device for some reason, orcontent.tex +3
-1- even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and +ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver hasacknowledged all
Quoted 7 lines
Quoted 7 lines
the
Quoted 6 lines
Quoted 6 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] WhenVIRTIO_F_SUSPENDis negotiated,indicates
Quoted 23 lines
Quoted 23 lines
Quoted 4 lines
content.tex
Quoted 4 lines
content.tex +2
-0+that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 11 lines
Quoted 11 lines
an error from which it can't recover.
content.tex +8
-0+ +\item[FAILED (128)] Indicates that something went wrong in +the guest, + and it has given up on the device. This could be an +internal + error, or the driver didn't like the device for some +reason, or + even a fatal error during device operation. \end{description}Quoted 6 lines
Quoted 6 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
-The driver MUST NOT clear a
-\field{device status} bit. If the driver sets the FAILED bit,content.tex +4
-0+The driver MUST NOT clear a \field{device status} bit other +than SUSPEND except when setting \field{device status} to 0 +as a transport-specific way to initiate a reset. If the +driver sets the FAILED bit, the driver MUST later reset the device before attempting tore-
initialize.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The driver SHOULD NOT rely on completion of operations of a
@@
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities +of a Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
content.tex \section{Device
Quoted 2 lines
\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{DeviceCleanup}\label{sec:General
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Initialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
content.tex
Quoted 11 lines
+\section{Device Suspend}\label{sec:General Initialization
content.tex +10
-0+And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, +and can clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not setor
Quoted 18 lines
Quoted 18 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
content.tex
Quoted 3 lines
VIRTIO_F_SUSPENDis not negotiated.content.tex +2
-0+ +Once the driver sets SUSPEND to \field{device status} of thedevice:
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to +verify whether the
SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to +thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Do we agree that
a. suspending a device is non frequent operation (in order of Noperations/sec, where N is roughly in range of 10 or 100) per
device?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is
often.
Quoted 26 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
I think we don't have an intention of the frequency in the spec.
Sure.
Quoted 8 lines
Quoted 1 line
The spec only provides generic mechanisms and interfaces.
Sure.
Quoted 2 lines
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.
As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <50nsec of time.
Quoted 2 lines
The suspend operation for a net device to store hundreds of queues, RSS
table, flow filters, takes plenty of time (at least more than 50nsec :) ).
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Similarly for the GPU to store some MBs of memory takes more than 50nsec
of time, for example to store in a file for a software-based GPU device.
Quoted 1 line
So a device cannot respond back suspend=true in next 50nsec time.
It's OK for the device to take longer time to respond, the driver simply re-reads device status.
Sure, the issue is, when the driver re-reads, the device must present
suspend=false within 50nsec.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
(because device didn't suspend it yet).
As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly
differently.
why there is a constraint condition of the time?Because this is what driver does as you explained in [1] indicating "we must
be able to handle ..."
Quoted 3 lines
[1]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore
Quoted 1 line
.kernel.org%2Fvirtio-comment%2Fc4d5eed3-774b-4d35-a007-
f9dff28ce884%40
Quoted 1 line
amd.com%2FT%2F%23m6f081f96ef9dcea29c64a88b633eb21d50e8c410&dat a=05%7C0
Quoted 1 line
2%7Cparav%40nvidia.com%7C4a248d6982514660015808dccbf79c55%7C430 83d1572
Quoted 1 line
7340c1b7db39efd9ccc17a%7C0%7C0%7C638609511605195425%7CUnknown %7CTWFpbG
Quoted 1 line
Quoted 8 lines
Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 Mn0%
Quoted 1 line
3D%7C0%7C%7C%7C&sdata=Gj4oG8Z1O7wEl%2B7Mbud%2BbP31aMDMAFA H9ElHJSvBTUA%
Quoted 1 line
3D&reserved=0
The sentence is "we must be able to handle the scenario in which SUSPENDING is often", where do you see any constraint conditions of time?
If the device takes X usec, how can the device inform that, it suspending in progress without flipping the polarity?
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Are there any similar
constraining for other states like RESET orDRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.
DRIVER_OKdoes not suffer from it because it is async notification. A device may start slow afterDRIVER_OK.
SUSPEND operation cannot rely on such async behavior.The driver can set suspend and re-read & wait. This is a common routine in the driver.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.
Here a new functionality is being proposed and it has a chance for efficient
device implementation.
Quoted 1 line
Therefore the request is to improve it.
sure, as long as we confirm this new register apply for all device status transitions, not only for SUSPEND.
Alright, please do it as you suggested here.
Quoted 36 lines
Quoted 36 lines
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
More below.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the
software.
Quoted 1 line
Do you agree?
The device emulator does not know anything about whether trapped
or
Quoted 7 lines
Quoted 7 lines
not.
Quoted 6 lines
Quoted 6 lines
Trapping and Emulation is a hypervisor thing.
If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of
VM_EXIT&VM_ENTRY.Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Right. So a software wants to implement
device_statusas pure MMIOwrites. (and not
VM_EXIT).
This is not always true, there can beVM_EXITof pure emulated devices.I am not denying that
VM_EXITcan/cannot be there. I am saying, the proposal forcesVM_EXITbased approach in myunderstanding of this patch.
[
MARKER_1]Quoted 4 lines
Quoted 1 line
If that is not true, may be can you please explain how this can be
implemented without
VM_EXIT?Quoted 2 lines
We should have an interface that can be done with and without
VM_EXIT
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
method at least for any new additions.
VM_EXITis out of spec, it is a hypervisor and the processor thing.You keep repeating
VM_EXISTis out of spec.
I already replied at [MARKER_1], sure it is out of spec, the current approachforces one to do
VM_EXITbased approach.Quoted 1 line
And if not, please explain, how can it be achieved?
It is not the current solution force the hypervisor & guest to perform
VM_EXIT.
I read it few times but couldn't parse it.
Can you please explain how your proposal works WITHOUT VM_EXIT? If not, you can say, that current proposal is limited to force VM_EXIT. Please explain.
Quoted 2 lines
In basic virtualization, any access to hypervisor emulated HW registers would be Traped and Emulated, means a
VM_EXITand aVM_ENTRY.
That was not my question, my question is how to achieve without VM_EXIT? You are suggesting a spec that forces VM_EXIT.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
In non-pass-through case, any register access are sensitive and will trigger
VM_EXIT. Like RESET orDRIVER_OKneeds to accessdevice_status, nothing different.I don't think you understood the point.
Let me repeat, The question is, if the device implementation wants toachieve the functionality without
VM_EXIT, what is the way? Map the register to guest address space, out of the spec.
And which component will flip the bit of suspend before driver reads it to indicate that device is still_suspending?
Quoted 22 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
The
HW registers are sensitive resource and any access to them need to be trapped and emulated.This does not apply to PCI PFs and VFs which are HW devices (mainly
PFs).
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
so this trap + emulation is narrow view that we better avoid.
This is how *basic* virtualization work, once access sensitive resource,
trap it.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
If you think this is the way forward, you should put forward in patch as
MUST requirement.
Quoted 2 lines
and that does not look right to me.
I hope you also don't mean to force this method to deviceQuoted 8 lines
Quoted 6 lines
implementations.
Quoted 1 line
Right?
Again,
VM_EXITis a hypervisor thing, out of spec. Whether there is aVM_EXITwhen setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different fromDRIVER_OK.Please avoid repeating the point that
VM_EXITis hypervosor thing. No one asked to put this in spec. Please re-read [MARKER1]. You probably
Quoted 8 lines
missed that.
Quoted 2 lines
SUSPEND is different than
DRIVER_OK. I explained the timing constraintsand the required circuitry needed to fulfill the proposal.
Quoted 1 line
And with the additional register, such complicated circuitry can be easily
avoided.
Please read QEMU code, that can help you find how the trap - emulate mechanism work for guest when it tries to access sensitive resources.
I am not suggesting trap and emulation. Please come out of this looping thoughts.
I repeat my question, please explain how does suggested solution works without VM_EXIT? From your response, I derive it cannot. Can you please confirm?
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
Means, if your virtualization needs to trap SUSPEND, it also needs to trap
DRIVER_OK, and don't assumeDRIVER_OKis faster than SUSPEND.
DRIVER_OKis by law of physics is faster than SUSPEND because it does notdemand the driver of reading back.
Quoted 1 line
There is no driver side loop to check if the device accepted
DRIVER_OKornot.
Quoted 1 line
Agree?
Why do you assume so?
I am not assuming. It is the current spec. :)
Please read the device initialization sequence, that does NOT say that driver must READ DRIVER_OK. Hence by law of physics it is faster.
Quoted 3 lines
In some vendor implementation,
DRIVER_OKcan be
slow. CanDRIVER_OKimplement a deferred initialization even deferred resource allocation? Does the spec say thatDRIVER_OKmust be faster?
No. it does not. But your suspend proposal says so that device must respond back before the next read arrives from the driver.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
Quoted 3 lines
And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return
suspend=true right after it was written.
Quoted 4 lines
A MMIO read will read it back, as suspend=true.
An alternative would be, to forward CPU loads and CPU stores to different
address.
Quoted 4 lines
However, this does not work for the hw based devices.
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 2 lines
In this example, the device cannot build special circuitry to answer
Quoted 8 lines
Quoted 4 lines
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation. why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.
When driver wrote, it wrote suspend=true, And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsecor less to answer suspend=false.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
And even worst, it needs to suspend=true when unsuspending within
50nsec when resuming is ongoing.
again, there is not a 50nsec constraining and please take a reference of howDRIVER_OKwork with virtualization.There is. The device is expected to return back the desired value to indicate
driver that suspend is ongoing.
Quoted 1 line
It is different than
DRIVER_OK.where does the spec or any code say 50nsec?
Your spec and your previous comment implies that when device is busy suspending, it must returned the toggled value than what driver set it. Isn't it? It means device needs to react before the next read arrives from the driver, isn't it?
I took that read as 50nsec example as you explained previously that driver will read very fast.
Quoted 137 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful commentsQuoted 5 lines
Quoted 5 lines
Quoted 3 lines
Unfortunately, not. His comment was that it is not specific to suspend. But here we are introducing a new interface and functionality that does not
need to suffer or follow anything that may not be efficient.
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
Quoted 1 line
Quoted 1 line
%2F&data=05%7C02%7Cparav%40nvidia.com%7C4a248d6982514660015808 dccbf79
Quoted 1 line
Quoted 1 line
c55%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6386095116052 07133%7
Quoted 1 line
Quoted 1 line
CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB TiI6Ik
Quoted 1 line
Quoted 1 line
1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VGOfVe0deSDilzufY7NH gU5RmciNh
Quoted 4 lines
Quoted 4 lines
dXMMQiVUSiP%2BKM%3D&reserved=0
Quoted 1 line
Quoted 1 line
e.kernel.org%2Fvirtio-comment%2F20240612082055-mutt-send-email-
mst%40
kernel.org%2FT%2F%23mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9&dat
Quoted 2 lines
Quoted 2 lines
a=05
%7C02%7Cparav%40nvidia.com%7C619c82b60b824ca03f8808dcc89beb4a%7
Quoted 2 lines
Quoted 2 lines
C43083
d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638605819247985989%7CUn
Quoted 2 lines
Quoted 2 lines
known%7C
TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
Quoted 2 lines
Quoted 2 lines
CJXV
CI6Mn0%3D%7C0%7C%7C%7C&sdata=VqmJmqt3k5tf3x4ihjE5Bd7u59GadOn
Quoted 27 lines
Quoted 27 lines
OaOfJ5lvG
Quoted 26 lines
Quoted 26 lines
1DE%3D&reserved=0
Quoted 25 lines
Quoted 25 lines
this is out of the spec anyway.
Thanks
Zhu LingshanQuoted 12 lines
Quoted 12 lines
Thanks
Zhu LingshanQuoted 10 lines
Quoted 10 lines
+\devicenormative{\subsection}{Device Suspend}{General
content.tex +3
-0+Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status} if it is +part of the ConfigurationSpace.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
content.tex +8
-0+ +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in its +Configuration Space while suspended. + +If changes occur in the Configuration Space while the +SUSPEND bit is set, the device MUST NOT send any +configuration changenotifications.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
content.tex
Quoted 5 lines
content.tex +2
-0+Instead, the device MUST send the notification after the +SUSPEND bit hasbeen cleared.
content.tex +2
-0+ +When the driver sets SUSPEND, the device MUST eithersuspend
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
content.tex
Quoted 5 lines
+itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND, the device MUST either resume normaloperation
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
content.tex
Quoted 4 lines
+or set
DEVICE_NEEDS_RESET.content.tex +2
-0+ +When the driver sets SUSPEND, the device SHOULD performthe
Quoted 36 lines
Quoted 36 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
+following actions before presenting that
the SUSPEND bit is set to 1 in the \field{device status}:content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item +Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:VirtioTransport Options}
Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that
+ the drivercan
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
content.tex +3
-0+ trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device + StatusField}.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+
\end{description}\drivernormative{\section}{Reserved Feature Bits}{Reserved
Feature Bits}
--
2.45.2
Quoted 256 lines · Parav Pandit
Quoted 256 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, September 3, 2024 2:36 PMQuoted 24 lines · Parav Pandit
Quoted 24 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:02 AMQuoted 20 lines · Parav Pandit
Quoted 20 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 15, 2024 1:53 PMQuoted 5 lines · Parav Pandit
Quoted 3 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:44 AMI removed the unreachable intel email id as every single email is bouncing
from it.
Quoted 2 lines
Please consider dropping that email from v8 as it will cause all reviewers
email to bounce.
Quoted 7 lines
Quoted 7 lines · Parav Pandit
Quoted 6 lines
Quoted 6 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, August 13, 2024 11:15 AMQuoted 2 lines · Parav Pandit
Hi Lingshan, David,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
From: Zhu Lingshan <[email protected]>
Sent: Thursday, August 1, 2024 5:05 PMThis commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDQuoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
which indicating whether the device support SUSPEND.
Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>--- content.tex | 75++++++++++++++++++++++++++++++++++++++++++++++-
Quoted 17 lines
Quoted 17 lines
----
Quoted 16 lines
Quoted 16 lines
--
1 file changed, 65 insertions(+), 10 deletions(-)Changes from V6:
- the device should hold its config interrupt while SUSPEND, and send config interrupt when the SUSPEND bit is cleared.
- while SUSPEND, the driver MUST NOT access Device Configuration Space
- minor changes.Changes from V5:
- the device should presentNEEDS_RESETif failed to suspend
- allow the driver access device status in the config space when suspended if it is implemented in config space.
- language improvementsQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Changes from V4:
- re-order the device status bits section
- kick vqs --> notify vqsChanges from V3:
- allow the driver clearing the SUSPEND bit to resume thedevice.
Quoted 97 lines
Quoted 97 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
- disallow access to config space while suspended.
content.tex index 0a62dce..2d1bee8 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -36,19 +36,22 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
this bit. For example, under Linux, drivers can be
loadablemodules.
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
\end{note}-\item[FAILED (128)] Indicates that something went wrong in the guest,
- and it has given up on the device. This could be an internal
- error, or the driver didn't like the device for some reason, orcontent.tex +3
-1- even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and +ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver hasacknowledged all
Quoted 7 lines
Quoted 7 lines
the
Quoted 6 lines
Quoted 6 lines
features it understands, and feature negotiation is complete.
-\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
- drive the device.
+\item[SUSPEND (16)] WhenVIRTIO_F_SUSPENDis negotiated,indicates
Quoted 23 lines
Quoted 23 lines
Quoted 4 lines
content.tex
Quoted 4 lines
content.tex +2
-0+that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device hasexperienced
Quoted 11 lines
Quoted 11 lines
an error from which it can't recover.
content.tex +8
-0+ +\item[FAILED (128)] Indicates that something went wrong in +the guest, + and it has given up on the device. This could be an +internal + error, or the driver didn't like the device for some +reason, or + even a fatal error during device operation. \end{description}Quoted 6 lines
Quoted 6 lines
The \field{device status} field starts out as 0, and is
reinitialized to 0 by @@ -
60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / DeviceInitialization}.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
-The driver MUST NOT clear a
-\field{device status} bit. If the driver sets the FAILED bit,content.tex +4
-0+The driver MUST NOT clear a \field{device status} bit other +than SUSPEND except when setting \field{device status} to 0 +as a transport-specific way to initiate a reset. If the +driver sets the FAILED bit, the driver MUST later reset the device before attempting tore-
initialize.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The driver SHOULD NOT rely on completion of operations of a
@@
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
-99,10
+103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities +of a Virtio Device
/ Feature B \begin{description}
\item[0 to 23, and 50 to 127] Feature bits for the specific
device type-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the +queue and
feature negotiation mechanisms-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for +future
extensions.
\end{description}Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
content.tex \section{Device
Quoted 2 lines
\begin{note}
content.tex +0
-0@@ -629,6 +633,53 @@ \section{DeviceCleanup}\label{sec:General
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Initialization And Device Operation /
Thus a driver MUST ensure a virtqueue isn't live (by device
reset) before removing exposed buffers.Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
content.tex
Quoted 11 lines
+\section{Device Suspend}\label{sec:General Initialization
content.tex +10
-0+And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, +and can clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General +Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not setor
Quoted 18 lines
Quoted 18 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
content.tex
Quoted 3 lines
VIRTIO_F_SUSPENDis not negotiated.content.tex +2
-0+ +Once the driver sets SUSPEND to \field{device status} of thedevice:
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to +verify whether the
SUSPEND bit is set.
+\item The driver MUST NOT make any more buffers available to +thedevice.
Quoted 6 lines
Quoted 6 lines
+\item The driver MUST NOT access any virtqueues or send +notifications for
any virtqueues.
+\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
+Hi Parva
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Do we agree that
a. suspending a device is non frequent operation (in order of Noperations/sec, where N is roughly in range of 10 or 100) per
device?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Ideally it should not be often in normal operations, but remember we can not restrict the behaviors of the driver, so we must be able to handle the scenario in which SUSPENDING is
often.
Quoted 26 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Sure. the intent is slow rate, but one can do at unexpected times. Do you agree?
I think we don't have an intention of the frequency in the spec.
Sure.
Quoted 8 lines
Quoted 1 line
The spec only provides generic mechanisms and interfaces.
Sure.
Quoted 2 lines
Don't assume it(or driver wants it to be) would be often or not, that depends on the driver.
As you rightly said : it cannot be assumed.
The driver will read the device status right after it wrote it. This typically is <50nsec of time.
Quoted 2 lines
The suspend operation for a net device to store hundreds of queues, RSS
table, flow filters, takes plenty of time (at least more than 50nsec :) ).
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Similarly for the GPU to store some MBs of memory takes more than 50nsec
of time, for example to store in a file for a software-based GPU device.
Quoted 1 line
So a device cannot respond back suspend=true in next 50nsec time.
It's OK for the device to take longer time to respond, the driver simply re-reads device status.
Sure, the issue is, when the driver re-reads, the device must present
suspend=false within 50nsec.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
(because device didn't suspend it yet).
As I explained in previous email, this requires building special circuitry. Such circuitry can be avoided if the suspend interface is done slightly
differently.
why there is a constraint condition of the time?Because this is what driver does as you explained in [1] indicating "we must
be able to handle ..."
Quoted 2 lines
[1]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore
Quoted 1 line
.kernel.org%2Fvirtio-comment%2Fc4d5eed3-774b-4d35-a007-
f9dff28ce884%40 amd.com%2FT%2F%23m6f081f96ef9dcea29c64a88b633eb21d50e8c410&dat a=05%7C0 2%7Cparav%40nvidia.com%7C4a248d6982514660015808dccbf79c55%7C430 83d1572 7340c1b7db39efd9ccc17a%7C0%7C0%7C638609511605195425%7CUnknown %7CTWFpbG Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 Mn0% 3D%7C0%7C%7C%7C&sdata=Gj4oG8Z1O7wEl%2B7Mbud%2BbP31aMDMAFA H9ElHJSvBTUA%
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
3D&reserved=0
The sentence is "we must be able to handle the scenario in which SUSPENDING is often", where do you see any constraint conditions of time?
If the device takes X usec, how can the device inform that, it suspending in progress without flipping the polarity?
Oh, I see your puzzle. The device does not inform that. It is VM_EXIT and VM_ENTRY, once VM_ENTRY, the guest vcpu aka the driver resume running
Quoted 58 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Are there any similar
constraining for other states like RESET orDRIVER_OK? Don't assume any other states transitions are faster than SUSPEND.
DRIVER_OKdoes not suffer from it because it is async notification. A device may start slow afterDRIVER_OK.
SUSPEND operation cannot rely on such async behavior.The driver can set suspend and re-read & wait. This is a common routine in the driver.
Quoted 8 lines
Quoted 4 lines
RESET also suffers from similar inefficiencies. But that is because it is inherited from the past.
Here a new functionality is being proposed and it has a chance for efficient
device implementation.
Quoted 1 line
Therefore the request is to improve it.
sure, as long as we confirm this new register apply for all device status transitions, not only for SUSPEND.
Alright, please do it as you suggested here.
Quoted 39 lines
Quoted 39 lines
Quoted 22 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
More below.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?
Trap and Emulation is the basic of virtualization, and how to pass-through a device is out of this spec.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Sure, I didn't suggest to put such things in the spec. My question is, whether to trap and emulate or not is a choice of the
software.
Quoted 1 line
Do you agree?
The device emulator does not know anything about whether trapped
or
Quoted 7 lines
Quoted 7 lines
not.
Quoted 6 lines
Quoted 6 lines
Trapping and Emulation is a hypervisor thing.
If here "software" refers to the device emulator, then Yes, it is not the emulator's decision. And the device should not be aware of
VM_EXIT&VM_ENTRY.Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Right. So a software wants to implement
device_statusas pure MMIOwrites. (and not
VM_EXIT).
This is not always true, there can beVM_EXITof pure emulated devices.I am not denying that
VM_EXITcan/cannot be there. I am saying, the proposal forcesVM_EXITbased approach in myunderstanding of this patch.
[
MARKER_1]Quoted 6 lines
Quoted 1 line
If that is not true, may be can you please explain how this can be
implemented without
VM_EXIT?Quoted 2 lines
We should have an interface that can be done with and without
VM_EXITmethod at least for any new additions.
VM_EXITis out of spec, it is a hypervisor and the processor thing.You keep repeating
VM_EXISTis out of spec.
I already replied at [MARKER_1], sure it is out of spec, the current approach
Quoted 8 lines
Quoted 4 lines
forces one to do
VM_EXITbased approach.Quoted 1 line
And if not, please explain, how can it be achieved?
It is not the current solution force the hypervisor & guest to perform
VM_EXIT.I read it few times but couldn't parse it.
Can you please explain how your proposal works WITHOUTVM_EXIT? If not, you can say, that current proposal is limited to forceVM_EXIT. Please explain.
It needs VM_EXIT, not only SUSPEND, all sensitive resource, like the config space access should be trapped.
Quoted 5 lines
Quoted 3 lines
In basic virtualization, any access to hypervisor emulated HW registers would be Traped and Emulated, means a
VM_EXITand aVM_ENTRY.That was not my question, my question is how to achieve without
VM_EXIT? You are suggesting a spec that forcesVM_EXIT.
It needs VM_EXIT, and there is a pairing VM_ENTRY
Quoted 228 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
In non-pass-through case, any register access are sensitive and will trigger
VM_EXIT. Like RESET orDRIVER_OKneeds to accessdevice_status, nothing different.I don't think you understood the point.
Let me repeat, The question is, if the device implementation wants toachieve the functionality without
VM_EXIT, what is the way? Map the register to guest address space, out of the spec.And which component will flip the bit of suspend before driver reads it to indicate that device is
still_suspending?Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
The
HW registers are sensitive resource and any access to them need to be trapped and emulated.This does not apply to PCI PFs and VFs which are HW devices (mainly
PFs).
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
so this trap + emulation is narrow view that we better avoid.
This is how *basic* virtualization work, once access sensitive resource,
trap it.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
If you think this is the way forward, you should put forward in patch as
MUST requirement.
Quoted 2 lines
and that does not look right to me.
I hope you also don't mean to force this method to deviceQuoted 8 lines
Quoted 6 lines
implementations.
Quoted 1 line
Right?
Again,
VM_EXITis a hypervisor thing, out of spec. Whether there is aVM_EXITwhen setting SUSPEND totally depends on the virtualization solution. And SUSPEND is nothing different fromDRIVER_OK.Please avoid repeating the point that
VM_EXITis hypervosor thing. No one asked to put this in spec. Please re-read [MARKER1]. You probablyQuoted 7 lines
missed that.
Quoted 1 line
SUSPEND is different than
DRIVER_OK. I explained the timing constraintsand the required circuitry needed to fulfill the proposal.
Quoted 1 line
And with the additional register, such complicated circuitry can be easily
avoided.
Please read QEMU code, that can help you find how the trap - emulate mechanism work for guest when it tries to access sensitive resources.I am not suggesting trap and emulation. Please come out of this looping thoughts.
I repeat my question, please explain how does suggested solution works without
VM_EXIT? From your response, I derive it cannot. Can you please confirm?Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
Means, if your virtualization needs to trap SUSPEND, it also needs to trap
DRIVER_OK, and don't assumeDRIVER_OKis faster than SUSPEND.
DRIVER_OKis by law of physics is faster than SUSPEND because it does notdemand the driver of reading back.
Quoted 1 line
There is no driver side loop to check if the device accepted
DRIVER_OKornot.
Quoted 1 line
Agree?
Why do you assume so?
I am not assuming. It is the current spec. :)
Please read the device initialization sequence, that does NOT say that driver must READDRIVER_OK. Hence by law of physics it is faster.Quoted 3 lines
In some vendor implementation,
DRIVER_OKcan be
slow. CanDRIVER_OKimplement a deferred initialization even deferred resource allocation? Does the spec say thatDRIVER_OKmust be faster?No. it does not. But your suspend proposal says so that device must respond back before the next read arrives from the driver.
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
Quoted 2 lines
And prefer to returning SUSPEND=true at slow pace. This means, the device implementation cannot immediately return
suspend=true right after it was written.
Quoted 4 lines
A MMIO read will read it back, as suspend=true.
An alternative would be, to forward CPU loads and CPU stores to different
address.
Quoted 4 lines
However, this does not work for the hw based devices.
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 2 lines
In this example, the device cannot build special circuitry to answer
Quoted 8 lines
Quoted 4 lines
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation. why? The device can just not to change the value of the SUSPEND bit before it has fully suspended.
When driver wrote, it wrote suspend=true, And device returns suspend=false while suspend is ongoing, right?
If yes, this is expensive because the device needs to operate within 50nsecor less to answer suspend=false.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
And even worst, it needs to suspend=true when unsuspending within
50nsec when resuming is ongoing.
again, there is not a 50nsec constraining and please take a reference of howDRIVER_OKwork with virtualization.There is. The device is expected to return back the desired value to indicate
driver that suspend is ongoing.
Quoted 1 line
It is different than
DRIVER_OK.where does the spec or any code say 50nsec?
Your spec and your previous comment implies that when device is busy suspending, it must returned the toggled value than what driver set it. Isn't it? It means device needs to react before the next read arrives from the driver, isn't it?
I took that read as 50nsec example as you explained previously that driver will read very fast.Quoted 132 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I think we have already discussed on this before in V5, and Jason has some insightful commentsQuoted 5 lines
Quoted 5 lines
Quoted 3 lines
Unfortunately, not. His comment was that it is not specific to suspend. But here we are introducing a new interface and functionality that does not
need to suffer or follow anything that may not be efficient.
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
%2F&data=05%7C02%7Cparav%40nvidia.com%7C4a248d6982514660015808 dccbf79 c55%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6386095116052 07133%7 CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB TiI6Ik 1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VGOfVe0deSDilzufY7NH gU5RmciNhQuoted 4 lines
Quoted 4 lines
dXMMQiVUSiP%2BKM%3D&reserved=0
Quoted 1 line
Quoted 1 line
e.kernel.org%2Fvirtio-comment%2F20240612082055-mutt-send-email-
mst%40
kernel.org%2FT%2F%23mc817fc6ca12ff0bcbae62b43b6146a177ecf13a9&dat
Quoted 2 lines
Quoted 2 lines
a=05
%7C02%7Cparav%40nvidia.com%7C619c82b60b824ca03f8808dcc89beb4a%7
Quoted 2 lines
Quoted 2 lines
C43083
d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638605819247985989%7CUn
Quoted 2 lines
Quoted 2 lines
known%7C
TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
Quoted 2 lines
Quoted 2 lines
CJXV
CI6Mn0%3D%7C0%7C%7C%7C&sdata=VqmJmqt3k5tf3x4ihjE5Bd7u59GadOn
Quoted 27 lines
Quoted 27 lines
OaOfJ5lvG
Quoted 26 lines
Quoted 26 lines
1DE%3D&reserved=0
Quoted 25 lines
Quoted 25 lines
this is out of the spec anyway.
Thanks
Zhu LingshanQuoted 12 lines
Quoted 12 lines
Thanks
Zhu LingshanQuoted 10 lines
Quoted 10 lines
+\devicenormative{\subsection}{Device Suspend}{General
content.tex +3
-0+Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or
VIRTIO_F_SUSPENDis not negotiated.content.tex +4
-0+ +The device MUST ignore all access to its Configuration Space +while suspended, except for \field{device status} if it is +part of the ConfigurationSpace.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex
Quoted 8 lines
content.tex +8
-0+ +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in its +Configuration Space while suspended. + +If changes occur in the Configuration Space while the +SUSPEND bit is set, the device MUST NOT send any +configuration changenotifications.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
content.tex
Quoted 5 lines
content.tex +2
-0+Instead, the device MUST send the notification after the +SUSPEND bit hasbeen cleared.
content.tex +2
-0+ +When the driver sets SUSPEND, the device MUST eithersuspend
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
content.tex
Quoted 5 lines
+itself or set
DEVICE_NEEDS_RESETif failed to suspend.content.tex +3
-0+ +If SUSPEND is set in \field{device status}, when the driver +clears SUSPEND, the device MUST either resume normaloperation
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
content.tex
Quoted 4 lines
+or set
DEVICE_NEEDS_RESET.content.tex +2
-0+ +When the driver sets SUSPEND, the device SHOULD performthe
Quoted 36 lines
Quoted 36 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
+following actions before presenting that
the SUSPEND bit is set to 1 in the \field{device status}:content.tex +7
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues \item +Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:VirtioTransport Options}
Virtio can use various different buses, thus the standard is
split @@ -872,6
+923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved +Feature
Bits}
\ref{devicenormative:Basic Facilities of a Virtio Device /
Feature Bits} for
handling features reserved for future use.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+ \item[VIRTIO_F_SUSPEND(42)] This feature indicates that
+ the drivercan
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
content.tex +3
-0+ trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device + StatusField}.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+
\end{description}\drivernormative{\section}{Reserved Feature Bits}{Reserved
Feature Bits}
--
2.45.2
Quoted 2 lines · Zhu Lingshan
Quoted 1 line
So a device cannot respond back suspend=true in next 50nsec time.
It's OK for the device to take longer time to respond, the driver simply re-reads device status.
You didn't bother saying in the spec that it must, though.
Signature · 1 line
MST
Quoted 4 lines · Michael S. Tsirkin
Quoted 2 lines · Zhu Lingshan
Quoted 1 line
So a device cannot respond back suspend=true in next 50nsec time.
It's OK for the device to take longer time to respond, the driver simply re-reads device status.
You didn't bother saying in the spec that it must, though.
I can add this re-read like how we handle FEATURE_OK
Quoted 1 line
Quoted 7 lines · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not suspended.
In this example, the device cannot build special circuitry to answer suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
Quoted 8 lines
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This is the default value.
3. When the device completes suspend, it changes the polarity of suspend_status=true.This has two main benefits:
[A] This will enable software-based devices to write data to slow files and does not have to forceVM_EXITs.[B] It also enables hw based devices to not build special circuitry to answer within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Signature · 1 line
MST
Quoted 24 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 1 line · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 1 line
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files and
Quoted 9 lines
does not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation.
And device_status new bit that Lingshan defined returns the status of this operation.
Quoted 27 lines · Parav Pandit
Quoted 27 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 1 line · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 1 line
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 8 lines
Quoted 6 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation.
Anddevice_statusnew bit that Lingshan defined returns the status of this operation.
Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progress
What I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)
How does this sound?
Signature · 1 line
MST
Quoted 65 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 8:38 PMQuoted 32 lines · Parav Pandit
Quoted 32 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 2 lines · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 2 lines
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
Quoted 8 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives
not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 4 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 3 lines
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 8 lines
Quoted 6 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation.
Anddevice_statusnew bit that Lingshan defined returns the status of thisoperation.
Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progressWhat I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)
Quoted 2 lines
How does this sound?
Yes.
Just that having command in different register than the status is better approach to me. As it allows steering command to different piece of hw from the status.
1. separate status register can be optimized by the device as its largely serving the reads.
2. modular
3. simplicity and clarity to developers
cmd_register:
0 = resume
1 = suspend
status:
0 = resumed.
1 = suspended
cmd status
0 0 operational (resumed), typical default.
1 0 suspend in progress (initiated)
1 1 suspended
0 1 resume initiated (still suspended)
Quoted 2 lines
--
MST
Quoted 49 lines · Michael S. Tsirkin
Quoted 23 lines · Parav Pandit
Quoted 23 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 1 line · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 1 line
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
Quoted 6 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 2 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 1 line
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 8 lines
Quoted 6 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation.
Anddevice_statusnew bit that Lingshan defined returns the status of this operation.Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progressWhat I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)
Quoted 1 line
How does this sound?
This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device.
When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.
But if you think we should add a new register which applying for all device_status transitions, NOT only for SUSPEND. we can surely do that.
Thanks
Quoted 1 line
Quoted 68 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AMQuoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Parav Pandit
Quoted 28 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 2 lines · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 2 lines
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation.
Anddevice_statusnew bit that Lingshan defined returns the status of thisoperation.
Quoted 17 lines
Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progressWhat I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)Quoted 1 line
How does this sound?
This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.
Quoted 5 lines
But if you think we should add a new register which applying for all
device_statustransitions, NOT only for SUSPEND. we can surely do that.Thanks
Quoted 1 line
New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.
Quoted 65 lines · Parav Pandit
Quoted 65 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AMQuoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Parav Pandit
Quoted 27 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 2 lines · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 2 lines
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation.
Anddevice_statusnew bit that Lingshan defined returns the status of thisoperation.
Quoted 16 lines
Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progressWhat I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)Quoted 1 line
How does this sound?
This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.
Quoted 6 lines
Quoted 4 lines
But if you think we should add a new register which applying for all
device_statustransitions, NOT only for SUSPEND. we can surely do that.Thanks
New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.
There should be no difference in how the driver handles SUSPEND and other device status like RESET.
If we want to add a new register, then it is not only for SUSPEND, but for all status transitions.
We need Michael to confirm we should implement this new register that apply to all device_status transitions for common interests.
Thanks
Quoted 1 line
Quoted 74 lines
From: Zhu Lingshan <[email protected]>
Sent: Tuesday, September 3, 2024 2:21 PMQuoted 10 lines · Parav Pandit
Quoted 10 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AMQuoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Parav Pandit
Quoted 3 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Parav Pandit
Quoted 2 lines
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 2 lines
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow
operation.
Quoted 48 lines
Quoted 48 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation. And
device_statusnew bit that Lingshan defined returns the status of thisoperation.
Quoted 16 lines
Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progressWhat I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)Quoted 1 line
How does this sound?
This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
But if you think we should add a new register which applying for all
device_statustransitions, NOT only for SUSPEND. we can surely do that.Thanks
New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.
There should be no difference in how the driver handles SUSPEND and other device status like RESET.
You still miss the fundamental point that I explained many times that is to avoid special time sensitive circuitry in the device by means of a new register. (not the driver).
Quoted 3 lines
If we want to add a new register, then it is not only for SUSPEND, but for all status transitions. We need Michael to confirm we should implement this new register that apply to all
device_statustransitions for common interests.
Sure. Look forward to his opinion.
Quoted 77 lines · Zhu Lingshan
Quoted 65 lines · Parav Pandit
Quoted 65 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AMQuoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Parav Pandit
Quoted 27 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 2 lines · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 2 lines
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation.
Anddevice_statusnew bit that Lingshan defined returns the status of thisoperation.
Quoted 16 lines
Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progressWhat I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)Quoted 1 line
How does this sound?
This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.Quoted 6 lines
Quoted 4 lines
But if you think we should add a new register which applying for all
device_statustransitions, NOT only for SUSPEND. we can surely do that.Thanks
New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.
There should be no difference in how the driver handles SUSPEND and other device status like RESET.
Quoted 6 lines
If we want to add a new register, then it is not only for SUSPEND, but for all status transitions.
We need Michael to confirm we should implement this new register that apply to alldevice_statustransitions for common interests.Thanks
There is a difference between SUSPEND and RESET. RESET is not a state. Thus a single bit is enough to signal "reset in progress".
I don't really see any other transitions that can take a long time. We can start with just suspend, and extend it later if appropriate.
Quoted 1 line
Quoted 1 line
Quoted 80 lines · Michael S. Tsirkin
Quoted 79 lines · Zhu Lingshan
Quoted 63 lines · Parav Pandit
Quoted 63 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AMQuoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Parav Pandit
Quoted 27 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 2 lines · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 2 lines
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation.
Anddevice_statusnew bit that Lingshan defined returns the status of thisoperation.
Quoted 15 lines
Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progressWhat I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)Quoted 1 line
How does this sound?
This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.Quoted 6 lines
Quoted 4 lines
But if you think we should add a new register which applying for all
device_statustransitions, NOT only for SUSPEND. we can surely do that.Thanks
New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.
There should be no difference in how the driver handles SUSPEND and other device status like RESET.
If we want to add a new register, then it is not only for SUSPEND, but for all status transitions.
We need Michael to confirm we should implement this new register that apply to alldevice_statustransitions for common interests.
Quoted 8 lines
Quoted 1 line
Thanks
There is a difference between SUSPEND and RESET. RESET is not a state. Thus a single bit is enough to signal "reset in progress".
I don't really see any other transitions that can take a long time. We can start with just suspend, and extend it later if appropriate.
Yes, that is what I mean, the register should not only work for SUSPEND. To be more specific, the definition should be:
// The device_status is still in a status transition #define DEVICE_STATUS_TRANSITION_IN_PROGRESS 0 // device status transition is done
#define DEVICE_STATUS_TRANSITION_DONE 1
They should not be defined as:
#define DEVICE_STATUS_SUSPEND_IN_PROGRESS 0
Thanks
Quoted 3 lines
Quoted 99 lines · Zhu Lingshan
Quoted 80 lines · Michael S. Tsirkin
Quoted 79 lines · Zhu Lingshan
Quoted 63 lines · Parav Pandit
Quoted 63 lines
From: Zhu Lingshan <[email protected]>
Sent: Friday, August 30, 2024 8:07 AMQuoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Parav Pandit
Quoted 27 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, August 15, 2024 4:23 PMQuoted 2 lines · Parav Pandit
That means, PCI HW needs to return suspend=0, until the device is not
suspended.
Quoted 2 lines
In this example, the device cannot build special circuitry to answer
suspend=true within 50nsec, or in other words building special circuitry to return suspend=false is too complex for the slow operation.
Quoted 7 lines
If this understanding of burden is clear,
The proposal is, can you please extend the interface such that,
1. driver writes suspend command.
2. driver readssuspend_status, and receives not_completed=(false). This isthe default value.
Quoted 1 line
3. When the device completes suspend, it changes the polarity of
suspend_status=true.
Quoted 3 lines
This has two main benefits:
[A] This will enable software-based devices to write data to slow files anddoes not have to force
VM_EXITs.Quoted 2 lines
[B] It also enables hw based devices to not build special circuitry to answer
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
within 50nsec, which can get very complicated for tens or hundreds of PCI PFs.
I read this several times, and I don't understand what is proposed. A special register for suspend/resume? Is this the difference?
Yes, a command register for suspend/resume operation.
Anddevice_statusnew bit that Lingshan defined returns the status of thisoperation.
Quoted 15 lines
Ugh, it's all quite messy IMHO.
We have 4 states:
- operational (resumed)
- suspend in progress
- suspended
- resume in progressWhat I'd do then is a two bit register.
To suspend:
- write suspend in progress
- re-read, waiting until suspended
To resume
- write resume in progress
- re-read, waiting until operational (resumed)Quoted 1 line
How does this sound?
This can work for sure. but is it a must?
I mean, the driver has its own knowledge of how it operate the device. When device presents SUSPEND == 0, It know whether the device is in normal operational state or in the progress of SUSPENDING.Quoted 6 lines
Quoted 4 lines
But if you think we should add a new register which applying for all
device_statustransitions, NOT only for SUSPEND. we can surely do that.Thanks
New register beyond suspend+resume can be useful too. For sure it will simplify the suspend + resume flow.
There should be no difference in how the driver handles SUSPEND and other device status like RESET.
If we want to add a new register, then it is not only for SUSPEND, but for all status transitions.
We need Michael to confirm we should implement this new register that apply to alldevice_statustransitions for common interests.Quoted 8 lines
Quoted 1 line
Thanks
There is a difference between SUSPEND and RESET. RESET is not a state. Thus a single bit is enough to signal "reset in progress".
I don't really see any other transitions that can take a long time. We can start with just suspend, and extend it later if appropriate.
Yes, that is what I mean, the register should not only work for SUSPEND. To be more specific, the definition should be:
// The
device_statusis still in a status transition #defineDEVICE_STATUS_TRANSITION_IN_PROGRESS0 // device status transition is done
#defineDEVICE_STATUS_TRANSITION_DONE1
Quoted 5 lines
They should not be defined as:
#defineDEVICE_STATUS_SUSPEND_IN_PROGRESS0Thanks
in case we add more commands down the line? ok, sure.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Parav Pandit
a. suspending a device is non frequent operation (in order of N operations/sec, where N is roughly in range of 10 or 100) per device? b. A software-based device may not always want to force
VM_EXITon read and write on thedevice_statusregister?
I just see where this is going. Parav, if you want a device with as little as possible memory on it, using DMA as much as possible, what you want is develop an alternative transport
that will let you control status without using memory accesses. This will also, with time, allow doing this for existing config space. We aren't blocking all spec development until you do, and the tradeoff of easy to debug, robust MMIO control versus complex but maybe cheaper for some hardware vendors DMA based access is up to the device.
Signature · 1 line
MST
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 13, 2024 1:22 PM
Removing intel email id again to avoid constant bouncing messages.
Quoted 10 lines · Parav Pandit
Quoted 1 line
a. suspending a device is non frequent operation (in order of N
operations/sec, where N is roughly in range of 10 or 100) per device?
Quoted 1 line
b. A software-based device may not always want to force
VM_EXITon readand write on the
device_statusregister?I just see where this is going. Parav, if you want a device with as little as possible memory on it, using DMA as much as possible, what you want is develop an alternative transport that will let you control status without using memory accesses.
No, there is no need of DMA here.
This is not about less memory either.
As I explained, the challenge is implementing suspend bits to operate at sub-microsecond level granularity.
Quoted 5 lines
This will also, with time, allow doing this for existing config space. We aren't blocking all spec development until you do, and the tradeoff of easy to debug, robust MMIO control versus complex but maybe cheaper for some hardware vendors DMA based access is up to the device.
No one is blocking the spec etc. Not sure why you say that.
The idea is to have suspend as command register, and its status to reflect in the existing device_status.
Last time you recommended to only make design suggestion and not go down the exact details of the bit and let author to fix it.
If you prefer to discuss the bit level details, it looks good to me to go to that details.
Quoted 2 lines
--
MST
Quoted 6 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 13, 2024 1:22 PMRemoving intel email id again to avoid constant bouncing messages.
Thanks.
Quoted 4 lines
Quoted 4 lines · Parav Pandit
Quoted 1 line
a. suspending a device is non frequent operation (in order of N
operations/sec, where N is roughly in range of 10 or 100) per device?
Quoted 1 line
b. A software-based device may not always want to force
VM_EXITon read
Quoted 8 lines
Quoted 6 lines
and write on the
device_statusregister?I just see where this is going. Parav, if you want a device with as little as possible memory on it, using DMA as much as possible, what you want is develop an alternative transport that will let you control status without using memory accesses.
No, there is no need of DMA here.
This is not about less memory either.
OK, sorry. I misunderstood.
Quoted 1 line
As I explained, the challenge is implementing suspend bits to operate at sub-microsecond level granularity.
Yes, I think the proposal is too vague on how all this happens. Replied separately.
Quoted 11 lines
Quoted 5 lines
This will also, with time, allow doing this for existing config space. We aren't blocking all spec development until you do, and the tradeoff of easy to debug, robust MMIO control versus complex but maybe cheaper for some hardware vendors DMA based access is up to the device.
No one is blocking the spec etc. Not sure why you say that.
The idea is to have suspend as command register, and its status to reflect in the existingdevice_status.Last time you recommended to only make design suggestion and not go down the exact details of the bit and let author to fix it.
If you prefer to discuss the bit level details, it looks good to me to go to that details.
Sorry, reacted too early.
Quoted 2 lines
Quoted 2 lines
--
MST
Review
Quoted 3 lines · Zhu Lingshan
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
(quoted diff) +2
-0+ +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.
Actually, it has no effect before DRIVER_OK, no? I would forbid that then.
Quoted 3 lines
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.
This is still vague, I commented on this several times. I think what you mean is that until it reads status with SUSPEND as 1, it does not consider SUSPEND set.
But, what happens if driver clears SUSPEND before it reads it as set? It would seem that should cancel suspend (which is a useful thing to support)? But this creates a problem as it breaks read/modify/write that some hypervisors assumed to be safe. I guess we need an extra SUSPEND_IN_PROGRESS bit then? A little too much for status, at this stage - maybe we need an extra register for this.
Quoted 19 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
content.tex +13
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the Configuration Space. + +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in +its Configuration Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND bit is set,
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 10 lines
content.tex +10
-0+the device MUST NOT send any configuration change notifications. +Instead, the device MUST send the notification after the SUSPEND bit has been cleared. + +When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND, +the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}:
what does "before presenting" mean? does it return SUSPEND as 0 after driver wrote 1 there and before it completed these actions?
Quoted 23 lines
content.tex +11
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
2.45.2
Quoted 6 lines · Michael S. Tsirkin
Review
Quoted 3 lines · Zhu Lingshan
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
(quoted diff) +2
-0+ +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.Actually, it has no effect before
DRIVER_OK, no? I would forbid that then.
There can be two cases:
1) debugging the device before DRIVER_OK
2) VM migration right after boot, and before DRIVER_OK. So it needs to migrate the device before DRIVER_OK
Quoted 10 lines
Quoted 4 lines
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.This is still vague, I commented on this several times. I think what you mean is that until it reads status with SUSPEND as 1, it does not consider SUSPEND set.
Quoted 6 lines
But, what happens if driver clears SUSPEND before it reads it as set? It would seem that should cancel suspend (which is a useful thing to support)? But this creates a problem as it breaks read/modify/write that some hypervisors assumed to be safe. I guess we need an extra
SUSPEND_IN_PROGRESSbit then? A little too much for status, at this stage - maybe we need an extra register for this.
if the driver reads SUSPEND bit == 0, with the knowledge of its own operations, it knows the device is either in a) normal operation 2). suspending in progress
Cancelling a SUSPEND right after setting SUSPEND is actually a RESUME. The ideal case is the driver waits until SUSPEND == 1, then clear it.
But there are various drivers, not all drivers follow our expectation, so there can be a corner case that how the driver know whether the device is resumed after clearing the SUSPEND bit or not finish suspending.
I think adding a new resister can surely fix this issue, however as you said, add a new status bit is overkill and where to place this new register is a new question.
So, to make things easier, how about suspend & resume the device sequentially:
If the driver has suspended the device by setting the SUSPEND bit to 1, it MUST NOT clear the SUSPEND bit before the device presenting the SUSPEND bit set as 1.
Quoted 24 lines
Quoted 24 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
content.tex +18
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the Configuration Space. + +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in +its Configuration Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND bit is set, +the device MUST NOT send any configuration change notifications. +Instead, the device MUST send the notification after the SUSPEND bit has been cleared. + +When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend. +
Quoted 8 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 5 lines
content.tex +5
-0+If SUSPEND is set in \field{device status}, when the driver clears SUSPEND, +the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}:what does "before presenting" mean? does it return SUSPEND as 0 after driver wrote 1 there and before it completed these actions?
Yes, it should return 0 if the suspending operation is still in progress,
It should not change SUSPEND bit value before finish.
Do you suggest we add:
The device MUST present SUSPEND bit set to 1 in \field{device status} once it has been suspended
It says: presenting that the SUSPEND bit is set to 1. Not a native speaker, but this looks clear to me.
Quoted 24 lines
Quoted 24 lines
content.tex +11
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
2.45.2
Quoted 4 lines · Zhu Lingshan
Quoted 1 line · Michael S. Tsirkin
On Thu, Aug 01, 2024 at 07:35:16PM +0800, Zhu Lingshan wrote:
Quoted 8 lines
Quoted 5 lines
Review
Quoted 3 lines
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
(quoted diff) +2
-0+ +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.Actually, it has no effect before
DRIVER_OK, no? I would forbid that then.There can be two cases:
1) debugging the device beforeDRIVER_OK
2) VM migration right after boot, and beforeDRIVER_OK. So it needs to migrate the device beforeDRIVER_OK
Sorry I don't understand. The only effect of SUSPEND as you specified it is not consuming buffers. Buffers are not consumed before DRIVER_OK, thus there is no point to set SUSPEND before DRIVER_OK.
Quoted 25 lines
Quoted 10 lines
Quoted 4 lines
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.This is still vague, I commented on this several times. I think what you mean is that until it reads status with SUSPEND as 1, it does not consider SUSPEND set.
Quoted 6 lines
But, what happens if driver clears SUSPEND before it reads it as set? It would seem that should cancel suspend (which is a useful thing to support)? But this creates a problem as it breaks read/modify/write that some hypervisors assumed to be safe. I guess we need an extra
SUSPEND_IN_PROGRESSbit then? A little too much for status, at this stage - maybe we need an extra register for this.if the driver reads SUSPEND bit == 0, with the knowledge of its own operations, it knows the device is either in a) normal operation 2). suspending in progress
Cancelling a SUSPEND right after setting SUSPEND is actually a RESUME. The ideal case is the driver waits until SUSPEND == 1, then clear it. But there are various drivers, not all drivers follow our expectation, so there can be a corner case that how the driver know whether the device is resumed after clearing the SUSPEND bit or not finish suspending.
Quoted 3 lines
I think adding a new resister can surely fix this issue, however as you said, add a new status bit is overkill and where to place this new register is a new question.
Place it after existing registers.
Quoted 5 lines
So, to make things easier, how about suspend & resume the device sequentially:
If the driver has suspended the device by setting the SUSPEND bit to 1, it MUST NOT clear the SUSPEND bit before the device presenting the SUSPEND bit set as 1.
I'm worried that it's fragile - one can no longer deduce the device state by reading the status.
Quoted 32 lines
Quoted 24 lines
Quoted 24 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
content.tex +18
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the Configuration Space. + +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in +its Configuration Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND bit is set, +the device MUST NOT send any configuration change notifications. +Instead, the device MUST send the notification after the SUSPEND bit has been cleared. + +When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend. +Quoted 8 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 5 lines
content.tex +5
-0+If SUSPEND is set in \field{device status}, when the driver clears SUSPEND, +the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}:what does "before presenting" mean? does it return SUSPEND as 0 after driver wrote 1 there and before it completed these actions?
Quoted 7 lines
Yes, it should return 0 if the suspending operation is still in progress,
It should not change SUSPEND bit value before finish.
Do you suggest we add:
The device MUST present SUSPEND bit set to 1 in \field{device status} once it has been suspendedIt says: presenting that the SUSPEND bit is set to 1. Not a native speaker, but this looks clear to me.
I don't think this is sufficient, you need to document all assumptions on both device and driver.
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
content.tex +11
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
2.45.2
Quoted 9 lines · Michael S. Tsirkin
Quoted 8 lines · Zhu Lingshan
Quoted 6 lines · Michael S. Tsirkin
Review
Quoted 3 lines · Zhu Lingshan
+\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend}
(quoted diff) +2
-0+ +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.Actually, it has no effect before
DRIVER_OK, no? I would forbid that then.
Quoted 7 lines
Quoted 3 lines
There can be two cases:
1) debugging the device beforeDRIVER_OK
2) VM migration right after boot, and beforeDRIVER_OK. So it needs to migrate the device beforeDRIVER_OKSorry I don't understand. The only effect of SUSPEND as you specified it is not consuming buffers. Buffers are not consumed before
DRIVER_OK, thus there is no point to set SUSPEND beforeDRIVER_OK.
Sorry for the late reply, I am very busy these two weeks.
There can be other operations except buffers before DRIVER_OK, like config interrupts. And the hypervisor may migrate VMs right after creating them, at that moment DRIVER_OK may not be set yet, we should not block this use case.
Quoted 24 lines
Quoted 24 lines
Quoted 9 lines
Quoted 4 lines
+Once the driver sets SUSPEND to \field{device status} of the device:
+\begin{itemize}
+\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.This is still vague, I commented on this several times. I think what you mean is that until it reads status with SUSPEND as 1, it does not consider SUSPEND set.
Quoted 6 lines
But, what happens if driver clears SUSPEND before it reads it as set? It would seem that should cancel suspend (which is a useful thing to support)? But this creates a problem as it breaks read/modify/write that some hypervisors assumed to be safe. I guess we need an extra
SUSPEND_IN_PROGRESSbit then? A little too much for status, at this stage - maybe we need an extra register for this.if the driver reads SUSPEND bit == 0, with the knowledge of its own operations, it knows the device is either in a) normal operation 2). suspending in progress
Cancelling a SUSPEND right after setting SUSPEND is actually a RESUME. The ideal case is the driver waits until SUSPEND == 1, then clear it. But there are various drivers, not all drivers follow our expectation, so there can be a corner case that how the driver know whether the device is resumed after clearing the SUSPEND bit or not finish suspending.
Quoted 4 lines
Quoted 3 lines
I think adding a new resister can surely fix this issue, however as you said, add a new status bit is overkill and where to place this new register is a new question.
Place it after existing registers.
I am still confused, SUSPEND and RESUME are both device state transition, just like RESET, FEATURES_OK and DRIVER_OK. I see current virtio works fine for these existing status without an extra register. And we can not assume SUSPEND / RESUME takes longer time than any other states.
So, do you want a new register for all state transitions? Apply to all device states, not only SUSPEND.
Quoted 9 lines
Quoted 6 lines
So, to make things easier, how about suspend & resume the device sequentially:
If the driver has suspended the device by setting the SUSPEND bit to 1, it MUST NOT clear the SUSPEND bit before the device presenting the SUSPEND bit set as 1.
I'm worried that it's fragile - one can no longer deduce the device state by reading the status.
It is the driver reads / sets device status, it has the knowledge of the device status I think. For example, if it sets SUSPEND and reads SUSPEND == 0, means SUSPEND in progress.
so I even fell this is actually not necessary .
Quoted 37 lines
Quoted 37 lines
Quoted 24 lines
Quoted 24 lines
+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space. +\end{itemize}
content.tex +18
-0+ +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the Configuration Space. + +A device MUST NOT send any notifications for any virtqeuues, +access any virtqueues, or modify any fields in +its Configuration Space while suspended. + +If changes occur in the Configuration Space while the SUSPEND bit is set, +the device MUST NOT send any configuration change notifications. +Instead, the device MUST send the notification after the SUSPEND bit has been cleared. + +When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend. +Quoted 8 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 675
Quoted 5 lines
content.tex +5
-0+If SUSPEND is set in \field{device status}, when the driver clears SUSPEND, +the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting that the SUSPEND bit is set to 1 in the \field{device status}:what does "before presenting" mean? does it return SUSPEND as 0 after driver wrote 1 there and before it completed these actions?
Yes, it should return 0 if the suspending operation is still in progress,
It should not change SUSPEND bit value before finish.
Do you suggest we add:
The device MUST present SUSPEND bit set to 1 in \field{device status} once it has been suspended
Quoted 4 lines
Quoted 2 lines
It says: presenting that the SUSPEND bit is set to 1. Not a native speaker, but this looks clear to me.
I don't think this is sufficient, you need to document all assumptions on both device and driver.
Would you please help me understanding what assumptions should be addressed here? IMHO, the driver knows what it is doing I think.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
content.tex +11
-0+ +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split@@ -872,6 +923,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
2.45.2