virtio: introduce SUSPEND bit in device status
PATCH RESEND v6 1/1 virtio: introduce SUSPEND bit in device status
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in device_status.
This commit also introduce a new feature bit VIRTIO_F_SUSPEND which indicating whether the device support SUSPEND.
--- content.tex | 71 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 61 insertions(+), 10 deletions(-)
content.tex +61 -10
@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by
@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a
@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}
@@ -629,6 +633,49 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers. +\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set. +\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or send notifications for any virtqueues. +\item The driver MUST NOT access Device Configuration Space except for the field \field {device status}, +if it is implemented in the Config Space. +\end{itemize} + +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the Configuration Space. + +A device MUST NOT send any notifications, access any virtqueues, or modify +any fields in its Configuration Space while suspended. + +When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND, +the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting SUSPEND bit in the \field{device status}: + +\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize} + \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split
@@ -872,6 +919,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.45.2
Quoted 10 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>
Your signature would normally be last.
Quoted 3 lines
--- content.tex | 71 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 61 insertions(+), 10 deletions(-)
At a high level, I think this is somewhat overspecified. Generally e.g. if driver is forbidden from accessing a field, then we do not also add specific requirements for devices - with no driver doing this, such functionality will remain untested and unused, and when we finally decide we need it it's not going to be there.
Similarly for when device is not touching a field - no reason for driver not to touch it.
Quoted 65 lines
content.tex +17
-10@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -629,6 +633,49 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers. +\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 648
Quoted 10 lines
content.tex +10
-0+SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.
If it's not set, then what? Device error, have to reset?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 650
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or send notifications for any virtqueues.
what are these "fields"? the area in memory where the vq lives? why is doing that a problem - you want device to be able to write something there?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 652
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT access Device Configuration Space except for the field \field {device status}, +if it is implemented in the Config Space.
what is "the Config Space"? device status is never in
a device configuration space, it is part of the common configuration. so what exactly are you forbidding here?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 657
Quoted 5 lines
content.tex +5
-0+\end{itemize} + +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.
what does must ignore SUSPEND mean? what do you want the device to do and when?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 2 lines
content.tex +2
-0+The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the Configuration Space.
again what does ignore mean? Configuration Space here means Device Configuration Space?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 663
Quoted 3 lines
content.tex +3
-0+ +A device MUST NOT send any notifications, access any virtqueues, or modify +any fields in its Configuration Space while suspended.
If a tree falls in a forest, and no one is around to hear it, does it still make a sound?
I think what matters is that if a configuration space changes while SUSPEND is set, then the device must not send a configuration change notification, and it must send a configuration change notification after SUSPEND bit has been cleared.
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 671
Quoted 7 lines
content.tex +7
-0+When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND, +the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting SUSPEND bit in the \field{device status}:
what does "before presenting" mean here? that a driver reading status will see 0 in this bit?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 677
Quoted 5 lines
content.tex +5
-0+\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}
Is there anything here that has not been said before, but as a MUST? Saying same thing twice once a MUST once a SHOULD is confusing IMHO.
Quoted 17 lines
content.tex +5
-0+ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split@@ -872,6 +919,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
2.45.2
Quoted 116 lines · Michael S. Tsirkin
Quoted 10 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>Your signature would normally be last.
Quoted 3 lines
--- content.tex | 71 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 61 insertions(+), 10 deletions(-)At a high level, I think this is somewhat overspecified. Generally e.g. if driver is forbidden from accessing a field, then we do not also add specific requirements for devices - with no driver doing this, such functionality will remain untested and unused, and when we finally decide we need it it's not going to be there.
Similarly for when device is not touching a field - no reason for driver not to touch it.Quoted 65 lines
content.tex +17
-10@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -629,6 +633,49 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers. +\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set thecontent.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 648
Quoted 10 lines
content.tex +10
-0+SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.If it's not set, then what? Device error, have to reset?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 650
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or send notifications for any virtqueues.what are these "fields"? the area in memory where the vq lives? why is doing that a problem - you want device to be able to write something there?
Quoted 6 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 652
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT access Device Configuration Space except for the field \field {device status}, +if it is implemented in the Config Space.what is "the Config Space"? device status is never in
a device configuration space, it is part of the common configuration. so what exactly are you forbidding here?
IIUC, referring to the common configuration here wouldn't make sense. The common configuration is part of the PCI transport specification, so in this part of the specification it should only be referred to via the "transport specific" phrasing. This wording appears to be taken from Cornelia's feedback on v5. Specifically:
Quoted 2 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 653
Quoted 1 line
content.tex +1
-0+\item The driver MUST NOT access Device Configuration Space....except for the status field, if it is part of the config space?
I asked for clarification [1], because Cornelia's feedback seems to contradict your feedback here and on the (unfortunately unarchived) v3 patch. What is the definitive statement agreed upon by both editors of the virtio specification as to whether or not the device status field is part of the device configuration space?
[1] https://lore.kernel.org/all/CAD=HUj4RBeLS4L=ehyPGtejeu+sGZ5j8PRtrK8AvxjEgEBd5ZA@mail.gmail.com/
-David
Quoted 120 lines · David Stevens
Quoted 119 lines · Michael S. Tsirkin
Quoted 10 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>Your signature would normally be last.
Quoted 3 lines
--- content.tex | 71 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 61 insertions(+), 10 deletions(-)At a high level, I think this is somewhat overspecified. Generally e.g. if driver is forbidden from accessing a field, then we do not also add specific requirements for devices - with no driver doing this, such functionality will remain untested and unused, and when we finally decide we need it it's not going to be there.
Similarly for when device is not touching a field - no reason for driver not to touch it.Quoted 65 lines
content.tex +17
-10@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -629,6 +633,49 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers. +\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set thecontent.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 648
Quoted 10 lines
content.tex +10
-0+SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.If it's not set, then what? Device error, have to reset?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 650
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or send notifications for any virtqueues.what are these "fields"? the area in memory where the vq lives? why is doing that a problem - you want device to be able to write something there?
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT access Device Configuration Space except for the field \field {device status}, +if it is implemented in the Config Space.
Quoted 8 lines
Quoted 3 lines
what is "the Config Space"? device status is never in
a device configuration space, it is part of the common configuration. so what exactly are you forbidding here?IIUC, referring to the common configuration here wouldn't make sense. The common configuration is part of the PCI transport specification, so in this part of the specification it should only be referred to via the "transport specific" phrasing.
Good point. We can easily fix this for MMIO though.
MMIO virtio devices provide a set of memory mapped control
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.
->
MMIO virtio devices provide memory mapped control
including a set of common configuration
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.
Less sure about CCW.
Maybe:
In addition to the basic channel commands, virtio-ccw defines a set of channel commands related to configuration and operation of virtio:
\begin{lstlisting}
#define CCW_CMD_SET_VQ 0x13
#define CCW_CMD_VDEV_RESET 0x33
#define CCW_CMD_SET_IND 0x43
#define CCW_CMD_SET_CONF_IND 0x53
#define CCW_CMD_SET_IND_ADAPTER 0x73
#define CCW_CMD_READ_FEAT 0x12
#define CCW_CMD_WRITE_FEAT 0x11
#define CCW_CMD_READ_CONF 0x22
#define CCW_CMD_WRITE_CONF 0x21
#define CCW_CMD_WRITE_STATUS 0x31
#define CCW_CMD_READ_VQ_CONF 0x32
#define CCW_CMD_SET_VIRTIO_REV 0x83
#define CCW_CMD_READ_STATUS 0x72
\end{lstlisting}
->
In addition to the basic channel commands, virtio-ccw defines channel commands related to configuration and operation of virtio - a set of commands to access common configuration of the device:
\begin{lstlisting}
#define CCW_CMD_SET_VQ 0x13
#define CCW_CMD_VDEV_RESET 0x33
#define CCW_CMD_SET_IND 0x43
#define CCW_CMD_SET_CONF_IND 0x53
#define CCW_CMD_SET_IND_ADAPTER 0x73
#define CCW_CMD_READ_FEAT 0x12
#define CCW_CMD_WRITE_FEAT 0x11
#define CCW_CMD_WRITE_STATUS 0x31
#define CCW_CMD_READ_VQ_CONF 0x32
#define CCW_CMD_SET_VIRTIO_REV 0x83
#define CCW_CMD_READ_STATUS 0x72
\end{lstlisting}
and additionally, two commands to access the device specification configuration space:
\begin{lstlisting}
#define CCW_CMD_READ_CONF 0x22
#define CCW_CMD_WRITE_CONF 0x21
\end{lstlisting}
And now we have common configuration defined for all transports. Cornelia, WDYT?
Quoted 12 lines
This wording appears to be taken
from Cornelia's feedback on v5. Specifically:Quoted 2 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 653
Quoted 1 line
content.tex +1
-0+\item The driver MUST NOT access Device Configuration Space....except for the status field, if it is part of the config space?
I asked for clarification [1], because Cornelia's feedback seems to contradict your feedback here and on the (unfortunately unarchived) v3 patch. What is the definitive statement agreed upon by both editors of the virtio specification as to whether or not the device status field is part of the device configuration space?
Quoted 3 lines
[1] https://lore.kernel.org/all/CAD=HUj4RBeLS4L=ehyPGtejeu+sGZ5j8PRtrK8AvxjEgEBd5ZA@mail.gmail.com/
-David
I don't know what did Cornelia mean. Cornelia?
Signature · 1 line
MST
Quoted 129 lines · Michael S. Tsirkin
Quoted 114 lines · David Stevens
Quoted 113 lines · Michael S. Tsirkin
Quoted 10 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>Your signature would normally be last.
Quoted 3 lines
--- content.tex | 71 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 61 insertions(+), 10 deletions(-)At a high level, I think this is somewhat overspecified. Generally e.g. if driver is forbidden from accessing a field, then we do not also add specific requirements for devices - with no driver doing this, such functionality will remain untested and unused, and when we finally decide we need it it's not going to be there.
Similarly for when device is not touching a field - no reason for driver not to touch it.Quoted 65 lines
content.tex +17
-10@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -629,6 +633,49 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers. +\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set thecontent.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 648
Quoted 10 lines
content.tex +10
-0+SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.If it's not set, then what? Device error, have to reset?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 650
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or send notifications for any virtqueues.what are these "fields"? the area in memory where the vq lives? why is doing that a problem - you want device to be able to write something there?
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT access Device Configuration Space except for the field \field {device status}, +if it is implemented in the Config Space.Quoted 7 lines
Quoted 3 lines
what is "the Config Space"? device status is never in
a device configuration space, it is part of the common configuration. so what exactly are you forbidding here?IIUC, referring to the common configuration here wouldn't make sense. The common configuration is part of the PCI transport specification, so in this part of the specification it should only be referred to via the "transport specific" phrasing.
Good point. We can easily fix this for MMIO though.
MMIO virtio devices provide a set of memory mapped control
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.
Quoted 6 lines
->
MMIO virtio devices provide memory mapped control
including a set of common configuration
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.
IMHO this may be a bit vague, PCI as a section about common configuration and a struct "struct virtio_pci_common_cfg " listed all contents. But what does CCW/MMIO common configuration contains? Shall we add new sections for them?
Quoted 77 lines
Less sure about CCW.
Maybe:In addition to the basic channel commands, virtio-ccw defines a set of channel commands related to configuration and operation of virtio:
\begin{lstlisting}
#defineCCW_CMD_SET_VQ0x13
#defineCCW_CMD_VDEV_RESET0x33
#defineCCW_CMD_SET_IND0x43
#defineCCW_CMD_SET_CONF_IND0x53
#defineCCW_CMD_SET_IND_ADAPTER0x73
#defineCCW_CMD_READ_FEAT0x12
#defineCCW_CMD_WRITE_FEAT0x11
#defineCCW_CMD_READ_CONF0x22
#defineCCW_CMD_WRITE_CONF0x21
#defineCCW_CMD_WRITE_STATUS0x31
#defineCCW_CMD_READ_VQ_CONF0x32
#defineCCW_CMD_SET_VIRTIO_REV0x83
#defineCCW_CMD_READ_STATUS0x72
\end{lstlisting}->
In addition to the basic channel commands, virtio-ccw defines channel commands related to configuration and operation of virtio - a set of commands to access common configuration of the device:
\begin{lstlisting}
#defineCCW_CMD_SET_VQ0x13
#defineCCW_CMD_VDEV_RESET0x33
#defineCCW_CMD_SET_IND0x43
#defineCCW_CMD_SET_CONF_IND0x53
#defineCCW_CMD_SET_IND_ADAPTER0x73
#defineCCW_CMD_READ_FEAT0x12
#defineCCW_CMD_WRITE_FEAT0x11
#defineCCW_CMD_WRITE_STATUS0x31
#defineCCW_CMD_READ_VQ_CONF0x32
#defineCCW_CMD_SET_VIRTIO_REV0x83
#defineCCW_CMD_READ_STATUS0x72
\end{lstlisting}and additionally, two commands to access the device specification configuration space:
\begin{lstlisting}
#defineCCW_CMD_READ_CONF0x22
#defineCCW_CMD_WRITE_CONF0x21
\end{lstlisting}And now we have common configuration defined for all transports. Cornelia, WDYT?
Quoted 11 lines
This wording appears to be taken
from Cornelia's feedback on v5. Specifically:Quoted 2 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 653
Quoted 1 line
content.tex +1
-0+\item The driver MUST NOT access Device Configuration Space....except for the status field, if it is part of the config space?
I asked for clarification [1], because Cornelia's feedback seems to contradict your feedback here and on the (unfortunately unarchived) v3 patch. What is the definitive statement agreed upon by both editors of the virtio specification as to whether or not the device status field is part of the device configuration space?
Quoted 3 lines
[1] https://lore.kernel.org/all/CAD=HUj4RBeLS4L=ehyPGtejeu+sGZ5j8PRtrK8AvxjEgEBd5ZA@mail.gmail.com/
-David
I don't know what did Cornelia mean. Cornelia?
Quoted 131 lines · Zhu Lingshan
Quoted 129 lines · Michael S. Tsirkin
Quoted 114 lines · David Stevens
Quoted 113 lines · Michael S. Tsirkin
Quoted 10 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>Your signature would normally be last.
Quoted 3 lines
--- content.tex | 71 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 61 insertions(+), 10 deletions(-)At a high level, I think this is somewhat overspecified. Generally e.g. if driver is forbidden from accessing a field, then we do not also add specific requirements for devices - with no driver doing this, such functionality will remain untested and unused, and when we finally decide we need it it's not going to be there.
Similarly for when device is not touching a field - no reason for driver not to touch it.Quoted 65 lines
content.tex +17
-10@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -629,6 +633,49 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers. +\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set thecontent.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 648
Quoted 10 lines
content.tex +10
-0+SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.If it's not set, then what? Device error, have to reset?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 650
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or send notifications for any virtqueues.what are these "fields"? the area in memory where the vq lives? why is doing that a problem - you want device to be able to write something there?
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT access Device Configuration Space except for the field \field {device status}, +if it is implemented in the Config Space.Quoted 7 lines
Quoted 3 lines
what is "the Config Space"? device status is never in
a device configuration space, it is part of the common configuration. so what exactly are you forbidding here?IIUC, referring to the common configuration here wouldn't make sense. The common configuration is part of the PCI transport specification, so in this part of the specification it should only be referred to via the "transport specific" phrasing.
Good point. We can easily fix this for MMIO though.
MMIO virtio devices provide a set of memory mapped control
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.
Quoted 8 lines
Quoted 6 lines
->
MMIO virtio devices provide memory mapped control
including a set of common configuration
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.IMHO this may be a bit vague, PCI as a section about common configuration and a struct "struct
virtio_pci_common_cfg" listed all contents. But what does CCW/MMIO common configuration contains? Shall we add new sections for them?
I think it's clear: everything that is not device-specific is common, and we have a table documenting that.
And again CCW does not have registers, it has commands. And below I suggested listing the common configuration commands.
Quoted 77 lines
Quoted 77 lines
Less sure about CCW.
Maybe:In addition to the basic channel commands, virtio-ccw defines a set of channel commands related to configuration and operation of virtio:
\begin{lstlisting}
#defineCCW_CMD_SET_VQ0x13
#defineCCW_CMD_VDEV_RESET0x33
#defineCCW_CMD_SET_IND0x43
#defineCCW_CMD_SET_CONF_IND0x53
#defineCCW_CMD_SET_IND_ADAPTER0x73
#defineCCW_CMD_READ_FEAT0x12
#defineCCW_CMD_WRITE_FEAT0x11
#defineCCW_CMD_READ_CONF0x22
#defineCCW_CMD_WRITE_CONF0x21
#defineCCW_CMD_WRITE_STATUS0x31
#defineCCW_CMD_READ_VQ_CONF0x32
#defineCCW_CMD_SET_VIRTIO_REV0x83
#defineCCW_CMD_READ_STATUS0x72
\end{lstlisting}->
In addition to the basic channel commands, virtio-ccw defines channel commands related to configuration and operation of virtio - a set of commands to access common configuration of the device:
\begin{lstlisting}
#defineCCW_CMD_SET_VQ0x13
#defineCCW_CMD_VDEV_RESET0x33
#defineCCW_CMD_SET_IND0x43
#defineCCW_CMD_SET_CONF_IND0x53
#defineCCW_CMD_SET_IND_ADAPTER0x73
#defineCCW_CMD_READ_FEAT0x12
#defineCCW_CMD_WRITE_FEAT0x11
#defineCCW_CMD_WRITE_STATUS0x31
#defineCCW_CMD_READ_VQ_CONF0x32
#defineCCW_CMD_SET_VIRTIO_REV0x83
#defineCCW_CMD_READ_STATUS0x72
\end{lstlisting}and additionally, two commands to access the device specification configuration space:
\begin{lstlisting}
#defineCCW_CMD_READ_CONF0x22
#defineCCW_CMD_WRITE_CONF0x21
\end{lstlisting}And now we have common configuration defined for all transports. Cornelia, WDYT?
Quoted 11 lines
This wording appears to be taken
from Cornelia's feedback on v5. Specifically:Quoted 2 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 653
Quoted 1 line
content.tex +1
-0+\item The driver MUST NOT access Device Configuration Space....except for the status field, if it is part of the config space?
I asked for clarification [1], because Cornelia's feedback seems to contradict your feedback here and on the (unfortunately unarchived) v3 patch. What is the definitive statement agreed upon by both editors of the virtio specification as to whether or not the device status field is part of the device configuration space?
Quoted 3 lines
[1] https://lore.kernel.org/all/CAD=HUj4RBeLS4L=ehyPGtejeu+sGZ5j8PRtrK8AvxjEgEBd5ZA@mail.gmail.com/
-David
I don't know what did Cornelia mean. Cornelia?
Quoted 138 lines · Michael S. Tsirkin
Quoted 137 lines · Zhu Lingshan
Quoted 135 lines · Michael S. Tsirkin
Quoted 114 lines · David Stevens
Quoted 113 lines · Michael S. Tsirkin
Quoted 10 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>Your signature would normally be last.
Quoted 3 lines
--- content.tex | 71 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 61 insertions(+), 10 deletions(-)At a high level, I think this is somewhat overspecified. Generally e.g. if driver is forbidden from accessing a field, then we do not also add specific requirements for devices - with no driver doing this, such functionality will remain untested and unused, and when we finally decide we need it it's not going to be there.
Similarly for when device is not touching a field - no reason for driver not to touch it.Quoted 65 lines
content.tex +17
-10@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -629,6 +633,49 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers. +\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set thecontent.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 648
Quoted 10 lines
content.tex +10
-0+SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.If it's not set, then what? Device error, have to reset?
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 650
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or send notifications for any virtqueues.what are these "fields"? the area in memory where the vq lives? why is doing that a problem - you want device to be able to write something there?
Quoted 2 lines
content.tex +2
-0+\item The driver MUST NOT access Device Configuration Space except for the field \field {device status}, +if it is implemented in the Config Space.Quoted 7 lines
Quoted 3 lines
what is "the Config Space"? device status is never in
a device configuration space, it is part of the common configuration. so what exactly are you forbidding here?IIUC, referring to the common configuration here wouldn't make sense. The common configuration is part of the PCI transport specification, so in this part of the specification it should only be referred to via the "transport specific" phrasing.
Good point. We can easily fix this for MMIO though.
MMIO virtio devices provide a set of memory mapped control
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.->
MMIO virtio devices provide memory mapped control
including a set of common configuration
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.
Quoted 7 lines
Quoted 2 lines
IMHO this may be a bit vague, PCI as a section about common configuration and a struct "struct
virtio_pci_common_cfg" listed all contents. But what does CCW/MMIO common configuration contains? Shall we add new sections for them?I think it's clear: everything that is not device-specific is common, and we have a table documenting that.
And again CCW does not have registers, it has commands. And below I suggested listing the common configuration commands.
OK, so would you like to add this editorial change or shall I work on it?
Therefore, in V7 I will rephrase fallback to V5: "The driver MUST NOT access Device Configuration Space" Any comments? @Michael @Cornelia Thanks Zhu Lingshan
Quoted 77 lines
Quoted 77 lines
Quoted 77 lines
Less sure about CCW.
Maybe:In addition to the basic channel commands, virtio-ccw defines a set of channel commands related to configuration and operation of virtio:
\begin{lstlisting}
#defineCCW_CMD_SET_VQ0x13
#defineCCW_CMD_VDEV_RESET0x33
#defineCCW_CMD_SET_IND0x43
#defineCCW_CMD_SET_CONF_IND0x53
#defineCCW_CMD_SET_IND_ADAPTER0x73
#defineCCW_CMD_READ_FEAT0x12
#defineCCW_CMD_WRITE_FEAT0x11
#defineCCW_CMD_READ_CONF0x22
#defineCCW_CMD_WRITE_CONF0x21
#defineCCW_CMD_WRITE_STATUS0x31
#defineCCW_CMD_READ_VQ_CONF0x32
#defineCCW_CMD_SET_VIRTIO_REV0x83
#defineCCW_CMD_READ_STATUS0x72
\end{lstlisting}->
In addition to the basic channel commands, virtio-ccw defines channel commands related to configuration and operation of virtio - a set of commands to access common configuration of the device:
\begin{lstlisting}
#defineCCW_CMD_SET_VQ0x13
#defineCCW_CMD_VDEV_RESET0x33
#defineCCW_CMD_SET_IND0x43
#defineCCW_CMD_SET_CONF_IND0x53
#defineCCW_CMD_SET_IND_ADAPTER0x73
#defineCCW_CMD_READ_FEAT0x12
#defineCCW_CMD_WRITE_FEAT0x11
#defineCCW_CMD_WRITE_STATUS0x31
#defineCCW_CMD_READ_VQ_CONF0x32
#defineCCW_CMD_SET_VIRTIO_REV0x83
#defineCCW_CMD_READ_STATUS0x72
\end{lstlisting}and additionally, two commands to access the device specification configuration space:
\begin{lstlisting}
#defineCCW_CMD_READ_CONF0x22
#defineCCW_CMD_WRITE_CONF0x21
\end{lstlisting}And now we have common configuration defined for all transports. Cornelia, WDYT?
Quoted 11 lines
This wording appears to be taken
from Cornelia's feedback on v5. Specifically:Quoted 2 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 653
Quoted 1 line
content.tex +1
-0+\item The driver MUST NOT access Device Configuration Space....except for the status field, if it is part of the config space?
I asked for clarification [1], because Cornelia's feedback seems to contradict your feedback here and on the (unfortunately unarchived) v3 patch. What is the definitive statement agreed upon by both editors of the virtio specification as to whether or not the device status field is part of the device configuration space?
Quoted 3 lines
[1] https://lore.kernel.org/all/CAD=HUj4RBeLS4L=ehyPGtejeu+sGZ5j8PRtrK8AvxjEgEBd5ZA@mail.gmail.com/
-David
I don't know what did Cornelia mean. Cornelia?
Quoted 55 lines · Michael S. Tsirkin
Quoted 6 lines · David Stevens
Quoted 5 lines · Michael S. Tsirkin
Quoted 2 lines · Zhu Lingshan
+\item The driver MUST NOT access Device Configuration Space except for the field \field {device status}, +if it is implemented in the Config Space.
Quoted 8 lines
Quoted 3 lines
what is "the Config Space"? device status is never in
a device configuration space, it is part of the common configuration. so what exactly are you forbidding here?IIUC, referring to the common configuration here wouldn't make sense. The common configuration is part of the PCI transport specification, so in this part of the specification it should only be referred to via the "transport specific" phrasing.
Good point. We can easily fix this for MMIO though.
MMIO virtio devices provide a set of memory mapped control
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.->
MMIO virtio devices provide memory mapped control
including a set of common configuration
registers followed by a device-specific configuration space,
described in the table~\ref{tab:Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}.Less sure about CCW.
Maybe:In addition to the basic channel commands, virtio-ccw defines a set of channel commands related to configuration and operation of virtio:
\begin{lstlisting}
#defineCCW_CMD_SET_VQ0x13
#defineCCW_CMD_VDEV_RESET0x33
#defineCCW_CMD_SET_IND0x43
#defineCCW_CMD_SET_CONF_IND0x53
#defineCCW_CMD_SET_IND_ADAPTER0x73
#defineCCW_CMD_READ_FEAT0x12
#defineCCW_CMD_WRITE_FEAT0x11
#defineCCW_CMD_READ_CONF0x22
#defineCCW_CMD_WRITE_CONF0x21
#defineCCW_CMD_WRITE_STATUS0x31
#defineCCW_CMD_READ_VQ_CONF0x32
#defineCCW_CMD_SET_VIRTIO_REV0x83
#defineCCW_CMD_READ_STATUS0x72
\end{lstlisting}
Quoted 7 lines
->
In addition to the basic channel commands, virtio-ccw defines channel commands related to configuration and operation of virtio - a set of commands to access common configuration of the device:
Hm, I'm not sure whether "common configuration" is the best wording here; resetting a device, for example, is not something I'd call configuration. If we don't manage to come up with anything better, I can live with it, though.
Quoted 40 lines
\begin{lstlisting}
#defineCCW_CMD_SET_VQ0x13
#defineCCW_CMD_VDEV_RESET0x33
#defineCCW_CMD_SET_IND0x43
#defineCCW_CMD_SET_CONF_IND0x53
#defineCCW_CMD_SET_IND_ADAPTER0x73
#defineCCW_CMD_READ_FEAT0x12
#defineCCW_CMD_WRITE_FEAT0x11
#defineCCW_CMD_WRITE_STATUS0x31
#defineCCW_CMD_READ_VQ_CONF0x32
#defineCCW_CMD_SET_VIRTIO_REV0x83
#defineCCW_CMD_READ_STATUS0x72
\end{lstlisting}and additionally, two commands to access the device specification configuration space:
\begin{lstlisting}
#defineCCW_CMD_READ_CONF0x22
#defineCCW_CMD_WRITE_CONF0x21
\end{lstlisting}And now we have common configuration defined for all transports. Cornelia, WDYT?
Quoted 12 lines
This wording appears to be taken
from Cornelia's feedback on v5. Specifically:Quoted 2 lines
Quoted 1 line
+\item The driver MUST NOT access Device Configuration Space.
...except for the status field, if it is part of the config space?
I asked for clarification [1], because Cornelia's feedback seems to contradict your feedback here and on the (unfortunately unarchived) v3 patch. What is the definitive statement agreed upon by both editors of the virtio specification as to whether or not the device status field is part of the device configuration space?
Quoted 5 lines
Quoted 3 lines
[1] https://lore.kernel.org/all/CAD=HUj4RBeLS4L=ehyPGtejeu+sGZ5j8PRtrK8AvxjEgEBd5ZA@mail.gmail.com/
-David
I don't know what did Cornelia mean. Cornelia?
What I meant (I think...) was that accessing the status field is fine, even if it is implemented as part of the config space.
Quoted 89 lines · Michael S. Tsirkin
Quoted 10 lines · Zhu Lingshan
This commit allows the driver to suspend the device by introducing a new status bit SUSPEND in
device_status.This commit also introduce a new feature bit
VIRTIO_F_SUSPENDwhich indicating whether the device support SUSPEND.Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by Eugenio Pérez <[email protected]>Signed-off-by Zhu Lingshan <[email protected]>Signed-off-by David Stevens <[email protected]>Your signature would normally be last.
Quoted 3 lines
--- content.tex | 71 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 61 insertions(+), 10 deletions(-)At a high level, I think this is somewhat overspecified. Generally e.g. if driver is forbidden from accessing a field, then we do not also add specific requirements for devices - with no driver doing this, such functionality will remain untested and unused, and when we finally decide we need it it's not going to be there.
Similarly for when device is not touching a field - no reason for driver not to touch it.Quoted 62 lines
content.tex +14
-10@@ -36,19 +36,22 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev this bit. For example, under Linux, drivers can be loadable modules. \end{note} -\item[FAILED (128)] Indicates that something went wrong in the guest, - and it has given up on the device. This could be an internal - error, or the driver didn't like the device for some reason, or - even a fatal error during device operation. +\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to + drive the device. \item[FEATURES_OK (8)] Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete. -\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to - drive the device. +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, indicates that the + device has been suspended by the driver. \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. + +\item[FAILED (128)] Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. \end{description} The \field{device status} field starts out as 0, and is reinitialized to 0 by@@ -60,8 +63,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev initialization sequence specified in \ref{sec:General Initialization And Device Operation / Device Initialization}. -The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, +The driver MUST NOT clear a \field{device status} bit other than SUSPEND +except when setting \field{device status} to 0 as a transport-specific way to +initiate a reset. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a@@ -99,10 +103,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 41] Feature bits reserved for extensions to the queue and +\item[24 to 42] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -629,6 +633,49 @@ \section{Device Cleanup}\label{sec:General Initialization And Device Operation / Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
Quoted 14 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 648
Quoted 13 lines
content.tex +13
-0+\section{Device Suspend}\label{sec:General Initialization And Device Operation / Device Suspend} + +When VIRTIO_F_SUSPEND is negotiated, the driver can set the +SUSPEND bit in \field{device status} to suspend a device, and can +clear the SUSPEND bit to resume a suspended device. + +\drivernormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The driver MUST NOT set SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated. + +Once the driver sets SUSPEND to \field{device status} of the device: +\begin{itemize} +\item The driver MUST re-read \field{device status} to verify whether the SUSPEND bit is set.If it's not set, then what? Device error, have to reset?
below in this patch:
"When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend"
Quoted 7 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 650
Quoted 4 lines
content.tex +2
-0+\item The driver MUST NOT make any more buffers available to the device. +\item The driver MUST NOT access any fields of any virtqueues or send notifications for any virtqueues.what are these "fields"? the area in memory where the vq lives? why is doing that a problem - you want device to be able to write something there?
To stabilize the vq states once suspended.
So how about change "fields" to "Descriptor Area and Driver Area", or just "virtqueues"?
Quoted 6 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 652
Quoted 3 lines
content.tex +2
-0+\item The driver MUST NOT access Device Configuration Space except for the field \field {device status}, +if it is implemented in the Config Space.what is "the Config Space"? device status is never in
a device configuration space, it is part of the common configuration. so what exactly are you forbidding here?
"common configuration" is PCI specific and actually "device status" and "device configuration space" are two separate sections in the spec.
And also for PCI, the device status is implemented in "common configuration".
Do you suggest we fall back to the last version:"The driver MUST NOT access Device Configuration Space"?
Quoted 7 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 657
Quoted 6 lines
content.tex +5
-0+\end{itemize} + +\devicenormative{\subsection}{Device Suspend}{General Initialization And Device Operation / Device Suspend} + +The device MUST ignore SUSPEND if FEATURES_OK is not set or VIRTIO_F_SUSPEND is not negotiated.what does must ignore SUSPEND mean? what do you want the device to do and when?
Do nothing and "ignore" is existing language in this spec:
The device MUST ignore the write-only flag (flags&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.
Quoted 7 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 660
Quoted 4 lines
content.tex +2
-0+The device MUST ignore all access to its Configuration Space while +suspended, except for \field{device status} if it is part of the Configuration Space.again what does ignore mean? Configuration Space here means Device Configuration Space?
Quoted 9 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 663
Quoted 3 lines
content.tex +3
-0+ +A device MUST NOT send any notifications, access any virtqueues, or modify +any fields in its Configuration Space while suspended.If a tree falls in a forest, and no one is around to hear it, does it still make a sound?
I think what matters is that if a configuration space changes while SUSPEND is set, then the device must not send a configuration change notification, and it must send a configuration change notification after SUSPEND bit has been cleared.
I will add this in V7.
Quoted 11 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 671
Quoted 9 lines
content.tex +7
-0+When the driver sets SUSPEND, the device MUST either suspend itself or set DEVICE_NEEDS_RESET if failed to suspend. + +If SUSPEND is set in \field{device status}, when the driver clears SUSPEND, +the device MUST either resume normal operation or set DEVICE_NEEDS_RESET. + +When the driver sets SUSPEND, +the device SHOULD perform the following actions before presenting SUSPEND bit in the \field{device status}:what does "before presenting" mean here? that a driver reading status will see 0 in this bit?
Presenting SUSPEND means setting the bit to 1. This is also existing spec language:
"device_feature The device uses this to report which feature bits it is offering to the driver: the driver writes to device_feature_select to select which feature bits are presented"
Is it clear if we say "presenting 1 in the SUSPEND bit"?
Quoted 8 lines
content.tex \section{Device Cleanup}\label{sec:General Initialization And Device Operation / · line 677
Quoted 6 lines
content.tex +5
-0+\begin{itemize} +\item Stop processing more buffers of any virtqueues +\item Wait until all buffers that are being processed have been used. +\item Send used buffer notifications to the driver. +\end{itemize}Is there anything here that has not been said before, but as a MUST? Saying same thing twice once a MUST once a SHOULD is confusing IMHO.
We have similar statements in the driver parts, but here is device normative section.
Quoted 19 lines
Quoted 19 lines
content.tex +5
-0+ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} Virtio can use various different buses, thus the standard is split@@ -872,6 +919,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the driver can + trigger suspending the device via the SUSPEND flag + See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
2.45.2