virtio: introduce VIRTIO_F_DEVICE_STOP
RFC PATCH 1/1 virtio: introduce VIRTIO_F_DEVICE_STOP
This patch introduces a new status bit DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.
--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)
content.tex +24 -2
@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. + \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. \end{description}
@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}. The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set.
@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev recover by issuing a reset. \end{note} +The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. + \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset. The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK. +The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED. + +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set. + \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.
@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates that the driver passes extra data (besides identifying the virtqueue) in its device notifications. + \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the + driver can stop and resume the device. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description}
Signature · 18 lines
2.25.1 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Quoted 18 lines · Jason Wang
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Signed-off-by Jason Wang <[email protected]>---
content.tex | 26 ++++++++++++++++++++++++--1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev\item[DRIVER_OK (4)] Indicates that the driver is set up and ready to
drive the device.+\item[DEVICE_STOPPED (32)] When
VIRTIO_F_DEVICE_STOPPEDis negotiated,
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 52
Quoted 2 lines
content.tex +2
-0+ indicates that the device has been stopped by the driver. +
Just for curiosity, why 32 and not 16?
Is there any rule for high bits and low bits of Device Status field?
Quoted 32 lines
\item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced
an error from which it can't recover.
\end{description}content.tex +0
-0@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev\ref{sec:General Initialization And Device Operation / Device Initialization}.
The driver MUST NOT clear a
-\field{device status} bit. If the driver sets the FAILED bit,content.tex +3
-1-the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize.The driver SHOULD NOT rely on completion of operations of a device if
DEVICE_NEEDS_RESETis set.content.tex +0
-0@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Devrecover by issuing a reset.
\end{note}content.tex +7
-0+The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. +\devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset.
The device MUST NOT consume buffers or send any used buffer notifications to the driver before
DRIVER_OK.
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 97
Quoted 8 lines
content.tex +8
-0+The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED. + +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set.
Maybe here we can specify if "DEVICE_STOPPED is set by the driver" or when the device set it after completing all the requests.
Quoted 13 lines
+
\label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD setDEVICE_NEEDS_RESETwhen it enters an error state that a reset is needed. IfDRIVER_OKis set, after it setsDEVICE_NEEDS_RESET, the device
MUST send a device configuration change notification to the driver.content.tex +2
-0@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates that the driver passes extra data (besides identifying the virtqueue) in its device notifications. + \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the + driver can stop and resume the device. See \ref{sec:Virtqueues / Driver
Quoted 3 lines
notifications}~\nameref{sec:Virtqueues / Driver notifications}.
\end{description}
Since the completion of in-flight requests can take time (I'm thinking of block devices), should we consider a notification when the device completes all the pending requests and set DEVICE_STOPPED.
Something similar to the notification when DEVICE_NEEDS_RESET is set.
Thanks,
Stefano
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 19 lines · Stefano Garzarella
Quoted 18 lines · Jason Wang
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Signed-off-by Jason Wang <[email protected]>---
content.tex | 26 ++++++++++++++++++++++++--1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0@@ -47,6 +47,9 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device.
Quoted 6 lines
Quoted 3 lines
+\item[DEVICE_STOPPED (32)] When
VIRTIO_F_DEVICE_STOPPEDis negotiated, + indicates that the device has been stopped by the driver. +Just for curiosity, why 32 and not 16?
Is there any rule for high bits and low bits of Device Status field?
I'm not sure. I just spot that DEVICE_NEEDS_RESET (with DEVICE prefix) is using 64.
Quoted 36 lines
Quoted 36 lines
\item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover.
\end{description}content.tex +0
-0@@ -58,8 +61,9 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}.
The driver MUST NOT clear a
-\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize.
+\field{device status} bit other than DEVICE_STOPPED. If thecontent.tex +2
-0+driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize.The driver SHOULD NOT rely on completion of operations of a device if
DEVICE_NEEDS_RESETis set.content.tex +0
-0@@ -70,12 +74,28 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
recover by issuing a reset.
\end{note}content.tex +7
-0+The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. +\devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset.
The device MUST NOT consume buffers or send any used buffer notifications to the driver before
DRIVER_OK.
Quoted 10 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 97
Quoted 8 lines
content.tex +8
-0+The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED. + +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set.Maybe here we can specify if "
DEVICE_STOPPEDis set by the driver" or when the device set it after completing all the requests.
Good point. Since the bit is offered by device, so "set by the device " might be better.
Quoted 18 lines
Quoted 7 lines
+
\label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD setDEVICE_NEEDS_RESETwhen it enters an error state that a reset is needed. IfDRIVER_OKis set, after it setsDEVICE_NEEDS_RESET, the device
MUST send a device configuration change notification to the driver.content.tex +0
-0@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}Quoted 8 lines
\item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates
that the driver passes extra data (besides identifying the virtqueue)
in its device notifications.
+ \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the
+ driver can stop and resume the device.
See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description}Since the completion of in-flight requests can take time (I'm thinking of block devices), should we consider a notification when the device completes all the pending requests and set
DEVICE_STOPPED.
Quoted 1 line
Something similar to the notification when
DEVICE_NEEDS_RESETis set.
I'm not sure, but it looks even more complicated. Driver may do a periodic polling or fail the operation (like migration) when the request is not completed soon.
Thanks
Quoted 4 lines
Thanks,
Stefano
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 80 lines · Jason Wang
Quoted 19 lines · Stefano Garzarella
Quoted 18 lines · Jason Wang
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Signed-off-by Jason Wang <[email protected]>---
content.tex | 26 ++++++++++++++++++++++++--1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0@@ -47,6 +47,9 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device.
Quoted 6 lines
Quoted 3 lines
+\item[DEVICE_STOPPED (32)] When
VIRTIO_F_DEVICE_STOPPEDis negotiated, + indicates that the device has been stopped by the driver. +Just for curiosity, why 32 and not 16?
Is there any rule for high bits and low bits of Device Status field?I'm not sure. I just spot that
DEVICE_NEEDS_RESET(with DEVICE prefix) is using 64.Quoted 36 lines
Quoted 36 lines
\item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover.
\end{description}content.tex +0
-0@@ -58,8 +61,9 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}.
The driver MUST NOT clear a
-\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize.
+\field{device status} bit other than DEVICE_STOPPED. If thecontent.tex +2
-0+driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize.The driver SHOULD NOT rely on completion of operations of a device if
DEVICE_NEEDS_RESETis set.content.tex +0
-0@@ -70,12 +74,28 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
recover by issuing a reset.
\end{note}content.tex +7
-0+The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. +\devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset.
The device MUST NOT consume buffers or send any used buffer notifications to the driver before
DRIVER_OK.Quoted 10 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 97
Quoted 8 lines
content.tex +8
-0+The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED. + +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set.Maybe here we can specify if "
DEVICE_STOPPEDis set by the driver" or when the device set it after completing all the requests.
Quoted 1 line
Good point. Since the bit is offered by device, so "set by the device " might be better.
Yeah, I agree.
Quoted 20 lines
Quoted 20 lines
Quoted 9 lines
+
\label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD setDEVICE_NEEDS_RESETwhen it enters an error state that a reset is needed. IfDRIVER_OKis set, after it setsDEVICE_NEEDS_RESET, the device
MUST send a device configuration change notification to the driver.content.tex +0
-0@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}Quoted 8 lines
\item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates
that the driver passes extra data (besides identifying the virtqueue)
in its device notifications.
+ \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the
+ driver can stop and resume the device.
See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description}Since the completion of in-flight requests can take time (I'm thinking of block devices), should we consider a notification when the device completes all the pending requests and set
DEVICE_STOPPED.
Quoted 5 lines
Quoted 1 line
Something similar to the notification when
DEVICE_NEEDS_RESETis set.I'm not sure, but it looks even more complicated. Driver may do a periodic polling or fail the operation (like migration) when the request is not completed soon.
I agree that colud be more complicated for the device point of view.
Since the driver is asking the device to stop, perhaps it makes sense for it to periodically check the status.
In case the driver decides to abort the operation (e.g. migration), can it clear the bit if the device hasn't set it yet or should it wait until the device has stopped? Maybe we should clarify this as well.
Thanks,
Stefano
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 111 lines · Stefano Garzarella
Quoted 83 lines · Jason Wang
Quoted 19 lines · Stefano Garzarella
Quoted 18 lines · Jason Wang
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Signed-off-by Jason Wang <[email protected]>---
content.tex | 26 ++++++++++++++++++++++++--1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0@@ -47,6 +47,9 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device.
Quoted 7 lines
Quoted 4 lines
+\item[DEVICE_STOPPED (32)] When
VIRTIO_F_DEVICE_STOPPEDis negotiated,
+ indicates that the device has been stopped by the driver. +Just for curiosity, why 32 and not 16?
Is there any rule for high bits and low bits of Device Status field?I'm not sure. I just spot that
DEVICE_NEEDS_RESET(with DEVICE prefix) is using 64.Quoted 37 lines
Quoted 37 lines
\item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover.
\end{description}content.tex +0
-0@@ -58,8 +61,9 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}.
The driver MUST NOT clear a
-\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize.
+\field{device status} bit other than DEVICE_STOPPED. If thecontent.tex +2
-0+driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize.The driver SHOULD NOT rely on completion of operations of a device if
DEVICE_NEEDS_RESETis set.content.tex +0
-0@@ -70,12 +74,28 @@ \section{\field{Device Status}Field}\label{sec:Basic Facilities of a Virtio Dev
recover by issuing a reset.
\end{note}content.tex +3
-0+The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set bythe
content.tex +4
-0+device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. +\devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset.
The device MUST NOT consume buffers or send any used buffer notifications to the driver before
DRIVER_OK.Quoted 11 lines
Quoted 9 lines
content.tex +7
-0+The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED. + +The device MUST keep the config space unchanged when
DEVICE_STOPPEDis
+set.Maybe here we can specify if "
DEVICE_STOPPEDis set by the driver" or when the device set it after completing all the requests.Quoted 1 line
Good point. Since the bit is offered by device, so "set by the device " might be better.
Yeah, I agree.
Quoted 23 lines
Quoted 21 lines
Quoted 9 lines
+
\label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD setDEVICE_NEEDS_RESETwhen it enters an error state that a reset is needed. IfDRIVER_OKis set, after it setsDEVICE_NEEDS_RESET, the device
MUST send a device configuration change notification to the driver.content.tex +0
-0@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}Quoted 8 lines
\item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates
that the driver passes extra data (besides identifying the virtqueue)
in its device notifications.
+ \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the
+ driver can stop and resume the device.
See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description}Since the completion of in-flight requests can take time (I'm thinking of block devices), should we consider a notification when the device completes all the pending requests and set
DEVICE_STOPPED.Something similar to the notification when
DEVICE_NEEDS_RESETis set.
Quoted 8 lines
Quoted 2 lines
I'm not sure, but it looks even more complicated. Driver may do a periodic polling or fail the operation (like migration) when the request is not completed soon.
I agree that colud be more complicated for the device point of view.
Since the driver is asking the device to stop, perhaps it makes sense for it to periodically check the status.In case the driver decides to abort the operation (e.g. migration), can it clear the bit if the device hasn't set it yet or should it wait until the device has stopped? Maybe we should clarify this as well.
Right, will add this.
Thanks
Quoted 4 lines
Thanks,
Stefano
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 4 lines · On Fri, 18 Dec 2020 12:23:02 +0800 Jason Wang <[email protected]> wrote:
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.
Can you please provide some more background information, or point me to the appropriate discussion?
I mean AFAIK migration already works without this driver initiated drain. What is the exact motivation? What about the big picture? I guess some agent in the guest would have to make the driver issue the DEVICE_STOP.
Quoted 10 lines
Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 52
Quoted 6 lines
content.tex +3
-0@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. +
AFAIU it is not only about indicating stopped, but also requesting to be stopped.
More importantly, that must not be set immediately, in a sense that the one side initiates some action by requesting the bit to be set, and the other side must not set the bit before the action is performed. We also seem to assume that every device implementation is capable of performing this trick. Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written, and about the completion, by reading the same value as written?
Quoted 25 lines
content.tex +9
-2\item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. \end{description}@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}. The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set.@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev recover by issuing a reset. \end{note} +The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device.
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 92
Quoted 10 lines
content.tex +4
-0+ \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset. The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK. +The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not
The when driver trying to set DEVICE_STOPPED is a bit soft as a duration. For example consider virtio-ccw, at the moment when the driver issues the ssch to set status, the device still does not know about it.
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 94
Quoted 2 lines
content.tex +2
-0+process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED.
I would like to have a more precise definition of 'new avail requests' and 'requests that is currently processing'.
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 97
Quoted 3 lines
content.tex +3
-0+ +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set.
Here you have the set by driver which is actually requesting the stop operation, and set by device which indicted that the stop operation was successfully performed by the device.
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6580
Quoted 13 lines
content.tex +3
-0+ \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates that the driver passes extra data (besides identifying the virtqueue) in its device notifications. + \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the + driver can stop and resume the device. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description}
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Fri, 18 Dec 2020 12:23:02 +0800 Jason Wang <[email protected]> wrote:
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.
Quoted 7 lines
Can you please provide some more background information, or point me to the appropriate discussion?
I mean AFAIK migration already works without this driver initiated drain. What is the exact motivation? What about the big picture? I guess some agent in the guest would have to make the driver issue the
DEVICE_STOP.
This is not necessary if the datapath is done inside qemu and when migration is initiated by qemu itself.
But it's a must for using virtio-device as a backend for emulated virtio devices (e.g vhost-vDPA). In this case, qemu needs to stop the device then it can safely synchronize the state from them.
Quoted 20 lines
Quoted 11 lines
Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 52
Quoted 6 lines
content.tex +3
-0@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. +AFAIU it is not only about indicating stopped, but also requesting to be stopped.
Quoted 3 lines
More importantly, that must not be set immediately, in a sense that the one side initiates some action by requesting the bit to be set, and the other side must not set the bit before the action is performed.
Yes.
Quoted 3 lines
We also
seem to assume that every device implementation is capable of performing this trick.
A dedicated feature bit is introduced for this.
Quoted 3 lines
Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written,
This is not ununsal in other devices. And in fact, the FEATURES_OK works like this:
"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the FEATURES_OK device status bit when the driver writes it.
"""
We've already had several hardware implementation of virtio-pci devices from different vendors. I didn't hear any complain about such kind of design.
Quoted 2 lines
and about the
completion, by reading the same value as written?
After after DEVICE_STOPPED is read from device the driver can assume the device is stopped.
Quoted 34 lines
Quoted 34 lines
content.tex +10
-2\item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. \end{description}@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}. The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set.@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev recover by issuing a reset. \end{note} +The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. + \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset. The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK.
Quoted 6 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 92
Quoted 3 lines
content.tex +3
-0+The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST notThe when driver trying to set
DEVICE_STOPPEDis a bit soft as a duration. For example consider virtio-ccw, at the moment when the driver issues the ssch to set status, the device still does not know about it.
I need more context on this, if it works like this, when or how can device know the status has been changed? (E.g how reset or other status bit is supposed to work?)
It looks like a transport limitation if we can't guarantee this. Similar issue were met in the PCIE Endpoint device, but it can be workaround by designing a new transport.
Quoted 5 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 94
Quoted 3 lines
content.tex +2
-0+process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED.I would like to have a more precise definition of 'new avail requests' and 'requests that is currently processing'.
Good point. How about something like:
The device MUST stop reading requests from descriptor area or driver area and MUST complete all in flight requests before setting DEVICE_STOPPED.
To be 100% accurate, it looks to me we need to mention device implementation internals or pseudo code. I start with "in flight" but Stefan wants a more accurate one. Reading the spec I found "in flight" has been used in:
"""
The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set. Note: For example, the driver can’t assume requests in flight will be completed if DEVICE_NEEDS_RESET is set, nor can it assume that they have not been completed. A good implementation will try to recover by issuing a reset.
"""
So we are probably fine. Any idea or suggestion are more than welcomed.
Quoted 7 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 97
Quoted 4 lines
content.tex +3
-0+ +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set.Here you have the set by driver which is actually requesting the stop operation, and set by device which indicted that the stop operation was successfully performed by the device.
Exactly.
Thanks
Quoted 14 lines
Quoted 6 lines
content.tex +1
-0+ \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6580
Quoted 8 lines
content.tex +2
-0@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates that the driver passes extra data (besides identifying the virtqueue) in its device notifications. + \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the + driver can stop and resume the device. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description}
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 18 lines · On Tue, 22 Dec 2020 10:36:41 +0800 Jason Wang <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Fri, 18 Dec 2020 12:23:02 +0800 Jason Wang <[email protected]> wrote:
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Quoted 7 lines
Can you please provide some more background information, or point me to the appropriate discussion?
I mean AFAIK migration already works without this driver initiated drain. What is the exact motivation? What about the big picture? I guess some agent in the guest would have to make the driver issue the
DEVICE_STOP.
Quoted 7 lines
This is not necessary if the datapath is done inside qemu and when migration is initiated by qemu itself.
But it's a must for using virtio-device as a backend for emulated virtio devices (e.g vhost-vDPA). In this case, qemu needs to stop the device then it can safely synchronize the state from them.
You say, in this case qemu needs to stop the device, which makes sense (it also has to do this when the datapath is implemented in qemu), but AFAIU DEVICE_STOPPED is initiated by the guest and not by qemu. I'm confused.
I'm still curious about how the different components in the stack (guest OS, qemu, vdpa-vhost in host kernel, the PCI function) are supposed to interact.
Quoted 29 lines
Quoted 21 lines
Quoted 12 lines
Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 52
Quoted 6 lines
content.tex +3
-0@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. +AFAIU it is not only about indicating stopped, but also requesting to be stopped.
Quoted 3 lines
More importantly, that must not be set immediately, in a sense that the one side initiates some action by requesting the bit to be set, and the other side must not set the bit before the action is performed.
Yes.
Quoted 7 lines
Quoted 3 lines
We also
seem to assume that every device implementation is capable of performing this trick.A dedicated feature bit is introduced for this.
This is not about the feature bit, but about the mechanism. But your subsequent answers explain, that this is nothing unusual, and then we should be fine.
Quoted 9 lines
Quoted 4 lines
Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written,
This is not ununsal in other devices. And in fact, the
FEATURES_OKworks like this:
Quoted 9 lines
"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the
FEATURES_OKdevice status bit when the driver writes it."""
Thanks for the pointer. I intend to have another look at how FEATURES_OK works, and how similar this is to DEVICE_STOPPED.
Quoted 5 lines
We've already had several hardware implementation of virtio-pci devices from different vendors. I didn't hear any complain about such kind of design.
Quoted 7 lines
Quoted 2 lines
and about the
completion, by reading the same value as written?After after
DEVICE_STOPPEDis read from device the driver can assume the device is stopped.
How is whether the device has already stopped or is still in the middle of stopping supposed/expected to affect the drivers behavior? Does the driver actually care?
Quoted 43 lines
Quoted 35 lines
Quoted 35 lines
content.tex +10
-2\item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. \end{description}@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}. The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set.@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev recover by issuing a reset. \end{note} +The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. + \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset. The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK.Quoted 6 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 92
Quoted 3 lines
content.tex +3
-0+The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST notThe when driver trying to set
DEVICE_STOPPEDis a bit soft as a duration. For example consider virtio-ccw, at the moment when the driver issues the ssch to set status, the device still does not know about it.
Quoted 3 lines
I need more context on this, if it works like this, when or how can device know the status has been changed? (E.g how reset or other status bit is supposed to work?)
I think we have a misunderstanding here. The device will eventually learn about the drivers request to write the status, it is just not instantaneous. It is like when you turn the light switch on, it takes some amount of time for the current to reach the lightbulb.
Quoted 5 lines
It looks like a transport limitation if we can't guarantee this. Similar issue were met in the PCIE Endpoint device, but it can be workaround by designing a new transport.
I don't think we have a problem with virtio-ccw here.
Quoted 20 lines
Quoted 6 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 94
Quoted 4 lines
content.tex +2
-0+process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED.I would like to have a more precise definition of 'new avail requests' and 'requests that is currently processing'.
Good point. How about something like:
The device MUST stop reading requests from descriptor area or driver area and MUST complete all in flight requests before setting
DEVICE_STOPPED.To be 100% accurate, it looks to me we need to mention device implementation internals or pseudo code. I start with "in flight" but Stefan wants a more accurate one. Reading the spec I found "in flight" has been used in:
"""
Quoted 8 lines
The driver SHOULD NOT rely on completion of operations of a device if
DEVICE_NEEDS_RESETis set. Note: For example, the driver can’t assume requests in flight will be completed ifDEVICE_NEEDS_RESETis set, nor can it assume that they have not been completed. A good implementation will try to recover by issuing a reset."""
My problem is, that one can look at this from the drivers or from the devices perspective. From the drivers perspective, every buffer that has been made available but haven't been marked as used, is essentially a request in flight.
From the device perspective, and probably from migration perspective, the in-flight requests are those that were submitted to some kind of a backend. I.e. available requests that are just sitting on the queue, but weren't submitted to the backend, can sit around, and get submitted after the migration. It seems that what we want is the following: while the DEVICE_STOPPED is set, there are no requests in the backend (i.e. all requests submitted prior have completed, and no more request are submitted to it).
BTW how about s/stop/drain? I think drain has more of this 'finish what you are doing' while stop can be 'stop in the middle of it, whatever you are doing'.
Quoted 4 lines
So we are probably fine. Any idea or suggestion are more than welcomed.
I will think about it some more.
Quoted 9 lines
Quoted 7 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 97
Quoted 4 lines
content.tex +3
-0+ +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set.Here you have the set by driver which is actually requesting the stop operation, and set by device which indicted that the stop operation was successfully performed by the device.
Quoted 4 lines
Exactly.
Thanks
Thank you for all the exlanations!
Regards,
Halil
[..]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 22 lines · Halil Pasic
Quoted 19 lines · On Tue, 22 Dec 2020 10:36:41 +0800 Jason Wang <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Fri, 18 Dec 2020 12:23:02 +0800 Jason Wang <[email protected]> wrote:
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Quoted 7 lines
Can you please provide some more background information, or point me to the appropriate discussion?
I mean AFAIK migration already works without this driver initiated drain. What is the exact motivation? What about the big picture? I guess some agent in the guest would have to make the driver issue the
DEVICE_STOP.This is not necessary if the datapath is done inside qemu and when migration is initiated by qemu itself.
Quoted 8 lines
Quoted 4 lines
But it's a must for using virtio-device as a backend for emulated virtio devices (e.g vhost-vDPA). In this case, qemu needs to stop the device then it can safely synchronize the state from them.
You say, in this case qemu needs to stop the device, which makes sense (it also has to do this when the datapath is implemented in qemu), but AFAIU
DEVICE_STOPPEDis initiated by the guest and not by qemu. I'm confused.
It's initiated by Qemu. Guest is unware of live migration.
Quoted 4 lines
I'm still curious about how the different components in the stack (guest OS, qemu, vdpa-vhost in host kernel, the PCI function) are supposed to interact.
It works like:
From Qemu point of view, vhost-vDPA is just another type of vhost backend. Qemu needs to stop virtio (vhost) before it can do migration. So we require vDPA devices to have the ability of stopping or pausing its datapath. If the vDPA device is by chance the virtio-PCI device, it needs an interface for receiving stop/resume command from the driver.
So the devce stop/resume command was sent from Qemu to vhost-VDPA, then to vDPA parent which could be a virtio-PCI device in this case.
Quoted 51 lines
Quoted 24 lines
Quoted 20 lines
Quoted 11 lines
Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 52
Quoted 6 lines
content.tex +3
-0@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. +AFAIU it is not only about indicating stopped, but also requesting to be stopped.
Quoted 3 lines
More importantly, that must not be set immediately, in a sense that the one side initiates some action by requesting the bit to be set, and the other side must not set the bit before the action is performed.
Quoted 9 lines
Yes.
Quoted 3 lines
We also
seem to assume that every device implementation is capable of performing this trick.A dedicated feature bit is introduced for this.
This is not about the feature bit, but about the mechanism. But your subsequent answers explain, that this is nothing unusual, and then we should be fine.
Quoted 14 lines
Quoted 3 lines
Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written,
This is not ununsal in other devices. And in fact, the
FEATURES_OKworks like this:"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the
FEATURES_OKdevice status bit when the driver writes it.
Quoted 4 lines
Quoted 2 lines
"""
Thanks for the pointer. I intend to have another look at how
FEATURES_OKworks, and how similar this is toDEVICE_STOPPED.
My understanding is that for both of them, driver can try to set the bit by writing to the status register but it's the device that decide when to set the bit.
Quoted 9 lines
Quoted 9 lines
We've already had several hardware implementation of virtio-pci devices from different vendors. I didn't hear any complain about such kind of design.
Quoted 2 lines
and about the
completion, by reading the same value as written?
Quoted 6 lines
Quoted 3 lines
After after
DEVICE_STOPPEDis read from device the driver can assume the device is stopped.How is whether the device has already stopped or is still in the middle of stopping supposed/expected to affect the drivers behavior? Does the driver actually care?
Yes, in the case of live migration. If device takes too longs time to be stopped. Qemu may simple fail the migration.
Quoted 41 lines
Quoted 41 lines
Quoted 40 lines
Quoted 27 lines
content.tex +9
-2\item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. \end{description}@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}. The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set.@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev recover by issuing a reset. \end{note} +The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device.content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 92
Quoted 10 lines
content.tex +4
-0+ \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset. The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK. +The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST notThe when driver trying to set
DEVICE_STOPPEDis a bit soft as a duration. For example consider virtio-ccw, at the moment when the driver issues the ssch to set status, the device still does not know about it.
Quoted 7 lines
Quoted 3 lines
I need more context on this, if it works like this, when or how can device know the status has been changed? (E.g how reset or other status bit is supposed to work?)
I think we have a misunderstanding here. The device will eventually learn about the drivers request to write the status, it is just not instantaneous. It is like when you turn the light switch on, it takes some amount of time for the current to reach the lightbulb.
I see, so it looks not a problem since device is not required to be stopped immediately if driver write to that bit.
Quoted 6 lines
Quoted 5 lines
It looks like a transport limitation if we can't guarantee this. Similar issue were met in the PCIE Endpoint device, but it can be workaround by designing a new transport.
I don't think we have a problem with virtio-ccw here.
Agree.
Quoted 24 lines
Quoted 24 lines
Quoted 5 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 94
Quoted 3 lines
content.tex +2
-0+process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED.I would like to have a more precise definition of 'new avail requests' and 'requests that is currently processing'.
Good point. How about something like:
The device MUST stop reading requests from descriptor area or driver area and MUST complete all in flight requests before setting
DEVICE_STOPPED.To be 100% accurate, it looks to me we need to mention device implementation internals or pseudo code. I start with "in flight" but Stefan wants a more accurate one. Reading the spec I found "in flight" has been used in:
"""
The driver SHOULD NOT rely on completion of operations of a device if
DEVICE_NEEDS_RESETis set. Note: For example, the driver can’t assume requests in flight will be completed ifDEVICE_NEEDS_RESETis set, nor can it assume that they have not been completed. A good implementation will try to recover by issuing a reset.
Quoted 6 lines
Quoted 2 lines
"""
My problem is, that one can look at this from the drivers or from the devices perspective. From the drivers perspective, every buffer that has been made available but haven't been marked as used, is essentially a request in flight.
Not a native speaker, but the sentence starts from "The device" so it should be the perspective of device.
Quoted 9 lines
From the device perspective, and probably from migration perspective, the
in-flight requests are those that were submitted to some kind of a backend. I.e. available requests that are just sitting on the queue, but weren't submitted to the backend, can sit around, and get submitted after the migration. It seems that what we want is the following: while theDEVICE_STOPPEDis set, there are no requests in the backend (i.e. all requests submitted prior have completed, and no more request are submitted to it).
Yes. but do we need to mention something like "backend" in the spec? It sounds like a implementation detail.
Quoted 4 lines
BTW how about s/stop/drain? I think drain has more of this 'finish what you are doing' while stop can be 'stop in the middle of it, whatever you are doing'.
It might be better. But drain may fit more for a queue not a device. We had stuffs other than queue e.g as Stefan pointed out, we should keep config space unchanged (and no config interrupt) after the device is stopped.
Quoted 5 lines
Quoted 4 lines
So we are probably fine. Any idea or suggestion are more than welcomed.
I will think about it some more.
Thanks
Quoted 26 lines
Quoted 8 lines
Quoted 7 lines
content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 97
Quoted 4 lines
content.tex +3
-0+ +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set.Here you have the set by driver which is actually requesting the stop operation, and set by device which indicted that the stop operation was successfully performed by the device.
Quoted 4 lines
Exactly.
Thanks
Thank you for all the exlanations!
Regards,
Halil[..]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 27 lines · On Tue, 22 Dec 2020 15:30:31 +0800 Jason Wang <[email protected]> wrote:
Quoted 26 lines · Halil Pasic
Quoted 23 lines · On Tue, 22 Dec 2020 10:36:41 +0800 Jason Wang <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Fri, 18 Dec 2020 12:23:02 +0800 Jason Wang <[email protected]> wrote:
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Quoted 7 lines
Can you please provide some more background information, or point me to the appropriate discussion?
I mean AFAIK migration already works without this driver initiated drain. What is the exact motivation? What about the big picture? I guess some agent in the guest would have to make the driver issue the
DEVICE_STOP.This is not necessary if the datapath is done inside qemu and when migration is initiated by qemu itself.
But it's a must for using virtio-device as a backend for emulated virtio devices (e.g vhost-vDPA). In this case, qemu needs to stop the device then it can safely synchronize the state from them.
Quoted 7 lines
Quoted 4 lines
You say, in this case qemu needs to stop the device, which makes sense (it also has to do this when the datapath is implemented in qemu), but AFAIU
DEVICE_STOPPEDis initiated by the guest and not by qemu. I'm confused.It's initiated by Qemu. Guest is unware of live migration.
But isn't setting DEVICE_STOPPED a _driver_ initiated process? That sounds like "guest" to me.
Quoted 10 lines
Quoted 6 lines
I'm still curious about how the different components in the stack (guest OS, qemu, vdpa-vhost in host kernel, the PCI function) are supposed to interact.
It works like:
Quoted 8 lines
From Qemu point of view, vhost-vDPA is just another type of vhost
backend. Qemu needs to stop virtio (vhost) before it can do migration. So we require vDPA devices to have the ability of stopping or pausing its datapath. If the vDPA device is by chance the virtio-PCI device, it needs an interface for receiving stop/resume command from the driver.So the devce stop/resume command was sent from Qemu to vhost-VDPA, then to vDPA parent which could be a virtio-PCI device in this case.
But QEMU implements the _device_, not the driver, doesn't it? And IIUC, vhost-VDPA and the vDPA parent are also on the device side. I feel like I'm missing something essential here.
Quoted 55 lines
Quoted 55 lines
Quoted 26 lines
Quoted 22 lines
Quoted 13 lines
Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 52
Quoted 6 lines
content.tex +3
-0@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. +AFAIU it is not only about indicating stopped, but also requesting to be stopped.
Quoted 3 lines
More importantly, that must not be set immediately, in a sense that the one side initiates some action by requesting the bit to be set, and the other side must not set the bit before the action is performed.
Quoted 9 lines
Yes.
Quoted 3 lines
We also
seem to assume that every device implementation is capable of performing this trick.A dedicated feature bit is introduced for this.
This is not about the feature bit, but about the mechanism. But your subsequent answers explain, that this is nothing unusual, and then we should be fine.
Quoted 16 lines
Quoted 3 lines
Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written,
This is not ununsal in other devices. And in fact, the
FEATURES_OKworks like this:"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the
FEATURES_OKdevice status bit when the driver writes it."""
Quoted 7 lines
Quoted 2 lines
Thanks for the pointer. I intend to have another look at how
FEATURES_OKworks, and how similar this is toDEVICE_STOPPED.My understanding is that for both of them, driver can try to set the bit by writing to the status register but it's the device that decide when to set the bit.
I think there's a difference: For FEATURES_OK, the driver can read back the status and knows that the feature combination is not accepted if it is not set. For DEVICE_STOPPED, it does not really know whether the device has started to stop yet. It only knows that the device is actually done stopping when DEVICE_STOPPED is set.
Do we need a different mechanism for the device to signal the driver that the device has actually been stopped? If the driver sees the STOPPED status after reading back, it knows that the device has acknowledged the request and is now stopping down. If it is not set, it means that the device has not honoured the request. Same for clearing it to resume.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 29 lines · Cornelia Huck
Quoted 26 lines · On Tue, 22 Dec 2020 15:30:31 +0800 Jason Wang <[email protected]> wrote:
Quoted 25 lines · Halil Pasic
Quoted 22 lines · On Tue, 22 Dec 2020 10:36:41 +0800 Jason Wang <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Fri, 18 Dec 2020 12:23:02 +0800 Jason Wang <[email protected]> wrote:
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Quoted 7 lines
Can you please provide some more background information, or point me to the appropriate discussion?
I mean AFAIK migration already works without this driver initiated drain. What is the exact motivation? What about the big picture? I guess some agent in the guest would have to make the driver issue the
DEVICE_STOP.This is not necessary if the datapath is done inside qemu and when migration is initiated by qemu itself.
But it's a must for using virtio-device as a backend for emulated virtio devices (e.g vhost-vDPA). In this case, qemu needs to stop the device then it can safely synchronize the state from them.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
You say, in this case qemu needs to stop the device, which makes sense (it also has to do this when the datapath is implemented in qemu), but AFAIU
DEVICE_STOPPEDis initiated by the guest and not by qemu. I'm confused.It's initiated by Qemu. Guest is unware of live migration.
But isn't setting
DEVICE_STOPPEDa _driver_ initiated process? That sounds like "guest" to me.
I think we need clarification on the "driver". Think the following setup:
Guest driver <-> Qemu <-> vhost-vdpa <-> vDPA parent (virtio-pci vDPA driver)
It's the Qemu that initiates the stop, and the request is forwarded to virtio-pci vDPA driver. So the process is transparent to the guest (driver).
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines
I'm still curious about how the different components in the stack (guest OS, qemu, vdpa-vhost in host kernel, the PCI function) are supposed to interact.
It works like:
From Qemu point of view, vhost-vDPA is just another type of vhost
backend. Qemu needs to stop virtio (vhost) before it can do migration. So we require vDPA devices to have the ability of stopping or pausing its datapath. If the vDPA device is by chance the virtio-PCI device, it needs an interface for receiving stop/resume command from the driver.
Quoted 3 lines
Quoted 2 lines
So the devce stop/resume command was sent from Qemu to vhost-VDPA, then to vDPA parent which could be a virtio-PCI device in this case.
But QEMU implements the _device_, not the driver, doesn't it?
The device is implemented with the co-operation between Qemu and vhost-vDPA. During migration qemu need to stop the virtio-net device, then vhost must be stopped as well.
Quoted 3 lines
And IIUC,
vhost-VDPA and the vDPA parent are also on the device side. I feel like I'm missing something essential here.
Virtio-PCI driver could be a vDPA parent as well in this case. So we need stop the virtio-pci device.
Quoted 55 lines
Quoted 55 lines
Quoted 54 lines
Quoted 25 lines
Quoted 22 lines
Quoted 13 lines
Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 52
Quoted 6 lines
content.tex +3
-0@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. +AFAIU it is not only about indicating stopped, but also requesting to be stopped.
Quoted 3 lines
More importantly, that must not be set immediately, in a sense that the one side initiates some action by requesting the bit to be set, and the other side must not set the bit before the action is performed.
Quoted 8 lines
Yes.
Quoted 3 lines
We also
seem to assume that every device implementation is capable of performing this trick.A dedicated feature bit is introduced for this.
This is not about the feature bit, but about the mechanism. But your subsequent answers explain, that this is nothing unusual, and then we should be fine.
Quoted 6 lines
Quoted 3 lines
Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written,
This is not ununsal in other devices. And in fact, the
FEATURES_OKworks like this:Quoted 9 lines
"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the
FEATURES_OKdevice status bit when the driver writes it."""
Thanks for the pointer. I intend to have another look at how
FEATURES_OKworks, and how similar this is toDEVICE_STOPPED.
Quoted 8 lines
Quoted 3 lines
My understanding is that for both of them, driver can try to set the bit by writing to the status register but it's the device that decide when to set the bit.
I think there's a difference: For
FEATURES_OK, the driver can read back the status and knows that the feature combination is not accepted if it is not set. ForDEVICE_STOPPED, it does not really know whether the device has started to stop yet. It only knows that the device is actually done stopping whenDEVICE_STOPPEDis set.
The only difference is that device may need sometime to be stopped. FEATURES_OK might not be the best example. Let's see how reset work which is more similar to DEVICE_STOPPED:
"""
After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.
"""
Quoted 7 lines
Do we need a different mechanism for the device to signal the driver that the device has actually been stopped? If the driver sees the STOPPED status after reading back, it knows that the device has acknowledged the request and is now stopping down. If it is not set, it means that the device has not honoured the request. Same for clearing it to resume.
I don't see much difference. Device reset doesn't have such notification and driver may simply poll or use a timer.
Thanks
Quoted 9 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 35 lines · On Tue, 22 Dec 2020 20:51:04 +0800 Jason Wang <[email protected]> wrote:
Quoted 34 lines · Cornelia Huck
Quoted 31 lines · On Tue, 22 Dec 2020 15:30:31 +0800 Jason Wang <[email protected]> wrote:
Quoted 29 lines · Halil Pasic
Quoted 15 lines · On Tue, 22 Dec 2020 10:36:41 +0800 Jason Wang <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Fri, 18 Dec 2020 12:23:02 +0800 Jason Wang <[email protected]> wrote:
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Quoted 7 lines
Can you please provide some more background information, or point me to the appropriate discussion?
I mean AFAIK migration already works without this driver initiated drain. What is the exact motivation? What about the big picture? I guess some agent in the guest would have to make the driver issue the
DEVICE_STOP.Quoted 7 lines
This is not necessary if the datapath is done inside qemu and when migration is initiated by qemu itself.
But it's a must for using virtio-device as a backend for emulated virtio devices (e.g vhost-vDPA). In this case, qemu needs to stop the device then it can safely synchronize the state from them.
You say, in this case qemu needs to stop the device, which makes sense (it also has to do this when the datapath is implemented in qemu), but AFAIU
DEVICE_STOPPEDis initiated by the guest and not by qemu. I'm confused.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
It's initiated by Qemu. Guest is unware of live migration.
But isn't setting
DEVICE_STOPPEDa _driver_ initiated process? That sounds like "guest" to me.I think we need clarification on the "driver".
Looking through the spec, I think we have never properly defined what we mean by "device" or "driver". Not sure what a good definition would look like. I would consider a "device" an entity that is present and is configured/used by a "driver". The "driver" is the means to actually configure/use a "device", i.e. the "initiating" part (even though some actions are initiated by the device once the driver has started interacting with it).
Quoted 7 lines
Think the following setup:
Guest driver <-> Qemu <-> vhost-vdpa <-> vDPA parent (virtio-pci vDPA driver)
It's the Qemu that initiates the stop, and the request is forwarded to virtio-pci vDPA driver. So the process is transparent to the guest (driver).
So it's
Guest driver <-> QEMU <-> vhost-vdpa <-> vDPA parent |<-- guest -->|<-------------- host -------------->|
but the vDPA parent is itself acting as a "driver" if you take my attempt at a definition above?
(Wouldn't that mean that the device in QEMU is configured/used from two entities?)
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 7 lines
I'm still curious about how the different components in the stack (guest OS, qemu, vdpa-vhost in host kernel, the PCI function) are supposed to interact.
It works like:
From Qemu point of view, vhost-vDPA is just another type of vhost
backend. Qemu needs to stop virtio (vhost) before it can do migration. So we require vDPA devices to have the ability of stopping or pausing its datapath. If the vDPA device is by chance the virtio-PCI device, it needs an interface for receiving stop/resume command from the driver.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
So the devce stop/resume command was sent from Qemu to vhost-VDPA, then to vDPA parent which could be a virtio-PCI device in this case.
But QEMU implements the _device_, not the driver, doesn't it?
The device is implemented with the co-operation between Qemu and vhost-vDPA. During migration qemu need to stop the virtio-net device, then vhost must be stopped as well.
Yes, it makes sense that any vhost parts need to be stopped as well.
Quoted 9 lines
Quoted 5 lines
And IIUC,
vhost-VDPA and the vDPA parent are also on the device side. I feel like I'm missing something essential here.Virtio-PCI driver could be a vDPA parent as well in this case. So we need stop the virtio-pci device.
Who is the "driver" in that case?
Quoted 71 lines
Quoted 57 lines
Quoted 57 lines
Quoted 56 lines
Quoted 27 lines
Quoted 24 lines
Quoted 15 lines
Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 52
Quoted 6 lines
content.tex +3
-0@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. +AFAIU it is not only about indicating stopped, but also requesting to be stopped.
Quoted 3 lines
More importantly, that must not be set immediately, in a sense that the one side initiates some action by requesting the bit to be set, and the other side must not set the bit before the action is performed.
Quoted 8 lines
Yes.
Quoted 3 lines
We also
seem to assume that every device implementation is capable of performing this trick.A dedicated feature bit is introduced for this.
This is not about the feature bit, but about the mechanism. But your subsequent answers explain, that this is nothing unusual, and then we should be fine.
Quoted 6 lines
Quoted 3 lines
Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written,
This is not ununsal in other devices. And in fact, the
FEATURES_OKworks like this:Quoted 9 lines
"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the
FEATURES_OKdevice status bit when the driver writes it."""
Thanks for the pointer. I intend to have another look at how
FEATURES_OKworks, and how similar this is toDEVICE_STOPPED.Quoted 8 lines
Quoted 3 lines
My understanding is that for both of them, driver can try to set the bit by writing to the status register but it's the device that decide when to set the bit.
I think there's a difference: For
FEATURES_OK, the driver can read back the status and knows that the feature combination is not accepted if it is not set. ForDEVICE_STOPPED, it does not really know whether the device has started to stop yet. It only knows that the device is actually done stopping whenDEVICE_STOPPEDis set.The only difference is that device may need sometime to be stopped.
FEATURES_OKmight not be the best example. Let's see how reset work which is more similar toDEVICE_STOPPED:
Quoted 6 lines
"""
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Hm, but that is PCI-specific writing of fields in the configuration structure, right? CCW uses a different method for resetting (an asynchronous channel command; channel commands create an interrupt when done); "write 0 to status for reset" is not universal.
Which makes me think: is using the device status a good universal method for stopping the device? If I look at it only from the CCW perspective, I'd have used a new channel command with a payload of stop/resume and only reflected the stopped state in the device status (i.e. read-only from the driver, and only changed by the device when it transitions to/from the stopped state.) Could PCI use a new field for that?
Quoted 10 lines
Quoted 8 lines
Do we need a different mechanism for the device to signal the driver that the device has actually been stopped? If the driver sees the STOPPED status after reading back, it knows that the device has acknowledged the request and is now stopping down. If it is not set, it means that the device has not honoured the request. Same for clearing it to resume.
Quoted 2 lines
I don't see much difference. Device reset doesn't have such notification and driver may simply poll or use a timer.
See above; CCW does not really use the device status in the same way. I think a notification would be useful.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 41 lines · Cornelia Huck
Quoted 38 lines · On Tue, 22 Dec 2020 20:51:04 +0800 Jason Wang <[email protected]> wrote:
Quoted 36 lines · Cornelia Huck
Quoted 26 lines · On Tue, 22 Dec 2020 15:30:31 +0800 Jason Wang <[email protected]> wrote:
Quoted 25 lines · Halil Pasic
Quoted 22 lines · On Tue, 22 Dec 2020 10:36:41 +0800 Jason Wang <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Fri, 18 Dec 2020 12:23:02 +0800 Jason Wang <[email protected]> wrote:
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Quoted 7 lines
Can you please provide some more background information, or point me to the appropriate discussion?
I mean AFAIK migration already works without this driver initiated drain. What is the exact motivation? What about the big picture? I guess some agent in the guest would have to make the driver issue the
DEVICE_STOP.This is not necessary if the datapath is done inside qemu and when migration is initiated by qemu itself.
But it's a must for using virtio-device as a backend for emulated virtio devices (e.g vhost-vDPA). In this case, qemu needs to stop the device then it can safely synchronize the state from them.
Quoted 5 lines
Quoted 4 lines
You say, in this case qemu needs to stop the device, which makes sense (it also has to do this when the datapath is implemented in qemu), but AFAIU
DEVICE_STOPPEDis initiated by the guest and not by qemu. I'm confused.It's initiated by Qemu. Guest is unware of live migration.
But isn't setting
DEVICE_STOPPEDa _driver_ initiated process? That sounds like "guest" to me.
Quoted 8 lines
Quoted 1 line
I think we need clarification on the "driver".
Looking through the spec, I think we have never properly defined what we mean by "device" or "driver". Not sure what a good definition would look like. I would consider a "device" an entity that is present and is configured/used by a "driver". The "driver" is the means to actually configure/use a "device", i.e. the "initiating" part (even though some actions are initiated by the device once the driver has started interacting with it).
Sorry for being unclear. I think the spec is fine since it's not limit to a host/guest interface, the clarification is only needed for the context of the use cases of DEVICE_STOP.
Quoted 10 lines
Quoted 8 lines
Think the following setup:
Guest driver <-> Qemu <-> vhost-vdpa <-> vDPA parent (virtio-pci vDPA driver)
It's the Qemu that initiates the stop, and the request is forwarded to virtio-pci vDPA driver. So the process is transparent to the guest (driver).
So it's
Quoted 8 lines
Guest driver <-> QEMU <-> vhost-vdpa <-> vDPA parent |<-- guest -->|<-------------- host -------------->|
but the vDPA parent is itself acting as a "driver" if you take my attempt at a definition above?
(Wouldn't that mean that the device in QEMU is configured/used from two entities?)
A more accurate version
Guest virtio-pci driver <-> virtio-pci device QEMU <-> vhost-vdpa <-> virtio pci vDPA driver <-> virtio-pci device
So we had two drivers:
1) The first that is running in the guest (it could be either L1 or L2)
2) The second that is running on the host (or L1 if the first is running in L2)
And we had two devices
1) The first that is emulated by QEMU which is running in either L0 or L1
2) The second could be either a physical device on L0 or emulated one in L1 (Qemu is running in L1)
The main use case for DEVICE_STOP the second device and driver. This driver is not the virtio-pci driver but a vDPA driver [1].
[1] https://lkml.org/lkml/2020/12/3/1589
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 7 lines
I'm still curious about how the different components in the stack (guest OS, qemu, vdpa-vhost in host kernel, the PCI function) are supposed to interact.
It works like:
From Qemu point of view, vhost-vDPA is just another type of vhost
backend. Qemu needs to stop virtio (vhost) before it can do migration. So we require vDPA devices to have the ability of stopping or pausing its datapath. If the vDPA device is by chance the virtio-PCI device, it needs an interface for receiving stop/resume command from the driver.Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
So the devce stop/resume command was sent from Qemu to vhost-VDPA, then to vDPA parent which could be a virtio-PCI device in this case.
But QEMU implements the _device_, not the driver, doesn't it?
The device is implemented with the co-operation between Qemu and vhost-vDPA. During migration qemu need to stop the virtio-net device, then vhost must be stopped as well.
Yes, it makes sense that any vhost parts need to be stopped as well.
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
And IIUC,
vhost-VDPA and the vDPA parent are also on the device side. I feel like I'm missing something essential here.Virtio-PCI driver could be a vDPA parent as well in this case. So we need stop the virtio-pci device.
Who is the "driver" in that case?
It's the virtio-PCI vDPA driver.
Quoted 71 lines
Quoted 71 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 27 lines
Quoted 24 lines
Quoted 15 lines
Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +0
-0content.tex \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev · line 52
Quoted 6 lines
content.tex +3
-0@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. +AFAIU it is not only about indicating stopped, but also requesting to be stopped.
Quoted 3 lines
More importantly, that must not be set immediately, in a sense that the one side initiates some action by requesting the bit to be set, and the other side must not set the bit before the action is performed.
Quoted 8 lines
Yes.
Quoted 3 lines
We also
seem to assume that every device implementation is capable of performing this trick.A dedicated feature bit is introduced for this.
This is not about the feature bit, but about the mechanism. But your subsequent answers explain, that this is nothing unusual, and then we should be fine.
Quoted 6 lines
Quoted 3 lines
Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written,
This is not ununsal in other devices. And in fact, the
FEATURES_OKworks like this:Quoted 9 lines
"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the
FEATURES_OKdevice status bit when the driver writes it."""
Thanks for the pointer. I intend to have another look at how
FEATURES_OKworks, and how similar this is toDEVICE_STOPPED.Quoted 8 lines
Quoted 3 lines
My understanding is that for both of them, driver can try to set the bit by writing to the status register but it's the device that decide when to set the bit.
I think there's a difference: For
FEATURES_OK, the driver can read back the status and knows that the feature combination is not accepted if it is not set. ForDEVICE_STOPPED, it does not really know whether the device has started to stop yet. It only knows that the device is actually done stopping whenDEVICE_STOPPEDis set.The only difference is that device may need sometime to be stopped.
FEATURES_OKmight not be the best example. Let's see how reset work which is more similar toDEVICE_STOPPED:"""
Quoted 8 lines
Quoted 4 lines
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Hm, but that is PCI-specific writing of fields in the configuration structure, right? CCW uses a different method for resetting (an asynchronous channel command; channel commands create an interrupt when done); "write 0 to status for reset" is not universal.
I may miss something. Even if CCW is using something asynchronously (e.g interrupt). It still present a synchronous API to the upper layer. E.g ccw_io_helper() is using wait_event()?
Quoted 3 lines
Which makes me think: is using the device status a good universal method for stopping the device?
This part I don't understand. Device stop is kind of similar to reset in this case. If reset can work what prevent stop work?
E.g:
ccw_io_helper(VIRTIO_CCW_DOING_WRITE_STATUS);
while (1) {
ccw_io_helper(VIRTIO_CCW_DOING_READ_STATUS);
if (status & DEVICE_STOPPED)
break;
}
Quoted 6 lines
If I look at it only from the CCW
perspective, I'd have used a new channel command with a payload of stop/resume and only reflected the stopped state in the device status (i.e. read-only from the driver, and only changed by the device when it transitions to/from the stopped state.) Could PCI use a new field for that?
PCI can use new filed. But I wonder how can that help for CCW.
Thanks
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Do we need a different mechanism for the device to signal the driver that the device has actually been stopped? If the driver sees the STOPPED status after reading back, it knows that the device has acknowledged the request and is now stopping down. If it is not set, it means that the device has not honoured the request. Same for clearing it to resume.
Quoted 4 lines
Quoted 2 lines
I don't see much difference. Device reset doesn't have such notification and driver may simply poll or use a timer.
See above; CCW does not really use the device status in the same way. I think a notification would be useful.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
On Wed, 23 Dec 2020 10:48:45 +0800
Jason Wang <[email protected]> wrote:
[..]
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 13 lines
Quoted 13 lines
Quoted 3 lines
Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written,
This is not ununsal in other devices. And in fact, the
FEATURES_OKworks like this:"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the
FEATURES_OKdevice status bit when the driver writes it.Quoted 4 lines
Quoted 2 lines
"""
Thanks for the pointer. I intend to have another look at how
FEATURES_OKworks, and how similar this is toDEVICE_STOPPED.My understanding is that for both of them, driver can try to set the bit by writing to the status register but it's the device that decide when to set the bit.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
I think there's a difference: For
FEATURES_OK, the driver can read back the status and knows that the feature combination is not accepted if it is not set. ForDEVICE_STOPPED, it does not really know whether the device has started to stop yet. It only knows that the device is actually done stopping whenDEVICE_STOPPEDis set.
I share Connie's concern. The config->set_status() is a synchronous API, that is right after the iowrite8 for pci comes back, or the channel program finishes executing for ccw, we can go, do a config->get_status() and know if the status was set succesfully.
Quoted 17 lines
Quoted 6 lines
Quoted 6 lines
The only difference is that device may need sometime to be stopped.
FEATURES_OKmight not be the best example. Let's see how reset work which is more similar toDEVICE_STOPPED:"""
Quoted 8 lines
Quoted 4 lines
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Hm, but that is PCI-specific writing of fields in the configuration structure, right? CCW uses a different method for resetting (an asynchronous channel command; channel commands create an interrupt when done); "write 0 to status for reset" is not universal.
Quoted 3 lines
I may miss something. Even if CCW is using something asynchronously (e.g interrupt). It still present a synchronous API to the upper layer. E.g
ccw_io_helper()is usingwait_event()?
Right. The reason why we have to wait for the completion of the channel program is exactly that. Furthermore if we have to fail setting FEATURES_OK we fail the channel program, that was supposed to do the operation with an unit check. So at the point where the set_status() is done, we already know if it worked or not. Of course the device can change it's status any time.
Since we have a synchronous API setting DEVICE_STOPPED would also have to block until all in-flight requests are completed. But this does not see to be what you want. I understood thatvirtio_add_status(dev, DEVICE_STOPPED) would return immediately, and so would a subsequent config->get_status(dev) without the DEVICE_STOPPED bit set, if there are still outstanding requests.
I don't think defining virtio_add_status(dev, FEATURES_OK) (followed by a readback) as synchronous and virtio_add_status(dev, DEVICE_STOPPED) (followed by a readback) as asynchronous is a good idea.
BTW regarding 'The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the FEATURES_OK device status bit when the driver writes it.' I'm a bit puzzled what 'must fail to set' actually means. From CPU perspective the 'set the FEATURE_OK status bit' boils down to an instruction generated for iowrite8. As far as I understand this instruction is not supposed to fail (on s390 there are multiple ways in which a pci store can fail). AFAIK x86 uses normal memory access instructions. I suppose when a store instruction does not fail, then the store must happen. Or is that only true for RAM and for PCI memory not?
Quoted 10 lines
Quoted 5 lines
Which makes me think: is using the device status a good universal method for stopping the device?
This part I don't understand. Device stop is kind of similar to reset in this case. If reset can work what prevent stop work?
Quoted 9 lines
E.g:
ccw_io_helper(VIRTIO_CCW_DOING_WRITE_STATUS);
while (1) {
ccw_io_helper(VIRTIO_CCW_DOING_READ_STATUS);
if (status &DEVICE_STOPPED)
break;
}
You are right I think. It can work for virtio-ccw, if setting DEVICE_STOPPED is synchronous. For reset we use the CCW_CMD_VDEV_RESET ccw command (see in the spec), but that is a design choice. I think setting the status to 0 via CCW_CMD_WRITE_STATUS should also work. The code below is from qemu and you should look for the invocation of virtio_ccw_reset_virtio(). I guess, that is an undocumented feature. And we can get away with setting the status first, and then doing the reset here, because the implementation guarantees that the driver won't see the new status value until the reset is actually done. (This is not a property of the architecture, but a property of the implementation).
case CCW_CMD_WRITE_STATUS:
if (check_len) {
if (ccw.count != sizeof(status)) {
ret = -EINVAL;
break;
}
} else if (ccw.count < sizeof(status)) {
/* Can't execute command. */
ret = -EINVAL;
break;
}
if (!ccw.cda) {
ret = -EFAULT;
} else {
ccw_dstream_read(&sch->cds, status);
if (!(status & VIRTIO_CONFIG_S_DRIVER_OK)) {
virtio_ccw_stop_ioeventfd(dev);
}
if (virtio_set_status(vdev, status) == 0) {
if (vdev->status == 0) {
virtio_ccw_reset_virtio(dev, vdev);
}
if (status & VIRTIO_CONFIG_S_DRIVER_OK) {
virtio_ccw_start_ioeventfd(dev);
}
sch->curr_status.scsw.count = ccw.count - sizeof(status);
ret = 0;
} else {
/* Trigger a command reject. */
ret = -ENOSYS;
}
}Quoted 10 lines
Quoted 7 lines
If I look at it only from the CCW
perspective, I'd have used a new channel command with a payload of stop/resume and only reflected the stopped state in the device status (i.e. read-only from the driver, and only changed by the device when it transitions to/from the stopped state.) Could PCI use a new field for that?PCI can use new filed. But I wonder how can that help for CCW.
Using a new mechanism to request the device to stop would help with the synchronous vs asynchronous stuff. I.e. we could keep the FEATURES_OK synchronous, and make REQUEST_STOP asynchronous.
A new PCI field would have the downside, of being PCI specific. This feature makes sense to me regardless of the transport. But I don't think it is likely to become relevant for the ccw any time soon.
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 3 lines · Halil Pasic
When driver is trying to set
DEVICE_STOPPED, the device MUST not process new avail requests and MUST complete all requests that is currently processing before settingDEVICE_STOPPED.
...
Quoted 2 lines
Since we have a synchronous API setting
DEVICE_STOPPEDwould also have to block until all in-flight requests are completed.
Judging from the surrounding discussion,
when you say complete you seem to mean "use", and I'm not sure how you define in flight, but
it seems there could be many of these (up to a full queue?) so waiting for all available buffers to be
used might indeed require an asynchronous interface, which gets complex very quickly.
However wouldn't it be possible for device to just cancel processing available buffers even if it started processing them? Some buffers could be in indeterminate state (e.g. we might not have a way to know how much data did device have time to write into a buffer).
Making it clearer what does "complete" mean here might help.
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Quoted 5 lines · On Sun, 27 Dec 2020 05:00:05 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 3 lines · Halil Pasic
When driver is trying to set
DEVICE_STOPPED, the device MUST not process new avail requests and MUST complete all requests that is currently processing before settingDEVICE_STOPPED.
Quoted 7 lines
...
Quoted 2 lines
Since we have a synchronous API setting
DEVICE_STOPPEDwould also have to block until all in-flight requests are completed.Judging from the surrounding discussion,
when you say complete you seem to mean "use", and
My intention was merely to paraphrase Jason's proposal which says:
+When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED.
Quoted 2 lines
I'm not sure how you define in flight, but
it seems there could be many of these (up to a full queue?)
In the follow-on discussion 'in-flight' emerged as an alternative to 'all requests that is currently processing' form the proposed text.
A large part of the discussion is, IMHO, about finding precise definitions, for what the quoted paragraph is trying to express.
Quoted 10 lines
so waiting for all available buffers to be used might indeed require an asynchronous interface, which gets complex very quickly.
However wouldn't it be possible for device to just cancel processing available buffers even if it started processing them? Some buffers could be in indeterminate state (e.g. we might not have a way to know how much data did device have time to write into a buffer).
Maybe Jason can answer this one.
Quoted 2 lines
Making it clearer what does "complete" mean here might help.
I think what we are trying to accomplish here, is avoiding, having non-standardised state in device (that can not be dropped) when migrating.
I'm still struggling with wrapping my mind around the problem. AFAIU migration and migratability is not really a feature of the virtio standard, but can be a feature of it's implementation
(e.g. QEMU & KVM), where the standard does help a great deal by having certain aspects of the operation and interaction nailed down.
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 19 lines · Halil Pasic
Quoted 10 lines · On Sun, 27 Dec 2020 05:00:05 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 3 lines · Halil Pasic
When driver is trying to set
DEVICE_STOPPED, the device MUST not process new avail requests and MUST complete all requests that is currently processing before settingDEVICE_STOPPED....
Quoted 2 lines
Since we have a synchronous API setting
DEVICE_STOPPEDwould also have to block until all in-flight requests are completed.Judging from the surrounding discussion,
when you say complete you seem to mean "use", andMy intention was merely to paraphrase Jason's proposal which says:
(quoted diff) +3
-0+When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED.
Quoted 11 lines
Quoted 2 lines
I'm not sure how you define in flight, but
it seems there could be many of these (up to a full queue?)In the follow-on discussion 'in-flight' emerged as an alternative to 'all requests that is currently processing' form the proposed text.
A large part of the discussion is, IMHO, about finding precise definitions, for what the quoted paragraph is trying to express.
Quoted 3 lines
so waiting for all available buffers to be used might indeed require an asynchronous interface, which gets complex very quickly.
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (0x0) to this QueueReady and MUST read the value back to ensure synchronization.
"""
For PCI it said:
"""
After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.
"""
Quoted 8 lines
Quoted 7 lines
However wouldn't it be possible for device to just cancel processing available buffers even if it started processing them? Some buffers could be in indeterminate state (e.g. we might not have a way to know how much data did device have time to write into a buffer).
Maybe Jason can answer this one.
For networking device, it should be possible (packet could be dropped or duplicated). But I'm not sure it works for other device. E.g for the block devices that needs to communicate with a remote backend. Waat's more important, my understadning is that the intermediate state is something that we need to avoid (hard to be migrated anyhow).
Quoted 7 lines
Quoted 3 lines
Making it clearer what does "complete" mean here might help.
I think what we are trying to accomplish here, is avoiding, having non-standardised state in device (that can not be dropped) when migrating.
Quoted 5 lines
I'm still struggling with wrapping my mind around the problem. AFAIU migration and migratability is not really a feature of the virtio standard, but can be a feature of it's implementation
(e.g. QEMU & KVM), where the standard does help a great deal by having certain aspects of the operation and interaction nailed down.
It looks like a must (at least from the level of semantics) for designing software API for vDPA. Using a standard spec may help to avoid subtle misunderstanding of different vendors.
Thanks
Quoted 4 lines
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 45 lines · Jason Wang
Quoted 19 lines · Halil Pasic
Quoted 10 lines · On Sun, 27 Dec 2020 05:00:05 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 3 lines · Halil Pasic
When driver is trying to set
DEVICE_STOPPED, the device MUST not process new avail requests and MUST complete all requests that is currently processing before settingDEVICE_STOPPED....
Quoted 2 lines
Since we have a synchronous API setting
DEVICE_STOPPEDwould also have to block until all in-flight requests are completed.Judging from the surrounding discussion,
when you say complete you seem to mean "use", andMy intention was merely to paraphrase Jason's proposal which says:
(quoted diff) +3
-0+When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED.Quoted 11 lines
Quoted 2 lines
I'm not sure how you define in flight, but
it seems there could be many of these (up to a full queue?)In the follow-on discussion 'in-flight' emerged as an alternative to 'all requests that is currently processing' form the proposed text.
A large part of the discussion is, IMHO, about finding precise definitions, for what the quoted paragraph is trying to express.
Quoted 3 lines
so waiting for all available buffers to be used might indeed require an asynchronous interface, which gets complex very quickly.
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (
0x0) to this QueueReady and MUST read the value back to ensure synchronization."""
Quoted 8 lines
For PCI it said:
"""
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Absolutely. It's ok for simple things.
However doing anything major like waiting for IO
in a tight loop is a problem.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
However wouldn't it be possible for device to just cancel processing available buffers even if it started processing them? Some buffers could be in indeterminate state (e.g. we might not have a way to know how much data did device have time to write into a buffer).
Quoted 8 lines
Quoted 1 line
Maybe Jason can answer this one.
For networking device, it should be possible (packet could be dropped or duplicated). But I'm not sure it works for other device. E.g for the block devices that needs to communicate with a remote backend. Waat's more important, my understadning is that the intermediate state is something that we need to avoid (hard to be migrated anyhow).
The difficulty is on the device side though, so why not say: if it wants to flush IO that's up to it.
Quoted 15 lines
Quoted 8 lines
Quoted 4 lines
Making it clearer what does "complete" mean here might help.
I think what we are trying to accomplish here, is avoiding, having non-standardised state in device (that can not be dropped) when migrating.
Quoted 5 lines
I'm still struggling with wrapping my mind around the problem. AFAIU migration and migratability is not really a feature of the virtio standard, but can be a feature of it's implementation
(e.g. QEMU & KVM), where the standard does help a great deal by having certain aspects of the operation and interaction nailed down.
Quoted 11 lines
It looks like a must (at least from the level of semantics) for designing software API for vDPA. Using a standard spec may help to avoid subtle misunderstanding of different vendors.
Thanks
Quoted 4 lines
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
----- Original Message -----
Quoted 52 lines · Jason Wang
Quoted 51 lines · Halil Pasic
Quoted 20 lines
Quoted 11 lines · On Sun, 27 Dec 2020 05:00:05 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 3 lines · Halil Pasic
When driver is trying to set
DEVICE_STOPPED, the device MUST not process new avail requests and MUST complete all requests that is currently processing before settingDEVICE_STOPPED....
Quoted 3 lines
Since we have a synchronous API setting
DEVICE_STOPPEDwould also have to
block until all in-flight requests are completed.Judging from the surrounding discussion,
when you say complete you seem to mean "use", andMy intention was merely to paraphrase Jason's proposal which says:
(quoted diff) +3
-0+When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED.Quoted 11 lines
Quoted 2 lines
I'm not sure how you define in flight, but
it seems there could be many of these (up to a full queue?)In the follow-on discussion 'in-flight' emerged as an alternative to 'all requests that is currently processing' form the proposed text.
A large part of the discussion is, IMHO, about finding precise definitions, for what the quoted paragraph is trying to express.
Quoted 3 lines
so waiting for all available buffers to be used might indeed require an asynchronous interface, which gets complex very quickly.
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (
0x0) to this QueueReady and MUST read the value back to ensure synchronization."""
For PCI it said:
"""
Quoted 8 lines
Quoted 4 lines
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Absolutely. It's ok for simple things.
However doing anything major like waiting for IO
in a tight loop is a problem.
It's mainly used for Qemu. And driver can choose to cancel the stop by trying to clear the bit. Does this work?
Quoted 21 lines
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 6 lines
However wouldn't it be possible for device to just cancel processing available buffers even if it started processing them? Some buffers could be in indeterminate state (e.g. we might not have a way to know how much data did device have time to write into a buffer).
Maybe Jason can answer this one.
Quoted 6 lines
For networking device, it should be possible (packet could be dropped or duplicated). But I'm not sure it works for other device. E.g for the block devices that needs to communicate with a remote backend. Waat's more important, my understadning is that the intermediate state is something that
we need to avoid (hard to be migrated anyhow).
Quoted 2 lines
The difficulty is on the device side though, so why not say: if it wants to flush IO that's up to it.
Do you mean to introduce a device specific way to flush IO? If yes, it actually introduces a new intermediate state implicitly:
1) device is stopped
2) device is stopped and IO is flushed
This looks more complicated than a single new state:
1) device is stopped and IO is flushed
Thanks
Quoted 20 lines
Quoted 20 lines
Quoted 9 lines
Quoted 5 lines
Making it clearer what does "complete" mean here might help.
I think what we are trying to accomplish here, is avoiding, having non-standardised state in device (that can not be dropped) when migrating.
Quoted 5 lines
I'm still struggling with wrapping my mind around the problem. AFAIU migration and migratability is not really a feature of the virtio standard, but can be a feature of it's implementation
(e.g. QEMU & KVM), where the standard does help a great deal by having certain aspects of the operation and interaction nailed down.It looks like a must (at least from the level of semantics) for designing software API for vDPA. Using a standard spec may help to avoid subtle misunderstanding of different vendors.
Quoted 9 lines
Quoted 7 lines
Thanks
Quoted 4 lines
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 22 lines · Jason Wang
Quoted 16 lines
Quoted 9 lines
Quoted 7 lines
Quoted 6 lines
However wouldn't it be possible for device to just cancel processing available buffers even if it started processing them? Some buffers could be in indeterminate state (e.g. we might not have a way to know how much data did device have time to write into a buffer).
Maybe Jason can answer this one.
Quoted 6 lines
For networking device, it should be possible (packet could be dropped or duplicated). But I'm not sure it works for other device. E.g for the block devices that needs to communicate with a remote backend. Waat's more important, my understadning is that the intermediate state is something that
we need to avoid (hard to be migrated anyhow).Quoted 2 lines
The difficulty is on the device side though, so why not say: if it wants to flush IO that's up to it.
Do you mean to introduce a device specific way to flush IO? If yes, it actually introduces a new intermediate state implicitly:
Quoted 8 lines
1) device is stopped
2) device is stopped and IO is flushedThis looks more complicated than a single new state:
1) device is stopped and IO is flushed
Thanks
I thought some more about it, if we call a buffer which was made available by driver but not yet used by device, a cancelled buffer, then maybe we can specify
the behaviour by explaining that it should be possible to retry any cancelled buffer, that is make it available again, and achieve the same effect as if it was used.
For example, any reads without side effects can be just ignored even if they might have modified guest memory.
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
However wouldn't it be possible for device to just cancel processing available buffers even if it started processing them? Some buffers could be in indeterminate state (e.g. we might not have a way to know how much data did device have time to write into a buffer).
Maybe Jason can answer this one.
Quoted 8 lines
Quoted 6 lines
For networking device, it should be possible (packet could be dropped or duplicated). But I'm not sure it works for other device. E.g for the block devices that needs to communicate with a remote backend. Waat's more important, my understadning is that the intermediate state is something that
we need to avoid (hard to be migrated anyhow).The difficulty is on the device side though, so why not say: if it wants to flush IO that's up to it.
Do you mean to introduce a device specific way to flush IO? If yes, it actually introduces a new intermediate state implicitly:
1) device is stopped
2) device is stopped and IO is flushed
Quoted 7 lines
Quoted 5 lines
This looks more complicated than a single new state:
1) device is stopped and IO is flushed
Thanks
I thought some more about it, if we call a buffer which was made available by driver but not yet used by device,
For "used by device" you meant posting entries to used ring?
Quoted 6 lines
a cancelled buffer, then maybe we can specify
the behaviour by explaining that it should be possible to retry any cancelled buffer, that is make it available again, and achieve the same effect as if it was used.
For example, any reads without side effects can be just ignored even if they might have modified guest memory.
Right, but how about the buffer that can't be retried? Or you meant we leave that to a specific device to handle?
THanks
Quoted 1 line
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 10 lines · On Mon, 28 Dec 2020 15:01:57 +0800 Jason Wang <[email protected]> wrote:
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (
0x0) to this QueueReady and MUST read the value back to ensure synchronization."""
I read the MMIO quote like a single read is sufficient to ensure synchronization. I.e. it does not require a loop which waits for the read to yield the expected value.
Quoted 9 lines
For PCI it said:
"""
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Yes, this does sound like a loop. And that's what Linux does. But this is transport (PCI) specific. On a spec level, a reset is a distinct operation from setting device status (to 0). It just happens to be mapped to the PCI transport as setting the status to 0.
For the channel IO transport it is mapped via CCW_CMD_VDEV_RESET while setting the status is mapped via CCW_CMD_WRITE_STATUS.
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 21 lines · Halil Pasic
Quoted 10 lines · On Mon, 28 Dec 2020 15:01:57 +0800 Jason Wang <[email protected]> wrote:
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (
0x0) to this QueueReady and MUST read the value back to ensure synchronization."""
I read the MMIO quote like a single read is sufficient to ensure synchronization. I.e. it does not require a loop which waits for the read to yield the expected value.
Quoted 4 lines
For PCI it said:
"""
Quoted 7 lines
Quoted 4 lines
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Yes, this does sound like a loop. And that's what Linux does. But this is transport (PCI) specific. On a spec level, a reset is a distinct operation from setting device status (to 0).
I'm not sure or it looks unclear to me for this point.
E.g the device reset is mentioned in "Device Status Field" belongs to "Basic Facilities of a Virtio Device". But there's no "Device Reset" in "Basic Facilities of a Virtio Device".
Quoted 2 lines
It just happens to be
mapped to the PCI transport as setting the status to 0.
MMIO did the same. And it makes sense since using a single API to configure/change device status looks simpler.
Quoted 3 lines
For the channel IO transport it is mapped via
CCW_CMD_VDEV_RESETwhile setting the status is mapped viaCCW_CMD_WRITE_STATUS.
Yes, but I think actually there's no limitation if we want to tread 0 as a reset command for CCW_CMD_WRITE_STATUS.
Thanks
Quoted 4 lines
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 31 lines · On Wed, 30 Dec 2020 16:15:47 +0800 Jason Wang <[email protected]> wrote:
Quoted 21 lines · Halil Pasic
Quoted 10 lines · On Mon, 28 Dec 2020 15:01:57 +0800 Jason Wang <[email protected]> wrote:
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (
0x0) to this QueueReady and MUST read the value back to ensure synchronization."""
I read the MMIO quote like a single read is sufficient to ensure synchronization. I.e. it does not require a loop which waits for the read to yield the expected value.
Quoted 4 lines
For PCI it said:
"""
Quoted 7 lines
Quoted 4 lines
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Yes, this does sound like a loop. And that's what Linux does. But this is transport (PCI) specific. On a spec level, a reset is a distinct operation from setting device status (to 0).
Quoted 5 lines
I'm not sure or it looks unclear to me for this point.
E.g the device reset is mentioned in "Device Status Field" belongs to "Basic Facilities of a Virtio Device". But there's no "Device Reset" in "Basic Facilities of a Virtio Device".
I think it should be, just to make clear what a driver-initiated reset of a device actually resets (and that the method for doing so is transport-specific.)
Quoted 11 lines
Quoted 4 lines
It just happens to be
mapped to the PCI transport as setting the status to 0.MMIO did the same. And it makes sense since using a single API to configure/change device status looks simpler.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
For the channel IO transport it is mapped via
CCW_CMD_VDEV_RESETwhile setting the status is mapped viaCCW_CMD_WRITE_STATUS.Yes, but I think actually there's no limitation if we want to tread 0 as a reset command for
CCW_CMD_WRITE_STATUS.
I'm not a fan of the "driver writes 0 to status to initiate a device reset" method, but we are stuck with it. I think it's actually not working well with two other requirements in the spec:
- "The driver MUST NOT clear a device status bit."
- "The device MUST initialize device status to 0 upon reset." (This suggests to me that a zero status is something set by the device as the result of a reset request by the driver, and _not_ set by the driver.)
Also, treating CCW_CMD_WRITE_STATUS with 0 as a reset request would be incompatible with older devices, wouldn't it?
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 33 lines · Cornelia Huck
Quoted 30 lines · On Wed, 30 Dec 2020 16:15:47 +0800 Jason Wang <[email protected]> wrote:
Quoted 28 lines · Halil Pasic
Quoted 10 lines · On Mon, 28 Dec 2020 15:01:57 +0800 Jason Wang <[email protected]> wrote:
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (
0x0) to this QueueReady and MUST read the value back to ensure synchronization."""
I read the MMIO quote like a single read is sufficient to ensure synchronization. I.e. it does not require a loop which waits for the read to yield the expected value.
Quoted 8 lines
For PCI it said:
"""
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Yes, this does sound like a loop. And that's what Linux does. But this is transport (PCI) specific. On a spec level, a reset is a distinct operation from setting device status (to 0).
Quoted 8 lines
Quoted 5 lines
I'm not sure or it looks unclear to me for this point.
E.g the device reset is mentioned in "Device Status Field" belongs to "Basic Facilities of a Virtio Device". But there's no "Device Reset" in "Basic Facilities of a Virtio Device".
I think it should be, just to make clear what a driver-initiated reset of a device actually resets (and that the method for doing so is transport-specific.)
Then we need some clarifications in the spec. It would be easily imply that reset is part of device status after reading "Basic Facilities of a Virtio Device".
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
It just happens to be
mapped to the PCI transport as setting the status to 0.MMIO did the same. And it makes sense since using a single API to configure/change device status looks simpler.
Quoted 2 lines
For the channel IO transport it is mapped via
CCW_CMD_VDEV_RESETwhile setting the status is mapped viaCCW_CMD_WRITE_STATUS.
Quoted 7 lines
Quoted 2 lines
Yes, but I think actually there's no limitation if we want to tread 0 as a reset command for
CCW_CMD_WRITE_STATUS.I'm not a fan of the "driver writes 0 to status to initiate a device reset" method, but we are stuck with it. I think it's actually not working well with two other requirements in the spec:
- "The driver MUST NOT clear a device status bit."
Yes, that's kind of conflict but it was how PCI works now ...
Quoted 4 lines
- "The device MUST initialize device status to 0 upon reset." (This suggests to me that a zero status is something set by the device as the result of a reset request by the driver, and _not_ set by the driver.)
Not a native speaker but we probably need to define "set" and "clear" mean, E.g:
Clear a bit from driver means write with that bit cleared. Set a bit from driver means write with that bit set. Clear a bit from device means read with that be cleared. Set a bit from device means read with that bit set.
So a question here is how to trigger the device set as we discussed before.
Quoted 3 lines
Also, treating
CCW_CMD_WRITE_STATUSwith 0 as a reset request would be incompatible with older devices, wouldn't it?
Probably, but we can make both methods work. Note that I'm not suggesting doing something like this. Just to point out it may work. And there's something not clear in the spec.
Thanks
Quoted 9 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 39 lines · On Tue, 12 Jan 2021 11:27:20 +0800 Jason Wang <[email protected]> wrote:
Quoted 38 lines · Cornelia Huck
Quoted 35 lines · On Wed, 30 Dec 2020 16:15:47 +0800 Jason Wang <[email protected]> wrote:
Quoted 21 lines · Halil Pasic
Quoted 10 lines · On Mon, 28 Dec 2020 15:01:57 +0800 Jason Wang <[email protected]> wrote:
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (
0x0) to this QueueReady and MUST read the value back to ensure synchronization."""
I read the MMIO quote like a single read is sufficient to ensure synchronization. I.e. it does not require a loop which waits for the read to yield the expected value.
Quoted 4 lines
For PCI it said:
"""
Quoted 7 lines
Quoted 4 lines
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Yes, this does sound like a loop. And that's what Linux does. But this is transport (PCI) specific. On a spec level, a reset is a distinct operation from setting device status (to 0).
I'm not sure or it looks unclear to me for this point.
E.g the device reset is mentioned in "Device Status Field" belongs to "Basic Facilities of a Virtio Device". But there's no "Device Reset" in "Basic Facilities of a Virtio Device".
Quoted 8 lines
Quoted 3 lines
I think it should be, just to make clear what a driver-initiated reset of a device actually resets (and that the method for doing so is transport-specific.)
Then we need some clarifications in the spec. It would be easily imply that reset is part of device status after reading "Basic Facilities of a Virtio Device".
I can try to come up with a patch.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
It just happens to be
mapped to the PCI transport as setting the status to 0.MMIO did the same. And it makes sense since using a single API to configure/change device status looks simpler.
Quoted 2 lines
For the channel IO transport it is mapped via
CCW_CMD_VDEV_RESETwhile setting the status is mapped viaCCW_CMD_WRITE_STATUS.Yes, but I think actually there's no limitation if we want to tread 0 as a reset command for
CCW_CMD_WRITE_STATUS.
Quoted 8 lines
Quoted 5 lines
I'm not a fan of the "driver writes 0 to status to initiate a device reset" method, but we are stuck with it. I think it's actually not working well with two other requirements in the spec:
- "The driver MUST NOT clear a device status bit."
Yes, that's kind of conflict but it was how PCI works now ...
Nod.
Quoted 8 lines
Quoted 6 lines
- "The device MUST initialize device status to 0 upon reset." (This suggests to me that a zero status is something set by the device as the result of a reset request by the driver, and _not_ set by the driver.)
Quoted 5 lines
Not a native speaker but we probably need to define "set" and "clear" mean, E.g:
Clear a bit from driver means write with that bit cleared. Set a bit from driver means write with that bit set.
Agreed.
Quoted 2 lines
Clear a bit from device means read with that be cleared. Set a bit from device means read with that bit set.
I always saw the status as an actual part of the device. I.e., the device setting or clearing a bit means that the device is modifying the status in the device. The driver doing a read will get the status as it currently is within the device. IOW, the device "controls" the status, and the driver can submit changes to the status.
[Maybe that's closer to how ccw operates with its commands for reading or writing the status. Are pci/mmio envisioning the status more like some kind of shared area?]
Quoted 5 lines
So a question here is how to trigger the device set as we discussed before.
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Also, treating
CCW_CMD_WRITE_STATUSwith 0 as a reset request would be incompatible with older devices, wouldn't it?Probably, but we can make both methods work. Note that I'm not suggesting doing something like this. Just to point out it may work. And there's something not clear in the spec.
I'm still not convinced that's the way to go. But first, we should be clear on how the status really is supposed to work.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 41 lines · Cornelia Huck
Quoted 38 lines · On Tue, 12 Jan 2021 11:27:20 +0800 Jason Wang <[email protected]> wrote:
Quoted 37 lines · Cornelia Huck
Quoted 34 lines · On Wed, 30 Dec 2020 16:15:47 +0800 Jason Wang <[email protected]> wrote:
Quoted 21 lines · Halil Pasic
Quoted 10 lines · On Mon, 28 Dec 2020 15:01:57 +0800 Jason Wang <[email protected]> wrote:
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (
0x0) to this QueueReady and MUST read the value back to ensure synchronization."""
I read the MMIO quote like a single read is sufficient to ensure synchronization. I.e. it does not require a loop which waits for the read to yield the expected value.
Quoted 4 lines
For PCI it said:
"""
Quoted 7 lines
Quoted 4 lines
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Yes, this does sound like a loop. And that's what Linux does. But this is transport (PCI) specific. On a spec level, a reset is a distinct operation from setting device status (to 0).
I'm not sure or it looks unclear to me for this point.
E.g the device reset is mentioned in "Device Status Field" belongs to "Basic Facilities of a Virtio Device". But there's no "Device Reset" in "Basic Facilities of a Virtio Device".
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
I think it should be, just to make clear what a driver-initiated reset of a device actually resets (and that the method for doing so is transport-specific.)
Then we need some clarifications in the spec. It would be easily imply that reset is part of device status after reading "Basic Facilities of a Virtio Device".
I can try to come up with a patch.
Thanks
Quoted 44 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
It just happens to be
mapped to the PCI transport as setting the status to 0.MMIO did the same. And it makes sense since using a single API to configure/change device status looks simpler.
Quoted 2 lines
For the channel IO transport it is mapped via
CCW_CMD_VDEV_RESETwhile setting the status is mapped viaCCW_CMD_WRITE_STATUS.Yes, but I think actually there's no limitation if we want to tread 0 as a reset command for
CCW_CMD_WRITE_STATUS.Quoted 7 lines
Quoted 5 lines
I'm not a fan of the "driver writes 0 to status to initiate a device reset" method, but we are stuck with it. I think it's actually not working well with two other requirements in the spec:
- "The driver MUST NOT clear a device status bit."
Yes, that's kind of conflict but it was how PCI works now ...
Nod.
Quoted 11 lines
Quoted 5 lines
- "The device MUST initialize device status to 0 upon reset." (This suggests to me that a zero status is something set by the device as the result of a reset request by the driver, and _not_ set by the driver.)
Not a native speaker but we probably need to define "set" and "clear" mean, E.g:
Clear a bit from driver means write with that bit cleared. Set a bit from driver means write with that bit set.
Agreed.
Quoted 2 lines
Clear a bit from device means read with that be cleared. Set a bit from device means read with that bit set.
I always saw the status as an actual part of the device. I.e., the device setting or clearing a bit means that the device is modifying the status in the device. The driver doing a read will get the status as it currently is within the device. IOW, the device "controls" the status, and the driver can submit changes to the status.
Quoted 3 lines
[Maybe that's closer to how ccw operates with its commands for reading or writing the status. Are pci/mmio envisioning the status more like some kind of shared area?]
My understanding is it's not a shared area, each MMIO read and write will generate a PCI transaction on the bus.
Some developers upstream told me that in the PCIE Endpoint device, the MMIO area could be a shared memory from the view of endpoint device which makes it very hard to make current virtio-pci transport work[1].
[1] https://lore.kernel.org/linux-pci/[email protected]/T/
Quoted 4 lines
Quoted 4 lines
So a question here is how to trigger the device set as we discussed before.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Also, treating
CCW_CMD_WRITE_STATUSwith 0 as a reset request would be incompatible with older devices, wouldn't it?Probably, but we can make both methods work. Note that I'm not suggesting doing something like this. Just to point out it may work. And there's something not clear in the spec.
I'm still not convinced that's the way to go. But first, we should be clear on how the status really is supposed to work.
Right, I meant we had two ways to make the implement aligned with the spec:
1) implement zero write as reset (and maybe keep the old way), so we don't need to modify the spec
2) clarify that transport specific device reset is part of the basic facilities
Either is fine I think.
Thanks
Quoted 9 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 46 lines · On Wed, 13 Jan 2021 10:52:32 +0800 Jason Wang <[email protected]> wrote:
Quoted 45 lines · Cornelia Huck
Quoted 42 lines · On Tue, 12 Jan 2021 11:27:20 +0800 Jason Wang <[email protected]> wrote:
Quoted 40 lines · Cornelia Huck
Quoted 26 lines · On Wed, 30 Dec 2020 16:15:47 +0800 Jason Wang <[email protected]> wrote:
Quoted 25 lines · Halil Pasic
Quoted 10 lines · On Mon, 28 Dec 2020 15:01:57 +0800 Jason Wang <[email protected]> wrote:
Some part of the virtio has enforced an asynchronous interface during reset:
For MMIO the spec said:
"""
To stop using the queue the driver MUST write zero (
0x0) to this QueueReady and MUST read the value back to ensure synchronization."""
I read the MMIO quote like a single read is sufficient to ensure synchronization. I.e. it does not require a loop which waits for the read to yield the expected value.
Quoted 8 lines
For PCI it said:
"""
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Quoted 8 lines
Quoted 3 lines
Yes, this does sound like a loop. And that's what Linux does. But this is transport (PCI) specific. On a spec level, a reset is a distinct operation from setting device status (to 0).
I'm not sure or it looks unclear to me for this point.
E.g the device reset is mentioned in "Device Status Field" belongs to "Basic Facilities of a Virtio Device". But there's no "Device Reset" in "Basic Facilities of a Virtio Device".
I think it should be, just to make clear what a driver-initiated reset of a device actually resets (and that the method for doing so is transport-specific.)
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Then we need some clarifications in the spec. It would be easily imply that reset is part of device status after reading "Basic Facilities of a Virtio Device".
I can try to come up with a patch.
Thanks
Attempt here:
https://lists.oasis-open.org/archives/virtio-comment/202101/msg00020.html
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 48 lines · Halil Pasic
On Wed, 23 Dec 2020 10:48:45 +0800
Jason Wang <[email protected]> wrote:
[..]Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 6 lines
Quoted 3 lines
Is it for hardware devices (e.g. PCI) standard to request an operation by writing some value into a register, and get feedback bout a non-completion by reading different value that written,
This is not ununsal in other devices. And in fact, the
FEATURES_OKworks like this:Quoted 9 lines
"""
The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the
FEATURES_OKdevice status bit when the driver writes it."""
Thanks for the pointer. I intend to have another look at how
FEATURES_OKworks, and how similar this is toDEVICE_STOPPED.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
My understanding is that for both of them, driver can try to set the bit by writing to the status register but it's the device that decide when to set the bit.
I think there's a difference: For
FEATURES_OK, the driver can read back the status and knows that the feature combination is not accepted if it is not set. ForDEVICE_STOPPED, it does not really know whether the device has started to stop yet. It only knows that the device is actually done stopping whenDEVICE_STOPPEDis set.I share Connie's concern. The
config->set_status()is a synchronous API, that is right after the iowrite8 for pci comes back, or the channel program finishes executing for ccw, we can go, do aconfig->get_status()and know if the status was set succesfully.Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
The only difference is that device may need sometime to be stopped.
FEATURES_OKmight not be the best example. Let's see how reset work which is more similar toDEVICE_STOPPED:"""
After writing 0 to
device_status, the driver MUST wait for a read ofdevice_statusto return 0 before reinitializing the device."""
Hm, but that is PCI-specific writing of fields in the configuration structure, right? CCW uses a different method for resetting (an asynchronous channel command; channel commands create an interrupt when done); "write 0 to status for reset" is not universal.
Quoted 8 lines
Quoted 3 lines
I may miss something. Even if CCW is using something asynchronously (e.g interrupt). It still present a synchronous API to the upper layer. E.g
ccw_io_helper()is usingwait_event()?Right. The reason why we have to wait for the completion of the channel program is exactly that. Furthermore if we have to fail setting
FEATURES_OKwe fail the channel program, that was supposed to do the operation with an unit check. So at the point where theset_status()is done, we already know if it worked or not.
The only difference is, for FEATURES_OK, driver know it won't work any more. But for DEVICE_STOPPED, driver know it might work sometime in the future.
Quoted 9 lines
Of course the device can
change it's status any time.Since we have a synchronous API setting
DEVICE_STOPPEDwould also have to block until all in-flight requests are completed. But this does not see to be what you want. I understood thatvirtio_add_status(dev, DEVICE_STOPPED)would return immediately, and so would a subsequentconfig->get_status(dev)without theDEVICE_STOPPEDbit set, if there are still outstanding requests.
Yes.
Quoted 4 lines
I don't think defining
virtio_add_status(dev, FEATURES_OK)(followed by a readback) as synchronous andvirtio_add_status(dev, DEVICE_STOPPED)(followed by a readback) as asynchronous is a good idea.
The main reason is that DEVICE_STOPPED is a one of the possible state. We'd better do this by extending the current device status state machine. Otherwise it would be more complicated (e.g using another interface).
It looks to me what you don't like is the driver API design? If yes, how about introduce something like virtio_try_add_status()? The difference is that it doesn't require a immediate result.
Quoted 5 lines
BTW regarding 'The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the
FEATURES_OKdevice status bit when the driver writes it.' I'm a bit puzzled what 'must fail to set' actually means.
Good point, my understanding for "set XX bit" means the whether or not the bit could be read by the driver. We need some clarification here.
Quoted 4 lines
From CPU perspective the 'set the
FEATURE_OK
status bit' boils down to an instruction generated for iowrite8. As far as I understand this instruction is not supposed to fail (on s390 there are multiple ways in which a pci store can fail).
Yes, my understanding is that it doesn't mean fail the instruction. As you said it's sometime impossible a specific transport.
Quoted 4 lines
AFAIK x86 uses normal
memory access instructions. I suppose when a store instruction does not fail, then the store must happen. Or is that only true for RAM and for PCI memory not?
AFAIK, PCI have non-posted write which can fail but the spec doesn't requires that.
Quoted 18 lines
Quoted 9 lines
Quoted 5 lines
Which makes me think: is using the device status a good universal method for stopping the device?
This part I don't understand. Device stop is kind of similar to reset in this case. If reset can work what prevent stop work?
Quoted 9 lines
E.g:
ccw_io_helper(VIRTIO_CCW_DOING_WRITE_STATUS); while (1) {
ccw_io_helper(VIRTIO_CCW_DOING_READ_STATUS); if (status &DEVICE_STOPPED)
break;
}
Quoted 6 lines
You are right I think. It can work for virtio-ccw, if setting
DEVICE_STOPPEDis synchronous. For reset we use theCCW_CMD_VDEV_RESETccw command (see in the spec), but that is a design choice. I think setting the status to 0 viaCCW_CMD_WRITE_STATUSshould also work. The code below is from qemu and you should look for the invocation ofvirtio_ccw_reset_virtio(). I guess, that is an undocumented feature.
Then I think it needs to be clarified in the spec.
Quoted 5 lines
And
we can get away with setting the status first, and then doing the reset here, because the implementation guarantees that the driver won't see the new status value until the reset is actually done. (This is not a property of the architecture, but a property of the implementation).
So did PCI:
case VIRTIO_PCI_STATUS:
if (!(val & VIRTIO_CONFIG_S_DRIVER_OK)) {
virtio_pci_stop_ioeventfd(proxy);
}
virtio_set_status(vdev, val & 0xFF);
if (val & VIRTIO_CONFIG_S_DRIVER_OK) {
virtio_pci_start_ioeventfd(proxy);
}
if (vdev->status == 0) {
virtio_pci_reset(DEVICE(proxy));
}
As you said, it's a implementation property since qemu implements a synchronous virtio_set_status(). But it might not be true for real hardware which may require more time or even hard to drain a queue in short time. That's why PCI spec mandate the driver to poll for the status after writing zero to device status.
Quoted 42 lines
case CCW_CMD_WRITE_STATUS: if (check_len) { if (ccw.count != sizeof(status)) { ret = -EINVAL; break; } } else if (ccw.count < sizeof(status)) { /* Can't execute command. */ ret = -EINVAL; break; } if (!ccw.cda) { ret = -EFAULT; } else { ccw_dstream_read(&sch->cds, status); if (!(status & VIRTIO_CONFIG_S_DRIVER_OK)) { virtio_ccw_stop_ioeventfd(dev); } if (virtio_set_status(vdev, status) == 0) { if (vdev->status == 0) { virtio_ccw_reset_virtio(dev, vdev); } if (status & VIRTIO_CONFIG_S_DRIVER_OK) { virtio_ccw_start_ioeventfd(dev); } sch->curr_status.scsw.count = ccw.count - sizeof(status); ret = 0; } else { /* Trigger a command reject. */ ret = -ENOSYS; } }Quoted 7 lines
Quoted 6 lines
If I look at it only from the CCW
perspective, I'd have used a new channel command with a payload of stop/resume and only reflected the stopped state in the device status (i.e. read-only from the driver, and only changed by the device when it transitions to/from the stopped state.) Could PCI use a new field for that?
Quoted 4 lines
Quoted 1 line
PCI can use new filed. But I wonder how can that help for CCW.
Using a new mechanism to request the device to stop would help with the synchronous vs asynchronous stuff. I.e. we could keep the
FEATURES_OKsynchronous, and makeREQUEST_STOPasynchronous.
Right, but as said before, it looks more like a part of device status state machine, using new mechanism might bring extra complexity and confusion.
Looking at CCW implementation of device status, I don't see any blocker for implementing it synchronously if we just wait for the stop of the device and then return?
Quoted 4 lines
A new PCI field would have the downside, of being PCI specific. This feature makes sense to me regardless of the transport. But I don't think it is likely to become relevant for the ccw any time soon.
Yes, it's better not limit the feature to a specific transport.
Thanks
Quoted 4 lines
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 1 line · On Mon, 28 Dec 2020 14:47:21 +0800 Jason Wang <[email protected]> wrote:
[..]
Quoted 11 lines
Quoted 5 lines
Right. The reason why we have to wait for the completion of the channel program is exactly that. Furthermore if we have to fail setting
FEATURES_OKwe fail the channel program, that was supposed to do the operation with an unit check. So at the point where theset_status()is done, we already know if it worked or not.The only difference is, for
FEATURES_OK, driver know it won't work any more. But forDEVICE_STOPPED, driver know it might work sometime in the future.
If the channel program fails with a command reject, that that ccw is not allowed to cause further side effects. I.e. we can not fail the channel program, but then make the operation succeed later.
Quoted 21 lines
Quoted 10 lines
Of course the device can
change it's status any time.Since we have a synchronous API setting
DEVICE_STOPPEDwould also have to block until all in-flight requests are completed. But this does not see to be what you want. I understood thatvirtio_add_status(dev, DEVICE_STOPPED)would return immediately, and so would a subsequentconfig->get_status(dev)without theDEVICE_STOPPEDbit set, if there are still outstanding requests.Yes.
Quoted 4 lines
I don't think defining
virtio_add_status(dev, FEATURES_OK)(followed by a readback) as synchronous andvirtio_add_status(dev, DEVICE_STOPPED)(followed by a readback) as asynchronous is a good idea.
Quoted 9 lines
The main reason is that
DEVICE_STOPPEDis a one of the possible state. We'd better do this by extending the current device status state machine. Otherwise it would be more complicated (e.g using another interface).It looks to me what you don't like is the driver API design? If yes, how about introduce something like
virtio_try_add_status()? The difference is that it doesn't require a immediate result.
Yes a new (Linux) driver API operation sounds good -- at least on the surface. I would rather call it virtio_add_status_async() than virtio_try_add_status() because for me 'try' tells that something may or may not work, but the outcome is decided when the function returns. E.g. try_lock(l) may or may not succeed the lock l, but when it returns you know if it did take l or not.
But how would you implement that new operation? I guess it would still have to boil down to config->set_status(dev), and to set_status(s), being asynchronous when s & DEVICE_STOPPED and synchronous otherwise (and in particular FEATURES_OK is added to the status).
Ultimately a set_status(b) boils down to a iowrite8(b) for the PCI transport, and to a channel program that references a single byte buffer with the value of b for Channel IO transport. Making that synchroneaous or asynchronous depending on the value of b does not sound right.
Maybe Conny can chime in on this later as well.
[..]
Quoted 9 lines
Quoted 4 lines
AFAIK x86 uses normal
memory access instructions. I suppose when a store instruction does not fail, then the store must happen. Or is that only true for RAM and for PCI memory not?AFAIK, PCI have non-posted write which can fail but the spec doesn't requires that.
So for a posted write a PCI device can just silently discard writes it does not 'like' (i.e. pretend there was no such transaction in the first place)?
[..]
Quoted 18 lines
So did PCI:
case
VIRTIO_PCI_STATUS:
if (!(val &VIRTIO_CONFIG_S_DRIVER_OK)) {virtio_pci_stop_ioeventfd(proxy); }
virtio_set_status(vdev, val & 0xFF);if (val &
VIRTIO_CONFIG_S_DRIVER_OK) {
virtio_pci_start_ioeventfd(proxy);
}if (
vdev->status== 0) {
virtio_pci_reset(DEVICE(proxy));
}
Quoted 6 lines
As you said, it's a implementation property since qemu implements a synchronous
virtio_set_status(). But it might not be true for real hardware which may require more time or even hard to drain a queue in short time. That's why PCI spec mandate the driver to poll for the status after writing zero to device status.
That is a PCI specific thing, and unfortunately my PCI skills are very limited.
Please compare
static void vp_reset(struct virtio_device *vdev)
{
vp_iowrite8(0, &vp_dev->common->device_status);
[..]
while (vp_ioread8(&vp_dev->common->device_status))
msleep(1);With
int virtio_finalize_features(struct virtio_device *dev) {
[..]
virtio_add_status(dev, VIRTIO_CONFIG_S_FEATURES_OK);
status = dev->config->get_status(dev);
if (!(status & VIRTIO_CONFIG_S_FEATURES_OK)) { The latter clearly assumes that the first get_status() will yield the final outcome, where the former clearly assumes that the effect of set_status() may be delayed beyond the first subsequent get_status().
Btw, if the device were to to fail resetting itself, what would you expect to happen? I would expect the device to present a status DEVICE_NEEDS_RESET, but that would effectively make vp_reset() getting stuck in that loop. Or does the spec guarantee that a reset must work eventually?
[..]
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
PCI can use new filed. But I wonder how can that help for CCW.
Using a new mechanism to request the device to stop would help with the synchronous vs asynchronous stuff. I.e. we could keep the
FEATURES_OKsynchronous, and makeREQUEST_STOPasynchronous.
Quoted 8 lines
Right, but as said before, it looks more like a part of device status state machine, using new mechanism might bring extra complexity and confusion.
Looking at CCW implementation of device status, I don't see any blocker for implementing it synchronously if we just wait for the stop of the device and then return?
I don't see a problem on the qemu side, but I do see a problem on the guest side. There is a reason why you want set DEVICE_STOPPED to be asynchronous.
Regards,
Halil
[..]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 44 lines · Halil Pasic
On Mon, 28 Dec 2020 14:47:21 +0800
Jason Wang <[email protected]> wrote:[..]
Quoted 10 lines
Quoted 5 lines
Right. The reason why we have to wait for the completion of the channel program is exactly that. Furthermore if we have to fail setting
FEATURES_OKwe fail the channel program, that was supposed to do the operation with an unit check. So at the point where theset_status()is done, we already know if it worked or not.The only difference is, for
FEATURES_OK, driver know it won't work any more. But forDEVICE_STOPPED, driver know it might work sometime in the future.If the channel program fails with a command reject, that that ccw is not allowed to cause further side effects. I.e. we can not fail the channel program, but then make the operation succeed later.
Quoted 17 lines
Quoted 9 lines
Of course the device can
change it's status any time.Since we have a synchronous API setting
DEVICE_STOPPEDwould also have to block until all in-flight requests are completed. But this does not see to be what you want. I understood thatvirtio_add_status(dev, DEVICE_STOPPED)would return immediately, and so would a subsequentconfig->get_status(dev)without theDEVICE_STOPPEDbit set, if there are still outstanding requests.Yes.
Quoted 3 lines
I don't think defining
virtio_add_status(dev, FEATURES_OK)(followed by a readback) as synchronous andvirtio_add_status(dev, DEVICE_STOPPED)(followed by a readback) as asynchronous is a good idea.Quoted 9 lines
The main reason is that
DEVICE_STOPPEDis a one of the possible state. We'd better do this by extending the current device status state machine. Otherwise it would be more complicated (e.g using another interface).It looks to me what you don't like is the driver API design? If yes, how about introduce something like
virtio_try_add_status()? The difference is that it doesn't require a immediate result.
Quoted 6 lines
Yes a new (Linux) driver API operation sounds good -- at least on the surface. I would rather call it
virtio_add_status_async()thanvirtio_try_add_status()because for me 'try' tells that something may or may not work, but the outcome is decided when the function returns. E.g.try_lock(l)may or may not succeed the lock l, but when it returns you know if it did take l or not.
I agree.
Quoted 5 lines
But how would you implement that new operation? I guess it would still have to boil down to
config->set_status(dev), and toset_status(s), being asynchronous when s &DEVICE_STOPPEDand synchronous otherwise (and in particularFEATURES_OKis added to the status).
Or it could be done via a dedicated config ops like how vp_reset() did.
Quoted 5 lines
Ultimately a
set_status(b)boils down to a iowrite8(b) for the PCI transport, and to a channel program that references a single byte buffer with the value of b for Channel IO transport. Making that synchroneaous or asynchronous depending on the value of b does not sound right.
Just to make sure we're in the same page. I meant e.g in CCW_CMD_WRITE_STATUS, qemu just propagate the request of device stop to device and return success. And it's the device that decide when it can set the DEVICE_STOPPED bit and then the next CCW_CMD_READ_STATUS could read the bit.
Quoted 9 lines
Maybe Conny can chime in on this later as well.
[..]
Quoted 5 lines
Quoted 4 lines
AFAIK x86 uses normal
memory access instructions. I suppose when a store instruction does not fail, then the store must happen. Or is that only true for RAM and for PCI memory not?
Quoted 6 lines
Quoted 3 lines
AFAIK, PCI have non-posted write which can fail but the spec doesn't requires that.
So for a posted write a PCI device can just silently discard writes it does not 'like' (i.e. pretend there was no such transaction in the first place)?
I might be wrong, but it's my understanding as well.
Quoted 41 lines
[..]
Quoted 17 lines
So did PCI:
case VIRTIO_PCI_STATUS: if (!(val & VIRTIO_CONFIG_S_DRIVER_OK)) { virtio_pci_stop_ioeventfd(proxy); } virtio_set_status(vdev, val & 0xFF); if (val & VIRTIO_CONFIG_S_DRIVER_OK) { virtio_pci_start_ioeventfd(proxy); } if (vdev->status == 0) { virtio_pci_reset(DEVICE(proxy)); }Quoted 6 lines
As you said, it's a implementation property since qemu implements a synchronous
virtio_set_status(). But it might not be true for real hardware which may require more time or even hard to drain a queue in short time. That's why PCI spec mandate the driver to poll for the status after writing zero to device status.That is a PCI specific thing, and unfortunately my PCI skills are very limited.
Please compare
static void
vp_reset(struct virtio_device *vdev){ vp_iowrite8(0, &vp_dev->common->device_status); [..] while (vp_ioread8(&vp_dev->common->device_status)) msleep(1);With
intvirtio_finalize_features(struct virtio_device *dev){
[..]
Quoted 8 lines
virtio_add_status(dev, VIRTIO_CONFIG_S_FEATURES_OK); status = dev->config->get_status(dev); if (!(status & VIRTIO_CONFIG_S_FEATURES_OK)) {The latter clearly assumes that the first
get_status()will yield the final outcome, where the former clearly assumes that the effect ofset_status()may be delayed beyond the first subsequentget_status().
I think the reason is that. For PCI, the spec forces the device to either set or clear the FEATURE_OK during write. But for reset, device may choose to delay setting status to 0 after the write.
Quoted 5 lines
Btw, if the device were to to fail resetting itself, what would you expect to happen? I would expect the device to present a status
DEVICE_NEEDS_RESET, but that would effectively makevp_reset()getting stuck in that loop.
Yes, then it looks like a recursion here.
Quoted 2 lines
Or does the spec guarantee
that a reset must work eventually?
My understanding is that the spec implies it must succeed eventually.
Quoted 12 lines
[..]
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
PCI can use new filed. But I wonder how can that help for CCW.
Using a new mechanism to request the device to stop would help with the synchronous vs asynchronous stuff. I.e. we could keep the
FEATURES_OKsynchronous, and makeREQUEST_STOPasynchronous.Right, but as said before, it looks more like a part of device status state machine, using new mechanism might bring extra complexity and confusion.
Quoted 7 lines
Quoted 4 lines
Looking at CCW implementation of device status, I don't see any blocker for implementing it synchronously if we just wait for the stop of the device and then return?
I don't see a problem on the qemu side, but I do see a problem on the guest side. There is a reason why you want set
DEVICE_STOPPEDto be asynchronous.
So my understanding is that. at least for current kernel virtio drivers, it doesn't need DEVICE_STOPPED since there's no actual use cases. So no need for inventing new helpers.
It would be only need for the vDPA drivers that is used by userspace via vhost-vDPA. In this case, we just provide the mechanism to write and read status, vDPA driver don't need to enforce any policy in the middle (e.g polling or not).
Thanks
Quoted 6 lines
Regards,
Halil[..]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 6 lines · On Tue, 22 Dec 2020 20:51:04 +0800 Jason Wang <[email protected]> wrote:
Quoted 6 lines
Quoted 6 lines
From Qemu point of view, vhost-vDPA is just another type of vhost
backend. Qemu needs to stop virtio (vhost) before it can do migration. So we require vDPA devices to have the ability of stopping or pausing its datapath. If the vDPA device is by chance the virtio-PCI device, it needs an interface for receiving stop/resume command from the driver.
Quoted 9 lines
Quoted 3 lines
Quoted 2 lines
So the devce stop/resume command was sent from Qemu to vhost-VDPA, then to vDPA parent which could be a virtio-PCI device in this case.
But QEMU implements the _device_, not the driver, doesn't it?
The device is implemented with the co-operation between Qemu and vhost-vDPA. During migration qemu need to stop the virtio-net device, then vhost must be stopped as well.
Sorry for the lazy/stupid question (it's been a while since I last understood that code), how does this work with vhost based virtio-net?
I don't see ioctl that stops the old classic vhost (no vdpa). I have a fuzzy remembrance that we 'stop' the notifiers (ioeventfd & irqfd) but I'm not sure.
And since we are at virtio-net, I ask myself how are 'new requests' and 'requests that is (the device) currently processing' defined for the receive functionality and the rx queue(s). Remember the latter ('in flight requests') need to be all completed. I mean 'in-flight' is pretty straight-forward for something like virtio-blk, or even for tx but I have a hard time with rx.
Quoted 8 lines
Quoted 4 lines
And IIUC,
vhost-VDPA and the vDPA parent are also on the device side. I feel like I'm missing something essential here.Virtio-PCI driver could be a vDPA parent as well in this case. So we need stop the virtio-pci device.
I used to think about vdpa like a vehicle to make partial virtio support in HW viable and easy. I.e. when I hear vdpa I have something like this in mind:
https://www.redhat.com/cms/managed-files/2019-10-02-vdpa-figure5.jpg
Of course the 'physical nic' from the linked picture can be a nic that supports both the virtio data plane and the virtio control plane (i.e. what you are referring to as a virtio-pci device). But do we still expect that device to be connected via vdpa?
The second question is not strictly on topic, but I'm still curious, what do we plan to do with a lower device that does not support virtio control plane? In that case we can't go via DEVICE_STOP. Does that mean, there has to be a vendor specific mechanism equivalent to the DEVICE_STOP mechanism, or otherwise we are non-migratable?
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 17 lines · Halil Pasic
Quoted 6 lines · On Tue, 22 Dec 2020 20:51:04 +0800 Jason Wang <[email protected]> wrote:
Quoted 6 lines
Quoted 6 lines
From Qemu point of view, vhost-vDPA is just another type of vhost
backend. Qemu needs to stop virtio (vhost) before it can do migration. So we require vDPA devices to have the ability of stopping or pausing its datapath. If the vDPA device is by chance the virtio-PCI device, it needs an interface for receiving stop/resume command from the driver.Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
So the devce stop/resume command was sent from Qemu to vhost-VDPA, then to vDPA parent which could be a virtio-PCI device in this case.
But QEMU implements the _device_, not the driver, doesn't it?
The device is implemented with the co-operation between Qemu and vhost-vDPA. During migration qemu need to stop the virtio-net device, then vhost must be stopped as well.
Quoted 6 lines
Sorry for the lazy/stupid question (it's been a while since I last understood that code), how does this work with vhost based virtio-net?
I don't see ioctl that stops the old classic vhost (no vdpa). I have a fuzzy remembrance that we 'stop' the notifiers (ioeventfd & irqfd) but I'm not sure.
Each vhost devices implements its own ioctl:
For vhost-net, it's VHOST_NET_SET_BACKEND, passing -1 as socket fd will stop the device. For vhost-vsock, it's VHOST_VSOCK_SET_RUNNING.
I don't check SCSI but it should have something similar.
Quoted 2 lines
And since we are at virtio-net,
The feature is not net specific.
Quoted 6 lines
I ask myself how are 'new requests' and
'requests that is (the device) currently processing' defined for the receive functionality and the rx queue(s). Remember the latter ('in flight requests') need to be all completed. I mean 'in-flight' is pretty straight-forward for something like virtio-blk, or even for tx but I have a hard time with rx.
I think it should be no difference from the view of virtqueue. It's the requests that have been read from avail ring but not added to the used ring.
Actually this ties to the visibility of virtqueue internal state. Usually device maintains a tail pointer (last_avail_idx). So in the simplest case when no indices wrap and no out-of-order, requests belongs to [last_avail_idx, used_idx) are considered as in-flight.
Quoted 12 lines
Quoted 7 lines
Quoted 4 lines
And IIUC,
vhost-VDPA and the vDPA parent are also on the device side. I feel like I'm missing something essential here.Virtio-PCI driver could be a vDPA parent as well in this case. So we need stop the virtio-pci device.
I used to think about vdpa like a vehicle to make partial virtio support in HW viable and easy. I.e. when I hear vdpa I have something like this in mind:
https://www.redhat.com/cms/managed-files/2019-10-02-vdpa-figure5.jpg
Quoted 4 lines
Of course the 'physical nic' from the linked picture can be a nic that supports both the virtio data plane and the virtio control plane (i.e. what you are referring to as a virtio-pci device). But do we still expect that device to be connected via vdpa?
Why not?[1] With the help of vhost-vDPA, we get the wire speed and live migration support for virtio-pci device.
Quoted 4 lines
The second question is not strictly on topic, but I'm still curious, what do we plan to do with a lower device that does not support virtio control plane?
Yes, the vDPA device just need to implement the same semantic not the same interface. You may refer mlx5e vDPA driver in the kernel source.
Quoted 3 lines
In that case we can't go via
DEVICE_STOP. Does that mean, there has to be a vendor specific mechanism equivalent to theDEVICE_STOPmechanism, or otherwise we are non-migratable?
Exactly.
Thanks
[1] https://lwn.net/Articles/838978/
Quoted 4 lines
Regards,
Halil
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 5 lines · On Thu, 24 Dec 2020 13:51:31 +0800 Jason Wang <[email protected]> wrote:
Quoted 3 lines · Halil Pasic
On Tue, 22 Dec 2020 20:51:04 +0800
Jason Wang <[email protected]> wrote:
[..]
Quoted 9 lines
Each vhost devices implements its own ioctl:
For vhost-net, it's
VHOST_NET_SET_BACKEND, passing -1 as socket fd will stop the device.
For vhost-vsock, it'sVHOST_VSOCK_SET_RUNNING.I don't check SCSI but it should have something similar.
Thanks!
Quoted 7 lines
Quoted 3 lines
And since we are at virtio-net,
The feature is not net specific.
Nod.
Quoted 9 lines
Quoted 7 lines
I ask myself how are 'new requests' and
'requests that is (the device) currently processing' defined for the receive functionality and the rx queue(s). Remember the latter ('in flight requests') need to be all completed. I mean 'in-flight' is pretty straight-forward for something like virtio-blk, or even for tx but I have a hard time with rx.
Quoted 8 lines
I think it should be no difference from the view of virtqueue. It's the requests that have been read from avail ring but not added to the used ring.
Actually this ties to the visibility of virtqueue internal state. Usually device maintains a tail pointer (last_avail_idx). So in the simplest case when no indices wrap and no out-of-order, requests belongs to [
last_avail_idx,used_idx) are considered as in-flight.
I wouldn't call it virtqueue internal state, but I get what you mean. The virtio specification is concerned with the device/driver interface. The driver does not and can not differentiate between buffers that are available but "haven't been read from avail ring", and those that have "been read from the avail ring". As you said last_avail_idx is a device private thing -- a device internal state.
I tend to say, that from a perspective of the driver, all requests that are available, and not yet used, are in-flight. So we have to be very careful when wording this requirement, to avoid misunderstandings. I don't think the first RFC is good enough. I will think some more about this. I'm under the impression that considering virtio console (serial) could be useful, as it provides a reliable duplex data transfer. I.e. some I/O is not driver initiated, but dropping packets ain't OK. So, the data may be in flight without the virtqueue buffer being in flight according to our latest definition ([last_avail_idx, used_idx)).
Sorry, I have the feeling I'm spouting out half baked thoughts. Thank you for bearing with me.
Quoted 13 lines
Quoted 13 lines
Quoted 8 lines
Quoted 5 lines
And IIUC,
vhost-VDPA and the vDPA parent are also on the device side. I feel like I'm missing something essential here.Virtio-PCI driver could be a vDPA parent as well in this case. So we need stop the virtio-pci device.
I used to think about vdpa like a vehicle to make partial virtio support in HW viable and easy. I.e. when I hear vdpa I have something like this in mind:
https://www.redhat.com/cms/managed-files/2019-10-02-vdpa-figure5.jpg
Quoted 8 lines
Quoted 4 lines
Of course the 'physical nic' from the linked picture can be a nic that supports both the virtio data plane and the virtio control plane (i.e. what you are referring to as a virtio-pci device). But do we still expect that device to be connected via vdpa?
Why not?[1] With the help of vhost-vDPA, we get the wire speed and live migration support for virtio-pci device.
Indeed.
Quoted 10 lines
Quoted 6 lines
The second question is not strictly on topic, but I'm still curious, what do we plan to do with a lower device that does not support virtio control plane?
Yes, the vDPA device just need to implement the same semantic not the same interface. You may refer mlx5e vDPA driver in the kernel source.
Thanks! I will have a look.
Regards,
Halil
[..]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 39 lines · Halil Pasic
Quoted 4 lines · On Thu, 24 Dec 2020 13:51:31 +0800 Jason Wang <[email protected]> wrote:
Quoted 3 lines · Halil Pasic
On Tue, 22 Dec 2020 20:51:04 +0800
Jason Wang <[email protected]> wrote:[..]
Quoted 8 lines
Each vhost devices implements its own ioctl:
For vhost-net, it's
VHOST_NET_SET_BACKEND, passing -1 as socket fd will stop the device.
For vhost-vsock, it'sVHOST_VSOCK_SET_RUNNING.I don't check SCSI but it should have something similar.
Thanks!
Quoted 4 lines
Quoted 1 line
And since we are at virtio-net,
The feature is not net specific.
Nod.
Quoted 7 lines
Quoted 6 lines
I ask myself how are 'new requests' and
'requests that is (the device) currently processing' defined for the receive functionality and the rx queue(s). Remember the latter ('in flight requests') need to be all completed. I mean 'in-flight' is pretty straight-forward for something like virtio-blk, or even for tx but I have a hard time with rx.Quoted 8 lines
I think it should be no difference from the view of virtqueue. It's the requests that have been read from avail ring but not added to the used ring.
Actually this ties to the visibility of virtqueue internal state. Usually device maintains a tail pointer (last_avail_idx). So in the simplest case when no indices wrap and no out-of-order, requests belongs to [
last_avail_idx,used_idx) are considered as in-flight.
Quoted 6 lines
I wouldn't call it virtqueue internal state, but I get what you mean. The virtio specification is concerned with the device/driver interface. The driver does not and can not differentiate between buffers that are available but "haven't been read from avail ring", and those that have "been read from the avail ring". As you said
last_avail_idxis a device private thing -- a device internal state.
Exactly.
Quoted 6 lines
I tend to say, that from a perspective of the driver, all requests that are available, and not yet used, are in-flight. So we have to be very careful when wording this requirement, to avoid misunderstandings. I don't think the first RFC is good enough. I will think some more about this.
Yes, I agree. The problem is that the spec doesn't describe how device work, so if we want to be more accurate, it might require some work not only for stop but also for e.g reset (something like in flight has been used by the spec in that case).
Quoted 5 lines
I'm under the impression that considering virtio console (serial) could be useful, as it provides a reliable duplex data transfer. I.e. some I/O is not driver initiated, but dropping packets ain't OK. So, the data may be in flight without the virtqueue buffer being in flight according to our latest definition ([
last_avail_idx,used_idx)).
Yes.
Quoted 2 lines
Sorry, I have the feeling I'm spouting out half baked thoughts. Thank you for bearing with me.
You're welcome and you comments are very helpful and appreciated.
Thanks
Quoted 25 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
And IIUC,
vhost-VDPA and the vDPA parent are also on the device side. I feel like I'm missing something essential here.Virtio-PCI driver could be a vDPA parent as well in this case. So we need stop the virtio-pci device.
I used to think about vdpa like a vehicle to make partial virtio support in HW viable and easy. I.e. when I hear vdpa I have something like this in mind:
https://www.redhat.com/cms/managed-files/2019-10-02-vdpa-figure5.jpgQuoted 7 lines
Quoted 4 lines
Of course the 'physical nic' from the linked picture can be a nic that supports both the virtio data plane and the virtio control plane (i.e. what you are referring to as a virtio-pci device). But do we still expect that device to be connected via vdpa?
Why not?[1] With the help of vhost-vDPA, we get the wire speed and live migration support for virtio-pci device.
Indeed.
Quoted 5 lines
Quoted 4 lines
The second question is not strictly on topic, but I'm still curious, what do we plan to do with a lower device that does not support virtio control plane?
Quoted 9 lines
Quoted 2 lines
Yes, the vDPA device just need to implement the same semantic not the same interface. You may refer mlx5e vDPA driver in the kernel source.
Thanks! I will have a look.
Regards,
Halil[..]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 11 lines · Jason Wang
Quoted 5 lines
I tend to say, that from a perspective of the driver, all requests that are available, and not yet used, are in-flight. So we have to be very careful when wording this requirement, to avoid misunderstandings. I don't think the first RFC is good enough. I will think some more about this.
Yes, I agree. The problem is that the spec doesn't describe how device work, so if we want to be more accurate, it might require some work not only for stop but also for e.g reset (something like in flight has been used by the spec in that case).
You probably mean the DEVICE_NEEDS_RESET description, right?
For example, the driver can't assume requests in flight will be completed if DEVICE_NEEDS_RESET is set, nor can it assume that they have not been completed. A good implementation will try to recover by issuing a reset.
yes, DEVICE_NEEDS_RESET is unfortunately underspecified which likely is related to the fact it is not widely implemented.
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Quoted 13 lines · Michael S. Tsirkin
Quoted 10 lines · Jason Wang
Quoted 5 lines
I tend to say, that from a perspective of the driver, all requests that are available, and not yet used, are in-flight. So we have to be very careful when wording this requirement, to avoid misunderstandings. I don't think the first RFC is good enough. I will think some more about this.
Yes, I agree. The problem is that the spec doesn't describe how device work, so if we want to be more accurate, it might require some work not only for stop but also for e.g reset (something like in flight has been used by the spec in that case).
You probably mean the
DEVICE_NEEDS_RESETdescription, right?
Quoted 7 lines
For example, the driver can't assume requests in flight will be completed if
DEVICE_NEEDS_RESETis set, nor can it assume that they have not been completed. A good implementation will try to recover by issuing a reset.yes,
DEVICE_NEEDS_RESETis unfortunately underspecified which likely is related to the fact it is not widely implemented.
For both NEEDS_RESET and device reset.
For NEEDS_RESET, we use "in flight" and "completed" without an actual definition.
For device reset, we don't even define what is the device expected (e.g how are "in flight" requests expected to be processed) to behave.
Thanks
Quoted 1 line
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 26 lines · Jason Wang
Quoted 13 lines · Michael S. Tsirkin
Quoted 10 lines · Jason Wang
Quoted 5 lines
I tend to say, that from a perspective of the driver, all requests that are available, and not yet used, are in-flight. So we have to be very careful when wording this requirement, to avoid misunderstandings. I don't think the first RFC is good enough. I will think some more about this.
Yes, I agree. The problem is that the spec doesn't describe how device work, so if we want to be more accurate, it might require some work not only for stop but also for e.g reset (something like in flight has been used by the spec in that case).
You probably mean the
DEVICE_NEEDS_RESETdescription, right?Quoted 7 lines
For example, the driver can't assume requests in flight will be completed if
DEVICE_NEEDS_RESETis set, nor can it assume that they have not been completed. A good implementation will try to recover by issuing a reset.yes,
DEVICE_NEEDS_RESETis unfortunately underspecified which likely is related to the fact it is not widely implemented.For both
NEEDS_RESETand device reset.
Quoted 7 lines
For
NEEDS_RESET, we use "in flight" and "completed" without an actual definition.For device reset, we don't even define what is the device expected (e.g how are "in flight" requests expected to be processed) to behave.
Thanks
Because device is expected to just stop:
None of the virtqueues
of a device are live once the device has been reset.
and it's driver's job to recover buffers:
Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
what happened to buffers which were not used?
I think it's a quality of implementation/device specific issue, e.g. for net:
for transmit, if device sends a packet without using the buffer, then driver will resend the packet leading to duplicates.
for receive, it's a packet drop, slightly less of a problem.
my question is whether such behaviour is sufficient for migration? if not can we really describe it generally? it's possible we need to describe it per device type.
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
----- Original Message -----
Quoted 38 lines · Jason Wang
Quoted 37 lines · Michael S. Tsirkin
Quoted 18 lines · Jason Wang
Quoted 8 lines
Quoted 7 lines
I tend to say, that from a perspective of the driver, all requests that
are available, and not yet used, are in-flight. So we have to be very careful when wording this requirement, to avoid misunderstandings. I don't think the first RFC is good enough. I will think some more about
this.Quoted 7 lines
Yes, I agree. The problem is that the spec doesn't describe how device work,
so if we want to be more accurate, it might require some work not only for
stop but also for e.g reset (something like in flight has been used by the
spec in that case).You probably mean the
DEVICE_NEEDS_RESETdescription, right?Quoted 7 lines
For example, the driver can't assume requests in flight will be completed if
DEVICE_NEEDS_RESETis set, nor can it assume that they have not been completed. A good implementation will try to recover by issuing a reset.yes,
DEVICE_NEEDS_RESETis unfortunately underspecified which likely is related to the fact it is not widely implemented.For both
NEEDS_RESETand device reset.For
NEEDS_RESET, we use "in flight" and "completed" without an actual definition.For device reset, we don't even define what is the device expected (e.g how are "in flight" requests expected to be processed) to behave.
Quoted 6 lines
Quoted 1 line
Thanks
Because device is expected to just stop:
None of the virtqueues
of a device are live once the device has been reset.
So it's something similar to what DEVICE_STOPPED wants.
Quoted 10 lines
and it's driver's job to recover buffers:
Thus a driver MUST ensure a virtqueue isn't live (by device reset) before removing exposed buffers.
what happened to buffers which were not used?
I think it's a quality of implementation/device specific issue, e.g. for net:
for transmit, if device sends a packet without using the buffer,
This sounds like an intermediate state we need to avoid in the migration. Can we mandate the device to make the buffer used in this case? For networking device, it should be not slow, just wait for the DMA to be completed is sufficient.
Quoted 2 lines
then driver will resend the packet leading to duplicates. for receive, it's a packet drop, slightly less of a problem.
For net yes.
Quoted 3 lines
my question is whether such behaviour is sufficient for migration?
I suspect it's not, we need to either
1) wait for the buffer to be used
or
2) fail the migration if the device is not stopped in a short time
Quoted 1 line
if not can we really describe it generally?
It looks like a virtqueue level issue, so we need to try.
Quoted 2 lines
it's possible we need to
describe it per device type.
Maybe.
Thanks
Quoted 12 lines
--
MSTThis publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 10 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 75 lines · Jason Wang
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +24
-3@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. + \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. \end{description}@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}. The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set.@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev recover by issuing a reset. \end{note} +The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. + \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset. The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK. +The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED. + +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set. + \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates that the driver passes extra data (besides identifying the virtqueue) in its device notifications. + \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the + driver can stop and resume the device. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description} --
Quoted 2 lines
2.25.1
Hi!
After implementing a small POC for this in QEMU/virtio-pci vdpa device, I think that the standard should specify what state the driver
[1] is allowed to change, if any, and what fields are valid for the driver to query (like device status or virtqueue state).
As a proposal, I think all the device config status should freeze (as "the device is not allowed to change") when paused, and the driver must be able to read all of it.
Regarding writing to the device config, the straightforward solution is to not allow them, and mandate the driver to go through a full reset cycle in my opinion. The only exception should be the queue
selector: Writing to it by the driver has the obvious consequences. This way, the device does not need to check for changes on the resume, it could be tricky. However maybe there are some use cases where to change just a field through this method could be faster than the whole device reset.
I can think in another approach: The device supports to write some fields (for example, just virtqueue addresses and virtqueue size, or some feature flags), but any other change would lead to the device to report DEVICE_NEEDS_RESET. This has the mentioned advantages of reducing latencies (as transactions) in case of device re-usage, but at the cost of increasing complexity. Any opinion about if it is worth it?
Thanks!
[1] From the device's point of view, so I mean QEMU here.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 79 lines · Eugenio Perez Martin
Quoted 74 lines · Jason Wang
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Signed-off-by Jason Wang <[email protected]>--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +24
-3@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. + \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. \end{description}@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}. The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set.@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev recover by issuing a reset. \end{note} +The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. + \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset. The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK. +The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED. + +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set. + \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates that the driver passes extra data (besides identifying the virtqueue) in its device notifications. + \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the + driver can stop and resume the device. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description} --Quoted 2 lines
2.25.1
Hi!
Quoted 8 lines
After implementing a small POC for this in QEMU/virtio-pci vdpa device, I think that the standard should specify what state the driver
[1] is allowed to change, if any, and what fields are valid for the driver to query (like device status or virtqueue state).As a proposal, I think all the device config status should freeze (as "the device is not allowed to change") when paused, and the driver must be able to read all of it.
Yes.
Quoted 10 lines
Regarding writing to the device config, the straightforward solution is to not allow them, and mandate the driver to go through a full reset cycle in my opinion. The only exception should be the queue
selector: Writing to it by the driver has the obvious consequences. This way, the device does not need to check for changes on the resume, it could be tricky. However maybe there are some use cases where to change just a field through this method could be faster than the whole device reset.
Quoted 7 lines
I can think in another approach: The device supports to write some fields (for example, just virtqueue addresses and virtqueue size, or some feature flags), but any other change would lead to the device to report
DEVICE_NEEDS_RESET. This has the mentioned advantages of reducing latencies (as transactions) in case of device re-usage, but at the cost of increasing complexity. Any opinion about if it is worth it?
I'm not sure we need to do this, and actually it's not specific to the stop state.
A more generic question is: do we allow the driver to change device/virtqueue status after DRIVER_OK is set? If yes, it's probably less interesting in limiting the behavior when stop is set. If not, we should clarify it and there's no need to teat stop as a specific case to deal with.
Thanks
Quoted 13 lines
Thanks!
[1] From the device's point of view, so I mean QEMU here.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 5 lines · Jason Wang
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.Signed-off-by Jason Wang <[email protected]>
I have been thinking about this whole approach.
While yes it's possible to have the hypervisor poke at the device while device is not reset, e.g. by blocking guest access to device, while device can still DMA into guest memory, this is at least unusual and seems somewhat fragile.
I think normally live migration would belong in a PF as opposed to the VF.
As such, how about for starters we prototype this
as part of a vendor specific capability of a PF?
This will also show-case how to add vendor specific capabilities to devices.
Longer term it would be nice to come up with all kind of PF related functionality in the spec too, but I think that should be part of a more comprehensive/extensible approach since it's not limited to live migration concerns. For example the way qemu handles control vq of a nic in software creates problems, couldn't we inject
the relevant commands through PF somehow?
Quoted 70 lines
--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +24
-2@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. + \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. \end{description}@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}. The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set.@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev recover by issuing a reset. \end{note} +The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. + \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset. The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK. +The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED. + +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set. + \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates that the driver passes extra data (besides identifying the virtqueue) in its device notifications. + \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the + driver can stop and resume the device. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description}Signature · 1 line
2.25.1
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Jason Wang
This patch introduces a new status bit
DEVICE_STOPPED. This will be used by the driver to stop and resume a device. The main user will be live migration support for virtio device.
Quoted 8 lines
Quoted 1 line
Signed-off-by Jason Wang <[email protected]>I have been thinking about this whole approach.
While yes it's possible to have the hypervisor poke at the device while device is not reset, e.g. by blocking guest access to device, while device can still DMA into guest memory, this is at least unusual and seems somewhat fragile.
So in the context of stop, I wonder if we can just forbid the DMA in this case?
Quoted 5 lines
I think normally live migration would belong in a PF as opposed to the VF.
As such, how about for starters we prototype this as part of a vendor specific capability of a PF?
Just to make sure I understand the proposal. Except for the differnt control path, I don't see any other differences:
Consider we use vendor specific capability, we still need to define the semantics like the VIRTIO_F_DEVICE_STOP in this case. E.g as you said, quiescence the DMA and other stuffs.
Quoted 11 lines
This will also show-case how to add vendor specific capabilities to devices.
Longer term it would be nice to come up with all kind of PF related functionality in the spec too, but I think that should be part of a more comprehensive/extensible approach since it's not limited to live migration concerns. For example the way qemu handles control vq of a nic in software creates problems, couldn't we inject
the relevant commands through PF somehow?
Yes, as we discussed in the past, I plan to send a RFC to introduce the admin virtqueue for PF. It could be used for such extensions.
Thanks
Quoted 70 lines
Quoted 70 lines
--- content.tex | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)content.tex +24
-2@@ -47,6 +47,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \item[DRIVER_OK (4)] Indicates that the driver is set up and ready to drive the device. +\item[DEVICE_STOPPED (32)] When VIRTIO_F_DEVICE_STOPPED is negotiated, + indicates that the device has been stopped by the driver. + \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has experienced an error from which it can't recover. \end{description}@@ -58,8 +61,9 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev \ref{sec:General Initialization And Device Operation / Device Initialization}. The driver MUST NOT clear a -\field{device status} bit. If the driver sets the FAILED bit, -the driver MUST later reset the device before attempting to re-initialize. +\field{device status} bit other than DEVICE_STOPPED. If the +driver sets the FAILED bit, the driver MUST later reset the device +before attempting to re-initialize. The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set.@@ -70,12 +74,28 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev recover by issuing a reset. \end{note} +The driver MUST NOT set or clear DEVICE_STOPPED when DRIVER_OK is not +set. In order to stop the device, the driver MUST set DEVICE_STOPPED +first and re-read status to check whether DEVICE_STOPPED is set by the +device. In order to resume the device, the driver MUST clear +DEVICE_STOPPED first and read status to ensure whether DEVICE_STOPPED +is cleared by the device. + \devicenormative{\subsection}{Device Status Field}{Basic Facilities of a Virtio Device / Device Status Field} The device MUST initialize \field{device status} to 0 upon reset. The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK. +The device MUST ignore DEVICE_STOPPED when DRIVER_OK is not set. + +When driver is trying to set DEVICE_STOPPED, the device MUST not +process new avail requests and MUST complete all requests that is +currently processing before setting DEVICE_STOPPED. + +The device MUST keep the config space unchanged when DEVICE_STOPPED is +set. + \label{sec:Basic Facilities of a Virtio Device / Device Status Field / DEVICENEEDSRESET}The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.@@ -6553,6 +6573,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_NOTIFICATION_DATA(38)] This feature indicates that the driver passes extra data (besides identifying the virtqueue) in its device notifications. + \item[VIRTIO_F_DEVICE_STOP(39)] This feature indicates that the + driver can stop and resume the device. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description}
Quoted 2 lines
Quoted 2 lines
Signature · 1 line
2.25.1
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/