VIRTIO_F_PARTIAL_ORDER for page fault handling
RFC PATCH 1/1 VIRTIO_F_PARTIAL_ORDER for page fault handling
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.
The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.
--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex +8 -1
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} Some devices always use descriptors in the same order in which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge might allow optimizations or simplify driver and/or device code. Each virtqueue can consist of up to 3 parts:
@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver passes extra data (besides identifying the virtqueue) in its device notifications. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. + \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates + that device has ability to indicate use of (some of) buffers by the device in the same + order in which they have been made available. \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +23 -0
@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors} only writing out a single used descriptor with the Buffer ID corresponding to the last descriptor in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_DESC_F_FLUSH flag in the used descriptor. +\begin{lstlisting} +#define VIRTQ_DESC_F_FLUSH 8 +\end{lstlisting} + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out a single used descriptor with the Buffer ID +corresponding to the last descriptor in the batch, +and VIRTQ_DESC_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_DESC_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. The driver needs to look up the used Buffer ID and calculate the batch size to be able to advance to where the next
split-ring.tex +24 -2
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ }; -/* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ - le32 id; + le16 id; +#define VIRTQ_USED_ELEM_F_FLUSH 0x8000 + le16 flags; /* Total length of the descriptor chain which was used (written to) */ le32 len; };
@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors} corresponding to the head entry of the descriptor chain describing the last buffer in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry. + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out single used ring entry with the \field{id} +corresponding to the head entry of the +descriptor chain describing the last buffer in the batch, +and VIRTQ_USED_ELEM_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. Accordingly, it increments the used \field{idx} by the size of the batch.
Signature · 7 lines
MST --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Quoted 45 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)content.tex +0
-0
content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} · line 303
Quoted 9 lines
content.tex +5
-1@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} Some devices always use descriptors in the same order in which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge
Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6140
Quoted 9 lines
content.tex +2
-0might allow optimizations or simplify driver and/or device code. Each virtqueue can consist of up to 3 parts:@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver passes extra data (besides identifying the virtqueue) in its device notifications. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. + \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates + that device has ability to indicate use of (some of) buffers by the device in the same
"use of a subset of buffers" ?
Quoted 9 lines
content.tex +1
-0+ order in which they have been made available. \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}packed-ring.tex +0
-0
packed-ring.tex \subsection{In-order use of descriptors} · line 290
Quoted 7 lines
packed-ring.tex +4
-0@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors} only writing out a single used descriptor with the Buffer ID corresponding to the last descriptor in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers
s/device/the device/
packed-ring.tex \subsection{In-order use of descriptors} · line 292
Quoted 2 lines
packed-ring.tex +2
-0+since the previous used buffer in the same order +in which they have been made available, device can set the
s/device/the device/
packed-ring.tex \subsection{In-order use of descriptors} · line 304
Quoted 12 lines
packed-ring.tex +12
-0+VIRTQ_DESC_F_FLUSH flag in the used descriptor. +\begin{lstlisting} +#define VIRTQ_DESC_F_FLUSH 8 +\end{lstlisting} + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out a single used descriptor with the Buffer ID +corresponding to the last descriptor in the batch, +and VIRTQ_DESC_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way
s/device/the device/
packed-ring.tex \subsection{In-order use of descriptors} · line 306
Quoted 2 lines
packed-ring.tex +2
-0+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH +flag set, as a result, considering the group of buffers
s/set, as/set. As/
Quoted 11 lines
packed-ring.tex +3
-0+used between two buffers with VIRTQ_DESC_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. The driver needs to look up the used Buffer ID and calculate the batch size to be able to advance to where the nextsplit-ring.tex +0
-0
split-ring.tex \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi · line 405
Quoted 10 lines
split-ring.tex +3
-2@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ }; -/* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ - le32 id; + le16 id; +#define VIRTQ_USED_ELEM_F_FLUSH 0x8000 + le16 flags;
This is a bit confusing for legacy, as this will only work for little-endian. The flags field will only be used for non-legacy, but I think the id field should still be treated as 32 bit for legacy. (We have a wholesale "it's guest endian when legacy" further up.)
split-ring.tex \subsection{In-order use of descriptors} · line 488
Quoted 11 lines
split-ring.tex +4
-0/* Total length of the descriptor chain which was used (written to) */ le32 len; };@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors} corresponding to the head entry of the descriptor chain describing the last buffer in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers
s/device/the device/
split-ring.tex \subsection{In-order use of descriptors} · line 490
Quoted 2 lines
split-ring.tex +2
-0+since the previous used buffer in the same order +in which they have been made available, device can set the
s/device/the device/
split-ring.tex \subsection{In-order use of descriptors} · line 495
Quoted 5 lines
split-ring.tex +5
-0+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry. + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out single used ring entry with the \field{id}
s/single/a single/
split-ring.tex \subsection{In-order use of descriptors} · line 500
Quoted 5 lines
split-ring.tex +5
-0+corresponding to the head entry of the +descriptor chain describing the last buffer in the batch, +and VIRTQ_USED_ELEM_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way
s/device/the device/
split-ring.tex \subsection{In-order use of descriptors} · line 502
Quoted 2 lines
split-ring.tex +2
-0+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH +flag set, as a result, considering the group of buffers
s/set, as/set. As/
split-ring.tex \subsection{In-order use of descriptors} · line 508
Quoted 6 lines
split-ring.tex +3
-0+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. Accordingly, it increments the used \field{idx} by the size of the batch.
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Quoted 49 lines · Cornelia Huck
Quoted 46 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)content.tex +0
-0@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
Quoted 11 lines
content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} · line 303
Quoted 8 lines
content.tex +5
-1Some devices always use descriptors in the same order in which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledgeDo these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
Good point. I think so, yes. Will document.
Quoted 11 lines
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6140
Quoted 9 lines
content.tex +2
-0might allow optimizations or simplify driver and/or device code. Each virtqueue can consist of up to 3 parts:@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver passes extra data (besides identifying the virtqueue) in its device notifications. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. + \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates + that device has ability to indicate use of (some of) buffers by the device in the same"use of a subset of buffers" ?
That sounds better, thanks!
Quoted 58 lines
Quoted 9 lines
content.tex +1
-0+ order in which they have been made available. \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}packed-ring.tex +0
-0packed-ring.tex \subsection{In-order use of descriptors} · line 290
Quoted 7 lines
packed-ring.tex +4
-0@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors} only writing out a single used descriptor with the Buffer ID corresponding to the last descriptor in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all bufferss/device/the device/
packed-ring.tex \subsection{In-order use of descriptors} · line 292
Quoted 2 lines
packed-ring.tex +2
-0+since the previous used buffer in the same order +in which they have been made available, device can set thes/device/the device/
packed-ring.tex \subsection{In-order use of descriptors} · line 304
Quoted 12 lines
packed-ring.tex +12
-0+VIRTQ_DESC_F_FLUSH flag in the used descriptor. +\begin{lstlisting} +#define VIRTQ_DESC_F_FLUSH 8 +\end{lstlisting} + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out a single used descriptor with the Buffer ID +corresponding to the last descriptor in the batch, +and VIRTQ_DESC_F_FLUSH set. + +Note that device is only allowed to batch buffers in this ways/device/the device/
packed-ring.tex \subsection{In-order use of descriptors} · line 306
Quoted 2 lines
packed-ring.tex +2
-0+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH +flag set, as a result, considering the group of bufferss/set, as/set. As/
Quoted 14 lines
packed-ring.tex +3
-0+used between two buffers with VIRTQ_DESC_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. The driver needs to look up the used Buffer ID and calculate the batch size to be able to advance to where the nextsplit-ring.tex +0
-0@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ };
Quoted 12 lines
split-ring.tex \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi · line 405
Quoted 7 lines
split-ring.tex +3
-2-/* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ - le32 id; + le16 id; +#define VIRTQ_USED_ELEM_F_FLUSH 0x8000 + le16 flags;This is a bit confusing for legacy, as this will only work for little-endian. The flags field will only be used for non-legacy, but I think the id field should still be treated as 32 bit for legacy. (We have a wholesale "it's guest endian when legacy" further up.)
True. How do you suggest handling this? Just say that id is OR with VIRTQ_USED_ELEM_F_FLUSH ? Or maybe just add struct virtq_used_elem_legacy?
Quoted 40 lines
split-ring.tex \subsection{In-order use of descriptors} · line 488
Quoted 11 lines
split-ring.tex +4
-0/* Total length of the descriptor chain which was used (written to) */ le32 len; };@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors} corresponding to the head entry of the descriptor chain describing the last buffer in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all bufferss/device/the device/
split-ring.tex \subsection{In-order use of descriptors} · line 490
Quoted 2 lines
split-ring.tex +2
-0+since the previous used buffer in the same order +in which they have been made available, device can set thes/device/the device/
split-ring.tex \subsection{In-order use of descriptors} · line 495
Quoted 5 lines
split-ring.tex +5
-0+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry. + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out single used ring entry with the \field{id}s/single/a single/
split-ring.tex \subsection{In-order use of descriptors} · line 500
Quoted 5 lines
split-ring.tex +5
-0+corresponding to the head entry of the +descriptor chain describing the last buffer in the batch, +and VIRTQ_USED_ELEM_F_FLUSH set. + +Note that device is only allowed to batch buffers in this ways/device/the device/
split-ring.tex \subsection{In-order use of descriptors} · line 502
Quoted 2 lines
split-ring.tex +2
-0+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH +flag set, as a result, considering the group of bufferss/set, as/set. As/
Quoted 6 lines
split-ring.tex \subsection{In-order use of descriptors} · line 508
Quoted 6 lines
split-ring.tex +3
-0+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. Accordingly, it increments the used \field{idx} by the size of the batch.
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Quoted 25 lines · On Tue, 11 Aug 2020 04:23:13 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 11 lines · Cornelia Huck
Quoted 8 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
split-ring.tex +0
-0@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ };Quoted 12 lines
split-ring.tex \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi · line 405
Quoted 7 lines
split-ring.tex +3
-2-/* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ - le32 id; + le16 id; +#define VIRTQ_USED_ELEM_F_FLUSH 0x8000 + le16 flags;This is a bit confusing for legacy, as this will only work for little-endian. The flags field will only be used for non-legacy, but I think the id field should still be treated as 32 bit for legacy. (We have a wholesale "it's guest endian when legacy" further up.)
Quoted 3 lines
True. How do you suggest handling this? Just say that id is OR with
VIRTQ_USED_ELEM_F_FLUSH? Or maybe just add structvirtq_used_elem_legacy?
I think introducing a legacy struct is the least confusing option.
Quoted 5 lines
Quoted 5 lines
split-ring.tex \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi · line 408
Quoted 5 lines
split-ring.tex +0
-0/* Total length of the descriptor chain which was used (written to) */ le32 len; };
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Hi Michael,
Short comment below.
BR,
-Lars
Quoted 59 lines
From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23Quoted 46 lines · Cornelia Huck
Quoted 43 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)content.tex +0
-0@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}Quoted 7 lines
content.tex
Quoted 7 lines
content.tex +0
-0Some devices always use descriptors in the same order in whichthey have been made available. These devices can offer the
content.tex +4
-1-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use descriptors +in the same order in which they have been made available. These +devices can offer the VIRTIO_F_PARTIAL_ORDER feature. If one of the +features VIRTIO_F_IN_ORDER or VIRTIO_F_PARTIAL_ORDER isnegotiated,
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
+this knowledge
Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
Good point. I think so, yes. Will document.
Isn't it more natural to think of VIRTIO_F_IN_ORDER as the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible let VIRTIO_F_IN_ORDER imply the new bit is set, while the new bit set by itself without VIRTIO_F_IN_ORDER set means only active control is offered. I guess a name like VIRTIO_F_CTRL_ORDER would be more appropriate with this interpretation.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 75 lines
From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54Quoted 4 lines
From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23Quoted 30 lines
Quoted 29 lines · Cornelia Huck
Quoted 26 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH
flag.
Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)content.tex index 91735e3..8494eb6 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}Quoted 8 lines
content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
Quoted 8 lines
content.tex +0
-0Some devices always use descriptors in the same order in whichthey have been made available. These devices can offer the
content.tex +5
-1-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER isnegotiated,
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
+this knowledge
Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
Good point. I think so, yes. Will document.
Quoted 8 lines
Isn't it more natural to think of
VIRTIO_F_IN_ORDERas the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible letVIRTIO_F_IN_ORDERimply the new bit is set, while the new bit set by itself withoutVIRTIO_F_IN_ORDERset means only active control is offered. I guess a name likeVIRTIO_F_CTRL_ORDERwould be more appropriate with this interpretation.
On second thought that might be a bit backwards - how about:
Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0 This proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1 Potential future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1
Quoted 12 lines
This publicly archived list offers a means to provide input to theOASIS Virtual I/O Device (VIRTIO) TC.In order to verify user consent to the Feedback License terms andto minimize spam in the list archive, subscription is requiredbefore posting.Subscribe: [email protected] open.orgUnsubscribe: [email protected]
help: [email protected] archive: https://lists.oasis-
open.org/archives/virtio-comment/Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdfList Guidelines: https://www.oasis-
open.org/policies-guidelines/mailing-listsCommittee: https://www.oasis-
open.org/committees/virtio/Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 84 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 75 lines
From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54Quoted 4 lines
From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23Quoted 30 lines
Quoted 29 lines · Cornelia Huck
Quoted 26 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH
flag.
Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)content.tex index 91735e3..8494eb6 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}Quoted 8 lines
content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
Quoted 8 lines
content.tex +0
-0Some devices always use descriptors in the same order in whichthey have been made available. These devices can offer the
content.tex +5
-1-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER isnegotiated,
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
+this knowledge
Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
Good point. I think so, yes. Will document.
Quoted 8 lines
Isn't it more natural to think of
VIRTIO_F_IN_ORDERas the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible letVIRTIO_F_IN_ORDERimply the new bit is set, while the new bit set by itself withoutVIRTIO_F_IN_ORDERset means only active control is offered. I guess a name likeVIRTIO_F_CTRL_ORDERwould be more appropriate with this interpretation.
Quoted 5 lines
On second thought that might be a bit backwards - how about:
Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0 This proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1 Potential future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1
What happens in the new device/old driver case?
- device offers IN_ORDER and PARTIAL_ORDER
- driver does not know PARTIAL_ORDER, accepts IN_ORDER
- device now only can do complete ordering
Maybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering IN_ORDER might not know about the new feature and its mechanism, and a device offering the new feature, but not IN_ORDER probably does so because it cannot support full IN_ORDER.
I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Quoted 113 lines
From: [email protected] <[email protected] open.org> On Behalf Of Cornelia Huck
Sent: 12. august 2020 14:51Quoted 12 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 12 lines
From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54Quoted 8 lines
From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23Quoted 3 lines · Cornelia Huck
On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has
been resolved.
Quoted 57 lines
Quoted 57 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH
flag.
Quoted 36 lines
Quoted 21 lines
Quoted 21 lines
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++-packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex
|
26
++++++++++++++++++++++++--3 files changed, 55 insertions(+), 3 deletions(-)content.tex index 91735e3..8494eb6 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}Quoted 8 lines
content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
Quoted 8 lines
content.tex +0
-0Some devices always use descriptors in the same order inwhich they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
content.tex +5
-0+VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER isnegotiated,
Quoted 4 lines
Quoted 1 line
+this knowledge
Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
Good point. I think so, yes. Will document.
Quoted 7 lines
Quoted 7 lines
Isn't it more natural to think of
VIRTIO_F_IN_ORDERas the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible letVIRTIO_F_IN_ORDERimply the new bit is set, while the new bit set by itself withoutVIRTIO_F_IN_ORDERset means only active control is offered. I guess a name likeVIRTIO_F_CTRL_ORDERwould be more appropriate with thisinterpretation.
Quoted 7 lines
Quoted 1 line
On second thought that might be a bit backwards - how about:
Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0 This
proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1 Potential future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1What happens in the new device/old driver case?
- device offersIN_ORDERandPARTIAL_ORDER
- driver does not knowPARTIAL_ORDER, acceptsIN_ORDER
- device now only can do complete orderingMaybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering
IN_ORDERmight not know about the new feature and its mechanism, and a device offering the new feature, but notIN_ORDERprobably does so because it cannot support fullIN_ORDER.
Quoted 3 lines
I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?
Yes, you are absolutely right. Keeping them as two mutually exclusive options is probably best.
Another aspect of the proposal is that the functionality is only described for the packed ring layout, but the concept should be equally applicable to the split ring. The split ring does not have flags for each used ring element, so the FLUSH flag functionality needs to be a bit different. One way is to add an IN_ORDER-flag to the virtq_used.flags, that allows the device to signal when it starts to adhere to IN_ORDER buffer use. The driver acknowledges this when it is ready to fulfil its part of the IN_ORDER restrictions by setting a new virtq_avail.flag: IN_ORDER_ACK. At this point both device and driver can assume IN_ORDER optimization until the device clears IN_ORDER. After clearing IN_ORDER, the device must wait until IN_ORDER_ACK is cleared before IN_ORDER can be asserted it again.
This is a little less fine-grained than the packed-ring solution with the FLUSH flag, but for infrequent scenarios like migration it should be OK.
Quoted 7 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 117 lines · Lars Ganrot
Quoted 113 lines
From: [email protected] <[email protected] open.org> On Behalf Of Cornelia Huck
Sent: 12. august 2020 14:51Quoted 12 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 12 lines
From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54Quoted 8 lines
From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23Quoted 3 lines · Cornelia Huck
On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has
been resolved.
Quoted 57 lines
Quoted 57 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH
flag.
Quoted 36 lines
Quoted 21 lines
Quoted 21 lines
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++-packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex
|
26
++++++++++++++++++++++++--3 files changed, 55 insertions(+), 3 deletions(-)content.tex index 91735e3..8494eb6 renamed from content.tex +0
-0100644
content.tex +0
-0@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}Quoted 8 lines
content.tex
Quoted 8 lines
content.tex +0
-0Some devices always use descriptors in the same order inwhich they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
content.tex +5
-0+VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER isnegotiated,
Quoted 4 lines
Quoted 1 line
+this knowledge
Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
Good point. I think so, yes. Will document.
Quoted 7 lines
Quoted 7 lines
Isn't it more natural to think of
VIRTIO_F_IN_ORDERas the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible letVIRTIO_F_IN_ORDERimply the new bit is set, while the new bit set by itself withoutVIRTIO_F_IN_ORDERset means only active control is offered. I guess a name likeVIRTIO_F_CTRL_ORDERwould be more appropriate with thisinterpretation.
Quoted 7 lines
Quoted 1 line
On second thought that might be a bit backwards - how about:
Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0 This
proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1 Potential future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1What happens in the new device/old driver case?
- device offersIN_ORDERandPARTIAL_ORDER
- driver does not knowPARTIAL_ORDER, acceptsIN_ORDER
- device now only can do complete orderingMaybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering
IN_ORDERmight not know about the new feature and its mechanism, and a device offering the new feature, but notIN_ORDERprobably does so because it cannot support fullIN_ORDER.Quoted 3 lines
I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?
Quoted 5 lines
Yes, you are absolutely right. Keeping them as two mutually exclusive options is probably best.
Another aspect of the proposal is that the functionality is only described for the packed ring layout, but the concept should be equally applicable to the split ring.
Pls take a look at the changes in split-ring.tex
Are they unclear?
Quoted 10 lines
The split ring does not have
flags for each used ring element, so the FLUSH flag functionality needs to be a bit different. One way is to add an IN_ORDER-flag to the virtq_used.flags, that allows the device to signal when it starts to adhere toIN_ORDERbuffer use. The driver acknowledges this when it is ready to fulfil its part of theIN_ORDERrestrictions by setting a new virtq_avail.flag:IN_ORDER_ACK. At this point both device and driver can assumeIN_ORDERoptimization until the device clearsIN_ORDER. After clearingIN_ORDER, the device must wait untilIN_ORDER_ACKis cleared beforeIN_ORDERcan be asserted it again.
It would be tricky to figure out which buffers does the handshake apply to. My proposal does it a bit differently, pls take a look.
Quoted 20 lines
This is a little less fine-grained than the packed-ring solution with the FLUSH flag, but for infrequent scenarios like migration it should be OK.
Quoted 7 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 1 line
Disclaimer: This email and any files transmitted with it may contain confidential information intended for the addressee(s) only. The information is not to be surrendered or copied to unauthorized persons. If you have received this communication in error, please notify the sender immediately and delete this e-mail from your system.
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Quoted 132 lines
From: [email protected] <[email protected] open.org> On Behalf Of Michael S. Tsirkin
Sent: 14. august 2020 01:18Quoted 16 lines · Lars Ganrot
Quoted 16 lines
From: [email protected] <[email protected] open.org> On Behalf Of Cornelia Huck
Sent: 12. august 2020 14:51Quoted 9 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 9 lines
From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54Quoted 5 lines
From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Cornelia Huck
Quoted 5 lines
Quoted 2 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of
order.
Quoted 76 lines
Quoted 76 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has
been resolved.
Quoted 68 lines
Quoted 68 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH
flag.
Quoted 33 lines
Quoted 23 lines
Quoted 23 lines
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++split-ring.tex
|
26
++++++++++++++++++++++++--3 files changed, 55 insertions(+), 3 deletions(-)content.tex index renamed from content.tex +0
-091735e3..8494eb6
100644content.tex +0
-0@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}Quoted 9 lines
content.tex
Quoted 9 lines
content.tex +0
-0Some devices always use descriptors in the same order inwhich they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
content.tex +5
-0+VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been +made available. These devices can offer the +VIRTIO_F_PARTIAL_ORDER feature. If one of the features +VIRTIO_F_IN_ORDER or VIRTIO_F_PARTIAL_ORDER isnegotiated,
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
+this knowledge
Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
Good point. I think so, yes. Will document.
Isn't it more natural to think of
VIRTIO_F_IN_ORDERas the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible letVIRTIO_F_IN_ORDERimply the new bit is set, while the new bit set by itself withoutVIRTIO_F_IN_ORDERset means only active control is offered. I guess a name likeVIRTIO_F_CTRL_ORDERwould be more appropriate with thisQuoted 6 lines
Quoted 6 lines
interpretation.
Quoted 5 lines
Quoted 1 line
On second thought that might be a bit backwards - how about:
Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0
This
Quoted 1 line
Quoted 1 line
Quoted 1 line
proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1
Potential
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1
What happens in the new device/old driver case?
- device offersIN_ORDERandPARTIAL_ORDER
- driver does not knowPARTIAL_ORDER, acceptsIN_ORDER
- device now only can do complete orderingQuoted 6 lines
Quoted 6 lines
Maybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering
IN_ORDERmight not know about the new feature and its mechanism, and a device offering the new feature, but notIN_ORDERprobably does so because it cannotsupport full
IN_ORDER.Quoted 5 lines
Quoted 4 lines
I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?
Quoted 9 lines
Quoted 1 line
Yes, you are absolutely right. Keeping them as two mutually exclusive
options is probably best.
Quoted 4 lines
Another aspect of the proposal is that the functionality is only described for the packed ring layout, but the concept should be equally applicable to the split ring.
Pls take a look at the changes in split-ring.tex Are they unclear?
I guess not. I failed to understand that the used-ring 32-bit ID field is divided into a 16-bit ID and a 16-bit flags. Sorry for the confusion.
Does this mean that for a device that always sets the FLUSH-flag, the PARTIAL_ORDER feature will be identical to IN_ORDER for both packed and split rings? The first write to a Used structure (after creation of the Virtqueue) would not have 2 FLUSH-flags, but would still fulfil the condition.
Quoted 45 lines
Quoted 10 lines
The split ring does not have
flags for each used ring element, so the FLUSH flag functionality needs to be a bit different. One way is to add an IN_ORDER-flag to the virtq_used.flags, that allows the device to signal when it starts to adhere toIN_ORDERbuffer use. The driver acknowledges this when it is ready to fulfil its part of theIN_ORDERrestrictions by setting a new virtq_avail.flag:IN_ORDER_ACK. At this point both device and driver can assumeIN_ORDERoptimization until the device clearsIN_ORDER. After clearingIN_ORDER, the device must wait untilIN_ORDER_ACKis cleared beforeIN_ORDERcan be asserted it again.It would be tricky to figure out which buffers does the handshake apply to. My proposal does it a bit differently, pls take a look.
Quoted 2 lines
This is a little less fine-grained than the packed-ring solution with the FLUSH
flag, but for infrequent scenarios like migration it should be OK.
Quoted 8 lines
Quoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 11 lines
Quoted 10 lines
Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 139 lines · Lars Ganrot
Quoted 139 lines
From: [email protected] <[email protected] open.org> On Behalf Of Michael S. Tsirkin
Sent: 14. august 2020 01:18Quoted 16 lines · Lars Ganrot
Quoted 16 lines
From: [email protected] <[email protected] open.org> On Behalf Of Cornelia Huck
Sent: 12. august 2020 14:51Quoted 9 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 9 lines
From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54Quoted 5 lines
From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Cornelia Huck
Quoted 5 lines
Quoted 2 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of
order.
Quoted 76 lines
Quoted 76 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has
been resolved.
Quoted 68 lines
Quoted 68 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH
flag.
Quoted 33 lines
Quoted 23 lines
Quoted 23 lines
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++split-ring.tex
|
26
++++++++++++++++++++++++--3 files changed, 55 insertions(+), 3 deletions(-)content.tex index renamed from content.tex +0
-091735e3..8494eb6
100644content.tex +0
-0@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}Quoted 9 lines
content.tex
Quoted 9 lines
content.tex +0
-0Some devices always use descriptors in the same order inwhich they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
content.tex +5
-0+VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been +made available. These devices can offer the +VIRTIO_F_PARTIAL_ORDER feature. If one of the features +VIRTIO_F_IN_ORDER or VIRTIO_F_PARTIAL_ORDER isnegotiated,
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
+this knowledge
Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
Good point. I think so, yes. Will document.
Isn't it more natural to think of
VIRTIO_F_IN_ORDERas the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible letVIRTIO_F_IN_ORDERimply the new bit is set, while the new bit set by itself withoutVIRTIO_F_IN_ORDERset means only active control is offered. I guess a name likeVIRTIO_F_CTRL_ORDERwould be more appropriate with thisQuoted 6 lines
Quoted 6 lines
interpretation.
Quoted 5 lines
Quoted 1 line
On second thought that might be a bit backwards - how about:
Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0
This
Quoted 1 line
Quoted 1 line
Quoted 1 line
proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1
Potential
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1
What happens in the new device/old driver case?
- device offersIN_ORDERandPARTIAL_ORDER
- driver does not knowPARTIAL_ORDER, acceptsIN_ORDER
- device now only can do complete orderingQuoted 6 lines
Quoted 6 lines
Maybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering
IN_ORDERmight not know about the new feature and its mechanism, and a device offering the new feature, but notIN_ORDERprobably does so because it cannotsupport full
IN_ORDER.Quoted 6 lines
Quoted 4 lines
I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?
Yes, you are absolutely right. Keeping them as two mutually exclusive
options is probably best.
Quoted 4 lines
Another aspect of the proposal is that the functionality is only described for the packed ring layout, but the concept should be equally applicable to the split ring.
Quoted 8 lines
Quoted 2 lines
Pls take a look at the changes in split-ring.tex Are they unclear?
I guess not. I failed to understand that the used-ring 32-bit ID field is divided into a 16-bit ID and a 16-bit flags. Sorry for the confusion.
Does this mean that for a device that always sets the FLUSH-flag, the
PARTIAL_ORDERfeature will be identical toIN_ORDERfor both packed and split rings?
Yes, I think so.
Quoted 3 lines
The first write to a Used structure (after creation
of the Virtqueue) would not have 2 FLUSH-flags, but would still fulfil the condition.
Needs thought.
Quoted 71 lines
Quoted 45 lines
Quoted 10 lines
The split ring does not have
flags for each used ring element, so the FLUSH flag functionality needs to be a bit different. One way is to add an IN_ORDER-flag to the virtq_used.flags, that allows the device to signal when it starts to adhere toIN_ORDERbuffer use. The driver acknowledges this when it is ready to fulfil its part of theIN_ORDERrestrictions by setting a new virtq_avail.flag:IN_ORDER_ACK. At this point both device and driver can assumeIN_ORDERoptimization until the device clearsIN_ORDER. After clearingIN_ORDER, the device must wait untilIN_ORDER_ACKis cleared beforeIN_ORDERcan be asserted it again.It would be tricky to figure out which buffers does the handshake apply to. My proposal does it a bit differently, pls take a look.
Quoted 2 lines
This is a little less fine-grained than the packed-ring solution with the FLUSH
flag, but for infrequent scenarios like migration it should be OK.
Quoted 8 lines
Quoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 11 lines
Quoted 10 lines
Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 74 lines · Lars Ganrot
Hi Michael,
Short comment below.
BR,
-Lars
Quoted 59 lines
From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23Quoted 46 lines · Cornelia Huck
Quoted 43 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)content.tex +0
-0@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}Quoted 7 lines
content.tex
Quoted 7 lines
content.tex +0
-0Some devices always use descriptors in the same order in whichthey have been made available. These devices can offer the
content.tex +4
-1-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use descriptors +in the same order in which they have been made available. These +devices can offer the VIRTIO_F_PARTIAL_ORDER feature. If one of the +features VIRTIO_F_IN_ORDER or VIRTIO_F_PARTIAL_ORDER isnegotiated,
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
+this knowledge
Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?
Good point. I think so, yes. Will document.
Quoted 8 lines
Isn't it more natural to think of
VIRTIO_F_IN_ORDERas the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible letVIRTIO_F_IN_ORDERimply the new bit is set, while the new bit set by itself withoutVIRTIO_F_IN_ORDERset means only active control is offered. I guess a name likeVIRTIO_F_CTRL_ORDERwould be more appropriate with this interpretation.
So how does the behaviour with both VIRTIO_F_IN_ORDER with VIRTIO_F_PARTIAL_ORDER differ from just VIRTIO_F_IN_ORDER? If all buffers are used in order, then what is the extra meaning of FLUSH?
Quoted 6 lines
Disclaimer: This email and any files transmitted with it may contain confidential information intended for the addressee(s) only. The information is not to be surrendered or copied to unauthorized persons. If you have received this communication in error, please notify the sender immediately and delete this e-mail from your system.
I think we are supposed to disregard this part as irrlevant, but just a reminder that you agreed that all material posted to the virtio mailing lists is governed by the relevant IPR rules. It is your responsibility not to post confidential information there.
Signature · 7 lines
MST --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Quoted 33 lines · Michael S. Tsirkin
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.
Quoted 6 lines
Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)
Hi Michael,
What is the status of this feature?
There is a Google Summer of Code project to implement VIRTIO_F_IN_ORDER in QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).
I wanted to check if you still want to pursue PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.
Thanks to Ilya Maximets for mentioning PARTIAL_ORDER!
Stefan
Quoted 116 lines
content.tex +8
-1@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} Some devices always use descriptors in the same order in which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge might allow optimizations or simplify driver and/or device code. Each virtqueue can consist of up to 3 parts:@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver passes extra data (besides identifying the virtqueue) in its device notifications. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. + \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates + that device has ability to indicate use of (some of) buffers by the device in the same + order in which they have been made available. \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}packed-ring.tex +23
-0@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors} only writing out a single used descriptor with the Buffer ID corresponding to the last descriptor in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_DESC_F_FLUSH flag in the used descriptor. +\begin{lstlisting} +#define VIRTQ_DESC_F_FLUSH 8 +\end{lstlisting} + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out a single used descriptor with the Buffer ID +corresponding to the last descriptor in the batch, +and VIRTQ_DESC_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_DESC_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. The driver needs to look up the used Buffer ID and calculate the batch size to be able to advance to where the nextsplit-ring.tex +24
-2@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ }; -/* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ - le32 id; + le16 id; +#define VIRTQ_USED_ELEM_F_FLUSH 0x8000 + le16 flags; /* Total length of the descriptor chain which was used (written to) */ le32 len; };@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors} corresponding to the head entry of the descriptor chain describing the last buffer in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry. + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out single used ring entry with the \field{id} +corresponding to the head entry of the +descriptor chain describing the last buffer in the batch, +and VIRTQ_USED_ELEM_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. Accordingly, it increments the used \field{idx} by the size of the batch.Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 45 lines · Stefan Hajnoczi
Quoted 33 lines · Michael S. Tsirkin
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Quoted 6 lines
Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)Hi Michael,
What is the status of this feature?
Quoted 8 lines
There is a Google Summer of Code project to implement
VIRTIO_F_IN_ORDERin QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).I wanted to check if you still want to pursue
PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.
If the status is close to being merged I'm totally in to implement either partial order first or to implement both in the project. In some sense it superseded IN_ORDER.
Thanks!
Quoted 116 lines
Thanks to Ilya Maximets for mentioning
PARTIAL_ORDER!Stefan
Quoted 112 lines
content.tex +8
-1@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} Some devices always use descriptors in the same order in which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge might allow optimizations or simplify driver and/or device code. Each virtqueue can consist of up to 3 parts:@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver passes extra data (besides identifying the virtqueue) in its device notifications. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. + \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates + that device has ability to indicate use of (some of) buffers by the device in the same + order in which they have been made available. \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}packed-ring.tex +23
-0@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors} only writing out a single used descriptor with the Buffer ID corresponding to the last descriptor in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_DESC_F_FLUSH flag in the used descriptor. +\begin{lstlisting} +#define VIRTQ_DESC_F_FLUSH 8 +\end{lstlisting} + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out a single used descriptor with the Buffer ID +corresponding to the last descriptor in the batch, +and VIRTQ_DESC_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_DESC_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. The driver needs to look up the used Buffer ID and calculate the batch size to be able to advance to where the nextsplit-ring.tex +24
-3@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ }; -/* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ - le32 id; + le16 id; +#define VIRTQ_USED_ELEM_F_FLUSH 0x8000 + le16 flags; /* Total length of the descriptor chain which was used (written to) */ le32 len; };@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors} corresponding to the head entry of the descriptor chain describing the last buffer in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry. + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out single used ring entry with the \field{id} +corresponding to the head entry of the +descriptor chain describing the last buffer in the batch, +and VIRTQ_USED_ELEM_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. Accordingly, it increments the used \field{idx} by the size of the batch. --MST
Quoted 4 lines
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 55 lines · Eugenio Perez Martin
Quoted 45 lines · Stefan Hajnoczi
Quoted 33 lines · Michael S. Tsirkin
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Quoted 6 lines
Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)Hi Michael,
What is the status of this feature?Quoted 8 lines
There is a Google Summer of Code project to implement
VIRTIO_F_IN_ORDERin QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).I wanted to check if you still want to pursue
PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.
Quoted 5 lines
If the status is close to being merged I'm totally in to implement either partial order first or to implement both in the project. In some sense it superseded
IN_ORDER.Thanks!
I think it's blocked on lack of implementation actually. TC was concerned whether there are any correctness issues. An implementation which can be stress-tested would resolve this.
Quoted 116 lines
Quoted 116 lines
Thanks to Ilya Maximets for mentioning
PARTIAL_ORDER!Stefan
Quoted 112 lines
content.tex +8
-1@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} Some devices always use descriptors in the same order in which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge might allow optimizations or simplify driver and/or device code. Each virtqueue can consist of up to 3 parts:@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver passes extra data (besides identifying the virtqueue) in its device notifications. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. + \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates + that device has ability to indicate use of (some of) buffers by the device in the same + order in which they have been made available. \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}packed-ring.tex +23
-0@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors} only writing out a single used descriptor with the Buffer ID corresponding to the last descriptor in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_DESC_F_FLUSH flag in the used descriptor. +\begin{lstlisting} +#define VIRTQ_DESC_F_FLUSH 8 +\end{lstlisting} + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out a single used descriptor with the Buffer ID +corresponding to the last descriptor in the batch, +and VIRTQ_DESC_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_DESC_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. The driver needs to look up the used Buffer ID and calculate the batch size to be able to advance to where the nextsplit-ring.tex +24
-3@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ }; -/* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ - le32 id; + le16 id; +#define VIRTQ_USED_ELEM_F_FLUSH 0x8000 + le16 flags; /* Total length of the descriptor chain which was used (written to) */ le32 len; };@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors} corresponding to the head entry of the descriptor chain describing the last buffer in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry. + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out single used ring entry with the \field{id} +corresponding to the head entry of the +descriptor chain describing the last buffer in the batch, +and VIRTQ_USED_ELEM_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. Accordingly, it increments the used \field{idx} by the size of the batch. --MST
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 61 lines · Michael S. Tsirkin
Quoted 59 lines · Eugenio Perez Martin
Quoted 45 lines · Stefan Hajnoczi
Quoted 33 lines · Michael S. Tsirkin
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Quoted 6 lines
Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)Hi Michael,
What is the status of this feature?Quoted 8 lines
There is a Google Summer of Code project to implement
VIRTIO_F_IN_ORDERin QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).I wanted to check if you still want to pursue
PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.If the status is close to being merged I'm totally in to implement either partial order first or to implement both in the project. In some sense it superseded
IN_ORDER.
Quoted 6 lines
Quoted 1 line
Thanks!
I think it's blocked on lack of implementation actually. TC was concerned whether there are any correctness issues. An implementation which can be stress-tested would resolve this.
Got it, so then it is worth trying to allocate some time to implement it IMO.
Thanks!
Quoted 117 lines
Quoted 117 lines
Quoted 117 lines
Thanks to Ilya Maximets for mentioning
PARTIAL_ORDER!Stefan
Quoted 112 lines
content.tex +8
-1@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} Some devices always use descriptors in the same order in which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge might allow optimizations or simplify driver and/or device code. Each virtqueue can consist of up to 3 parts:@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver passes extra data (besides identifying the virtqueue) in its device notifications. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. + \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates + that device has ability to indicate use of (some of) buffers by the device in the same + order in which they have been made available. \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}packed-ring.tex +23
-0@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors} only writing out a single used descriptor with the Buffer ID corresponding to the last descriptor in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_DESC_F_FLUSH flag in the used descriptor. +\begin{lstlisting} +#define VIRTQ_DESC_F_FLUSH 8 +\end{lstlisting} + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out a single used descriptor with the Buffer ID +corresponding to the last descriptor in the batch, +and VIRTQ_DESC_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_DESC_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. The driver needs to look up the used Buffer ID and calculate the batch size to be able to advance to where the nextsplit-ring.tex +24
-3@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ }; -/* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ - le32 id; + le16 id; +#define VIRTQ_USED_ELEM_F_FLUSH 0x8000 + le16 flags; /* Total length of the descriptor chain which was used (written to) */ le32 len; };@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors} corresponding to the head entry of the descriptor chain describing the last buffer in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry. + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out single used ring entry with the \field{id} +corresponding to the head entry of the +descriptor chain describing the last buffer in the batch, +and VIRTQ_USED_ELEM_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. Accordingly, it increments the used \field{idx} by the size of the batch. --MST
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 48 lines · Stefan Hajnoczi
Quoted 33 lines · Michael S. Tsirkin
Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.
As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.
Page faults allow more features such as THP, auto-NUMA, live migration.
Out of order is of course already possible, however,
IN_ORDERis currently required for descriptor batching where device marks a whole batch of buffers used in one go.The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:
Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.
Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to
IN_ORDERfrom the driver implementation POV.Quoted 6 lines
Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 9 ++++++++- packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex | 26 ++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-)Hi Michael,
What is the status of this feature?There is a Google Summer of Code project to implement
VIRTIO_F_IN_ORDERin QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).
Quoted 7 lines
I wanted to check if you still want to pursue
PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.Thanks to Ilya Maximets for mentioning
PARTIAL_ORDER!Stefan
Yes I do. Would be great if we had someone who can work on this!
Quoted 112 lines
Quoted 112 lines
content.tex +8
-1@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} Some devices always use descriptors in the same order in which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge +VIRTIO_F_IN_ORDER feature. Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If one of the features VIRTIO_F_IN_ORDER or +VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge might allow optimizations or simplify driver and/or device code. Each virtqueue can consist of up to 3 parts:@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver passes extra data (besides identifying the virtqueue) in its device notifications. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. + \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates + that device has ability to indicate use of (some of) buffers by the device in the same + order in which they have been made available. \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}packed-ring.tex +23
-0@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors} only writing out a single used descriptor with the Buffer ID corresponding to the last descriptor in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_DESC_F_FLUSH flag in the used descriptor. +\begin{lstlisting} +#define VIRTQ_DESC_F_FLUSH 8 +\end{lstlisting} + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out a single used descriptor with the Buffer ID +corresponding to the last descriptor in the batch, +and VIRTQ_DESC_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_DESC_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. The driver needs to look up the used Buffer ID and calculate the batch size to be able to advance to where the nextsplit-ring.tex +24
-2@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ }; -/* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ - le32 id; + le16 id; +#define VIRTQ_USED_ELEM_F_FLUSH 0x8000 + le16 flags; /* Total length of the descriptor chain which was used (written to) */ le32 len; };@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors} corresponding to the head entry of the descriptor chain describing the last buffer in the batch. +Other devices sometimes use +descriptors in the same order in which they have been made +available. These devices can offer the VIRTIO_F_PARTIAL_ORDER +feature. If negotiated, whenever device has used all buffers +since the previous used buffer in the same order +in which they have been made available, device can set the +VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry. + +This knowledge allows +devices to notify the use of a batch of buffers to the driver by +only writing out single used ring entry with the \field{id} +corresponding to the head entry of the +descriptor chain describing the last buffer in the batch, +and VIRTQ_USED_ELEM_F_FLUSH set. + +Note that device is only allowed to batch buffers in this way +if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH +flag set, as a result, considering the group of buffers +used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set, +either all of them constitute a batch, or none at all. + The device then skips forward in the ring according to the size of the batch. Accordingly, it increments the used \field{idx} by the size of the batch.Signature · 1 line
MST
Quoted 4 lines
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]