Skip to contentSkip to conversation

VIRTIO_F_PARTIAL_ORDER for page fault handling

RFC PATCH In review Michael S. Tsirkin 5 participants 17 messages Aug 10, 2020 – Mar 30, 2022

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 VIRTIO_F_PARTIAL_ORDER for page fault handling

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++
 split-ring.tex  | 26 ++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex +8 -1
index 91735e3..8494eb6 100644
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 
 Some devices always use descriptors in the same order in which
 they have been made available. These devices can offer the
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge
 might allow optimizations or simplify driver and/or device code.
 
 Each virtqueue can consist of up to 3 parts:
@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver passes extra data (besides identifying the virtqueue)
   in its device notifications.
   See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
+  \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates
+  that device has ability to indicate use of (some of) buffers by the device in the same
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +23 -0
index ea92543..a120a19 100644
@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors}
 only writing out a single used descriptor with the Buffer ID
 corresponding to the last descriptor in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_DESC_F_FLUSH flag in the used descriptor.
+\begin{lstlisting}
+#define VIRTQ_DESC_F_FLUSH      8
+\end{lstlisting}
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch,
+and VIRTQ_DESC_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_DESC_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. The driver needs to look up the used Buffer ID and
 calculate the batch size to be able to advance to where the next
split-ring.tex +24 -2
index 123ac9f..cf197f8 100644
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi
         le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
 };
 
-/* le32 is used here for ids for padding reasons. */
 struct virtq_used_elem {
         /* Index of start of used descriptor chain. */
-        le32 id;
+        le16 id;
+#define VIRTQ_USED_ELEM_F_FLUSH  0x8000
+        le16 flags;
         /* Total length of the descriptor chain which was used (written to) */
         le32 len;
 };
@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors}
 corresponding to the head entry of the
 descriptor chain describing the last buffer in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry.
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out single used ring entry with the \field{id}
+corresponding to the head entry of the
+descriptor chain describing the last buffer in the batch,
+and VIRTQ_USED_ELEM_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. Accordingly, it increments the used \field{idx} by the
 size of the batch.
Signature · 7 lines
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
#1 lore raw
Quoted 45 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++
 split-ring.tex  | 26 ++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex +0 -0
index 91735e3..8494eb6 100644

content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} · line 303

Quoted 9 lines
content.tex +5 -1
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 
 Some devices always use descriptors in the same order in which
 they have been made available. These devices can offer the
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6140

Quoted 9 lines
content.tex +2 -0
 might allow optimizations or simplify driver and/or device code.
 
 Each virtqueue can consist of up to 3 parts:
@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver passes extra data (besides identifying the virtqueue)
   in its device notifications.
   See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
+  \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates
+  that device has ability to indicate use of (some of) buffers by the device in the same

"use of a subset of buffers" ?

Quoted 9 lines
content.tex +1 -0
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +0 -0
index ea92543..a120a19 100644

packed-ring.tex \subsection{In-order use of descriptors} · line 290

Quoted 7 lines
packed-ring.tex +4 -0
@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors}
 only writing out a single used descriptor with the Buffer ID
 corresponding to the last descriptor in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers

s/device/the device/

packed-ring.tex \subsection{In-order use of descriptors} · line 292

Quoted 2 lines
packed-ring.tex +2 -0
+since the previous used buffer in the same order
+in which they have been made available, device can set the

s/device/the device/

packed-ring.tex \subsection{In-order use of descriptors} · line 304

Quoted 12 lines
packed-ring.tex +12 -0
+VIRTQ_DESC_F_FLUSH flag in the used descriptor.
+\begin{lstlisting}
+#define VIRTQ_DESC_F_FLUSH      8
+\end{lstlisting}
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch,
+and VIRTQ_DESC_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way

s/device/the device/

packed-ring.tex \subsection{In-order use of descriptors} · line 306

Quoted 2 lines
packed-ring.tex +2 -0
+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH
+flag set, as a result, considering the group of buffers

s/set, as/set. As/

Quoted 11 lines
packed-ring.tex +3 -0
+used between two buffers with VIRTQ_DESC_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. The driver needs to look up the used Buffer ID and
 calculate the batch size to be able to advance to where the next
split-ring.tex +0 -0
index 123ac9f..cf197f8 100644

split-ring.tex \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi · line 405

Quoted 10 lines
split-ring.tex +3 -2
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi
         le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
 };
 
-/* le32 is used here for ids for padding reasons. */
 struct virtq_used_elem {
         /* Index of start of used descriptor chain. */
-        le32 id;
+        le16 id;
+#define VIRTQ_USED_ELEM_F_FLUSH  0x8000
+        le16 flags;

This is a bit confusing for legacy, as this will only work for little-endian. The flags field will only be used for non-legacy, but I think the id field should still be treated as 32 bit for legacy. (We have a wholesale "it's guest endian when legacy" further up.)

split-ring.tex \subsection{In-order use of descriptors} · line 488

Quoted 11 lines
split-ring.tex +4 -0
         /* Total length of the descriptor chain which was used (written to) */
         le32 len;
 };
@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors}
 corresponding to the head entry of the
 descriptor chain describing the last buffer in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers

s/device/the device/

split-ring.tex \subsection{In-order use of descriptors} · line 490

Quoted 2 lines
split-ring.tex +2 -0
+since the previous used buffer in the same order
+in which they have been made available, device can set the

s/device/the device/

split-ring.tex \subsection{In-order use of descriptors} · line 495

Quoted 5 lines
split-ring.tex +5 -0
+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry.
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out single used ring entry with the \field{id}

s/single/a single/

split-ring.tex \subsection{In-order use of descriptors} · line 500

Quoted 5 lines
split-ring.tex +5 -0
+corresponding to the head entry of the
+descriptor chain describing the last buffer in the batch,
+and VIRTQ_USED_ELEM_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way

s/device/the device/

split-ring.tex \subsection{In-order use of descriptors} · line 502

Quoted 2 lines
split-ring.tex +2 -0
+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH
+flag set, as a result, considering the group of buffers

s/set, as/set. As/

split-ring.tex \subsection{In-order use of descriptors} · line 508

Quoted 6 lines
split-ring.tex +3 -0
+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. Accordingly, it increments the used \field{idx} by the
 size of the batch.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

#2 lore raw
Quoted 49 lines · Cornelia Huck
Quoted 46 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++
 split-ring.tex  | 26 ++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex +0 -0
index 91735e3..8494eb6 100644
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 
Quoted 11 lines

content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} · line 303

Quoted 8 lines
content.tex +5 -1
 Some devices always use descriptors in the same order in which
 they have been made available. These devices can offer the
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

Good point. I think so, yes. Will document.

Quoted 11 lines

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6140

Quoted 9 lines
content.tex +2 -0
 might allow optimizations or simplify driver and/or device code.
 
 Each virtqueue can consist of up to 3 parts:
@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver passes extra data (besides identifying the virtqueue)
   in its device notifications.
   See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
+  \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates
+  that device has ability to indicate use of (some of) buffers by the device in the same

"use of a subset of buffers" ?

That sounds better, thanks!

Quoted 58 lines
Quoted 9 lines
content.tex +1 -0
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +0 -0
index ea92543..a120a19 100644

packed-ring.tex \subsection{In-order use of descriptors} · line 290

Quoted 7 lines
packed-ring.tex +4 -0
@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors}
 only writing out a single used descriptor with the Buffer ID
 corresponding to the last descriptor in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers

s/device/the device/

packed-ring.tex \subsection{In-order use of descriptors} · line 292

Quoted 2 lines
packed-ring.tex +2 -0
+since the previous used buffer in the same order
+in which they have been made available, device can set the

s/device/the device/

packed-ring.tex \subsection{In-order use of descriptors} · line 304

Quoted 12 lines
packed-ring.tex +12 -0
+VIRTQ_DESC_F_FLUSH flag in the used descriptor.
+\begin{lstlisting}
+#define VIRTQ_DESC_F_FLUSH      8
+\end{lstlisting}
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch,
+and VIRTQ_DESC_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way

s/device/the device/

packed-ring.tex \subsection{In-order use of descriptors} · line 306

Quoted 2 lines
packed-ring.tex +2 -0
+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH
+flag set, as a result, considering the group of buffers

s/set, as/set. As/

Quoted 14 lines
packed-ring.tex +3 -0
+used between two buffers with VIRTQ_DESC_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. The driver needs to look up the used Buffer ID and
 calculate the batch size to be able to advance to where the next
split-ring.tex +0 -0
index 123ac9f..cf197f8 100644
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi
         le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
 };
 
Quoted 12 lines

split-ring.tex \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi · line 405

Quoted 7 lines
split-ring.tex +3 -2
-/* le32 is used here for ids for padding reasons. */
 struct virtq_used_elem {
         /* Index of start of used descriptor chain. */
-        le32 id;
+        le16 id;
+#define VIRTQ_USED_ELEM_F_FLUSH  0x8000
+        le16 flags;

This is a bit confusing for legacy, as this will only work for little-endian. The flags field will only be used for non-legacy, but I think the id field should still be treated as 32 bit for legacy. (We have a wholesale "it's guest endian when legacy" further up.)

True. How do you suggest handling this? Just say that id is OR with VIRTQ_USED_ELEM_F_FLUSH ? Or maybe just add struct virtq_used_elem_legacy?

Quoted 40 lines

split-ring.tex \subsection{In-order use of descriptors} · line 488

Quoted 11 lines
split-ring.tex +4 -0
         /* Total length of the descriptor chain which was used (written to) */
         le32 len;
 };
@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors}
 corresponding to the head entry of the
 descriptor chain describing the last buffer in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers

s/device/the device/

split-ring.tex \subsection{In-order use of descriptors} · line 490

Quoted 2 lines
split-ring.tex +2 -0
+since the previous used buffer in the same order
+in which they have been made available, device can set the

s/device/the device/

split-ring.tex \subsection{In-order use of descriptors} · line 495

Quoted 5 lines
split-ring.tex +5 -0
+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry.
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out single used ring entry with the \field{id}

s/single/a single/

split-ring.tex \subsection{In-order use of descriptors} · line 500

Quoted 5 lines
split-ring.tex +5 -0
+corresponding to the head entry of the
+descriptor chain describing the last buffer in the batch,
+and VIRTQ_USED_ELEM_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way

s/device/the device/

split-ring.tex \subsection{In-order use of descriptors} · line 502

Quoted 2 lines
split-ring.tex +2 -0
+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH
+flag set, as a result, considering the group of buffers

s/set, as/set. As/

Quoted 6 lines

split-ring.tex \subsection{In-order use of descriptors} · line 508

Quoted 6 lines
split-ring.tex +3 -0
+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. Accordingly, it increments the used \field{idx} by the
 size of the batch.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

#3 lore raw
Quoted 25 lines · On Tue, 11 Aug 2020 04:23:13 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 11 lines · Cornelia Huck
Quoted 8 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
split-ring.tex +0 -0
index 123ac9f..cf197f8 100644
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi
         le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
 };
 
Quoted 12 lines

split-ring.tex \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi · line 405

Quoted 7 lines
split-ring.tex +3 -2
-/* le32 is used here for ids for padding reasons. */
 struct virtq_used_elem {
         /* Index of start of used descriptor chain. */
-        le32 id;
+        le16 id;
+#define VIRTQ_USED_ELEM_F_FLUSH  0x8000
+        le16 flags;  

This is a bit confusing for legacy, as this will only work for little-endian. The flags field will only be used for non-legacy, but I think the id field should still be treated as 32 bit for legacy. (We have a wholesale "it's guest endian when legacy" further up.)

Quoted 3 lines

True. How do you suggest handling this? Just say that id is OR with VIRTQ_USED_ELEM_F_FLUSH ? Or maybe just add struct virtq_used_elem_legacy?

I think introducing a legacy struct is the least confusing option.

Quoted 5 lines
Quoted 5 lines

split-ring.tex \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi · line 408

Quoted 5 lines
split-ring.tex +0 -0
         /* Total length of the descriptor chain which was used (written to) */
         le32 len;
 };

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

#4 lore raw
In reply to Michael S. Tsirkin

Hi Michael,

Short comment below.

BR,

-Lars

Quoted 59 lines

From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23

Quoted 46 lines · Cornelia Huck
Quoted 43 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++  split-ring.tex  | 26
++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex +0 -0
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities
 of a Virtio Device / Virtqueues}
 
Quoted 7 lines

content.tex

Quoted 7 lines
content.tex +0 -0
 Some devices always use descriptors in the same order in which

they have been made available. These devices can offer the

content.tex +4 -1
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use descriptors
+in the same order in which they have been made available. These
+devices can offer the VIRTIO_F_PARTIAL_ORDER feature. If one of the
+features VIRTIO_F_IN_ORDER or VIRTIO_F_PARTIAL_ORDER is

negotiated,

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

+this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

Good point. I think so, yes. Will document.

Isn't it more natural to think of VIRTIO_F_IN_ORDER as the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible let VIRTIO_F_IN_ORDER imply the new bit is set, while the new bit set by itself without VIRTIO_F_IN_ORDER set means only active control is offered. I guess a name like VIRTIO_F_CTRL_ORDER would be more appropriate with this interpretation.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw
Quoted 75 lines

From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54

Quoted 4 lines

From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23

Quoted 30 lines
Quoted 29 lines · Cornelia Huck
Quoted 26 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH

flag.

Quoted 29 lines
Quoted 20 lines
Quoted 20 lines

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++  split-ring.tex  |
26
++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex index 91735e3..8494eb6 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic
 Facilities of a Virtio Device / Virtqueues}
 
Quoted 8 lines

content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}

Quoted 8 lines
content.tex +0 -0
 Some devices always use descriptors in the same order in which

they have been made available. These devices can offer the

content.tex +5 -1
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is

negotiated,

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

+this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

Good point. I think so, yes. Will document.

Quoted 8 lines

Isn't it more natural to think of VIRTIO_F_IN_ORDER as the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible let VIRTIO_F_IN_ORDER imply the new bit is set, while the new bit set by itself without VIRTIO_F_IN_ORDER set means only active control is offered. I guess a name like VIRTIO_F_CTRL_ORDER would be more appropriate with this interpretation.

On second thought that might be a bit backwards - how about:

Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0 This proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1 Potential future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1

Quoted 12 lines

This publicly archived list offers a means to provide input to theOASIS Virtual I/O Device (VIRTIO) TC.In order to verify user consent to the Feedback License terms andto minimize spam in the list archive, subscription is requiredbefore posting.Subscribe: [email protected] open.orgUnsubscribe: [email protected]
help: [email protected] archive: https://lists.oasis-
open.org/archives/virtio-comment/Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdfList Guidelines: https://www.oasis-
open.org/policies-guidelines/mailing-listsCommittee: https://www.oasis-
open.org/committees/virtio/Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#6 lore raw
Quoted 84 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 75 lines

From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54

Quoted 4 lines

From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23

Quoted 30 lines
Quoted 29 lines · Cornelia Huck
Quoted 26 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH

flag.

Quoted 29 lines
Quoted 20 lines
Quoted 20 lines

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++  split-ring.tex  |
26
++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex index 91735e3..8494eb6 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic
 Facilities of a Virtio Device / Virtqueues}
 
Quoted 8 lines

content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}

Quoted 8 lines
content.tex +0 -0
 Some devices always use descriptors in the same order in which

they have been made available. These devices can offer the

content.tex +5 -1
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is  

negotiated,

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

+this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

Good point. I think so, yes. Will document.

Quoted 8 lines

Isn't it more natural to think of VIRTIO_F_IN_ORDER as the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible let VIRTIO_F_IN_ORDER imply the new bit is set, while the new bit set by itself without VIRTIO_F_IN_ORDER set means only active control is offered. I guess a name like VIRTIO_F_CTRL_ORDER would be more appropriate with this interpretation.

Quoted 5 lines

On second thought that might be a bit backwards - how about:

Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0 This proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1 Potential future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1

What happens in the new device/old driver case?
- device offers IN_ORDER and PARTIAL_ORDER
- driver does not know PARTIAL_ORDER, accepts IN_ORDER
- device now only can do complete ordering

Maybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering IN_ORDER might not know about the new feature and its mechanism, and a device offering the new feature, but not IN_ORDER probably does so because it cannot support full IN_ORDER.

I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

#7 lore raw
Quoted 113 lines

From: [email protected] <[email protected] open.org> On Behalf Of Cornelia Huck
Sent: 12. august 2020 14:51

Quoted 12 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 12 lines

From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54

Quoted 8 lines

From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23

Quoted 3 lines · Cornelia Huck

On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has

been resolved.

Quoted 57 lines
Quoted 57 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH

flag.

Quoted 36 lines
Quoted 21 lines
Quoted 21 lines

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-

packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex
|
26
++++++++++++++++++++++++--

 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex index 91735e3..8494eb6 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic
 Facilities of a Virtio Device / Virtqueues}
 
Quoted 8 lines

content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}

Quoted 8 lines
content.tex +0 -0
 Some devices always use descriptors in the same order in

which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge

content.tex +5 -0
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is

negotiated,

Quoted 4 lines
Quoted 1 line

+this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

Good point. I think so, yes. Will document.

Quoted 7 lines
Quoted 7 lines

Isn't it more natural to think of VIRTIO_F_IN_ORDER as the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible let VIRTIO_F_IN_ORDER imply the new bit is set, while the new bit set by itself without VIRTIO_F_IN_ORDER set means only active control is offered. I guess a name like VIRTIO_F_CTRL_ORDER would be more appropriate with this

interpretation.

Quoted 7 lines
Quoted 1 line

On second thought that might be a bit backwards - how about:

Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0 This
proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1 Potential future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1

What happens in the new device/old driver case?
- device offers IN_ORDER and PARTIAL_ORDER
- driver does not know PARTIAL_ORDER, accepts IN_ORDER
- device now only can do complete ordering

Maybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering IN_ORDER might not know about the new feature and its mechanism, and a device offering the new feature, but not IN_ORDER probably does so because it cannot support full IN_ORDER.

Quoted 3 lines

I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?

Yes, you are absolutely right. Keeping them as two mutually exclusive options is probably best.

Another aspect of the proposal is that the functionality is only described for the packed ring layout, but the concept should be equally applicable to the split ring. The split ring does not have flags for each used ring element, so the FLUSH flag functionality needs to be a bit different. One way is to add an IN_ORDER-flag to the virtq_used.flags, that allows the device to signal when it starts to adhere to IN_ORDER buffer use. The driver acknowledges this when it is ready to fulfil its part of the IN_ORDER restrictions by setting a new virtq_avail.flag: IN_ORDER_ACK. At this point both device and driver can assume IN_ORDER optimization until the device clears IN_ORDER. After clearing IN_ORDER, the device must wait until IN_ORDER_ACK is cleared before IN_ORDER can be asserted it again.

This is a little less fine-grained than the packed-ring solution with the FLUSH flag, but for infrequent scenarios like migration it should be OK.

Quoted 7 lines

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#8 lore raw
Quoted 117 lines · Lars Ganrot
Quoted 113 lines

From: [email protected] <[email protected] open.org> On Behalf Of Cornelia Huck
Sent: 12. august 2020 14:51

Quoted 12 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 12 lines

From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54

Quoted 8 lines

From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23

Quoted 3 lines · Cornelia Huck

On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has

been resolved.

Quoted 57 lines
Quoted 57 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH

flag.

Quoted 36 lines
Quoted 21 lines
Quoted 21 lines

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-

packed-ring.tex | 23 +++++++++++++++++++++++ split-ring.tex
|
26
++++++++++++++++++++++++--

 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex index 91735e3..8494eb6 renamed from content.tex +0 -0

100644

content.tex +0 -0
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic
 Facilities of a Virtio Device / Virtqueues}
 
Quoted 8 lines

content.tex

Quoted 8 lines
content.tex +0 -0
 Some devices always use descriptors in the same order in

which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge

content.tex +5 -0
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is

negotiated,

Quoted 4 lines
Quoted 1 line

+this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

Good point. I think so, yes. Will document.

Quoted 7 lines
Quoted 7 lines

Isn't it more natural to think of VIRTIO_F_IN_ORDER as the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible let VIRTIO_F_IN_ORDER imply the new bit is set, while the new bit set by itself without VIRTIO_F_IN_ORDER set means only active control is offered. I guess a name like VIRTIO_F_CTRL_ORDER would be more appropriate with this

interpretation.

Quoted 7 lines
Quoted 1 line

On second thought that might be a bit backwards - how about:

Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0 This
proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1 Potential future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1

What happens in the new device/old driver case?
- device offers IN_ORDER and PARTIAL_ORDER
- driver does not know PARTIAL_ORDER, accepts IN_ORDER
- device now only can do complete ordering

Maybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering IN_ORDER might not know about the new feature and its mechanism, and a device offering the new feature, but not IN_ORDER probably does so because it cannot support full IN_ORDER.

Quoted 3 lines

I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?

Quoted 5 lines

Yes, you are absolutely right. Keeping them as two mutually exclusive options is probably best.

Another aspect of the proposal is that the functionality is only described for the packed ring layout, but the concept should be equally applicable to the split ring.

Pls take a look at the changes in split-ring.tex
Are they unclear?

Quoted 10 lines

The split ring does not have
flags for each used ring element, so the FLUSH flag functionality needs to be a bit different. One way is to add an IN_ORDER-flag to the virtq_used.flags, that allows the device to signal when it starts to adhere to IN_ORDER buffer use. The driver acknowledges this when it is ready to fulfil its part of the IN_ORDER restrictions by setting a new virtq_avail.flag: IN_ORDER_ACK. At this point both device and driver can assume IN_ORDER optimization until the device clears IN_ORDER. After clearing IN_ORDER, the device must wait until IN_ORDER_ACK is cleared before IN_ORDER can be asserted it again.

It would be tricky to figure out which buffers does the handshake apply to. My proposal does it a bit differently, pls take a look.

Quoted 20 lines

This is a little less fine-grained than the packed-ring solution with the FLUSH flag, but for infrequent scenarios like migration it should be OK.

Quoted 7 lines

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Quoted 1 line

Disclaimer: This email and any files transmitted with it may contain confidential information intended for the addressee(s) only. The information is not to be surrendered or copied to unauthorized persons. If you have received this communication in error, please notify the sender immediately and delete this e-mail from your system.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

#9 lore raw
Quoted 132 lines

From: [email protected] <[email protected] open.org> On Behalf Of Michael S. Tsirkin
Sent: 14. august 2020 01:18

Quoted 16 lines · Lars Ganrot
Quoted 16 lines

From: [email protected] <[email protected] open.org> On Behalf Of Cornelia Huck
Sent: 12. august 2020 14:51

Quoted 9 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 9 lines

From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54

Quoted 5 lines

From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Cornelia Huck
Quoted 5 lines
Quoted 2 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of

order.

Quoted 76 lines
Quoted 76 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has

been resolved.

Quoted 68 lines
Quoted 68 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH

flag.

Quoted 33 lines
Quoted 23 lines
Quoted 23 lines

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++

split-ring.tex
|
26
++++++++++++++++++++++++--

 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex index renamed from content.tex +0 -0

91735e3..8494eb6
100644

content.tex +0 -0
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic
 Facilities of a Virtio Device / Virtqueues}
 
Quoted 9 lines

content.tex

Quoted 9 lines
content.tex +0 -0
 Some devices always use descriptors in the same order in

which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge

content.tex +5 -0
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been
+made available. These devices can offer the
+VIRTIO_F_PARTIAL_ORDER feature. If one of the features
+VIRTIO_F_IN_ORDER or VIRTIO_F_PARTIAL_ORDER is

negotiated,

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

+this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

Good point. I think so, yes. Will document.

Isn't it more natural to think of VIRTIO_F_IN_ORDER as the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible let VIRTIO_F_IN_ORDER imply the new bit is set, while the new bit set by itself without VIRTIO_F_IN_ORDER set means only active control is offered. I guess a name like VIRTIO_F_CTRL_ORDER would be more appropriate with this

Quoted 6 lines
Quoted 6 lines

interpretation.

Quoted 5 lines
Quoted 1 line

On second thought that might be a bit backwards - how about:

Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0

This

Quoted 1 line
Quoted 1 line
Quoted 1 line

proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1

Potential

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1

What happens in the new device/old driver case?
- device offers IN_ORDER and PARTIAL_ORDER
- driver does not know PARTIAL_ORDER, accepts IN_ORDER
- device now only can do complete ordering

Quoted 6 lines
Quoted 6 lines

Maybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering IN_ORDER might not know about the new feature and its mechanism, and a device offering the new feature, but not IN_ORDER probably does so because it cannot

support full IN_ORDER.

Quoted 5 lines
Quoted 4 lines

I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?

Quoted 9 lines
Quoted 1 line

Yes, you are absolutely right. Keeping them as two mutually exclusive

options is probably best.

Quoted 4 lines

Another aspect of the proposal is that the functionality is only described for the packed ring layout, but the concept should be equally applicable to the split ring.

Pls take a look at the changes in split-ring.tex Are they unclear?

I guess not. I failed to understand that the used-ring 32-bit ID field is divided into a 16-bit ID and a 16-bit flags. Sorry for the confusion.

Does this mean that for a device that always sets the FLUSH-flag, the PARTIAL_ORDER feature will be identical to IN_ORDER for both packed and split rings? The first write to a Used structure (after creation of the Virtqueue) would not have 2 FLUSH-flags, but would still fulfil the condition.

Quoted 45 lines
Quoted 10 lines

The split ring does not have
flags for each used ring element, so the FLUSH flag functionality needs to be a bit different. One way is to add an IN_ORDER-flag to the virtq_used.flags, that allows the device to signal when it starts to adhere to IN_ORDER buffer use. The driver acknowledges this when it is ready to fulfil its part of the IN_ORDER restrictions by setting a new virtq_avail.flag: IN_ORDER_ACK. At this point both device and driver can assume IN_ORDER optimization until the device clears IN_ORDER. After clearing IN_ORDER, the device must wait until IN_ORDER_ACK is cleared before IN_ORDER can be asserted it again.

It would be tricky to figure out which buffers does the handshake apply to. My proposal does it a bit differently, pls take a look.

Quoted 2 lines

This is a little less fine-grained than the packed-ring solution with the FLUSH

flag, but for infrequent scenarios like migration it should be OK.

Quoted 8 lines
Quoted 8 lines

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 11 lines
Quoted 10 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#10 lore raw
Quoted 139 lines · Lars Ganrot
Quoted 139 lines

From: [email protected] <[email protected] open.org> On Behalf Of Michael S. Tsirkin
Sent: 14. august 2020 01:18

Quoted 16 lines · Lars Ganrot
Quoted 16 lines

From: [email protected] <[email protected] open.org> On Behalf Of Cornelia Huck
Sent: 12. august 2020 14:51

Quoted 9 lines · On Tue, 11 Aug 2020 15:43:44 +0000 Lars Ganrot <[email protected]> wrote:
Quoted 9 lines

From: [email protected] <[email protected] open.org> On Behalf Of Lars Ganrot
Sent: 11. august 2020 16:54

Quoted 5 lines

From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Cornelia Huck
Quoted 5 lines
Quoted 2 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of

order.

Quoted 76 lines
Quoted 76 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has

been resolved.

Quoted 68 lines
Quoted 68 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as
follows:

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH

flag.

Quoted 33 lines
Quoted 23 lines
Quoted 23 lines

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++

split-ring.tex
|
26
++++++++++++++++++++++++--

 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex index renamed from content.tex +0 -0

91735e3..8494eb6
100644

content.tex +0 -0
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic
 Facilities of a Virtio Device / Virtqueues}
 
Quoted 9 lines

content.tex

Quoted 9 lines
content.tex +0 -0
 Some devices always use descriptors in the same order in

which they have been made available. These devices can offer the -VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge

content.tex +5 -0
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been
+made available. These devices can offer the
+VIRTIO_F_PARTIAL_ORDER feature. If one of the features
+VIRTIO_F_IN_ORDER or VIRTIO_F_PARTIAL_ORDER is

negotiated,

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

+this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

Good point. I think so, yes. Will document.

Isn't it more natural to think of VIRTIO_F_IN_ORDER as the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible let VIRTIO_F_IN_ORDER imply the new bit is set, while the new bit set by itself without VIRTIO_F_IN_ORDER set means only active control is offered. I guess a name like VIRTIO_F_CTRL_ORDER would be more appropriate with this

Quoted 6 lines
Quoted 6 lines

interpretation.

Quoted 5 lines
Quoted 1 line

On second thought that might be a bit backwards - how about:

Legacy case: VIRTIO_F_IN_ORDER==0/1 + VIRTIO_F_ORDER_RELAX==0

This

Quoted 1 line
Quoted 1 line
Quoted 1 line

proposal: VIRTIO_F_IN_ORDER==1 + VIRTIO_F_ORDER_RELAX==1

Potential

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

future use: VIRTIO_F_???_ORDER==1 + VIRTIO_F_ORDER_RELAX==0/1

What happens in the new device/old driver case?
- device offers IN_ORDER and PARTIAL_ORDER
- driver does not know PARTIAL_ORDER, accepts IN_ORDER
- device now only can do complete ordering

Quoted 6 lines
Quoted 6 lines

Maybe I don't understand the purpose of the new feature correctly, but I thought it was for those devices that don't do full in-order, but can do it for a subset of buffers? As such, the two features can't really imply each other: a device offering IN_ORDER might not know about the new feature and its mechanism, and a device offering the new feature, but not IN_ORDER probably does so because it cannot

support full IN_ORDER.

Quoted 6 lines
Quoted 4 lines

I think it makes the most sense if the device can offer both flags, but the driver must only accept at most one of them?

Yes, you are absolutely right. Keeping them as two mutually exclusive

options is probably best.

Quoted 4 lines

Another aspect of the proposal is that the functionality is only described for the packed ring layout, but the concept should be equally applicable to the split ring.

Quoted 8 lines
Quoted 2 lines

Pls take a look at the changes in split-ring.tex Are they unclear?

I guess not. I failed to understand that the used-ring 32-bit ID field is divided into a 16-bit ID and a 16-bit flags. Sorry for the confusion.

Does this mean that for a device that always sets the FLUSH-flag, the PARTIAL_ORDER feature will be identical to IN_ORDER for both packed and split rings?

Yes, I think so.

Quoted 3 lines

The first write to a Used structure (after creation
of the Virtqueue) would not have 2 FLUSH-flags, but would still fulfil the condition.

Needs thought.

Quoted 71 lines
Quoted 45 lines
Quoted 10 lines

The split ring does not have
flags for each used ring element, so the FLUSH flag functionality needs to be a bit different. One way is to add an IN_ORDER-flag to the virtq_used.flags, that allows the device to signal when it starts to adhere to IN_ORDER buffer use. The driver acknowledges this when it is ready to fulfil its part of the IN_ORDER restrictions by setting a new virtq_avail.flag: IN_ORDER_ACK. At this point both device and driver can assume IN_ORDER optimization until the device clears IN_ORDER. After clearing IN_ORDER, the device must wait until IN_ORDER_ACK is cleared before IN_ORDER can be asserted it again.

It would be tricky to figure out which buffers does the handshake apply to. My proposal does it a bit differently, pls take a look.

Quoted 2 lines

This is a little less fine-grained than the packed-ring solution with the FLUSH

flag, but for infrequent scenarios like migration it should be OK.

Quoted 8 lines
Quoted 8 lines

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 11 lines
Quoted 10 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#11 lore raw
In reply to Lars Ganrot
Quoted 74 lines · Lars Ganrot

Hi Michael,

Short comment below.

BR,

-Lars

Quoted 59 lines

From: [email protected] <[email protected]> On Behalf Of Michael S. Tsirkin
Sent: 11. august 2020 10:23

Quoted 46 lines · Cornelia Huck
Quoted 43 lines · On Mon, 10 Aug 2020 12:15:15 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++  split-ring.tex  | 26
++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)
content.tex +0 -0
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities
 of a Virtio Device / Virtqueues}
 
Quoted 7 lines

content.tex

Quoted 7 lines
content.tex +0 -0
 Some devices always use descriptors in the same order in which

they have been made available. These devices can offer the

content.tex +4 -1
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use descriptors
+in the same order in which they have been made available. These
+devices can offer the VIRTIO_F_PARTIAL_ORDER feature. If one of the
+features VIRTIO_F_IN_ORDER or VIRTIO_F_PARTIAL_ORDER is

negotiated,

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

+this knowledge

Do these two features conflict with each other? I.e., at most one of them may be negotiated (or offered?) at a time?

Good point. I think so, yes. Will document.

Quoted 8 lines

Isn't it more natural to think of VIRTIO_F_IN_ORDER as the simple case which always maintains ordered access, while the new feature flag allows active control of when descriptors are ordered and when not? To make it backward compatible let VIRTIO_F_IN_ORDER imply the new bit is set, while the new bit set by itself without VIRTIO_F_IN_ORDER set means only active control is offered. I guess a name like VIRTIO_F_CTRL_ORDER would be more appropriate with this interpretation.

So how does the behaviour with both VIRTIO_F_IN_ORDER with VIRTIO_F_PARTIAL_ORDER differ from just VIRTIO_F_IN_ORDER? If all buffers are used in order, then what is the extra meaning of FLUSH?

Quoted 6 lines

Disclaimer: This email and any files transmitted with it may contain confidential information intended for the addressee(s) only. The information is not to be surrendered or copied to unauthorized persons. If you have received this communication in error, please notify the sender immediately and delete this e-mail from your system.

I think we are supposed to disregard this part as irrlevant, but just a reminder that you agreed that all material posted to the virtio mailing lists is governed by the relevant IPR rules. It is your responsibility not to post confidential information there.

Signature · 7 lines
MST


---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 
#12 lore raw
In reply to Michael S. Tsirkin
Quoted 33 lines · Michael S. Tsirkin

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

Quoted 6 lines
---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++
 split-ring.tex  | 26 ++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)

Hi Michael,
What is the status of this feature?

There is a Google Summer of Code project to implement VIRTIO_F_IN_ORDER in QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).

I wanted to check if you still want to pursue PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.

Thanks to Ilya Maximets for mentioning PARTIAL_ORDER!

Stefan

Quoted 116 lines
content.tex +8 -1
index 91735e3..8494eb6 100644
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 
 Some devices always use descriptors in the same order in which
 they have been made available. These devices can offer the
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge
 might allow optimizations or simplify driver and/or device code.
 
 Each virtqueue can consist of up to 3 parts:
@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver passes extra data (besides identifying the virtqueue)
   in its device notifications.
   See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
+  \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates
+  that device has ability to indicate use of (some of) buffers by the device in the same
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +23 -0
index ea92543..a120a19 100644
@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors}
 only writing out a single used descriptor with the Buffer ID
 corresponding to the last descriptor in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_DESC_F_FLUSH flag in the used descriptor.
+\begin{lstlisting}
+#define VIRTQ_DESC_F_FLUSH      8
+\end{lstlisting}
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch,
+and VIRTQ_DESC_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_DESC_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. The driver needs to look up the used Buffer ID and
 calculate the batch size to be able to advance to where the next
split-ring.tex +24 -2
index 123ac9f..cf197f8 100644
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi
         le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
 };
 
-/* le32 is used here for ids for padding reasons. */
 struct virtq_used_elem {
         /* Index of start of used descriptor chain. */
-        le32 id;
+        le16 id;
+#define VIRTQ_USED_ELEM_F_FLUSH  0x8000
+        le16 flags;
         /* Total length of the descriptor chain which was used (written to) */
         le32 len;
 };
@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors}
 corresponding to the head entry of the
 descriptor chain describing the last buffer in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry.
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out single used ring entry with the \field{id}
+corresponding to the head entry of the
+descriptor chain describing the last buffer in the batch,
+and VIRTQ_USED_ELEM_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. Accordingly, it increments the used \field{idx} by the
 size of the batch.
Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#13 lore raw
Quoted 45 lines · Stefan Hajnoczi
Quoted 33 lines · Michael S. Tsirkin

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

Quoted 6 lines
---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++
 split-ring.tex  | 26 ++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)

Hi Michael,
What is the status of this feature?

Quoted 8 lines

There is a Google Summer of Code project to implement VIRTIO_F_IN_ORDER in QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).

I wanted to check if you still want to pursue PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.

If the status is close to being merged I'm totally in to implement either partial order first or to implement both in the project. In some sense it superseded IN_ORDER.

Thanks!

Quoted 116 lines

Thanks to Ilya Maximets for mentioning PARTIAL_ORDER!

Stefan

Quoted 112 lines
content.tex +8 -1
index 91735e3..8494eb6 100644
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 
 Some devices always use descriptors in the same order in which
 they have been made available. These devices can offer the
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge
 might allow optimizations or simplify driver and/or device code.
 
 Each virtqueue can consist of up to 3 parts:
@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver passes extra data (besides identifying the virtqueue)
   in its device notifications.
   See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
+  \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates
+  that device has ability to indicate use of (some of) buffers by the device in the same
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +23 -0
index ea92543..a120a19 100644
@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors}
 only writing out a single used descriptor with the Buffer ID
 corresponding to the last descriptor in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_DESC_F_FLUSH flag in the used descriptor.
+\begin{lstlisting}
+#define VIRTQ_DESC_F_FLUSH      8
+\end{lstlisting}
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch,
+and VIRTQ_DESC_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_DESC_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. The driver needs to look up the used Buffer ID and
 calculate the batch size to be able to advance to where the next
split-ring.tex +24 -3
index 123ac9f..cf197f8 100644
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi
         le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
 };
 
-/* le32 is used here for ids for padding reasons. */
 struct virtq_used_elem {
         /* Index of start of used descriptor chain. */
-        le32 id;
+        le16 id;
+#define VIRTQ_USED_ELEM_F_FLUSH  0x8000
+        le16 flags;
         /* Total length of the descriptor chain which was used (written to) */
         le32 len;
 };
@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors}
 corresponding to the head entry of the
 descriptor chain describing the last buffer in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry.
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out single used ring entry with the \field{id}
+corresponding to the head entry of the
+descriptor chain describing the last buffer in the batch,
+and VIRTQ_USED_ELEM_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. Accordingly, it increments the used \field{idx} by the
 size of the batch.
--

MST

Quoted 4 lines
Quoted 4 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#14 lore raw
Quoted 55 lines · Eugenio Perez Martin
Quoted 45 lines · Stefan Hajnoczi
Quoted 33 lines · Michael S. Tsirkin

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

Quoted 6 lines
---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++
 split-ring.tex  | 26 ++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)

Hi Michael,
What is the status of this feature?

Quoted 8 lines

There is a Google Summer of Code project to implement VIRTIO_F_IN_ORDER in QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).

I wanted to check if you still want to pursue PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.

Quoted 5 lines

If the status is close to being merged I'm totally in to implement either partial order first or to implement both in the project. In some sense it superseded IN_ORDER.

Thanks!

I think it's blocked on lack of implementation actually. TC was concerned whether there are any correctness issues. An implementation which can be stress-tested would resolve this.

Quoted 116 lines
Quoted 116 lines

Thanks to Ilya Maximets for mentioning PARTIAL_ORDER!

Stefan

Quoted 112 lines
content.tex +8 -1
index 91735e3..8494eb6 100644
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 
 Some devices always use descriptors in the same order in which
 they have been made available. These devices can offer the
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge
 might allow optimizations or simplify driver and/or device code.
 
 Each virtqueue can consist of up to 3 parts:
@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver passes extra data (besides identifying the virtqueue)
   in its device notifications.
   See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
+  \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates
+  that device has ability to indicate use of (some of) buffers by the device in the same
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +23 -0
index ea92543..a120a19 100644
@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors}
 only writing out a single used descriptor with the Buffer ID
 corresponding to the last descriptor in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_DESC_F_FLUSH flag in the used descriptor.
+\begin{lstlisting}
+#define VIRTQ_DESC_F_FLUSH      8
+\end{lstlisting}
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch,
+and VIRTQ_DESC_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_DESC_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. The driver needs to look up the used Buffer ID and
 calculate the batch size to be able to advance to where the next
split-ring.tex +24 -3
index 123ac9f..cf197f8 100644
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi
         le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
 };
 
-/* le32 is used here for ids for padding reasons. */
 struct virtq_used_elem {
         /* Index of start of used descriptor chain. */
-        le32 id;
+        le16 id;
+#define VIRTQ_USED_ELEM_F_FLUSH  0x8000
+        le16 flags;
         /* Total length of the descriptor chain which was used (written to) */
         le32 len;
 };
@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors}
 corresponding to the head entry of the
 descriptor chain describing the last buffer in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry.
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out single used ring entry with the \field{id}
+corresponding to the head entry of the
+descriptor chain describing the last buffer in the batch,
+and VIRTQ_USED_ELEM_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. Accordingly, it increments the used \field{idx} by the
 size of the batch.
--

MST

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#15 lore raw
Quoted 61 lines · Michael S. Tsirkin
Quoted 59 lines · Eugenio Perez Martin
Quoted 45 lines · Stefan Hajnoczi
Quoted 33 lines · Michael S. Tsirkin

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

Quoted 6 lines
---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++
 split-ring.tex  | 26 ++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)

Hi Michael,
What is the status of this feature?

Quoted 8 lines

There is a Google Summer of Code project to implement VIRTIO_F_IN_ORDER in QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).

I wanted to check if you still want to pursue PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.

If the status is close to being merged I'm totally in to implement either partial order first or to implement both in the project. In some sense it superseded IN_ORDER.

Quoted 6 lines
Quoted 1 line

Thanks!

I think it's blocked on lack of implementation actually. TC was concerned whether there are any correctness issues. An implementation which can be stress-tested would resolve this.

Got it, so then it is worth trying to allocate some time to implement it IMO.

Thanks!

Quoted 117 lines
Quoted 117 lines
Quoted 117 lines

Thanks to Ilya Maximets for mentioning PARTIAL_ORDER!

Stefan

Quoted 112 lines
content.tex +8 -1
index 91735e3..8494eb6 100644
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 
 Some devices always use descriptors in the same order in which
 they have been made available. These devices can offer the
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge
 might allow optimizations or simplify driver and/or device code.
 
 Each virtqueue can consist of up to 3 parts:
@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver passes extra data (besides identifying the virtqueue)
   in its device notifications.
   See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
+  \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates
+  that device has ability to indicate use of (some of) buffers by the device in the same
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +23 -0
index ea92543..a120a19 100644
@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors}
 only writing out a single used descriptor with the Buffer ID
 corresponding to the last descriptor in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_DESC_F_FLUSH flag in the used descriptor.
+\begin{lstlisting}
+#define VIRTQ_DESC_F_FLUSH      8
+\end{lstlisting}
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch,
+and VIRTQ_DESC_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_DESC_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. The driver needs to look up the used Buffer ID and
 calculate the batch size to be able to advance to where the next
split-ring.tex +24 -3
index 123ac9f..cf197f8 100644
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi
         le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
 };
 
-/* le32 is used here for ids for padding reasons. */
 struct virtq_used_elem {
         /* Index of start of used descriptor chain. */
-        le32 id;
+        le16 id;
+#define VIRTQ_USED_ELEM_F_FLUSH  0x8000
+        le16 flags;
         /* Total length of the descriptor chain which was used (written to) */
         le32 len;
 };
@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors}
 corresponding to the head entry of the
 descriptor chain describing the last buffer in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry.
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out single used ring entry with the \field{id}
+corresponding to the head entry of the
+descriptor chain describing the last buffer in the batch,
+and VIRTQ_USED_ELEM_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. Accordingly, it increments the used \field{idx} by the
 size of the batch.
--

MST

Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#16 lore raw
In reply to Stefan Hajnoczi
Quoted 48 lines · Stefan Hajnoczi
Quoted 33 lines · Michael S. Tsirkin

Devices that normally use buffers in order can benefit from ability to temporarily switch to handle some buffers out of order.

As a case in point, a networking device might handle RX buffers in order normally. However, should an access to an RX buffer cause a page fault (e.g. when using PRI), the device could benefit from ability to temporarily keep using following buffers in the ring (possibly with higher overhead) until the fault has been resolved.

Page faults allow more features such as THP, auto-NUMA, live migration.

Out of order is of course already possible, however, IN_ORDER is currently required for descriptor batching where device marks a whole batch of buffers used in one go.

The idea behind this proposal is to relax that requirement, allowing batching without asking device to be in orde rat all times, as follows:

Device uses buffers in any order. Eventually when device detects that it has used all previously outstanding buffers, it sets a FLUSH flag on the last buffer used. If it set this flag on the last buffer used previously, and now uses a batch of descriptors in-order, it can now signal the last buffer used again setting the FLUSH flag.

Driver can detect in-order when it sees two FLUSH flags one after another. In other respects the feature is similar to IN_ORDER from the driver implementation POV.

Quoted 6 lines
---
 content.tex     |  9 ++++++++-
 packed-ring.tex | 23 +++++++++++++++++++++++
 split-ring.tex  | 26 ++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 3 deletions(-)

Hi Michael,
What is the status of this feature?

There is a Google Summer of Code project to implement VIRTIO_F_IN_ORDER in QEMU and Linux
(https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices).

Quoted 7 lines

I wanted to check if you still want to pursue PARTIAL_ORDER. Maybe Stefano and Eugenio (the mentors) will find a way to connect it to the GSoC project.

Thanks to Ilya Maximets for mentioning PARTIAL_ORDER!

Stefan

Yes I do. Would be great if we had someone who can work on this!

Quoted 112 lines
Quoted 112 lines
content.tex +8 -1
index 91735e3..8494eb6 100644
@@ -296,7 +296,11 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 
 Some devices always use descriptors in the same order in which
 they have been made available. These devices can offer the
-VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge
+VIRTIO_F_IN_ORDER feature.  Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If one of the features VIRTIO_F_IN_ORDER or
+VIRTIO_F_PARTIAL_ORDER is negotiated, this knowledge
 might allow optimizations or simplify driver and/or device code.
 
 Each virtqueue can consist of up to 3 parts:
@@ -6132,6 +6136,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver passes extra data (besides identifying the virtqueue)
   in its device notifications.
   See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
+  \item[VIRTIO_F_PARTIAL_ORDER(39)] This feature indicates
+  that device has ability to indicate use of (some of) buffers by the device in the same
+  order in which they have been made available.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +23 -0
index ea92543..a120a19 100644
@@ -284,6 +284,29 @@ \subsection{In-order use of descriptors}
 only writing out a single used descriptor with the Buffer ID
 corresponding to the last descriptor in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_DESC_F_FLUSH flag in the used descriptor.
+\begin{lstlisting}
+#define VIRTQ_DESC_F_FLUSH      8
+\end{lstlisting}
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out a single used descriptor with the Buffer ID
+corresponding to the last descriptor in the batch,
+and VIRTQ_DESC_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used descriptor also has the VIRTQ_DESC_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_DESC_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. The driver needs to look up the used Buffer ID and
 calculate the batch size to be able to advance to where the next
split-ring.tex +24 -2
index 123ac9f..cf197f8 100644
@@ -398,10 +398,11 @@ \subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Devi
         le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */
 };
 
-/* le32 is used here for ids for padding reasons. */
 struct virtq_used_elem {
         /* Index of start of used descriptor chain. */
-        le32 id;
+        le16 id;
+#define VIRTQ_USED_ELEM_F_FLUSH  0x8000
+        le16 flags;
         /* Total length of the descriptor chain which was used (written to) */
         le32 len;
 };
@@ -481,6 +482,27 @@ \subsection{In-order use of descriptors}
 corresponding to the head entry of the
 descriptor chain describing the last buffer in the batch.
 
+Other devices sometimes use
+descriptors in the same order in which they have been made
+available. These devices can offer the VIRTIO_F_PARTIAL_ORDER
+feature. If negotiated, whenever device has used all buffers
+since the previous used buffer in the same order
+in which they have been made available, device can set the
+VIRTQ_USED_ELEM_F_FLUSH flag in the used ring entry.
+
+This knowledge allows
+devices to notify the use of a batch of buffers to the driver by
+only writing out single used ring entry with the \field{id}
+corresponding to the head entry of the
+descriptor chain describing the last buffer in the batch,
+and VIRTQ_USED_ELEM_F_FLUSH set.
+
+Note that device is only allowed to batch buffers in this way
+if the previous used ring entry also has the VIRTQ_USED_ELEM_F_FLUSH
+flag set, as a result, considering the group of buffers
+used between two buffers with VIRTQ_USED_ELEM_F_FLUSH set,
+either all of them constitute a batch, or none at all.
+
 The device then skips forward in the ring according to the size of
 the batch. Accordingly, it increments the used \field{idx} by the
 size of the batch.
Signature · 1 line
MST
Quoted 4 lines
Quoted 4 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]