Skip to contentSkip to conversation

Add VIRTIO_RING_F_INDIRECT_SIZE and queue_indirect_size

PATCH Superseded Christian Schoenebeck 3 participants 25 messages Dec 14, 2021 – Feb 21, 2022

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#1 lore raw

PATCH 1/2 Add VIRTIO_RING_F_INDIRECT_SIZE

In reply to Christian Schoenebeck

This new feature flag allows to decouple the maximum amount of descriptors in indirect descriptor tables from the Queue Size.

---
 content.tex    | 21 +++++++++++++++++++++
 split-ring.tex |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
content.tex +21 -0
index 5d112af..0aa4842 100644
@@ -6693,6 +6693,27 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_RING_F_INDIRECT_SIZE(40)] This feature indicates that the
+  maximum amount of descriptors in indirect descriptor tables is independent
+  from the Queue Size.
+
+  If this feature bit is not negotiated, then a driver MUST NOT create a
+  descriptor chain longer than the Queue Size, which then also applies to
+  indirect descriptor tables as in \ref{sec:Basic Facilities of a Virtio
+  Device / Virtqueues / The Virtqueue Descriptor Table / Indirect
+  Descriptors}. Which means without this feature, the Queue Size limits
+  both the maximum amount of slots in the vring, as well as the actual
+  bulk data size being transmitted per vring slot.
+
+  With this feature enabled, the Queue Size only limits the maximum amount
+  of slots in the vring, but does not limit the actual bulk data size
+  being transmitted when indirect descriptors are used. Decoupling these
+  two configuration parameters this way not only allows much larger bulk data
+  being transferred per vring slot, but also avoids complicated synchronization
+  mechanisms if the device only supports a very small amount of vring slots. Due
+  to the 16-bit size of a descriptor's "next" field there is still an absolute
+  limit of $2^{16}$ descriptors per indirect descriptor table.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
split-ring.tex +1 -1
index bfef62d..ae2aeb8 100644
@@ -269,7 +269,7 @@ \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi
 one table per descriptor).
 
 A driver MUST NOT create a descriptor chain longer than the Queue Size of
-the device.
+the device unless VIRTIO_RING_F_INDIRECT_SIZE has been negotiated.
 
 A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
 in \field{flags}.
Signature · 1 line
2.20.1
#8 lore raw

PATCH 2/2 Add common configuration field "queue_indirect_size"

In reply to Christian Schoenebeck

This new common configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.

---
 content.tex    | 25 ++++++++++++++++++++++++-
 split-ring.tex |  3 +++
 2 files changed, 27 insertions(+), 1 deletion(-)
content.tex +24 -1
index 0aa4842..e3cfcae 100644
@@ -859,6 +859,7 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         le64 queue_driver;              /* read-write */
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
+        le32 queue_indirect_size;       /* read-write */
 };
 \end{lstlisting}
 
@@ -938,6 +939,16 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         may benefit from providing another value, for example an internal virtqueue
         identifier, or an internal offset related to the virtqueue number.
         \end{note}
+
+\item[\field{queue_indirect_size}]
+        This field is used to negotiate the maximum amount of descriptors per
+        vring slot as in \ref{sec:Basic Facilities of a Virtio Device /
+        Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors} if
+        and only if the VIRTIO_RING_F_INDIRECT_SIZE feature has been negotiated.
+
+        The device specifies its maximum supported number of descriptors per
+        vring slot. If the driver requires fewer descriptors, it writes its
+        lower value to inform the device of the reduced resource requirements.
 \end{description}
 
 \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -1003,6 +1014,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
 The driver MUST NOT write a 0 to \field{queue_enable}.
 
+The driver SHOULD write to \field{queue_indirect_size} if its maximum number of
+descriptors per vring slot is lower than that reported by the device.
+
+The driver MUST NOT write a higher value to \field{queue_indirect_size} than the
+one it reads from the device.
+
 \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 
 The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
@@ -6712,7 +6729,13 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   being transferred per vring slot, but also avoids complicated synchronization
   mechanisms if the device only supports a very small amount of vring slots. Due
   to the 16-bit size of a descriptor's "next" field there is still an absolute
-  limit of $2^{16}$ descriptors per indirect descriptor table.
+  limit of $2^{16}$ descriptors per indirect descriptor table. However the
+  actual maximum amount supported by either device or driver might be less,
+  and therefore the common configuration field "queue_indirect_size" MUST exist
+  if VIRTIO_RING_F_INDIRECT_SIZE was negotiated to subsequently
+  negotiate the actual amount of maximum indirect descriptors supported
+  by both sides, as described in \ref{sec:Virtio Transport Options / Virtio
+  Over PCI Bus / PCI Device Layout / Common configuration structure layout}.
 
 \end{description}
 
split-ring.tex +3 -0
index ae2aeb8..d8f66c1 100644
@@ -270,6 +270,9 @@ \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi
 
 A driver MUST NOT create a descriptor chain longer than the Queue Size of
 the device unless VIRTIO_RING_F_INDIRECT_SIZE has been negotiated.
+Furthermore if VIRTIO_RING_F_INDIRECT_SIZE has been negotiated then the number
+of descriptors per vring slot MUST NOT exceed the value negotiated by common
+configuration field "queue_indirect_size".
 
 A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
 in \field{flags}.
Signature · 1 line
2.20.1
#0 lore raw

cover Add VIRTIO_RING_F_INDIRECT_SIZE and queue_indirect_size

This is a follow-up of:
[0/2]: https://lists.oasis-open.org/archives/virtio-comment/202111/msg00089.html
[1/2]: https://lists.oasis-open.org/archives/virtio-comment/202112/msg00005.html
[2/2]: https://lists.oasis-open.org/archives/virtio-comment/202112/msg00006.html

These two patches introduce a more fine graded control over the maximum length of indirect descriptor tables.

Associated Github Task:
https://github.com/oasis-tcs/virtio-spec/issues/122

Changes:

* VIRTIO_RING_F_LARGE_INDIRECT_DESC -> VIRTIO_RING_F_INDIRECT_SIZE

    [patch 1], [patch 2]

* queue_indirect_size field: 16 bit -> 32 bit [patch 2]

* Semantic change: "per indirect descriptor table" -> "per vring slot".

    [patch 2]

* Revised first paragraph of VIRTIO_RING_F_INDIRECT_SIZE(40) to be more clear that it is not only about exceeding the queue size
("This feature indicates ... "). [patch 1]

* Drop mentioning "revision 3". [patch 2]

* revised commit log [patch 1]

* Grammar: "... but does not limit the actual bulk data size ...". [patch 1]

* Grammar: "... if the device ...". [patch 1]

* queue_indirect_size field may only be used if VIRTIO_RING_F_INDIRECT_SIZE was used. [patch 2]

* Drop usage of SHOULD/MUST in common description. [patch 2]

* Grammar: had been -> was [patch 2]

* Add requirements for field 'queue_indirect_size' to driver normative section. [patch 2]

Christian Schoenebeck (2):
Add VIRTIO_RING_F_INDIRECT_SIZE
Add common configuration field "queue_indirect_size"

 content.tex    | 44 ++++++++++++++++++++++++++++++++++++++++++++
 split-ring.tex |  5 ++++-
 2 files changed, 48 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
#2 lore raw

[virtio-comment] Re: [PATCH 1/2] Add VIRTIO_RING_F_INDIRECT_SIZE

In reply to Christian Schoenebeck
Quoted 15 lines · Christian Schoenebeck

This new feature flag allows to decouple the maximum amount of descriptors in indirect descriptor tables from the Queue Size.

---
 content.tex    | 21 +++++++++++++++++++++
 split-ring.tex |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
content.tex +0 -0
index 5d112af..0aa4842 100644

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6700

Quoted 8 lines
content.tex +5 -0
@@ -6693,6 +6693,27 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_RING_F_INDIRECT_SIZE(40)] This feature indicates that the
+  maximum amount of descriptors in indirect descriptor tables is independent
+  from the Queue Size.
+
+  If this feature bit is not negotiated, then a driver MUST NOT create a

"MUST NOT" does only belong into normative sections... maybe you can reword the last sentence in this paragraph a bit and move the "MUST NOT" part into a normative section?

Quoted 24 lines
content.tex +16 -0
+  descriptor chain longer than the Queue Size, which then also applies to
+  indirect descriptor tables as in \ref{sec:Basic Facilities of a Virtio
+  Device / Virtqueues / The Virtqueue Descriptor Table / Indirect
+  Descriptors}. Which means without this feature, the Queue Size limits
+  both the maximum amount of slots in the vring, as well as the actual
+  bulk data size being transmitted per vring slot.
+
+  With this feature enabled, the Queue Size only limits the maximum amount
+  of slots in the vring, but does not limit the actual bulk data size
+  being transmitted when indirect descriptors are used. Decoupling these
+  two configuration parameters this way not only allows much larger bulk data
+  being transferred per vring slot, but also avoids complicated synchronization
+  mechanisms if the device only supports a very small amount of vring slots. Due
+  to the 16-bit size of a descriptor's "next" field there is still an absolute
+  limit of $2^{16}$ descriptors per indirect descriptor table.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
split-ring.tex +0 -0
index bfef62d..ae2aeb8 100644

split-ring.tex \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi · line 272

Quoted 5 lines
split-ring.tex +1 -1
@@ -269,7 +269,7 @@ \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi
 one table per descriptor).
 
 A driver MUST NOT create a descriptor chain longer than the Queue Size of
-the device.
+the device unless VIRTIO_RING_F_INDIRECT_SIZE has been negotiated.

This might already be enough of a normative statement, though.

split-ring.tex \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi · line 275

Quoted 3 lines
split-ring.tex +0 -0
 A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
 in \field{flags}.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#9 lore raw

[virtio-comment] Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

In reply to Christian Schoenebeck
Quoted 15 lines · Christian Schoenebeck

This new common configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.

---
 content.tex    | 25 ++++++++++++++++++++++++-
 split-ring.tex |  3 +++
 2 files changed, 27 insertions(+), 1 deletion(-)
content.tex +0 -0
index 0aa4842..e3cfcae 100644

content.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 862

Quoted 4 lines
content.tex +1 -0
@@ -859,6 +859,7 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         le64 queue_driver;              /* read-write */
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
+        le32 queue_indirect_size;       /* read-write */

[Note for below: this "common" configuration structure layout is actually PCI-specific, it is only common between the different device types.]

Quoted 21 lines
content.tex +10 -0
 };
 \end{lstlisting}
 
@@ -938,6 +939,16 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         may benefit from providing another value, for example an internal virtqueue
         identifier, or an internal offset related to the virtqueue number.
         \end{note}
+
+\item[\field{queue_indirect_size}]
+        This field is used to negotiate the maximum amount of descriptors per
+        vring slot as in \ref{sec:Basic Facilities of a Virtio Device /
+        Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors} if
+        and only if the VIRTIO_RING_F_INDIRECT_SIZE feature has been negotiated.
+
+        The device specifies its maximum supported number of descriptors per
+        vring slot. If the driver requires fewer descriptors, it writes its
+        lower value to inform the device of the reduced resource requirements.
 \end{description}
 
 \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}

content.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 1018

Quoted 5 lines
content.tex +2 -0
@@ -1003,6 +1014,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
 The driver MUST NOT write a 0 to \field{queue_enable}.
 
+The driver SHOULD write to \field{queue_indirect_size} if its maximum number of
+descriptors per vring slot is lower than that reported by the device.

Maybe

"If the driver's maximum number of descriptors per vring slot is lower than the maximum value reported by the device, it SHOULD write that number to \field{queue_indirect_size}."

?

Quoted 27 lines
content.tex +11 -1
+
+The driver MUST NOT write a higher value to \field{queue_indirect_size} than the
+one it reads from the device.
+
 \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 
 The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
@@ -6712,7 +6729,13 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   being transferred per vring slot, but also avoids complicated synchronization
   mechanisms if the device only supports a very small amount of vring slots. Due
   to the 16-bit size of a descriptor's "next" field there is still an absolute
-  limit of $2^{16}$ descriptors per indirect descriptor table.
+  limit of $2^{16}$ descriptors per indirect descriptor table. However the
+  actual maximum amount supported by either device or driver might be less,
+  and therefore the common configuration field "queue_indirect_size" MUST exist
+  if VIRTIO_RING_F_INDIRECT_SIZE was negotiated to subsequently
+  negotiate the actual amount of maximum indirect descriptors supported
+  by both sides, as described in \ref{sec:Virtio Transport Options / Virtio
+  Over PCI Bus / PCI Device Layout / Common configuration structure layout}.
 
 \end{description}
 
split-ring.tex +0 -0
index ae2aeb8..d8f66c1 100644

split-ring.tex \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi · line 275

Quoted 6 lines
split-ring.tex +3 -0
@@ -270,6 +270,9 @@ \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi
 
 A driver MUST NOT create a descriptor chain longer than the Queue Size of
 the device unless VIRTIO_RING_F_INDIRECT_SIZE has been negotiated.
+Furthermore if VIRTIO_RING_F_INDIRECT_SIZE has been negotiated then the number
+of descriptors per vring slot MUST NOT exceed the value negotiated by common
+configuration field "queue_indirect_size".

As mentioned above, the "common configuration layout" is actually PCI-specific; other transports will use different mechanism. Maybe it would make sense to reword the whole paragraph and add it in patch 1?

"If VIRTIO_RING_F_INDIRECT_SIZE has not been negotiated, the driver MUST NOT create a descriptor chain longer than the Queue Size of the device.

If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated, the number of descriptors per vring slot MUST NOT exceed the negotiated Queue Indirect Size."

I also wonder whether we need a device normative statement as well, something like:

"With VIRTIO_RING_F_INDIRECT_SIZE, the device MUST provide the maximum Queue Indirect Size for the number of descriptors per vring slot. It MUST allow the driver to set a lower value."

Maybe we also need to mention that the mechanism is transport specific?

Also, this is only for split ring; does packed ring need any updates?

split-ring.tex \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi · line 278

Quoted 3 lines
split-ring.tex +0 -0
 A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT
 in \field{flags}.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#3 lore raw

[virtio-comment] Re: [PATCH 1/2] Add VIRTIO_RING_F_INDIRECT_SIZE

In reply to Cornelia Huck
Quoted 23 lines · On Dienstag, 14. Dezember 2021 17:59:51 CET Cornelia Huck wrote:
Quoted 22 lines · Christian Schoenebeck

This new feature flag allows to decouple the maximum amount of descriptors in indirect descriptor tables from the Queue Size.

---

 content.tex    | 21 +++++++++++++++++++++
 split-ring.tex |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
content.tex +0 -0
index 5d112af..0aa4842 100644
@@ -6693,6 +6693,27 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved

Feature Bits}>
transport specific.
For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.>
+ \item[VIRTIO_RING_F_INDIRECT_SIZE(40)] This feature indicates that the

Quoted 8 lines

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6700

Quoted 4 lines

+ maximum amount of descriptors in indirect descriptor tables is independent + from the Queue Size.

content.tex +2 -0
+
+  If this feature bit is not negotiated, then a driver MUST NOT create a

"MUST NOT" does only belong into normative sections... maybe you can reword the last sentence in this paragraph a bit and move the "MUST NOT" part into a normative section?

I lack creativity in this case. Suggestions appreciated. On doubt I would just delete the entire paragraph here ("If this feature ... transmitted per vring slot.").

Quoted 31 lines
Quoted 31 lines
content.tex +8 -0
+  descriptor chain longer than the Queue Size, which then also applies to
+  indirect descriptor tables as in \ref{sec:Basic Facilities of a Virtio
+  Device / Virtqueues / The Virtqueue Descriptor Table / Indirect
+  Descriptors}. Which means without this feature, the Queue Size limits
+  both the maximum amount of slots in the vring, as well as the actual
+  bulk data size being transmitted per vring slot.
+
+  With this feature enabled, the Queue Size only limits the maximum

amount

content.tex +3 -0
+  of slots in the vring, but does not limit the actual bulk data size
+  being transmitted when indirect descriptors are used. Decoupling these
+  two configuration parameters this way not only allows much larger bulk

data + being transferred per vring slot, but also avoids complicated synchronization + mechanisms if the device only supports a very small amount of vring slots. Due + to the 16-bit size of a descriptor's "next" field there is still an absolute + limit of $2^{16}$ descriptors per indirect descriptor table.

content.tex +1 -0
+
 
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
split-ring.tex +0 -0
index bfef62d..ae2aeb8 100644
@@ -269,7 +269,7 @@ \subsubsection{Indirect Descriptors}\label{sec:Basic

Facilities of a Virtio Devi>
one table per descriptor).

Quoted 9 lines

split-ring.tex \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi · line 272

Quoted 4 lines

A driver MUST NOT create a descriptor chain longer than the Queue Size of

split-ring.tex +1 -1
-the device.
+the device unless VIRTIO_RING_F_INDIRECT_SIZE has been negotiated.

This might already be enough of a normative statement, though.

Quoted 2 lines

A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT in \field{flags}.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#10 lore raw

Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

In reply to Cornelia Huck
Quoted 22 lines · On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck wrote:
Quoted 21 lines · Christian Schoenebeck

This new common configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.

---

 content.tex    | 25 ++++++++++++++++++++++++-
 split-ring.tex |  3 +++
 2 files changed, 27 insertions(+), 1 deletion(-)
content.tex +0 -0
index 0aa4842..e3cfcae 100644
@@ -859,6 +859,7 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>

         le64 queue_driver;              /* read-write */
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
Quoted 5 lines
Quoted 1 line

+ le32 queue_indirect_size; /* read-write */

[Note for below: this "common" configuration structure layout is actually PCI-specific, it is only common between the different device types.]

True

Quoted 36 lines
Quoted 31 lines

};
\end{lstlisting}

content.tex +0 -0
@@ -938,6 +939,16 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>
may benefit from providing another value, for example an internal virtqueue
identifier, or an internal offset related to the virtqueue number.
\end{note}

content.tex +3 -0
+
+\item[\field{queue_indirect_size}]
+        This field is used to negotiate the maximum amount of descriptors

per + vring slot as in \ref{sec:Basic Facilities of a Virtio
Device / + Virtqueues / The Virtqueue Descriptor Table / Indirect
Descriptors} if + and only if the VIRTIO_RING_F_INDIRECT_SIZE
feature has been negotiated. +
+ The device specifies its maximum supported number of descriptors
per + vring slot. If the driver requires fewer descriptors, it
writes its + lower value to inform the device of the reduced
resource requirements.>
\end{description}

\devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}>

content.tex +0 -0
@@ -1003,6 +1014,12 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>
The driver MUST NOT write a 0 to \field{queue_enable}.

Quoted 3 lines

+The driver SHOULD write to \field{queue_indirect_size} if its maximum number of +descriptors per vring slot is lower than that reported by the device.

Maybe

Quoted 5 lines

"If the driver's maximum number of descriptors per vring slot is lower than the maximum value reported by the device, it SHOULD write that number to \field{queue_indirect_size}."

?

I actually just used Stefan's wording here and extended it with semantically required components, i.e. "vring slot" and "\field{queue_indirect_size}":
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00006.html

For me both versions are fine.

Quoted 47 lines
Quoted 39 lines
content.tex +2 -0
+
+The driver MUST NOT write a higher value to \field{queue_indirect_size}

than the +one it reads from the device.

content.tex +1 -0
+
 
 \subsubsection{Notification structure layout}\label{sec:Virtio Transport
 Options / Virtio Over PCI Bus / PCI Device Layout / Notification
 capability}
 
 The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
@@ -6712,7 +6729,13 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved

Feature Bits}>
being transferred per vring slot, but also avoids complicated synchronization mechanisms if the device only supports a very small amount of vring slots. Due to the 16-bit size of a descriptor's "next" field there is still an absolute>

content.tex +1 -1
-  limit of $2^{16}$ descriptors per indirect descriptor table.
+  limit of $2^{16}$ descriptors per indirect descriptor table. However

the
+ actual maximum amount supported by either device or driver might be less, + and therefore the common configuration field "queue_indirect_size" MUST exist + if VIRTIO_RING_F_INDIRECT_SIZE was negotiated to subsequently

content.tex +2 -0
+  negotiate the actual amount of maximum indirect descriptors supported
+  by both sides, as described in \ref{sec:Virtio Transport Options /

Virtio + Over PCI Bus / PCI Device Layout / Common configuration structure layout}.>
\end{description}

split-ring.tex +0 -0
index ae2aeb8..d8f66c1 100644
@@ -270,6 +270,9 @@ \subsubsection{Indirect Descriptors}\label{sec:Basic

Facilities of a Virtio Devi>
A driver MUST NOT create a descriptor chain longer than the Queue Size of the device unless VIRTIO_RING_F_INDIRECT_SIZE has been negotiated.

Quoted 3 lines

+Furthermore if VIRTIO_RING_F_INDIRECT_SIZE has been negotiated then the number +of descriptors per vring slot MUST NOT exceed the value negotiated by common +configuration field "queue_indirect_size".

As mentioned above, the "common configuration layout" is actually PCI-specific; other transports will use different mechanism. Maybe it would make sense to reword the whole paragraph and add it in patch 1?

Quoted 6 lines

"If VIRTIO_RING_F_INDIRECT_SIZE has not been negotiated, the driver MUST NOT create a descriptor chain longer than the Queue Size of the device.

If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated, the number of descriptors per vring slot MUST NOT exceed the negotiated Queue Indirect Size."

That simplifies the statements, yes.

About the term "Queue Indirect Size": I understand your point about the field being PCI specific, but for somebody who just reads the virtio spec for the first time, how would you know what "Queue Indirect Size" means?

Quoted 6 lines

I also wonder whether we need a device normative statement as well, something like:

"With VIRTIO_RING_F_INDIRECT_SIZE, the device MUST provide the maximum Queue Indirect Size for the number of descriptors per vring slot. It MUST allow the driver to set a lower value."

Makes sense.

Quoted 1 line

Maybe we also need to mention that the mechanism is transport specific?

I wouldn't do that. Instead this would probably be extended for other transports in future appropriately.

Quoted 1 line

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so I could not say reliably all the parts that shall be updated for the packed ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a transport- specific and/or device-specific value. If not limited, the maximum number of descriptors in a list is the virt queue size."

However the question is, would anybody want large descriptor chains with the packaged ring in the first place? If I understand it correctly, the benefits of the packed ring over the split ring only manifest for devices that interchange a very large number of rather small bulk data (e.g. network devices), no?

Quoted 2 lines
Quoted 2 lines

A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT in \field{flags}.

#4 lore raw

Re: [virtio-comment] Re: [PATCH 1/2] Add VIRTIO_RING_F_INDIRECT_SIZE

In reply to Christian Schoenebeck
Quoted 33 lines · Christian Schoenebeck
Quoted 23 lines · On Dienstag, 14. Dezember 2021 17:59:51 CET Cornelia Huck wrote:
Quoted 22 lines · Christian Schoenebeck

This new feature flag allows to decouple the maximum amount of descriptors in indirect descriptor tables from the Queue Size.

---

 content.tex    | 21 +++++++++++++++++++++
 split-ring.tex |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
content.tex +0 -0
index 5d112af..0aa4842 100644
@@ -6693,6 +6693,27 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved

Feature Bits}>
transport specific.
For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.>
+ \item[VIRTIO_RING_F_INDIRECT_SIZE(40)] This feature indicates that the

Quoted 8 lines

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6700

Quoted 4 lines

+ maximum amount of descriptors in indirect descriptor tables is independent + from the Queue Size.

content.tex +2 -0
+
+  If this feature bit is not negotiated, then a driver MUST NOT create a

"MUST NOT" does only belong into normative sections... maybe you can reword the last sentence in this paragraph a bit and move the "MUST NOT" part into a normative section?

Quoted 3 lines

I lack creativity in this case. Suggestions appreciated. On doubt I would just delete the entire paragraph here ("If this feature ... transmitted per vring slot.").

Maybe

"Without this feature bit, the queue size limits the length of the descriptor chain, including indirect descriptor tables, i.e. both the maximum amound of slots in the vring and the actual bulk data size transmitted per vring slot."

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
content.tex +8 -0
+  descriptor chain longer than the Queue Size, which then also applies to
+  indirect descriptor tables as in \ref{sec:Basic Facilities of a Virtio
+  Device / Virtqueues / The Virtqueue Descriptor Table / Indirect
+  Descriptors}. Which means without this feature, the Queue Size limits
+  both the maximum amount of slots in the vring, as well as the actual
+  bulk data size being transmitted per vring slot.
+
+  With this feature enabled, the Queue Size only limits the maximum

amount

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
content.tex +3 -0
+  of slots in the vring, but does not limit the actual bulk data size
+  being transmitted when indirect descriptors are used. Decoupling these
+  two configuration parameters this way not only allows much larger bulk

data + being transferred per vring slot, but also avoids complicated synchronization + mechanisms if the device only supports a very small amount of vring slots. Due + to the 16-bit size of a descriptor's "next" field there is still an absolute + limit of $2^{16}$ descriptors per indirect descriptor table.

I think the second paragraph can stay like this.

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
content.tex +1 -0
+
 
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
split-ring.tex +0 -0
index bfef62d..ae2aeb8 100644
@@ -269,7 +269,7 @@ \subsubsection{Indirect Descriptors}\label{sec:Basic

Facilities of a Virtio Devi>
one table per descriptor).

A driver MUST NOT create a descriptor chain longer than the Queue Size of

Quoted 7 lines
Quoted 7 lines

split-ring.tex \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi · line 272

Quoted 2 lines
split-ring.tex +1 -1
-the device.
+the device unless VIRTIO_RING_F_INDIRECT_SIZE has been negotiated.

This might already be enough of a normative statement, though.

Quoted 2 lines

A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT in \field{flags}.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#11 lore raw

[virtio-comment] Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

In reply to Christian Schoenebeck
Quoted 71 lines · Christian Schoenebeck
Quoted 22 lines · On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck wrote:
Quoted 21 lines · Christian Schoenebeck

This new common configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.

---

 content.tex    | 25 ++++++++++++++++++++++++-
 split-ring.tex |  3 +++
 2 files changed, 27 insertions(+), 1 deletion(-)
content.tex +0 -0
index 0aa4842..e3cfcae 100644
@@ -859,6 +859,7 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>

         le64 queue_driver;              /* read-write */
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
Quoted 5 lines
Quoted 1 line

+ le32 queue_indirect_size; /* read-write */

[Note for below: this "common" configuration structure layout is actually PCI-specific, it is only common between the different device types.]

True

Quoted 40 lines
Quoted 31 lines

};
\end{lstlisting}

content.tex +0 -0
@@ -938,6 +939,16 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>
may benefit from providing another value, for example an internal virtqueue
identifier, or an internal offset related to the virtqueue number.
\end{note}

content.tex +3 -0
+
+\item[\field{queue_indirect_size}]
+        This field is used to negotiate the maximum amount of descriptors

per + vring slot as in \ref{sec:Basic Facilities of a Virtio
Device / + Virtqueues / The Virtqueue Descriptor Table / Indirect
Descriptors} if + and only if the VIRTIO_RING_F_INDIRECT_SIZE
feature has been negotiated. +
+ The device specifies its maximum supported number of descriptors
per + vring slot. If the driver requires fewer descriptors, it
writes its + lower value to inform the device of the reduced
resource requirements.>
\end{description}

\devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}>

content.tex +0 -0
@@ -1003,6 +1014,12 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>
The driver MUST NOT write a 0 to \field{queue_enable}.

Quoted 3 lines

+The driver SHOULD write to \field{queue_indirect_size} if its maximum number of +descriptors per vring slot is lower than that reported by the device.

Maybe

"If the driver's maximum number of descriptors per vring slot is lower than the maximum value reported by the device, it SHOULD write that number to \field{queue_indirect_size}."

Quoted 7 lines
Quoted 1 line

?

I actually just used Stefan's wording here and extended it with semantically required components, i.e. "vring slot" and "\field{queue_indirect_size}":
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00006.html

For me both versions are fine.

No strong opinion here, feel free to keep it like this if you prefer.

Quoted 24 lines
Quoted 17 lines
Quoted 9 lines
split-ring.tex +0 -0
index ae2aeb8..d8f66c1 100644
@@ -270,6 +270,9 @@ \subsubsection{Indirect Descriptors}\label{sec:Basic

Facilities of a Virtio Devi>
A driver MUST NOT create a descriptor chain longer than the Queue Size of the device unless VIRTIO_RING_F_INDIRECT_SIZE has been negotiated.

Quoted 3 lines

+Furthermore if VIRTIO_RING_F_INDIRECT_SIZE has been negotiated then the number +of descriptors per vring slot MUST NOT exceed the value negotiated by common +configuration field "queue_indirect_size".

As mentioned above, the "common configuration layout" is actually PCI-specific; other transports will use different mechanism. Maybe it would make sense to reword the whole paragraph and add it in patch 1?

Quoted 6 lines

"If VIRTIO_RING_F_INDIRECT_SIZE has not been negotiated, the driver MUST NOT create a descriptor chain longer than the Queue Size of the device.

If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated, the number of descriptors per vring slot MUST NOT exceed the negotiated Queue Indirect Size."

Quoted 5 lines

That simplifies the statements, yes.

About the term "Queue Indirect Size": I understand your point about the field being PCI specific, but for somebody who just reads the virtio spec for the first time, how would you know what "Queue Indirect Size" means?

If we follow my suggestion below, it is described in the device normative statement. (Not sure if we actually do the same for Queue Size already? That's a similar case.)

Quoted 8 lines
Quoted 7 lines

I also wonder whether we need a device normative statement as well, something like:

"With VIRTIO_RING_F_INDIRECT_SIZE, the device MUST provide the maximum Queue Indirect Size for the number of descriptors per vring slot. It MUST allow the driver to set a lower value."

Quoted 6 lines

Makes sense.

Quoted 1 line

Maybe we also need to mention that the mechanism is transport specific?

I wouldn't do that. Instead this would probably be extended for other transports in future appropriately.

The mechanics, not providing the value... something like

"The maximum Queue Indirect Size is provided by the device in a transport-specific way."

(There should already be some precedence for other values.)

Quoted 13 lines
Quoted 2 lines

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so I could not say reliably all the parts that shall be updated for the packed ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a transport- specific and/or device-specific value. If not limited, the maximum number of descriptors in a list is the virt queue size."

Quoted 5 lines

However the question is, would anybody want large descriptor chains with the packaged ring in the first place? If I understand it correctly, the benefits of the packed ring over the split ring only manifest for devices that interchange a very large number of rather small bulk data (e.g. network devices), no?

If we think that the feature does not make sense for packed ring, they should probably conflict with each other. Otherwise, I think we need at least a statement that the higher limit does not take effect for packed ring, or touch all the places where it would be relevant.

What do others think?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#12 lore raw

Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

Quoted 3 lines · On Donnerstag, 23. Dezember 2021 12:03:50 CET Cornelia Huck wrote:
Quoted 2 lines · Christian Schoenebeck
Quoted 1 line · On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck wrote:

On Tue, Dec 14 2021, Christian Schoenebeck <[email protected]>

Quoted 169 lines · wrote:
Quoted 80 lines
Quoted 22 lines
Quoted 22 lines

This new common configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.

---

 content.tex    | 25 ++++++++++++++++++++++++-
 split-ring.tex |  3 +++
 2 files changed, 27 insertions(+), 1 deletion(-)
content.tex +0 -0
index 0aa4842..e3cfcae 100644
@@ -859,6 +859,7 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>

         le64 queue_driver;              /* read-write */
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
Quoted 5 lines
Quoted 1 line

+ le32 queue_indirect_size; /* read-write */

[Note for below: this "common" configuration structure layout is actually PCI-specific, it is only common between the different device types.]

True

Quoted 50 lines
Quoted 37 lines

};
\end{lstlisting}

content.tex +0 -0
@@ -938,6 +939,16 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>

may benefit from providing another value, for example an internal
virtqueue
identifier, or an internal offset related to the virtqueue number.
\end{note}

content.tex +3 -0
+
+\item[\field{queue_indirect_size}]
+        This field is used to negotiate the maximum amount of

descriptors
per + vring slot as in \ref{sec:Basic Facilities of a Virtio
Device / + Virtqueues / The Virtqueue Descriptor Table /
Indirect
Descriptors} if + and only if the VIRTIO_RING_F_INDIRECT_SIZE
feature has been negotiated. +
+ The device specifies its maximum supported number of
descriptors
per + vring slot. If the driver requires fewer descriptors, it
writes its + lower value to inform the device of the reduced
resource requirements.>

\end{description}

\devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}>

content.tex +0 -0
@@ -1003,6 +1014,12 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>

Quoted 6 lines

The driver MUST NOT write a 0 to \field{queue_enable}.

+The driver SHOULD write to \field{queue_indirect_size} if its maximum number of +descriptors per vring slot is lower than that reported by the
device.

Maybe

"If the driver's maximum number of descriptors per vring slot is lower than the maximum value reported by the device, it SHOULD write that number to \field{queue_indirect_size}."

Quoted 8 lines
Quoted 1 line

?

I actually just used Stefan's wording here and extended it with semantically required components, i.e. "vring slot" and "\field{queue_indirect_size}":
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00006.html

For me both versions are fine.

No strong opinion here, feel free to keep it like this if you prefer.

Quoted 28 lines
Quoted 21 lines
Quoted 8 lines
split-ring.tex +0 -0
index ae2aeb8..d8f66c1 100644
@@ -270,6 +270,9 @@ \subsubsection{Indirect

Descriptors}\label{sec:Basic
Facilities of a Virtio Devi>

Quoted 8 lines

A driver MUST NOT create a descriptor chain longer than the Queue Size of
the device unless VIRTIO_RING_F_INDIRECT_SIZE has been negotiated.

+Furthermore if VIRTIO_RING_F_INDIRECT_SIZE has been negotiated then the
number +of descriptors per vring slot MUST NOT exceed the value negotiated by common +configuration field "queue_indirect_size".

As mentioned above, the "common configuration layout" is actually PCI-specific; other transports will use different mechanism. Maybe it would make sense to reword the whole paragraph and add it in patch 1?

Quoted 6 lines

"If VIRTIO_RING_F_INDIRECT_SIZE has not been negotiated, the driver MUST NOT create a descriptor chain longer than the Queue Size of the device.

If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated, the number of descriptors per vring slot MUST NOT exceed the negotiated Queue Indirect Size."

Quoted 5 lines

That simplifies the statements, yes.

About the term "Queue Indirect Size": I understand your point about the field being PCI specific, but for somebody who just reads the virtio spec for the first time, how would you know what "Queue Indirect Size" means?

If we follow my suggestion below, it is described in the device normative statement. (Not sure if we actually do the same for Queue Size already? That's a similar case.)

Quoted 7 lines
Quoted 6 lines

I also wonder whether we need a device normative statement as well, something like:

"With VIRTIO_RING_F_INDIRECT_SIZE, the device MUST provide the maximum Queue Indirect Size for the number of descriptors per vring slot. It MUST allow the driver to set a lower value."

Quoted 6 lines

Makes sense.

Quoted 1 line

Maybe we also need to mention that the mechanism is transport specific?

I wouldn't do that. Instead this would probably be extended for other transports in future appropriately.

The mechanics, not providing the value... something like

"The maximum Queue Indirect Size is provided by the device in a transport-specific way."

(There should already be some precedence for other values.)

Quoted 13 lines
Quoted 1 line

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so I could not say reliably all the parts that shall be updated for the packed ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a transport-
specific and/or device-specific value. If not limited, the maximum number of descriptors in a list is the virt queue size."

Quoted 5 lines

However the question is, would anybody want large descriptor chains with the packaged ring in the first place? If I understand it correctly, the benefits of the packed ring over the split ring only manifest for devices that interchange a very large number of rather small bulk data (e.g. network devices), no?

Quoted 6 lines

If we think that the feature does not make sense for packed ring, they should probably conflict with each other. Otherwise, I think we need at least a statement that the higher limit does not take effect for packed ring, or touch all the places where it would be relevant.

What do others think?

It would indeed be very useful if other people express their opinion about this issue (packed ring scenario) as well before I continue on this issue.

Probably the fact that my patches never made it through to the list were not necessarily supporting this. Should I contact somebody regarding this ML issue? Do members of the other ML also read this virtio-comment list?

I tried to compensate the current situation by updating the corresponding issue description on Github in a very defailed and verbose way:
https://github.com/oasis-tcs/virtio-spec/issues/122

If nobody replies early January, I would suggest to continue by ignoring the packed ring. Because if somebody wants this for packed ring in future, this can still be added to the specs without breaking things, because this feature is negotiated per queue, not for the entire device.

Best regards,
Christian Schoenebeck

#13 lore raw

[virtio-comment] Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

Quoted 27 lines · Christian Schoenebeck
Quoted 26 lines · On Donnerstag, 23. Dezember 2021 12:03:50 CET Cornelia Huck wrote:
Quoted 14 lines · Christian Schoenebeck
Quoted 1 line · On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck wrote:

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so I could not say reliably all the parts that shall be updated for the packed ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a transport-
specific and/or device-specific value. If not limited, the maximum number of descriptors in a list is the virt queue size."

Quoted 5 lines

However the question is, would anybody want large descriptor chains with the packaged ring in the first place? If I understand it correctly, the benefits of the packed ring over the split ring only manifest for devices that interchange a very large number of rather small bulk data (e.g. network devices), no?

If we think that the feature does not make sense for packed ring, they should probably conflict with each other. Otherwise, I think we need at least a statement that the higher limit does not take effect for packed ring, or touch all the places where it would be relevant.

Quoted 8 lines
Quoted 1 line

What do others think?

It would indeed be very useful if other people express their opinion about this issue (packed ring scenario) as well before I continue on this issue.

Probably the fact that my patches never made it through to the list were not necessarily supporting this. Should I contact somebody regarding this ML issue? Do members of the other ML also read this virtio-comment list?

Yes, this situation is very unsatisfactory :( (I have contacted the people running this list, but there have not yet been any fixes...)

Not sure which other lists would be appropriate to cc: -- maybe [email protected], but that one also suffers from DKIM issues :(

Quoted 4 lines

I tried to compensate the current situation by updating the corresponding issue description on Github in a very defailed and verbose way:
https://github.com/oasis-tcs/virtio-spec/issues/122

Thanks. Hopefully me quoting this makes it more visible (I tried to quote more than I usually would in my other replies already...)

Just to feature it more prominently for people who collapse quotes:

https://github.com/oasis-tcs/virtio-spec/issues/122

Quoted 5 lines

If nobody replies early January, I would suggest to continue by ignoring the packed ring. Because if somebody wants this for packed ring in future, this can still be added to the specs without breaking things, because this feature is negotiated per queue, not for the entire device.

The problem is that we need to specify what is supposed to happen if packed ring *and* this feature are negotiated. If we do not want to add statements for the packed ring case, my suggestion would be
- make packed ring and this feature mutually exclusive
- add a new feature bit that works with packed ring later, if we think it is useful

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#14 lore raw

Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

Quoted 3 lines · On Montag, 3. Januar 2022 14:21:13 CET Cornelia Huck wrote:
Quoted 2 lines · Christian Schoenebeck
Quoted 1 line · On Donnerstag, 23. Dezember 2021 12:03:50 CET Cornelia Huck wrote:

On Wed, Dec 15 2021, Christian Schoenebeck <[email protected]>

Quoted 32 lines · wrote:
Quoted 32 lines
Quoted 31 lines
Quoted 17 lines · On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck wrote:
Quoted 1 line

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so I
could not say reliably all the parts that shall be updated for the packed
ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a transport-
specific and/or device-specific value. If not limited, the maximum number
of descriptors in a list is the virt queue size."

Quoted 7 lines

However the question is, would anybody want large descriptor chains with
the packaged ring in the first place? If I understand it correctly, the benefits of the packed ring over the split ring only manifest for devices
that interchange a very large number of rather small bulk data (e.g. network devices), no?

If we think that the feature does not make sense for packed ring, they should probably conflict with each other. Otherwise, I think we need at least a statement that the higher limit does not take effect for packed ring, or touch all the places where it would be relevant.

What do others think?

Quoted 8 lines
Quoted 6 lines

It would indeed be very useful if other people express their opinion about this issue (packed ring scenario) as well before I continue on this issue.

Probably the fact that my patches never made it through to the list were not necessarily supporting this. Should I contact somebody regarding this ML issue? Do members of the other ML also read this virtio-comment list?

Yes, this situation is very unsatisfactory :( (I have contacted the people running this list, but there have not yet been any fixes...)

Only my emails with patches are refused by the list. All my other emails are accepted. So not sure if the cause is really DKIM or something else. Maybe the admins can suggest a workaround for me.

Quoted 3 lines

Not sure which other lists would be appropriate to cc: -- maybe [email protected], but that one also suffers from DKIM issues :(

What I thought was wether subscribers of virtio-dev would typically read virtio-comment as well. Because AFAICS people who more frequently deal with virtio for their companies rather seem to post to virtio-dev.

Quoted 16 lines
Quoted 3 lines

I tried to compensate the current situation by updating the corresponding issue description on Github in a very defailed and verbose way:
https://github.com/oasis-tcs/virtio-spec/issues/122

Thanks. Hopefully me quoting this makes it more visible (I tried to quote more than I usually would in my other replies already...)

Just to feature it more prominently for people who collapse quotes:

https://github.com/oasis-tcs/virtio-spec/issues/122

Quoted 4 lines

If nobody replies early January, I would suggest to continue by ignoring the packed ring. Because if somebody wants this for packed ring in future, this can still be added to the specs without breaking things, because this feature is negotiated per queue, not for the entire device.

Quoted 6 lines

The problem is that we need to specify what is supposed to happen if packed ring *and* this feature are negotiated. If we do not want to add statements for the packed ring case, my suggestion would be
- make packed ring and this feature mutually exclusive
- add a new feature bit that works with packed ring later, if we think it is useful

Mja, I have to correct myself on that: I wrote in my previous email that this was negotiated per queue. That's false of course as all virtio features are currently negotiated for the entire device.

So you are right, if this new indirect size feature was negotiated then it would apply to both a) split rings and b) packed rings a device might have. Which is unfortunate.

Stefan, you are aware about this circumstance as well, right? Because I remember we originally had a discussion on qemu-devel where you wanted to have this configurable per queue, not per device.

Best regards,
Christian Schoenebeck

#5 lore raw

Re: [PATCH 1/2] Add VIRTIO_RING_F_INDIRECT_SIZE

In reply to Christian Schoenebeck
Quoted 15 lines · Christian Schoenebeck

This new feature flag allows to decouple the maximum amount of descriptors in indirect descriptor tables from the Queue Size.

---
 content.tex    | 21 +++++++++++++++++++++
 split-ring.tex |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
content.tex +0 -0
index 5d112af..0aa4842 100644

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6696

Quoted 4 lines
content.tex +1 -0
@@ -6693,6 +6693,27 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_RING_F_INDIRECT_SIZE(40)] This feature indicates that the

Another feature bit was added in the meantime:

\item[VIRTIO_F_RING_RESET(40)] This feature indicates

                            ^^

Please use 41 and update the text that says:

\item[24 to 40] Feature bits reserved for extensions to the queue and feature negotiation mechanisms

\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.

#6 lore raw

[virtio-comment] Re: [PATCH 1/2] Add VIRTIO_RING_F_INDIRECT_SIZE

Quoted 24 lines · Stefan Hajnoczi
Quoted 15 lines · Christian Schoenebeck

This new feature flag allows to decouple the maximum amount of descriptors in indirect descriptor tables from the Queue Size.

---
 content.tex    | 21 +++++++++++++++++++++
 split-ring.tex |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
content.tex +0 -0
index 5d112af..0aa4842 100644

content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6696

Quoted 4 lines
content.tex +1 -0
@@ -6693,6 +6693,27 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_RING_F_INDIRECT_SIZE(40)] This feature indicates that the

Another feature bit was added in the meantime:

\item[VIRTIO_F_RING_RESET(40)] This feature indicates

Quoted 8 lines
                            ^^

Please use 41 and update the text that says:

\item[24 to 40] Feature bits reserved for extensions to the queue and feature negotiation mechanisms

\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.

Indeed... I'm wondering whether we could automate updating this with some clever variables?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#15 lore raw

[virtio-comment] Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

In reply to Christian Schoenebeck
Quoted 85 lines · Christian Schoenebeck
Quoted 3 lines · On Montag, 3. Januar 2022 14:21:13 CET Cornelia Huck wrote:
Quoted 2 lines · Christian Schoenebeck
Quoted 1 line · On Donnerstag, 23. Dezember 2021 12:03:50 CET Cornelia Huck wrote:

On Wed, Dec 15 2021, Christian Schoenebeck <[email protected]>

Quoted 32 lines · wrote:
Quoted 32 lines
Quoted 31 lines
Quoted 17 lines · On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck wrote:
Quoted 1 line

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so I
could not say reliably all the parts that shall be updated for the packed
ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a transport-
specific and/or device-specific value. If not limited, the maximum number
of descriptors in a list is the virt queue size."

Quoted 7 lines

However the question is, would anybody want large descriptor chains with
the packaged ring in the first place? If I understand it correctly, the benefits of the packed ring over the split ring only manifest for devices
that interchange a very large number of rather small bulk data (e.g. network devices), no?

If we think that the feature does not make sense for packed ring, they should probably conflict with each other. Otherwise, I think we need at least a statement that the higher limit does not take effect for packed ring, or touch all the places where it would be relevant.

What do others think?

Quoted 8 lines
Quoted 6 lines

It would indeed be very useful if other people express their opinion about this issue (packed ring scenario) as well before I continue on this issue.

Probably the fact that my patches never made it through to the list were not necessarily supporting this. Should I contact somebody regarding this ML issue? Do members of the other ML also read this virtio-comment list?

Yes, this situation is very unsatisfactory :( (I have contacted the people running this list, but there have not yet been any fixes...)

Only my emails with patches are refused by the list. All my other emails are accepted. So not sure if the cause is really DKIM or something else. Maybe the admins can suggest a workaround for me.

Quoted 3 lines

Not sure which other lists would be appropriate to cc: -- maybe [email protected], but that one also suffers from DKIM issues :(

What I thought was wether subscribers of virtio-dev would typically read virtio-comment as well. Because AFAICS people who more frequently deal with virtio for their companies rather seem to post to virtio-dev.

Quoted 16 lines
Quoted 3 lines

I tried to compensate the current situation by updating the corresponding issue description on Github in a very defailed and verbose way:
https://github.com/oasis-tcs/virtio-spec/issues/122

Thanks. Hopefully me quoting this makes it more visible (I tried to quote more than I usually would in my other replies already...)

Just to feature it more prominently for people who collapse quotes:

https://github.com/oasis-tcs/virtio-spec/issues/122

Quoted 4 lines

If nobody replies early January, I would suggest to continue by ignoring the packed ring. Because if somebody wants this for packed ring in future, this can still be added to the specs without breaking things, because this feature is negotiated per queue, not for the entire device.

Quoted 6 lines

The problem is that we need to specify what is supposed to happen if packed ring *and* this feature are negotiated. If we do not want to add statements for the packed ring case, my suggestion would be
- make packed ring and this feature mutually exclusive
- add a new feature bit that works with packed ring later, if we think it is useful

Mja, I have to correct myself on that: I wrote in my previous email that this was negotiated per queue. That's false of course as all virtio features are currently negotiated for the entire device.

Quoted 7 lines

So you are right, if this new indirect size feature was negotiated then it would apply to both a) split rings and b) packed rings a device might have. Which is unfortunate.

Stefan, you are aware about this circumstance as well, right? Because I remember we originally had a discussion on qemu-devel where you wanted to have this configurable per queue, not per device.

Regarding packed virtqueues, there are multiple reasons for using them. Recently someone asked about reducing complexity in VIRTIO implementations and one of the suggestions that Michael Tsirkin and I both made independently was to use the packed layout instead of split layout. I don't think we should assume packed virtqueues are only used in scenarios that don't need INDIRECT_SIZE.

Extending INDIRECT_SIZE to packed virtqueues looks straightforward and does not require many changes to the Packed Virtqueues section.

The reason I pushed for per-virtqueue Indirect Size values is because multi-virtqueue devices have specific purposes for each virtqueue. The Queue Size field is per-virtqueue already across all transports. This way a device with control virtqueue can allocate fewer resources (smaller Queue Size and Indirect Size) to it than to the data path virtqueues that carry I/O data buffers.

I don't think it's necessary to support INDIRECT_SIZE on only a subset of a device's virtqueues. If the device doesn't want to "enable" INDIRECT_SIZE on a specific virtqueue it should just report Queue Size as the Indirect Size value?

Stefan

#20 lore raw

Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

In reply to Christian Schoenebeck
Quoted 15 lines · Christian Schoenebeck

This new common configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.

---
 content.tex    | 25 ++++++++++++++++++++++++-
 split-ring.tex |  3 +++
 2 files changed, 27 insertions(+), 1 deletion(-)
content.tex +0 -0
index 0aa4842..e3cfcae 100644

content.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 862

Quoted 4 lines
content.tex +1 -0
@@ -859,6 +859,7 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         le64 queue_driver;              /* read-write */
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
+        le32 queue_indirect_size;       /* read-write */

I see no other unaligned fields in this struct, so I think a le16 padding field is needed for safety:

           le64 queue_device;              /* read-write */
           le16 queue_notify_data;         /* read-only for driver */
  +        le16 reserved;                  /* no access */
  +        le32 queue_indirect_size;       /* read-write */
Quoted 7 lines
content.tex +0 -0
 };
 \end{lstlisting}
 
@@ -938,6 +939,16 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         may benefit from providing another value, for example an internal virtqueue
         identifier, or an internal offset related to the virtqueue number.
         \end{note}

content.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 951

Quoted 10 lines
content.tex +10 -0
+
+\item[\field{queue_indirect_size}]
+        This field is used to negotiate the maximum amount of descriptors per
+        vring slot as in \ref{sec:Basic Facilities of a Virtio Device /
+        Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors} if
+        and only if the VIRTIO_RING_F_INDIRECT_SIZE feature has been negotiated.
+
+        The device specifies its maximum supported number of descriptors per
+        vring slot. If the driver requires fewer descriptors, it writes its
+        lower value to inform the device of the reduced resource requirements.

"vring slot" is a little vague, it means "indirect descriptor table"? The two paragraphs could use "maximum number of descriptors per indirect descriptor table" instead of referring to vring slots to imply indirect descriptor tables.

#24 lore raw
In reply to Christian Schoenebeck
Quoted 7 lines · Christian Schoenebeck

This is a follow-up of:
[0/2]: https://lists.oasis-open.org/archives/virtio-comment/202111/msg00089.html
[1/2]: https://lists.oasis-open.org/archives/virtio-comment/202112/msg00005.html
[2/2]: https://lists.oasis-open.org/archives/virtio-comment/202112/msg00006.html

These two patches introduce a more fine graded control over the maximum length of indirect descriptor tables.

Sorry for the long delay, I am very behind on reviews!

Stefan

#7 lore raw

Re: [PATCH 1/2] Add VIRTIO_RING_F_INDIRECT_SIZE

In reply to Cornelia Huck
Quoted 8 lines · On Montag, 24. Januar 2022 14:14:07 CET Cornelia Huck wrote:
Quoted 7 lines · Stefan Hajnoczi
Quoted 6 lines · Christian Schoenebeck

This new feature flag allows to decouple the maximum amount of descriptors in indirect descriptor tables from the Queue Size.

---

Adding Dominique Martinet on CC from now on, for being informed about progress on this virtio issue for Linux kernel side.

Quoted 21 lines
Quoted 21 lines
Quoted 10 lines
 content.tex    | 21 +++++++++++++++++++++
 split-ring.tex |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
content.tex +0 -0
index 5d112af..0aa4842 100644
@@ -6693,6 +6693,27 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
Quoted 7 lines

Feature Bits}>>
transport specific.
For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.>>
+ \item[VIRTIO_RING_F_INDIRECT_SIZE(40)] This feature indicates that the

Another feature bit was added in the meantime:
\item[VIRTIO_F_RING_RESET(40)] This feature indicates

Quoted 8 lines
Quoted 8 lines
                            ^^

Please use 41 and update the text that says:
\item[24 to 40] Feature bits reserved for extensions to the queue and feature negotiation mechanisms

\item[41 to 49, and 128 and above] Feature bits reserved for future extensions.

Sure, I'll take care of that. As I am always rebasing to latest git version of the spec, I guess I would have realized it anyway.

Quoted 2 lines

Indeed... I'm wondering whether we could automate updating this with some clever variables?

Well, for LaTeX there is

	\newcommand{\newCommandName}{1234}

and

	\def\variable{1234}

Both solutions have pros and cons. But not sure I would want to rack my brain on that. I just manually update the feature bit number for now :)

Best regards,
Christian Schoenebeck

#16 lore raw

Re: [virtio-comment] Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

In reply to Stefan Hajnoczi
Quoted 3 lines · On Montag, 24. Januar 2022 14:39:28 CET Stefan Hajnoczi wrote:
Quoted 2 lines · Christian Schoenebeck
Quoted 1 line · On Montag, 3. Januar 2022 14:21:13 CET Cornelia Huck wrote:

On Wed, Dec 29 2021, Christian Schoenebeck <[email protected]>

Quoted 118 lines · wrote:
Quoted 102 lines
Quoted 2 lines
Quoted 2 lines · On Donnerstag, 23. Dezember 2021 12:03:50 CET Cornelia Huck wrote:
Quoted 1 line

On Wed, Dec 15 2021, Christian Schoenebeck <[email protected]>

Quoted 51 lines · wrote:
Quoted 44 lines
Quoted 37 lines
Quoted 19 lines · On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck wrote:
Quoted 2 lines

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so
I
could not say reliably all the parts that shall be updated for the packed
ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a transport-
specific and/or device-specific value. If not limited, the maximum number
of descriptors in a list is the virt queue size."

Quoted 9 lines

However the question is, would anybody want large descriptor chains with
the packaged ring in the first place? If I understand it correctly, the
benefits of the packed ring over the split ring only manifest for devices
that interchange a very large number of rather small bulk data (e.g.
network devices), no?

If we think that the feature does not make sense for packed ring, they
should probably conflict with each other. Otherwise, I think we need at
least a statement that the higher limit does not take effect for packed
ring, or touch all the places where it would be relevant.

Quoted 1 line

What do others think?

It would indeed be very useful if other people express their opinion about
this issue (packed ring scenario) as well before I continue on this issue.

Quoted 6 lines

Probably the fact that my patches never made it through to the list were
not necessarily supporting this. Should I contact somebody regarding this
ML issue? Do members of the other ML also read this virtio-comment list?

Quoted 2 lines

Yes, this situation is very unsatisfactory :( (I have contacted the people running this list, but there have not yet been any fixes...)

Only my emails with patches are refused by the list. All my other emails are accepted. So not sure if the cause is really DKIM or something else. Maybe the admins can suggest a workaround for me.

Quoted 3 lines

Not sure which other lists would be appropriate to cc: -- maybe [email protected], but that one also suffers from DKIM issues :(

What I thought was wether subscribers of virtio-dev would typically read virtio-comment as well. Because AFAICS people who more frequently deal with
virtio for their companies rather seem to post to virtio-dev.

Quoted 19 lines
Quoted 4 lines

I tried to compensate the current situation by updating the corresponding
issue description on Github in a very defailed and verbose way:
https://github.com/oasis-tcs/virtio-spec/issues/122

Thanks. Hopefully me quoting this makes it more visible (I tried to quote more than I usually would in my other replies already...)

Just to feature it more prominently for people who collapse quotes:

https://github.com/oasis-tcs/virtio-spec/issues/122

Quoted 6 lines

If nobody replies early January, I would suggest to continue by ignoring
the packed ring. Because if somebody wants this for packed ring in future, this can still be added to the specs without breaking things, because this feature is negotiated per queue, not for the entire device.

Quoted 7 lines

The problem is that we need to specify what is supposed to happen if packed ring *and* this feature are negotiated. If we do not want to add statements for the packed ring case, my suggestion would be
- make packed ring and this feature mutually exclusive
- add a new feature bit that works with packed ring later, if we think

it is useful

Mja, I have to correct myself on that: I wrote in my previous email that this was negotiated per queue. That's false of course as all virtio features are currently negotiated for the entire device.

Quoted 8 lines

So you are right, if this new indirect size feature was negotiated then it would apply to both a) split rings and b) packed rings a device might have.
Which is unfortunate.

Stefan, you are aware about this circumstance as well, right? Because I remember we originally had a discussion on qemu-devel where you wanted to have this configurable per queue, not per device.

Regarding packed virtqueues, there are multiple reasons for using them. Recently someone asked about reducing complexity in VIRTIO implementations and one of the suggestions that Michael Tsirkin and I both made independently was to use the packed layout instead of split layout. I don't think we should assume packed virtqueues are only used in scenarios that don't need INDIRECT_SIZE.

Quoted 2 lines

Extending INDIRECT_SIZE to packed virtqueues looks straightforward and does not require many changes to the Packed Virtqueues section.

Yes I agree, it does make sense to apply this new INDIRECT_SIZE feature to packed queues as well.

Quoted 11 lines

The reason I pushed for per-virtqueue Indirect Size values is because multi-virtqueue devices have specific purposes for each virtqueue. The Queue Size field is per-virtqueue already across all transports. This way a device with control virtqueue can allocate fewer resources (smaller Queue Size and Indirect Size) to it than to the data path virtqueues that carry I/O data buffers.

I don't think it's necessary to support INDIRECT_SIZE on only a subset of a device's virtqueues. If the device doesn't want to "enable" INDIRECT_SIZE on a specific virtqueue it should just report Queue Size as the Indirect Size value?

Or another idea: what about adding this new indirect size field to the struct that holds the queue size field already, then this would become transport- independent (i.e. the discussed PCI config field would be unnecessary), and it would be a feature configurable per queue. Or is that struct inappropriate for feature negotiation purposes?

Best regards,
Christian Schoenebeck

#17 lore raw

Re: [virtio-comment] Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

Quoted 141 lines · Christian Schoenebeck
Quoted 3 lines · On Montag, 24. Januar 2022 14:39:28 CET Stefan Hajnoczi wrote:
Quoted 2 lines · Christian Schoenebeck
Quoted 1 line · On Montag, 3. Januar 2022 14:21:13 CET Cornelia Huck wrote:

On Wed, Dec 29 2021, Christian Schoenebeck <[email protected]>

Quoted 118 lines · wrote:
Quoted 102 lines
Quoted 2 lines
Quoted 2 lines · On Donnerstag, 23. Dezember 2021 12:03:50 CET Cornelia Huck wrote:
Quoted 1 line

On Wed, Dec 15 2021, Christian Schoenebeck <[email protected]>

Quoted 51 lines · wrote:
Quoted 44 lines
Quoted 37 lines
Quoted 19 lines · On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck wrote:
Quoted 2 lines

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so
I
could not say reliably all the parts that shall be updated for the packed
ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a transport-
specific and/or device-specific value. If not limited, the maximum number
of descriptors in a list is the virt queue size."

Quoted 9 lines

However the question is, would anybody want large descriptor chains with
the packaged ring in the first place? If I understand it correctly, the
benefits of the packed ring over the split ring only manifest for devices
that interchange a very large number of rather small bulk data (e.g.
network devices), no?

If we think that the feature does not make sense for packed ring, they
should probably conflict with each other. Otherwise, I think we need at
least a statement that the higher limit does not take effect for packed
ring, or touch all the places where it would be relevant.

Quoted 1 line

What do others think?

It would indeed be very useful if other people express their opinion about
this issue (packed ring scenario) as well before I continue on this issue.

Quoted 6 lines

Probably the fact that my patches never made it through to the list were
not necessarily supporting this. Should I contact somebody regarding this
ML issue? Do members of the other ML also read this virtio-comment list?

Quoted 2 lines

Yes, this situation is very unsatisfactory :( (I have contacted the people running this list, but there have not yet been any fixes...)

Only my emails with patches are refused by the list. All my other emails are accepted. So not sure if the cause is really DKIM or something else. Maybe the admins can suggest a workaround for me.

Quoted 3 lines

Not sure which other lists would be appropriate to cc: -- maybe [email protected], but that one also suffers from DKIM issues :(

What I thought was wether subscribers of virtio-dev would typically read virtio-comment as well. Because AFAICS people who more frequently deal with
virtio for their companies rather seem to post to virtio-dev.

Quoted 19 lines
Quoted 4 lines

I tried to compensate the current situation by updating the corresponding
issue description on Github in a very defailed and verbose way:
https://github.com/oasis-tcs/virtio-spec/issues/122

Thanks. Hopefully me quoting this makes it more visible (I tried to quote more than I usually would in my other replies already...)

Just to feature it more prominently for people who collapse quotes:

https://github.com/oasis-tcs/virtio-spec/issues/122

Quoted 6 lines

If nobody replies early January, I would suggest to continue by ignoring
the packed ring. Because if somebody wants this for packed ring in future, this can still be added to the specs without breaking things, because this feature is negotiated per queue, not for the entire device.

Quoted 7 lines

The problem is that we need to specify what is supposed to happen if packed ring *and* this feature are negotiated. If we do not want to add statements for the packed ring case, my suggestion would be
- make packed ring and this feature mutually exclusive
- add a new feature bit that works with packed ring later, if we think

it is useful

Mja, I have to correct myself on that: I wrote in my previous email that this was negotiated per queue. That's false of course as all virtio features are currently negotiated for the entire device.

Quoted 8 lines

So you are right, if this new indirect size feature was negotiated then it would apply to both a) split rings and b) packed rings a device might have.
Which is unfortunate.

Stefan, you are aware about this circumstance as well, right? Because I remember we originally had a discussion on qemu-devel where you wanted to have this configurable per queue, not per device.

Regarding packed virtqueues, there are multiple reasons for using them. Recently someone asked about reducing complexity in VIRTIO implementations and one of the suggestions that Michael Tsirkin and I both made independently was to use the packed layout instead of split layout. I don't think we should assume packed virtqueues are only used in scenarios that don't need INDIRECT_SIZE.

Quoted 2 lines

Extending INDIRECT_SIZE to packed virtqueues looks straightforward and does not require many changes to the Packed Virtqueues section.

Yes I agree, it does make sense to apply this new INDIRECT_SIZE feature to packed queues as well.

Quoted 11 lines

The reason I pushed for per-virtqueue Indirect Size values is because multi-virtqueue devices have specific purposes for each virtqueue. The Queue Size field is per-virtqueue already across all transports. This way a device with control virtqueue can allocate fewer resources (smaller Queue Size and Indirect Size) to it than to the data path virtqueues that carry I/O data buffers.

I don't think it's necessary to support INDIRECT_SIZE on only a subset of a device's virtqueues. If the device doesn't want to "enable" INDIRECT_SIZE on a specific virtqueue it should just report Queue Size as the Indirect Size value?

Quoted 5 lines

Or another idea: what about adding this new indirect size field to the struct that holds the queue size field already, then this would become transport- independent (i.e. the discussed PCI config field would be unnecessary), and it would be a feature configurable per queue. Or is that struct inappropriate for feature negotiation purposes?

There is no transport-independent struct that contains Queue Size. It is handled by struct virtio_pci_common_cfg->queue_size, the MMIO QueueNum register, and the CCW struct vq_config_block->max_num and struct vq_info_block->num fields.

Stefan

#18 lore raw

Re: [virtio-comment] Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

Quoted 13 lines · On Montag, 24. Januar 2022 17:41:09 CET Stefan Hajnoczi wrote:
Quoted 12 lines · Christian Schoenebeck
Quoted 3 lines · On Montag, 24. Januar 2022 14:39:28 CET Stefan Hajnoczi wrote:
Quoted 2 lines · Christian Schoenebeck
Quoted 1 line · On Montag, 3. Januar 2022 14:21:13 CET Cornelia Huck wrote:

On Wed, Dec 29 2021, Christian Schoenebeck <[email protected]>

Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines · On Donnerstag, 23. Dezember 2021 12:03:50 CET Cornelia Huck wrote:
Quoted 2 lines

On Wed, Dec 15 2021, Christian Schoenebeck
<[email protected]>

Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck

Quoted 158 lines · wrote:
Quoted 152 lines
Quoted 134 lines
Quoted 121 lines
Quoted 61 lines
Quoted 52 lines
Quoted 43 lines
Quoted 21 lines
Quoted 2 lines

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so
I
could not say reliably all the parts that shall be updated for the
packed
ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a
transport-
specific and/or device-specific value. If not limited, the maximum
number
of descriptors in a list is the virt queue size."

Quoted 12 lines

However the question is, would anybody want large descriptor chains
with
the packaged ring in the first place? If I understand it correctly,
the
benefits of the packed ring over the split ring only manifest for
devices
that interchange a very large number of rather small bulk data (e.g.
network devices), no?

If we think that the feature does not make sense for packed ring, they
should probably conflict with each other. Otherwise, I think we need
at
least a statement that the higher limit does not take effect for packed
ring, or touch all the places where it would be relevant.

Quoted 1 line

What do others think?

It would indeed be very useful if other people express their opinion
about
this issue (packed ring scenario) as well before I continue on this
issue.

Quoted 8 lines

Probably the fact that my patches never made it through to the list
were
not necessarily supporting this. Should I contact somebody regarding
this
ML issue? Do members of the other ML also read this virtio-comment list?

Quoted 2 lines

Yes, this situation is very unsatisfactory :( (I have contacted the people running this list, but there have not yet been any fixes...)

Only my emails with patches are refused by the list. All my other emails
are accepted. So not sure if the cause is really DKIM or something else.
Maybe the admins can suggest a workaround for me.

Quoted 3 lines

Not sure which other lists would be appropriate to cc: -- maybe [email protected], but that one also suffers from DKIM issues :(

What I thought was wether subscribers of virtio-dev would typically read
virtio-comment as well. Because AFAICS people who more frequently deal with
virtio for their companies rather seem to post to virtio-dev.

Quoted 28 lines
Quoted 4 lines

I tried to compensate the current situation by updating the corresponding
issue description on Github in a very defailed and verbose way:
https://github.com/oasis-tcs/virtio-spec/issues/122

Thanks. Hopefully me quoting this makes it more visible (I tried to quote more than I usually would in my other replies already...)

Just to feature it more prominently for people who collapse quotes:

https://github.com/oasis-tcs/virtio-spec/issues/122

Quoted 7 lines

If nobody replies early January, I would suggest to continue by ignoring
the packed ring. Because if somebody wants this for packed ring in future, this can still be added to the specs without breaking things,
because this feature is negotiated per queue, not for the entire device.

The problem is that we need to specify what is supposed to happen if packed ring *and* this feature are negotiated. If we do not want to add
statements for the packed ring case, my suggestion would be
- make packed ring and this feature mutually exclusive
- add a new feature bit that works with packed ring later, if we think

Quoted 1 line

it is useful

Mja, I have to correct myself on that: I wrote in my previous email that
this was negotiated per queue. That's false of course as all virtio features are currently negotiated for the entire device.

So you are right, if this new indirect size feature was negotiated then it
would apply to both a) split rings and b) packed rings a device might have.
Which is unfortunate.

Quoted 5 lines

Stefan, you are aware about this circumstance as well, right? Because I
remember we originally had a discussion on qemu-devel where you wanted to
have this configurable per queue, not per device.

Regarding packed virtqueues, there are multiple reasons for using them. Recently someone asked about reducing complexity in VIRTIO implementations and one of the suggestions that Michael Tsirkin and I both made independently was to use the packed layout instead of split layout. I don't think we should assume packed virtqueues are only used in scenarios that don't need INDIRECT_SIZE.

Quoted 2 lines

Extending INDIRECT_SIZE to packed virtqueues looks straightforward and does not require many changes to the Packed Virtqueues section.

Yes I agree, it does make sense to apply this new INDIRECT_SIZE feature to packed queues as well.

Quoted 11 lines

The reason I pushed for per-virtqueue Indirect Size values is because multi-virtqueue devices have specific purposes for each virtqueue. The Queue Size field is per-virtqueue already across all transports. This way a device with control virtqueue can allocate fewer resources (smaller Queue Size and Indirect Size) to it than to the data path virtqueues that carry I/O data buffers.

I don't think it's necessary to support INDIRECT_SIZE on only a subset of a device's virtqueues. If the device doesn't want to "enable" INDIRECT_SIZE on a specific virtqueue it should just report Queue Size as the Indirect Size value?

Quoted 5 lines

Or another idea: what about adding this new indirect size field to the struct that holds the queue size field already, then this would become transport- independent (i.e. the discussed PCI config field would be unnecessary), and it would be a feature configurable per queue. Or is that struct inappropriate for feature negotiation purposes?

Quoted 4 lines

There is no transport-independent struct that contains Queue Size. It is handled by struct virtio_pci_common_cfg->queue_size, the MMIO QueueNum register, and the CCW struct vq_config_block->max_num and struct vq_info_block->num fields.

Aaah, I just realized I was completely confused about
struct virtio_pci_common_cfg: so far I assumed this were all entirely device global settings.

The respective queue is first selected via 'queue_select' field and then the subsequent fields reflect the selected queue. Now I got it.

Ok fine, then the current draft actually already negotiated the indirect size for each queue independently already, without me even being aware. ;-)

I guess I know all I need then for the next round of patches. Maybe I also come along for changes for MMIO and CCW side, we'll see.

Thanks!

Best regards,
Christian Schoenebeck

#19 lore raw

Re: [virtio-comment] Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

Quoted 182 lines · Christian Schoenebeck
Quoted 13 lines · On Montag, 24. Januar 2022 17:41:09 CET Stefan Hajnoczi wrote:
Quoted 12 lines · Christian Schoenebeck
Quoted 3 lines · On Montag, 24. Januar 2022 14:39:28 CET Stefan Hajnoczi wrote:
Quoted 2 lines · Christian Schoenebeck
Quoted 1 line · On Montag, 3. Januar 2022 14:21:13 CET Cornelia Huck wrote:

On Wed, Dec 29 2021, Christian Schoenebeck <[email protected]>

Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines · On Donnerstag, 23. Dezember 2021 12:03:50 CET Cornelia Huck wrote:
Quoted 2 lines

On Wed, Dec 15 2021, Christian Schoenebeck
<[email protected]>

Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Dienstag, 14. Dezember 2021 18:20:28 CET Cornelia Huck

Quoted 158 lines · wrote:
Quoted 152 lines
Quoted 134 lines
Quoted 121 lines
Quoted 61 lines
Quoted 52 lines
Quoted 43 lines
Quoted 21 lines
Quoted 2 lines

Also, this is only for split ring; does packed ring need any updates?

I have not reviewed the packed ring as much as I did the split ring, so
I
could not say reliably all the parts that shall be updated for the
packed
ring. There are some obvious parts like:

2.7.5 Scatter-Gather Support

"The device limits the number of descriptors in a list through a
transport-
specific and/or device-specific value. If not limited, the maximum
number
of descriptors in a list is the virt queue size."

Quoted 12 lines

However the question is, would anybody want large descriptor chains
with
the packaged ring in the first place? If I understand it correctly,
the
benefits of the packed ring over the split ring only manifest for
devices
that interchange a very large number of rather small bulk data (e.g.
network devices), no?

If we think that the feature does not make sense for packed ring, they
should probably conflict with each other. Otherwise, I think we need
at
least a statement that the higher limit does not take effect for packed
ring, or touch all the places where it would be relevant.

Quoted 1 line

What do others think?

It would indeed be very useful if other people express their opinion
about
this issue (packed ring scenario) as well before I continue on this
issue.

Quoted 8 lines

Probably the fact that my patches never made it through to the list
were
not necessarily supporting this. Should I contact somebody regarding
this
ML issue? Do members of the other ML also read this virtio-comment list?

Quoted 2 lines

Yes, this situation is very unsatisfactory :( (I have contacted the people running this list, but there have not yet been any fixes...)

Only my emails with patches are refused by the list. All my other emails
are accepted. So not sure if the cause is really DKIM or something else.
Maybe the admins can suggest a workaround for me.

Quoted 3 lines

Not sure which other lists would be appropriate to cc: -- maybe [email protected], but that one also suffers from DKIM issues :(

What I thought was wether subscribers of virtio-dev would typically read
virtio-comment as well. Because AFAICS people who more frequently deal with
virtio for their companies rather seem to post to virtio-dev.

Quoted 28 lines
Quoted 4 lines

I tried to compensate the current situation by updating the corresponding
issue description on Github in a very defailed and verbose way:
https://github.com/oasis-tcs/virtio-spec/issues/122

Thanks. Hopefully me quoting this makes it more visible (I tried to quote more than I usually would in my other replies already...)

Just to feature it more prominently for people who collapse quotes:

https://github.com/oasis-tcs/virtio-spec/issues/122

Quoted 7 lines

If nobody replies early January, I would suggest to continue by ignoring
the packed ring. Because if somebody wants this for packed ring in future, this can still be added to the specs without breaking things,
because this feature is negotiated per queue, not for the entire device.

The problem is that we need to specify what is supposed to happen if packed ring *and* this feature are negotiated. If we do not want to add
statements for the packed ring case, my suggestion would be
- make packed ring and this feature mutually exclusive
- add a new feature bit that works with packed ring later, if we think

Quoted 1 line

it is useful

Mja, I have to correct myself on that: I wrote in my previous email that
this was negotiated per queue. That's false of course as all virtio features are currently negotiated for the entire device.

So you are right, if this new indirect size feature was negotiated then it
would apply to both a) split rings and b) packed rings a device might have.
Which is unfortunate.

Quoted 5 lines

Stefan, you are aware about this circumstance as well, right? Because I
remember we originally had a discussion on qemu-devel where you wanted to
have this configurable per queue, not per device.

Regarding packed virtqueues, there are multiple reasons for using them. Recently someone asked about reducing complexity in VIRTIO implementations and one of the suggestions that Michael Tsirkin and I both made independently was to use the packed layout instead of split layout. I don't think we should assume packed virtqueues are only used in scenarios that don't need INDIRECT_SIZE.

Quoted 2 lines

Extending INDIRECT_SIZE to packed virtqueues looks straightforward and does not require many changes to the Packed Virtqueues section.

Yes I agree, it does make sense to apply this new INDIRECT_SIZE feature to packed queues as well.

Quoted 11 lines

The reason I pushed for per-virtqueue Indirect Size values is because multi-virtqueue devices have specific purposes for each virtqueue. The Queue Size field is per-virtqueue already across all transports. This way a device with control virtqueue can allocate fewer resources (smaller Queue Size and Indirect Size) to it than to the data path virtqueues that carry I/O data buffers.

I don't think it's necessary to support INDIRECT_SIZE on only a subset of a device's virtqueues. If the device doesn't want to "enable" INDIRECT_SIZE on a specific virtqueue it should just report Queue Size as the Indirect Size value?

Quoted 5 lines

Or another idea: what about adding this new indirect size field to the struct that holds the queue size field already, then this would become transport- independent (i.e. the discussed PCI config field would be unnecessary), and it would be a feature configurable per queue. Or is that struct inappropriate for feature negotiation purposes?

Quoted 4 lines

There is no transport-independent struct that contains Queue Size. It is handled by struct virtio_pci_common_cfg->queue_size, the MMIO QueueNum register, and the CCW struct vq_config_block->max_num and struct vq_info_block->num fields.

Aaah, I just realized I was completely confused about
struct virtio_pci_common_cfg: so far I assumed this were all entirely device global settings.

Quoted 8 lines

The respective queue is first selected via 'queue_select' field and then the subsequent fields reflect the selected queue. Now I got it.

Ok fine, then the current draft actually already negotiated the indirect size for each queue independently already, without me even being aware. ;-)

I guess I know all I need then for the next round of patches. Maybe I also come along for changes for MMIO and CCW side, we'll see.

Awesome!

Stefan

#21 lore raw

Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

In reply to Stefan Hajnoczi
Quoted 8 lines · On Montag, 24. Januar 2022 14:53:45 CET Stefan Hajnoczi wrote:
Quoted 7 lines · Christian Schoenebeck

This new common configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.

---

Quoted 5 lines

content.tex

Quoted 5 lines
 content.tex    | 25 ++++++++++++++++++++++++-
 split-ring.tex |  3 +++
 2 files changed, 27 insertions(+), 1 deletion(-)
content.tex +0 -0

Stefan, while preparing the next version of patches for this issue (#122), I noticed that I missed your email here ...

Quoted 14 lines
Quoted 10 lines

index 0aa4842..e3cfcae 100644

content.tex +0 -0
@@ -859,6 +859,7 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>

         le64 queue_driver;              /* read-write */
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */

+ le32 queue_indirect_size; /* read-write */

I see no other unaligned fields in this struct, so I think a le16 padding field is needed for safety:

Quoted 5 lines
           le64 queue_device;              /* read-write */
           le16 queue_notify_data;         /* read-only for driver */
  +        le16 reserved;                  /* no access */
  +        le32 queue_indirect_size;       /* read-write */

That's no longer needed as there's now the new 'queue_reset' field, i.e.:

content.tex +1 -0
index 685525d..da57d5d 100644
@@ -902,6 +902,7 @@ \subsubsection{Common configuration structure layout}
\label{sec:Virtio Transport
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
         le16 queue_reset;               /* read-write */
+        le32 queue_indirect_size;       /* read-write */
 };
 \end{lstlisting}
 
Quoted 11 lines
Quoted 11 lines

};
\end{lstlisting}

content.tex +0 -0
@@ -938,6 +939,16 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>
may benefit from providing another value, for example an internal virtqueue
identifier, or an internal offset related to the virtqueue number.
\end{note}

Quoted 15 lines
Quoted 11 lines
content.tex +3 -0
+
+\item[\field{queue_indirect_size}]
+        This field is used to negotiate the maximum amount of descriptors

per + vring slot as in \ref{sec:Basic Facilities of a Virtio
Device / + Virtqueues / The Virtqueue Descriptor Table / Indirect
Descriptors} if + and only if the VIRTIO_RING_F_INDIRECT_SIZE
feature has been negotiated. +
+ The device specifies its maximum supported number of descriptors
per + vring slot. If the driver requires fewer descriptors, it
writes its + lower value to inform the device of the reduced
resource requirements.

"vring slot" is a little vague, it means "indirect descriptor table"? The two paragraphs could use "maximum number of descriptors per indirect descriptor table" instead of referring to vring slots to imply indirect descriptor tables.

The intended phrasing was intended to reflect that "Queue Indirect Size" is meant to be the *sum* of all indirect descriptors:
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00008.html

Just to avoid that I am missing something here; as of now, there can only be two indirect tables per round-trip message, right?

Best regards,
Christian Schoenebeck

#22 lore raw

Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

Quoted 80 lines · Christian Schoenebeck
Quoted 8 lines · On Montag, 24. Januar 2022 14:53:45 CET Stefan Hajnoczi wrote:
Quoted 7 lines · Christian Schoenebeck

This new common configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.

---

Quoted 5 lines

content.tex

Quoted 5 lines
 content.tex    | 25 ++++++++++++++++++++++++-
 split-ring.tex |  3 +++
 2 files changed, 27 insertions(+), 1 deletion(-)
content.tex +0 -0

Stefan, while preparing the next version of patches for this issue (#122), I noticed that I missed your email here ...

Quoted 14 lines
Quoted 10 lines

index 0aa4842..e3cfcae 100644

content.tex +0 -0
@@ -859,6 +859,7 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>

         le64 queue_driver;              /* read-write */
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */

+ le32 queue_indirect_size; /* read-write */

I see no other unaligned fields in this struct, so I think a le16 padding field is needed for safety:

Quoted 5 lines
           le64 queue_device;              /* read-write */
           le16 queue_notify_data;         /* read-only for driver */
  +        le16 reserved;                  /* no access */
  +        le32 queue_indirect_size;       /* read-write */

That's no longer needed as there's now the new 'queue_reset' field, i.e.:

content.tex +0 -0
index 685525d..da57d5d 100644
@@ -902,6 +902,7 @@ \subsubsection{Common configuration structure layout}

\label{sec:Virtio Transport

         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
         le16 queue_reset;               /* read-write */
+        le32 queue_indirect_size;       /* read-write */

};
\end{lstlisting}

Quoted 11 lines
Quoted 11 lines

};
\end{lstlisting}

content.tex +0 -0
@@ -938,6 +939,16 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>
may benefit from providing another value, for example an internal virtqueue
identifier, or an internal offset related to the virtqueue number.
\end{note}

Quoted 15 lines
Quoted 11 lines
content.tex +3 -0
+
+\item[\field{queue_indirect_size}]
+        This field is used to negotiate the maximum amount of descriptors

per + vring slot as in \ref{sec:Basic Facilities of a Virtio
Device / + Virtqueues / The Virtqueue Descriptor Table / Indirect
Descriptors} if + and only if the VIRTIO_RING_F_INDIRECT_SIZE
feature has been negotiated. +
+ The device specifies its maximum supported number of descriptors
per + vring slot. If the driver requires fewer descriptors, it
writes its + lower value to inform the device of the reduced
resource requirements.

"vring slot" is a little vague, it means "indirect descriptor table"? The two paragraphs could use "maximum number of descriptors per indirect descriptor table" instead of referring to vring slots to imply indirect descriptor tables.

Quoted 6 lines

The intended phrasing was intended to reflect that "Queue Indirect Size" is meant to be the *sum* of all indirect descriptors:
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00008.html

Just to avoid that I am missing something here; as of now, there can only be two indirect tables per round-trip message, right?

No, just one indirect descriptor table. An indirect descriptor table may contain both IN and OUT descriptors so it can handle a round-trip message.

2.6.5.3.1 Driver Requirements: Indirect Descriptors says:

A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT in flags.

It's unclear whether this statement is referring to 1) one descriptor,
2) to all descriptors in a buffer, or 3) to all descriptors ever made available by the driver. QEMU's hw/virtio.c:virtqueue_pop() shows that the interpretation is #2. So if a buffer uses an indirect descriptor table then it has exactly 1 descriptor in the virtqueue's descriptor table.

Stefan

#23 lore raw

Re: [PATCH 2/2] Add common configuration field "queue_indirect_size"

Quoted 38 lines · On Montag, 21. Februar 2022 11:33:45 CET Stefan Hajnoczi wrote:
Quoted 37 lines · Christian Schoenebeck
Quoted 22 lines · On Montag, 24. Januar 2022 14:53:45 CET Stefan Hajnoczi wrote:
Quoted 21 lines · Christian Schoenebeck
(quoted diff) +0 -0
@@ -938,6 +939,16 @@ \subsubsection{Common configuration structure

layout}\label{sec:Virtio Transport>

may benefit from providing another value, for example an internal
virtqueue
identifier, or an internal offset related to the virtqueue number.
\end{note}

(quoted diff) +3 -0
+
+\item[\field{queue_indirect_size}]
+        This field is used to negotiate the maximum amount of

descriptors
per + vring slot as in \ref{sec:Basic Facilities of a Virtio
Device / + Virtqueues / The Virtqueue Descriptor Table /
Indirect
Descriptors} if + and only if the VIRTIO_RING_F_INDIRECT_SIZE
feature has been negotiated. +
+ The device specifies its maximum supported number of
descriptors

Quoted 8 lines
Quoted 3 lines

per + vring slot. If the driver requires fewer descriptors, it
writes its + lower value to inform the device of the reduced
resource requirements.

"vring slot" is a little vague, it means "indirect descriptor table"? The two paragraphs could use "maximum number of descriptors per indirect descriptor table" instead of referring to vring slots to imply indirect descriptor tables.

The intended phrasing was intended to reflect that "Queue Indirect Size" is
meant to be the *sum* of all indirect descriptors:
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00008.html

Quoted 6 lines
Quoted 2 lines

Just to avoid that I am missing something here; as of now, there can only be two indirect tables per round-trip message, right?

No, just one indirect descriptor table. An indirect descriptor table may contain both IN and OUT descriptors so it can handle a round-trip message.

Ok, you are right. I thought it would pick (per round-trip message) 2 pre- allocated descriptors (directly from the queue), and that those 2 descriptors would then point to a separate indirect table each.

But it is like you said: just one pre-allocated descriptor, pointing to exactly one indirect table, and inside that indirect table there are first the indirect descriptors (pages) for device<-driver request message, followed by indirect descriptors (pages) for device->driver response message. And this order (request first, response second) is implied by the QEMU code as well.

The names for "in" and "out" in the code are a bit confusing BTW.

Quoted 4 lines

2.6.5.3.1 Driver Requirements: Indirect Descriptors says:

A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT in
flags.

Yes, I was aware about that and that should have made it clear to me already. I had a calculation error while debugging the code which mislead me.

Quoted 8 lines

It's unclear whether this statement is referring to 1) one descriptor,
2) to all descriptors in a buffer, or 3) to all descriptors ever made available by the driver. QEMU's hw/virtio.c:virtqueue_pop() shows that the interpretation is #2. So if a buffer uses an indirect descriptor table then it has exactly 1 descriptor in the virtqueue's descriptor table.

Stefan

Yes, it is 2).

Ok, I'll change the phrasing as you suggested to "per indirect descriptor table" and will send a new version this week.

Thanks!

Best regards,
Christian Schoenebeck