Add VIRTIO_RING_F_INDIRECT_SIZE and queue_indirect_size
v2 cover Add VIRTIO_RING_F_INDIRECT_SIZE and queue_indirect_size
This is a follow-up of:
[1/2]: https://lists.oasis-open.org/archives/virtio-comment/202112/msg00034.html
[2/2]: https://lists.oasis-open.org/archives/virtio-comment/202112/msg00035.html
These 4 patches introduce a more fine graded control over the maximum length of indirect descriptor tables.
Associated Github Task:
https://github.com/oasis-tcs/virtio-spec/issues/122
Latest Version (all patches):
https://github.com/cschoenebeck/virtio-spec/commits/long_indirect_descr
v1 -> v2:
* Update reserved bits section [patch 1].
* Add requirement to device normative section [patch 1].
* Use the general new term "Queue Indirect Size" where possible instead of mentioning a bus specific field name [patch 1].
* Simplified phrasing of requirements in split ring section [patch 1].
* Move bus-independent changes from patch 2 to patch 1.
* There is only one indirect descriptor table allowed per buffer (round-trip message), so use the term "per indirect descriptor table" for the semantic of "Queue Indirect Size" in patch 2.
* Simplify phrasing of requirements in patch 2.
* MMIO: First shot [NEW patch 3].
* CCW: First shot [NEW patch 4].
Christian Schoenebeck (4):
Add VIRTIO_RING_F_INDIRECT_SIZE
Add PCI configuration field "queue_indirect_size" Add MMIO configuration register "QueueIndirectNum" Add CCW configuration field "indirect_num" to vq_info_block
content.tex | 77 ++++++++++++++++++++++++++++++++++++++++++++++--- packed-ring.tex | 2 +- split-ring.tex | 8 +++-- 3 files changed, 80 insertions(+), 7 deletions(-)
Signature · 21 lines
2.30.2 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
PATCH v2 1/4 Add VIRTIO_RING_F_INDIRECT_SIZE
This new feature flag allows to decouple the maximum amount of descriptors in indirect descriptor tables from the Queue Size.
The new term "Queue Indirect Size" is introduced for this purpose, which is a bus specific configuration whose negotiation is further specified with subsequent patches.
--- content.tex | 32 ++++++++++++++++++++++++++++++-- packed-ring.tex | 2 +- split-ring.tex | 8 ++++++-- 3 files changed, 37 insertions(+), 5 deletions(-)
content.tex +30 -2
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}
@@ -1051,6 +1051,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport present either a value of 0 or a power of 2 in \field{queue_size}. +If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated, the device MUST provide the +Queue Indirect Size supported by device, which is a transport specific +configuration. It MUST allow the driver to set a lower value. + \drivernormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout} The driver MUST NOT write to \field{device_feature}, \field{num_queues}, \field{config_generation}, \field{queue_notify_off} or \field{queue_notify_data}.
@@ -6847,6 +6851,30 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_RING_F_INDIRECT_SIZE(41)] This feature indicates that the + Queue Indirect Size, i.e. the maximum amount of descriptors in indirect + descriptor tables, is independent from the Queue Size. + + Without this feature, the Queue Size limits the length of the descriptor + chain, including indirect descriptor tables as in \ref{sec:Basic Facilities of + a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect + Descriptors}, i.e. both the maximum amount of slots in the vring and the + actual bulk data size transmitted per vring slot. + + With this feature enabled, the Queue Size only limits the maximum amount + of slots in the vring, but does not limit the actual bulk data size + being transmitted when indirect descriptors are used. Decoupling these + two configuration parameters this way not only allows much larger bulk data + being transferred per vring slot, but also avoids complicated synchronization + mechanisms if the device only supports a very small amount of vring slots. Due + to the 16-bit size of a descriptor's "next" field there is still an absolute + limit of $2^{16}$ descriptors per indirect descriptor table. However the + actual maximum amount supported by either device or driver might be less, + and therefore the bus specific Queue Indirect Size value MUST additionally + be negotiated if VIRTIO_RING_F_INDIRECT_SIZE was negotiated to subsequently + negotiate the actual amount of maximum indirect descriptors supported + by both sides. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
packed-ring.tex +1 -1
@@ -195,7 +195,7 @@ \subsection{Scatter-Gather Support} The device limits the number of descriptors in a list through a transport-specific and/or device-specific value. If not limited, the maximum number of descriptors in a list is the virt queue -size. +size unless the VIRTIO_RING_F_INDIRECT_SIZE feature has been negotiated. \subsection{Next Flag: Descriptor Chaining} \label{sec:Packed Virtqueues / Next Flag: Descriptor Chaining}
split-ring.tex +6 -2
@@ -268,8 +268,12 @@ \subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Devi set the VIRTQ_DESC_F_INDIRECT flag within an indirect descriptor (ie. only one table per descriptor). -A driver MUST NOT create a descriptor chain longer than the Queue Size of -the device. +If VIRTIO_RING_F_INDIRECT_SIZE has not been negotiated, the driver MUST +NOT create a descriptor chain longer than the Queue Size of the device. + +If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated, the number of +descriptors per indirect descriptor table MUST NOT exceed the negotiated +Queue Indirect Size. A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT in \field{flags}.
Signature · 21 lines
2.30.2 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
PATCH v2 2/4 Add PCI configuration field "queue_indirect_size"
In reply to Christian SchoenebeckThis new PCI configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
--- content.tex | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
content.tex +18 -0
@@ -902,6 +902,7 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport le64 queue_device; /* read-write */ le16 queue_notify_data; /* read-only for driver */ le16 queue_reset; /* read-write */ + le32 queue_indirect_size; /* read-write */ }; \end{lstlisting}
@@ -987,6 +988,17 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport This field exists only if VIRTIO_F_RING_RESET has been negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}). +\item[\field{queue_indirect_size}] + This field is used to negotiate the maximum number of descriptors per + indirect descriptor table as in \ref{sec:Basic Facilities of a Virtio + Device / Virtqueues / The Virtqueue Descriptor Table / Indirect + Descriptors} if and only if the VIRTIO_RING_F_INDIRECT_SIZE feature has + been negotiated. + + The device specifies its maximum number of descriptors per indirect + descriptor table. If the driver requires fewer descriptors, it writes + its lower value to inform the device of the reduced resource + requirements. \end{description} \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -1081,6 +1093,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport values to different values than those that were used before the queue reset. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}). +The driver SHOULD write to \field{queue_indirect_size} if its maximum number of +descriptors per vring slot is lower than that reported by the device. + +The driver MUST NOT write a higher value to \field{queue_indirect_size} than the +one it reads from the device. + \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability} The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
Signature · 1 line
2.30.2
PATCH v2 3/4 Add MMIO configuration register "QueueIndirectNum"
In reply to Christian SchoenebeckThis new MMIO configuration register allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
--- content.tex | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
content.tex +20 -2
@@ -2225,8 +2225,9 @@ \subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over M \hline \mmioreg{QueueSel}{Virtual queue index}{0x030}{W}{% Writing to this register selects the virtual queue that the - following operations on the \field{QueueNumMax}, \field{QueueNum}, \field{QueueAlign} - and \field{QueuePFN} registers apply to. The index + following operations on the \field{QueueNumMax}, \field{QueueNum}, + \field{QueueAlign}, \field{QueuePFN} and \field{QueueIndirectNum} registers + apply to. The index number of the first queue is zero (0x0). . }
@@ -2282,6 +2283,19 @@ \subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over M Also see \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}. } \hline + \mmioreg{QueueIndirectNum}{Virtual queue indirect size}{0x074}{RW}{% + This register is used to negotiate the maximum number of descriptors per + indirect descriptor table as in \ref{sec:Basic Facilities of a Virtio + Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors} + if and only if the VIRTIO_RING_F_INDIRECT_SIZE feature has been negotiated. + + The device specifies its maximum number of descriptors per indirect + descriptor table. If the driver requires fewer descriptors, it writes + its lower value to inform the device of the reduced resource requirements. + + This applies to the queue selected by writing to \field{QueueSel}. + } + \hline \mmioreg{Config}{Configuration space}{0x100+}{RW}{} \hline \end{longtable}
@@ -2314,6 +2328,10 @@ \subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over M \item Notify the device about the queue size by writing the size to \field{QueueNum}. +\item If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated, read the maximum + length of indirect descriptor tables by reading \field{QueueIndirectNum}. + Write to lower this value if necessary. + \item Notify the device about the used alignment by writing its value in bytes to \field{QueueAlign}.
Signature · 1 line
2.30.2
PATCH v2 4/4 Add CCW configuration field "indirect_num" to vq_info_block
In reply to Christian SchoenebeckThis new CCW configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
--- content.tex | 5 +++++ 1 file changed, 5 insertions(+)
content.tex +5 -0
@@ -2599,6 +2599,7 @@ \subsubsection{Configuring a Virtqueue}\label{sec:Virtio Transport Options / Vir be16 num; be64 driver; be64 device; + be32 indirect_num; }; \end{lstlisting}
@@ -2607,6 +2608,10 @@ \subsubsection{Configuring a Virtqueue}\label{sec:Virtio Transport Options / Vir available area and used area for queue \field{index}, respectively. The actual virtqueue size (number of allocated buffers) is transmitted in \field{num}. +If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated then \field{indirect_num} +reflects the maximum length of indirect descriptor tables for queue +\field{index}. + \devicenormative{\paragraph}{Configuring a Virtqueue}{Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue} \field{res0} is reserved and MUST be ignored by the device.
Signature · 1 line
2.30.2
Quoted 6 lines · On Montag, 21. Februar 2022 17:22:18 CET Christian Schoenebeck wrote:
This is a follow-up of:
[1/2]:
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00034.html
[2/2]:
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00035.html
Quoted 8 lines
These 4 patches introduce a more fine graded control over the maximum length of indirect descriptor tables.
Associated Github Task:
https://github.com/oasis-tcs/virtio-spec/issues/122Latest Version (all patches):
https://github.com/cschoenebeck/virtio-spec/commits/long_indirect_descr
Ping.
Best regards,
Christian Schoenebeck
Quoted 29 lines
v1 -> v2:
* Update reserved bits section [patch 1].
* Add requirement to device normative section [patch 1].
* Use the general new term "Queue Indirect Size" where possible instead of
mentioning a bus specific field name [patch 1].* Simplified phrasing of requirements in split ring section [patch 1].
* Move bus-independent changes from patch 2 to patch 1.
* There is only one indirect descriptor table allowed per buffer (round-trip message), so use the term "per indirect descriptor table"
for the semantic of "Queue Indirect Size" in patch 2.* Simplify phrasing of requirements in patch 2.
* MMIO: First shot [NEW patch 3].
* CCW: First shot [NEW patch 4].
Christian Schoenebeck (4):
AddVIRTIO_RING_F_INDIRECT_SIZE
Add PCI configuration field "queue_indirect_size" Add MMIO configuration register "QueueIndirectNum" Add CCW configuration field "indirect_num" tovq_info_block
Quoted 4 lines
content.tex | 77 ++++++++++++++++++++++++++++++++++++++++++++++--- packed-ring.tex | 2 +- split-ring.tex | 8 +++-- 3 files changed, 80 insertions(+), 7 deletions(-)
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v2 1/4] Add VIRTIO_RING_F_INDIRECT_SIZE
In reply to Christian SchoenebeckQuoted 14 lines · Christian Schoenebeck
content.tex +10
-0@@ -6847,6 +6851,30 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_RING_F_INDIRECT_SIZE(41)] This feature indicates that the + Queue Indirect Size, i.e. the maximum amount of descriptors in indirect + descriptor tables, is independent from the Queue Size. + + Without this feature, the Queue Size limits the length of the descriptor + chain, including indirect descriptor tables as in \ref{sec:Basic Facilities of + a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect + Descriptors}, i.e. both the maximum amount of slots in the vring and the + actual bulk data size transmitted per vring slot. +
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6873
Quoted 10 lines
content.tex +10
-0+ With this feature enabled, the Queue Size only limits the maximum amount + of slots in the vring, but does not limit the actual bulk data size + being transmitted when indirect descriptors are used. Decoupling these + two configuration parameters this way not only allows much larger bulk data + being transferred per vring slot, but also avoids complicated synchronization + mechanisms if the device only supports a very small amount of vring slots. Due + to the 16-bit size of a descriptor's "next" field there is still an absolute + limit of $2^{16}$ descriptors per indirect descriptor table. However the + actual maximum amount supported by either device or driver might be less, + and therefore the bus specific Queue Indirect Size value MUST additionally
s/bus/transport/
Otherwise:
Re: [virtio-comment] [PATCH v2 2/4] Add PCI configuration field "queue_indirect_size"
In reply to Christian SchoenebeckQuoted 8 lines · Christian Schoenebeck
This new PCI configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
Signed-off-by Christian Schoenebeck <[email protected]>--- content.tex | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
Re: [PATCH v2 3/4] Add MMIO configuration register "QueueIndirectNum"
In reply to Christian SchoenebeckQuoted 9 lines · Christian Schoenebeck
This new MMIO configuration register allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
Signed-off-by Christian Schoenebeck <[email protected]Signed-off-by Christian Schoenebeck <[email protected]>--- content.tex | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
Re: [PATCH v2 4/4] Add CCW configuration field "indirect_num" to vq_info_block
In reply to Christian SchoenebeckQuoted 23 lines · Christian Schoenebeck
This new CCW configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
Signed-off-by Christian Schoenebeck <[email protected]Signed-off-by Christian Schoenebeck <[email protected]>--- content.tex | 5 +++++ 1 file changed, 5 insertions(+)content.tex +1
-0@@ -2599,6 +2599,7 @@ \subsubsection{Configuring a Virtqueue}\label{sec:Virtio Transport Options / Vir be16 num; be64 driver; be64 device; + be32 indirect_num; }; \end{lstlisting}
content.tex \subsubsection{Configuring a Virtqueue}\label{sec:Virtio Transport Options / Vir · line 2613
Quoted 6 lines
content.tex +3
-0@@ -2607,6 +2608,10 @@ \subsubsection{Configuring a Virtqueue}\label{sec:Virtio Transport Options / Vir available area and used area for queue \field{index}, respectively. The actual virtqueue size (number of allocated buffers) is transmitted in \field{num}. +If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated then \field{indirect_num} +reflects the maximum length of indirect descriptor tables for queue +\field{index}.
I think the transfer direction of CCW_CMD_SET_VQ struct vq_info_block is driver-to-device. So it allows the driver to set the Queue Indirect Size, but how does the driver query the device's maximum Queue Indirect Size value?
Quoted 10 lines · Christian Schoenebeck
Quoted 9 lines · On Montag, 21. Februar 2022 17:22:18 CET Christian Schoenebeck wrote:
This is a follow-up of:
[1/2]:
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00034.html
[2/2]:
https://lists.oasis-open.org/archives/virtio-comment/202112/msg00035.htmlThese 4 patches introduce a more fine graded control over the maximum length of indirect descriptor tables.
Quoted 7 lines
Quoted 5 lines
Associated Github Task:
https://github.com/oasis-tcs/virtio-spec/issues/122Latest Version (all patches):
https://github.com/cschoenebeck/virtio-spec/commits/long_indirect_descrPing.
Sorry for the delay.
This looks very close. I posted a CCW question but it might just be my lack of familiarity with CCW :).
Stefan
Re: [PATCH v2 4/4] Add CCW configuration field "indirect_num" to vq_info_block
In reply to Stefan HajnocziQuoted 7 lines · On Donnerstag, 10. März 2022 16:26:46 CET Stefan Hajnoczi wrote:
Quoted 6 lines · Christian Schoenebeck
This new CCW configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
Signed-off-by Christian Schoenebeck <[email protected]Signed-off-by Christian Schoenebeck <[email protected]>
I'll fix those accidentally added double SOBs BTW.
Quoted 20 lines
Quoted 20 lines
---
content.tex | 5 +++++ 1 file changed, 5 insertions(+)content.tex +0
-0@@ -2599,6 +2599,7 @@ \subsubsection{Configuring aVirtqueue}\label{sec:Virtio Transport Options / Vir>
be16 num; be64 driver; be64 device;+ be32
indirect_num;};
\end{lstlisting}
Quoted 13 lines
Quoted 8 lines
content.tex +0
-0@@ -2607,6 +2608,10 @@ \subsubsection{Configuring aVirtqueue}\label{sec:Virtio Transport Options / Vir>
available area and used area for queue \field{index}, respectively. The actual virtqueue size (number of allocated buffers) is transmitted in \field{num}.>
+IfVIRTIO_RING_F_INDIRECT_SIZEhas been negotiated then \field{indirect_num} +reflects the maximum length of indirect descriptor tables for queue +\field{index}.I think the transfer direction of
CCW_CMD_SET_VQstructvq_info_blockis driver-to-device. So it allows the driver to set the Queue Indirect Size, but how does the driver query the device's maximum Queue Indirect Size value?
Ah, seems you are right Stefan. I assumed this was similar to PCI.
So the way to go on CCW would be an additional (equally named?) field in struct vq_info_block (CCW_CMD_READ_VQ_CONF) for device -> driver?
Cornelia?
Best regards,
Christian Schoenebeck
[virtio-comment] Re: [PATCH v2 4/4] Add CCW configuration field "indirect_num" to vq_info_block
In reply to Stefan HajnocziQuoted 27 lines · Stefan Hajnoczi
Quoted 26 lines · Christian Schoenebeck
This new CCW configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
Signed-off-by Christian Schoenebeck <[email protected]Signed-off-by Christian Schoenebeck <[email protected]>--- content.tex | 5 +++++ 1 file changed, 5 insertions(+)content.tex +1
-0@@ -2599,6 +2599,7 @@ \subsubsection{Configuring a Virtqueue}\label{sec:Virtio Transport Options / Vir be16 num; be64 driver; be64 device; + be32 indirect_num; }; \end{lstlisting}@@ -2607,6 +2608,10 @@ \subsubsection{Configuring a Virtqueue}\label{sec:Virtio Transport Options / Vir available area and used area for queue \field{index}, respectively. The actual virtqueue size (number of allocated buffers) is transmitted in \field{num}.
Quoted 8 lines
content.tex \subsubsection{Configuring a Virtqueue}\label{sec:Virtio Transport Options / Vir · line 2613
Quoted 3 lines
content.tex +3
-0+If VIRTIO_RING_F_INDIRECT_SIZE has been negotiated then \field{indirect_num} +reflects the maximum length of indirect descriptor tables for queue +\field{index}.I think the transfer direction of
CCW_CMD_SET_VQstructvq_info_blockis driver-to-device. So it allows the driver to set the Queue Indirect Size, but how does the driver query the device's maximum Queue Indirect Size value?
[cc:ing Halil in case he has any further comments]
You're right, CCW_CMD_SET_VQ + vq_info_block is driver-to-device. The driver will obtain information about a queue via CCW_CMD_READ_VQ_CONF + vq_config_block, so a max_indirect_num field needs to be added there as well, I think.
Moreover, we're changing the length of the ccw payload. Extending at the end is generally fine, but the device and the driver need to agree on what the expected payload is. We basically have two options here:
* Make it depend on the feature bit being negotiated. This works because virtqueue discovery needs to be done only after feature negotiation has completed. However, this will get a bit awkward if we need to add another field depending on a new feature bit: negotiating that hypothetical feature would imply that the indirect num fields would be present, but not valid, if the indirect feature had not been negotiated. Not a showstopper, but looks a bit odd.
* Tie it to a new ccw revision (3) and make offering the feature bit dependant upon revision 3 or later being negotiated. This has the advantage that ccw revisions always build on each other (so no awkwardness for future extension) and the drawback of introducing another transport-specific prereq.
If we can live with the possible awkwardness of future extensions, tying the size of the structures to feature bits might be the preferable way.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v2 1/4] Add VIRTIO_RING_F_INDIRECT_SIZE
In reply to Stefan HajnocziQuoted 31 lines · On Donnerstag, 10. März 2022 15:57:28 CET Stefan Hajnoczi wrote:
Quoted 21 lines · Christian Schoenebeck
(quoted diff) +0
-0@@ -6847,6 +6851,30 @@ \chapter{Reserved Feature Bits}\label{sec:ReservedFeature Bits}>
that the driver can reset a queue individually.
See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.>content.tex +5
-0+ \item[VIRTIO_RING_F_INDIRECT_SIZE(41)] This feature indicates that the + Queue Indirect Size, i.e. the maximum amount of descriptors in indirect + descriptor tables, is independent from the Queue Size. + + Without this feature, the Queue Size limits the length of thedescriptor
+ chain, including indirect descriptor tables as in \ref{sec:Basic Facilities of + a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect + Descriptors}, i.e. both the maximum amount of slots in the vring and the + actual bulk data size transmitted per vring slot.(quoted diff) +2
-0+ + With this feature enabled, the Queue Size only limits the maximumamount
content.tex +3
-0+ of slots in the vring, but does not limit the actual bulk data size + being transmitted when indirect descriptors are used. Decoupling these + two configuration parameters this way not only allows much larger bulkQuoted 7 lines
data + being transferred per vring slot, but also avoids complicated synchronization + mechanisms if the device only supports a very small amount of vring slots. Due + to the 16-bit size of a descriptor's "next" field there is still an absolute + limit of $2^{16}$ descriptors per indirect descriptor table. However the + actual maximum amount supported by either device or driver might be less, + and therefore the bus specific Queue Indirect Size value MUST additionally
s/bus/transport/
Quoted 2 lines
Otherwise:
Reviewed-by Stefan Hajnoczi <[email protected]>
I'll change that in v3. Thanks!
Best regards,
Christian Schoenebeck
Re: [PATCH v2 4/4] Add CCW configuration field "indirect_num" to vq_info_block
In reply to Cornelia HuckQuoted 66 lines · On Donnerstag, 10. März 2022 17:09:25 CET Cornelia Huck wrote:
Quoted 27 lines · Stefan Hajnoczi
Quoted 26 lines · Christian Schoenebeck
This new CCW configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
Signed-off-by Christian Schoenebeck <[email protected]Signed-off-by Christian Schoenebeck <[email protected]>---
content.tex | 5 +++++ 1 file changed, 5 insertions(+)content.tex +0
-0@@ -2599,6 +2599,7 @@ \subsubsection{Configuring aVirtqueue}\label{sec:Virtio Transport Options / Vir>>
be16 num; be64 driver; be64 device;+ be32
indirect_num;};
\end{lstlisting}Quoted 13 lines
Quoted 8 lines
content.tex +0
-0@@ -2607,6 +2608,10 @@ \subsubsection{Configuring aVirtqueue}\label{sec:Virtio Transport Options / Vir>>
available area and used area for queue \field{index}, respectively. The actual virtqueue size (number of allocated buffers) is transmitted in \field{num}.>>
+IfVIRTIO_RING_F_INDIRECT_SIZEhas been negotiated then \field{indirect_num} +reflects the maximum length of indirect descriptor tables for queue +\field{index}.I think the transfer direction of
CCW_CMD_SET_VQstructvq_info_blockis driver-to-device. So it allows the driver to set the Queue Indirect Size, but how does the driver query the device's maximum Queue Indirect Size value?[cc:ing Halil in case he has any further comments]
You're right,
CCW_CMD_SET_VQ+vq_info_blockis driver-to-device. The driver will obtain information about a queue viaCCW_CMD_READ_VQ_CONF+vq_config_block, so amax_indirect_numfield needs to be added there as well, I think.Moreover, we're changing the length of the ccw payload. Extending at the end is generally fine, but the device and the driver need to agree on what the expected payload is. We basically have two options here:
* Make it depend on the feature bit being negotiated. This works because virtqueue discovery needs to be done only after feature negotiation has completed. However, this will get a bit awkward if we need to add another field depending on a new feature bit: negotiating that hypothetical feature would imply that the indirect num fields would be present, but not valid, if the indirect feature had not been negotiated. Not a showstopper, but looks a bit odd.
* Tie it to a new ccw revision (3) and make offering the feature bit dependant upon revision 3 or later being negotiated. This has the advantage that ccw revisions always build on each other (so no awkwardness for future extension) and the drawback of introducing another transport-specific prereq.
Quoted 2 lines
If we can live with the possible awkwardness of future extensions, tying the size of the structures to feature bits might be the preferable way.
Really? My intuitive pick would rather be option 2 (CCW revision). But I'll go for whatever the common opinion is on this CCW issue.
Best regards,
Christian Schoenebeck
[virtio-comment] Re: [PATCH v2 4/4] Add CCW configuration field "indirect_num" to vq_info_block
Quoted 67 lines · Christian Schoenebeck
Quoted 66 lines · On Donnerstag, 10. März 2022 17:09:25 CET Cornelia Huck wrote:
Quoted 27 lines · Stefan Hajnoczi
Quoted 26 lines · Christian Schoenebeck
This new CCW configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
Signed-off-by Christian Schoenebeck <[email protected]Signed-off-by Christian Schoenebeck <[email protected]>---
content.tex | 5 +++++ 1 file changed, 5 insertions(+)content.tex +0
-0@@ -2599,6 +2599,7 @@ \subsubsection{Configuring aVirtqueue}\label{sec:Virtio Transport Options / Vir>>
be16 num; be64 driver; be64 device;+ be32
indirect_num;};
\end{lstlisting}Quoted 13 lines
Quoted 8 lines
content.tex +0
-0@@ -2607,6 +2608,10 @@ \subsubsection{Configuring aVirtqueue}\label{sec:Virtio Transport Options / Vir>>
available area and used area for queue \field{index}, respectively. The actual virtqueue size (number of allocated buffers) is transmitted in \field{num}.>>
+IfVIRTIO_RING_F_INDIRECT_SIZEhas been negotiated then \field{indirect_num} +reflects the maximum length of indirect descriptor tables for queue +\field{index}.I think the transfer direction of
CCW_CMD_SET_VQstructvq_info_blockis driver-to-device. So it allows the driver to set the Queue Indirect Size, but how does the driver query the device's maximum Queue Indirect Size value?[cc:ing Halil in case he has any further comments]
You're right,
CCW_CMD_SET_VQ+vq_info_blockis driver-to-device. The driver will obtain information about a queue viaCCW_CMD_READ_VQ_CONF+vq_config_block, so amax_indirect_numfield needs to be added there as well, I think.Moreover, we're changing the length of the ccw payload. Extending at the end is generally fine, but the device and the driver need to agree on what the expected payload is. We basically have two options here:
* Make it depend on the feature bit being negotiated. This works because virtqueue discovery needs to be done only after feature negotiation has completed. However, this will get a bit awkward if we need to add another field depending on a new feature bit: negotiating that hypothetical feature would imply that the indirect num fields would be present, but not valid, if the indirect feature had not been negotiated. Not a showstopper, but looks a bit odd.
* Tie it to a new ccw revision (3) and make offering the feature bit dependant upon revision 3 or later being negotiated. This has the advantage that ccw revisions always build on each other (so no awkwardness for future extension) and the drawback of introducing another transport-specific prereq.
Quoted 5 lines
Quoted 2 lines
If we can live with the possible awkwardness of future extensions, tying the size of the structures to feature bits might be the preferable way.
Really? My intuitive pick would rather be option 2 (CCW revision). But I'll go for whatever the common opinion is on this CCW issue.
Either would work; that's why I'd like to have Halil's opinion as well.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v2 4/4] Add CCW configuration field "indirect_num" to vq_info_block
Quoted 56 lines · On Fri, 11 Mar 2022 10:21:54 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 54 lines · Christian Schoenebeck
Quoted 53 lines · On Donnerstag, 10. März 2022 17:09:25 CET Cornelia Huck wrote:
Quoted 27 lines · Stefan Hajnoczi
Quoted 26 lines · Christian Schoenebeck
This new CCW configuration field allows to negotiate a more fine graded maximum lenght of indirect descriptor chains.
Signed-off-by Christian Schoenebeck <[email protected]Signed-off-by Christian Schoenebeck <[email protected]>---
content.tex | 5 +++++ 1 file changed, 5 insertions(+)content.tex +0
-0@@ -2599,6 +2599,7 @@ \subsubsection{Configuring aVirtqueue}\label{sec:Virtio Transport Options / Vir>>
be16 num; be64 driver; be64 device;+ be32
indirect_num;};
\end{lstlisting}Quoted 13 lines
Quoted 8 lines
content.tex +0
-0@@ -2607,6 +2608,10 @@ \subsubsection{Configuring aVirtqueue}\label{sec:Virtio Transport Options / Vir>>
available area and used area for queue \field{index}, respectively. The actual virtqueue size (number of allocated buffers) is transmitted in \field{num}.>>
+IfVIRTIO_RING_F_INDIRECT_SIZEhas been negotiated then \field{indirect_num} +reflects the maximum length of indirect descriptor tables for queue +\field{index}.I think the transfer direction of
CCW_CMD_SET_VQstructvq_info_blockis driver-to-device. So it allows the driver to set the Queue Indirect Size, but how does the driver query the device's maximum Queue Indirect Size value?[cc:ing Halil in case he has any further comments]
You're right,
CCW_CMD_SET_VQ+vq_info_blockis driver-to-device. The driver will obtain information about a queue viaCCW_CMD_READ_VQ_CONF+vq_config_block, so amax_indirect_numfield needs to be added there as well, I think.Moreover, we're changing the length of the ccw payload. Extending at the end is generally fine, but the device and the driver need to agree on what the expected payload is. We basically have two options here:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
* Make it depend on the feature bit being negotiated. This works because virtqueue discovery needs to be done only after feature negotiation has completed. However, this will get a bit awkward if we need to add another field depending on a new feature bit: negotiating that hypothetical feature would imply that the indirect num fields would be present, but not valid, if the indirect feature had not been negotiated. Not a showstopper, but looks a bit odd.
I think can get very ugly. Present but not valid is not the problem in my opinion. In fact we would do the same: if the feature is not negotiated, the field is invalid. But making the layout and size depend on a feature is tricky, because one can't tell: if feature x is negotiated the layout is that and that for the reasons you pointed out.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
* Tie it to a new ccw revision (3) and make offering the feature bit dependant upon revision 3 or later being negotiated. This has the advantage that ccw revisions always build on each other (so no awkwardness for future extension) and the drawback of introducing another transport-specific prereq.
I think this is inherent. If the transport does not support the new CCW layout there is no way we can make this work. AFAIR the transport can just fence the feature if the transport prereq is not met.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
If we can live with the possible awkwardness of future extensions, tying the size of the structures to feature bits might be the preferable way.
Really? My intuitive pick would rather be option 2 (CCW revision). But I'll go for whatever the common opinion is on this CCW issue.
Either would work; that's why I'd like to have Halil's opinion as well.
Halil was catching up with the history :) Sorry for being late to the party.
I tend to agree with Christian: this impacts when a CCW should be accepted or rejected, and is thus not much unlike adding a new ccw. Furthermore I don't think mix and match (which is kind of the default for feature bits) works well for layout of structs.
But let me take a step back, and ask some more general questions?
1) What is the objective behind the mechanism which can be used by the driver to tell the device its maximum. I believe the indirect descriptor table is always allocated by the guest, so the guest has no reason to fear larger than its max. The only thing I can think of is some pools of buffers allocated and maintained by the device, where each buffer is the size of max payload. Is this what we have in mind here?
2) Not so long ago we had a discussion about introducing a common (device agnostic) configuration space (a.k.a. misc config). The indirect table size might be a good candidate for that as well. We essentially want the very same functionality for all the transports, it is just that we have no vehicle at the moment to do this in an uniform way. Opinions?
Regards,
Halil
[virtio-comment] Re: [PATCH v2 4/4] Add CCW configuration field "indirect_num" to vq_info_block
Quoted 7 lines · On Freitag, 11. März 2022 12:15:40 CET Halil Pasic wrote:
But let me take a step back, and ask some more general questions?
1) What is the objective behind the mechanism which can be used by the driver to tell the device its maximum. I believe the indirect descriptor table is always allocated by the guest, so the guest has no reason to fear larger than its max. The only thing I can think of is some pools of buffers allocated and maintained by the device, where each buffer is the size of max payload. Is this what we have in mind here?
As described at the end on the associated virtio-spec github issue [1]:
"
Linux drivers
Since torvalds/linux@44ed808 the Linux kernel ignores if individual drivers exceed the Queue Size (and violating the specs). Likewise there are some devices which use device specific config to negotiate a max. bulk data size being lower than the Queue Size. The proposed spec changes would address both use cases in a clean way.
QEMU
QEMU tolerates as well if guests drivers exceed the Queue Size. However currently it has a hard coded limit of max. 1024 memory segments:
#define VIRTQUEUE_MAX_SIZE 1024
Which limits the max. buik data size per virtio message to slightly below 4MB
"
So ATM when you run a Linux driver that exceeds QEMU's current limit of 1024 segments, you would get the following fatal QEMU error [2]:
qemu-system-x86_64: virtio: too many write descriptors in indirect table
QEMU currently allocates the required space for the descriptors on the stack and uses that hard coded value for the allocation size. Now even if you would change that (which is a delicate bunch of code), you would still break compatibility with Linux guests running on older QEMU hosts.
[1] https://github.com/oasis-tcs/virtio-spec/issues/122
[2] https://lore.kernel.org/netdev/[email protected]/
Quoted 8 lines
2) Not so long ago we had a discussion about introducing a common (device agnostic) configuration space (a.k.a. misc config). The indirect table size might be a good candidate for that as well. We essentially want the very same functionality for all the transports, it is just that we have no vehicle at the moment to do this in an uniform way. Opinions?
Regards,
Halil
I need to leave that to other people to comment on, as I am completely unaware about these plans. Sounds like a long-term plan to me though.
Best regards,
Christian Schoenebeck
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/