VIRTIO_F_RING_PACKED_USED: packing used descriptors
PATCH 1/1 VIRTIO_F_RING_PACKED_USED: packing used descriptors
Sometimes the device needs to only write out a single used descriptor after processing a batch of multiple available descriptors. This can happen when using
descriptor chaining or with in-order
use of descriptors. In this case, the device writes out a single used descriptor with the buffer id of the last descriptor in the batch.
When this happens, used descriptors stop being contiguous in memory. As a result each used descriptor needs a separate memory write. For devices on the pci express bus, this incurs overhead of a separate write trasaction (e.g. pci express headers etc).
This patch proposes a new feature: VIRTIO_F_RING_PACKED_USED. The idea is that both device and driver count all of the descriptors in a batch as if all of them are being written out when used. The resulting batch counter controls wrap around to beginning of the ring, which then serves as a synchronization point.
--- content.tex | 3 +++ packed-ring.tex | 27 +++++++++++++++++++++++---- 2 files changed, 26 insertions(+), 4 deletions(-)
content.tex +3 -0
@@ -6247,6 +6247,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_ORDER_PLATFORM(36)] This feature indicates that memory accesses by the driver and the device are ordered in a way described by the platform. + \item[VIRTIO_F_RING_PACKED_USED(37)] This feature indicates + support for packed write out of used descriptors. Requires + VIRTIO_F_RING_PACKED. If this feature bit is negotiated, the ordering in effect for any memory accesses by the driver that need to be ordered in a specific way
packed-ring.tex +23 -4
@@ -131,17 +131,36 @@ \subsection{Polling of available and used descriptors} poll (or test) a single location in memory: the next device descriptor after the one they processed previously, in circular order. +\subsection{Skipping Used Descriptors} +\label{sec:Packed Virtqueues / Skipping Used Descriptors} Sometimes the device needs to only write out a single used descriptor after processing a batch of multiple available descriptors. As described in more detail below, this can happen when using descriptor chaining or with in-order -use of descriptors. In this case, the device writes out a used -descriptor with the buffer id of the last descriptor in the group. -After processing the used descriptor, both device and driver then +use of descriptors. In this case, the device writes out a single used +descriptor with the buffer id of the last descriptor in the batch. + +If VIRTIO_F_RING_PACKED_USED has not been negotiated, +after processing the used descriptor, both device and driver then skip forward in the ring the number of the remaining descriptors -in the group until processing (reading for the driver and writing +in the batch until processing (reading for the driver and writing for the device) the next used descriptor. +If VIRTIO_F_RING_PACKED_USED has been negotiated, +both the device and the driver are expected to maintain, +internally, a 16-bit used batch counter initialized to 0. +Each time a batch of descriptors is used, the batch counter +is incremented by the number of descriptors in the batch. +If after processing the used descriptor, the batch counter +does not exceed the ring size, the next used descriptor +is written immediately adjacent to the current one, in ring +order. +When after processing the used descriptor, the batch counter +exceeds the ring size, the batch counter is decremented by the +ring size; both device and driver then skip to the +beginning (offset 0) of the ring until processing (reading for +the driver and writing for the device) the next used descriptor. + \subsection{Write Flag} \label{sec:Packed Virtqueues / Write Flag}
Signature · 7 lines
MST --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
packed-ring.tex \subsection{Polling of available and used descriptors} · line 162
Quoted 5 lines · Michael S. Tsirkin
packed-ring.tex +5
-0+When after processing the used descriptor, the batch counter +exceeds the ring size, the batch counter is decremented by the +ring size; both device and driver then skip to the +beginning (offset 0) of the ring until processing (reading for +the driver and writing for the device) the next used descriptor.
This may be clearer:
both device and driver then use ring index 0 for the next used descriptor
Or:
The device writes the next used descriptor at ring index 0. The driver reads the next used descriptor from ring index 0.
I'm not sure I understand this scheme. Is this pseudo-code correct?
batch_counter += num_used_descs
if batch_counter > ring_size:
batch_counter -= ring_size
used_idx = 0
Can you share an explanation of how this scheme ensures that the driver's next avail descriptor index never overtakes the device's next used descriptor index?
Previously it was obvious that the two never cross because the used descriptor index jumped by the number of avail descriptors that were processed by the device.
Stefan
Quoted 17 lines · Stefan Hajnoczi
packed-ring.tex \subsection{Polling of available and used descriptors} · line 162
Quoted 5 lines · Michael S. Tsirkin
packed-ring.tex +5
-0+When after processing the used descriptor, the batch counter +exceeds the ring size, the batch counter is decremented by the +ring size; both device and driver then skip to the +beginning (offset 0) of the ring until processing (reading for +the driver and writing for the device) the next used descriptor.This may be clearer:
both device and driver then use ring index 0 for the next used descriptor
Or:
The device writes the next used descriptor at ring index 0. The driver reads the next used descriptor from ring index 0.
Quoted 6 lines
I'm not sure I understand this scheme. Is this pseudo-code correct?
batch_counter+=num_used_descs
ifbatch_counter>ring_size:
batch_counter-=ring_size
used_idx= 0
yes
Quoted 3 lines
Can you share an explanation of how this scheme ensures that the driver's next avail descriptor index never overtakes the device's next used descriptor index?
Simply because to overtake it, it needs to wrap around to 0. That happens when batch counter wraps over to 0, if we make device wrap at that point all is well.
Quoted 5 lines
Previously it was obvious that the two never cross because the used descriptor index jumped by the number of avail descriptors that were processed by the device.
Stefan
it still does that, just at the time of wrap around, not after each descriptor.
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Quoted 28 lines · Michael S. Tsirkin
Quoted 17 lines · Stefan Hajnoczi
packed-ring.tex \subsection{Polling of available and used descriptors} · line 162
Quoted 5 lines · Michael S. Tsirkin
packed-ring.tex +5
-0+When after processing the used descriptor, the batch counter +exceeds the ring size, the batch counter is decremented by the +ring size; both device and driver then skip to the +beginning (offset 0) of the ring until processing (reading for +the driver and writing for the device) the next used descriptor.This may be clearer:
both device and driver then use ring index 0 for the next used descriptor
Or:
The device writes the next used descriptor at ring index 0. The driver reads the next used descriptor from ring index 0.
Quoted 6 lines
I'm not sure I understand this scheme. Is this pseudo-code correct?
batch_counter+=num_used_descs
ifbatch_counter>ring_size:
batch_counter-=ring_size
used_idx= 0yes
Quoted 7 lines
Quoted 3 lines
Can you share an explanation of how this scheme ensures that the driver's next avail descriptor index never overtakes the device's next used descriptor index?
Simply because to overtake it, it needs to wrap around to 0. That happens when batch counter wraps over to 0, if we make device wrap at that point all is well.
What confuses me is that batch_counter is not reset to 0. It is set to batch_counter -= ring_size.
I guess I need to get my crayons and draw pictures :-).
Stefan
Quoted 37 lines · Stefan Hajnoczi
Quoted 28 lines · Michael S. Tsirkin
Quoted 17 lines · Stefan Hajnoczi
packed-ring.tex \subsection{Polling of available and used descriptors} · line 162
Quoted 5 lines · Michael S. Tsirkin
packed-ring.tex +5
-0+When after processing the used descriptor, the batch counter +exceeds the ring size, the batch counter is decremented by the +ring size; both device and driver then skip to the +beginning (offset 0) of the ring until processing (reading for +the driver and writing for the device) the next used descriptor.This may be clearer:
both device and driver then use ring index 0 for the next used descriptor
Or:
The device writes the next used descriptor at ring index 0. The driver reads the next used descriptor from ring index 0.
Quoted 6 lines
I'm not sure I understand this scheme. Is this pseudo-code correct?
batch_counter+=num_used_descs
ifbatch_counter>ring_size:
batch_counter-=ring_size
used_idx= 0yes
Quoted 7 lines
Quoted 3 lines
Can you share an explanation of how this scheme ensures that the driver's next avail descriptor index never overtakes the device's next used descriptor index?
Simply because to overtake it, it needs to wrap around to 0. That happens when batch counter wraps over to 0, if we make device wrap at that point all is well.
Quoted 6 lines
What confuses me is that
batch_counteris not reset to 0. It is set tobatch_counter-=ring_size.I guess I need to get my crayons and draw pictures :-).
Stefan
The idea is that when all available buffers have been used then batch counter of the device matches the next avail index of the driver.
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php