packed ring virtio-net backends support
v2 cover packed ring virtio-net backends support
From: Wei Xu <[email protected]>
v1->v2:
- fix patchew complaint
- only set/get last_avail_idx/wrap_counter for vhost migration(Maxime)
- replace 'out_num' and 'in_num' with 'elem_entries'in packed_pop()(Maxime)
- set last used idx/wrap_counter to last avail ones when flushing(Maxime)
- replace '*host_has_feature()' with '*vdev_has_feature()' for ioctl(Maxime)
- replace going through indirect descriptors with desc.len/sizeof(desc)(btw)
- add new subsection for packed ring(Jason)
rfc v3 -> v1
- migration support for both userspace and vhost-net, need tweak vhost ioctl() to make it work(the code is pasted in the commit message of vhost migration patch #13).
Note:
the high 32-bit guest feature bit is saved as a subsection for virtio devices which makes packed ring feature bit check unusable when loading the saved per-queue variables(this is done before loading subsection which is the last action for device during migration), so I save and load all the things generally for now, any idea to fix this?
- Fixed comments from Jason for rfc v3 sorted by patch #, two comments I didn't take were(from patch) listed here:
09: - introduce new API(virtqueue_fill_n()).
- Didn't take it since userspace backend does not support batching, so only one element is popped and current API should be enough.
06 & 07: Refactor split and packed pop()/get_avail_bytes().
- the duplicated code interwined with split/packed ring specific things and it might make it unclear, so I only extracted the few common parts out side rcu and keep the others separate.
The other revised comments:
02: - reuse current 'avail/used' for 'driver/device' in VRingMemoryRegionCache.
- remove event_idx since shadow_avail_idx works.
03: - move size recalculation to a separate patch.
- keep 'avail/used' in current calculation function name.
- initialize 'desc' memory region as 'false' for 1.0('true' for 1.1)
04: - delete 'event_idx'
05: - rename 'wc' to wrap_counter.
06: - converge common part outside rcu section for 1.0/1.1.
- move memory barrier for the first 'desc' in between checking flag and read other fields.
- remove unnecessary memory barriers for indirect descriptors.
- no need to destroy indirect memory cache since it is generally done before return from the function.
- remove redundant maximum chained descriptors limitation check.
- there are some differences(desc name, wrap idx/counter, flags) between split and packed rings, so keep them separate for now.
- amend the comment when recording index and wrap counter for a kick from guest.
07: - calculate fields in descriptor instead of read it when filling.
- put memory barrier correctly before filling the flags in descriptor.
- replace full memory barrier with a write barrier in fill.
- shift to read descriptor flags and descriptor necessarily and separately in packed_pop().
- correct memory barrier in packed_pop() as in packed_fill().
08: - reuse 'shadow_avail_idx' instead of adding a new 'event_idx'.
- use the compact and verified vring_packed_need_event() version for vhost net/user.
12: - remove the odd cherry-pick comment.
- used bit '15' for wrap_counters.
rfc v2->v3
- addressed performance issue
- fixed feedback from v2
rfc v1->v2
- sync to tiwei's v5
- reuse memory cache function with 1.0
- dropped detach patch and notification helper(04 & 05 in v1)
- guest virtio-net driver unload/reload support
- event suppression support(not tested)
- addressed feedback from v1
Wei Xu (15):
virtio: introduce packed ring definitions
virtio: redefine structure & memory cache for packed ring
virtio: expand offset calculation for packed ring
virtio: add memory region init for packed ring
virtio: init wrap counter for packed ring
virtio: init and desc empty check for packed ring
virtio: get avail bytes check for packed ring
virtio: fill/flush/pop for packed ring
virtio: event suppression support for packed ring
virtio-net: fill head desc after done all in a chain
virtio: add userspace migration for packed ring
virtio: add vhost-net migration for packed ring
virtio: packed ring feature bit for userspace backend
vhost: enable packed ring
virtio: enable packed ring via a new command line
hw/net/vhost_net.c | 2 + hw/net/virtio-net.c | 11 +- hw/virtio/virtio.c | 774 +++++++++++++++++++++++-- include/hw/virtio/virtio.h | 4 +- include/standard-headers/linux/virtio_config.h | 15 + include/standard-headers/linux/virtio_ring.h | 43 ++ 6 files changed, 800 insertions(+), 49 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v2 1/15 virtio: introduce packed ring definitions
From: Wei Xu <[email protected]>
Quoted 1 line
From 1.1 spec.
--- include/standard-headers/linux/virtio_config.h | 15 +++++++++ include/standard-headers/linux/virtio_ring.h | 43 ++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)
include/standard-headers/linux/virtio_config.h +15 -0
@@ -75,6 +75,21 @@*/ #define VIRTIO_F_IOMMU_PLATFORM 33 +/* This feature indicates support for the packed virtqueue layout. */ +#define VIRTIO_F_RING_PACKED 34 + +/* Enable events */ +#define RING_EVENT_FLAGS_ENABLE 0x0 +/* Disable events */ +#define RING_EVENT_FLAGS_DISABLE 0x1 +/* +* * Enable events for a specific descriptor +* * (as specified by Descriptor Ring Change Event Offset/Wrap Counter). +* * Only valid if VIRTIO_F_RING_EVENT_IDX has been negotiated. +* */ +#define RING_EVENT_FLAGS_DESC 0x2 +/* The value 0x3 is reserved */ +/* * Does the device support Single Root I/O Virtualization? */
include/standard-headers/linux/virtio_ring.h +43 -0
59 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH v2 01/15] virtio: introduce packed ring definitions
Quoted 5 lines · On Wed, Jan 16, 2019 at 01:31:06PM -0500, [email protected] wrote:
From: Wei Xu <[email protected]>
Quoted 1 line
From 1.1 spec.
Signed-off-by Wei Xu <[email protected]>
So pls don't open-code this in qemu. These headers should come from Linux using scripts/update-linux-headers.sh.
Quoted 15 lines
--- include/standard-headers/linux/virtio_config.h | 15 +++++++++ include/standard-headers/linux/virtio_ring.h | 43 ++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)include/standard-headers/linux/virtio_config.h +2
-0@@ -75,6 +75,21 @@*/ #define VIRTIO_F_IOMMU_PLATFORM 33 +/* This feature indicates support for the packed virtqueue layout. */ +#define VIRTIO_F_RING_PACKED 34
include/standard-headers/linux/virtio_config.h line 89
Quoted 10 lines
include/standard-headers/linux/virtio_config.h +10
-0+ +/* Enable events */ +#define RING_EVENT_FLAGS_ENABLE 0x0 +/* Disable events */ +#define RING_EVENT_FLAGS_DISABLE 0x1 +/* +* * Enable events for a specific descriptor +* * (as specified by Descriptor Ring Change Event Offset/Wrap Counter). +* * Only valid if VIRTIO_F_RING_EVENT_IDX has been negotiated. +* */
Above is formatted very strangely. Prefix RING_ is also probably not a good one. Linux uses VRING.
Quoted 50 lines
include/standard-headers/linux/virtio_config.h +3
-0+#define RING_EVENT_FLAGS_DESC 0x2 +/* The value 0x3 is reserved */ +/* * Does the device support Single Root I/O Virtualization? */ include/standard-headers/linux/virtio_ring.h +22
-0@@ -42,6 +42,10 @@/* This means the buffer contains a list of buffer descriptors. */ #define VRING_DESC_F_INDIRECT 4 +/* Mark a descriptor as available or used. */ +#define VRING_DESC_F_AVAIL (1ul << 7) +#define VRING_DESC_F_USED (1ul << 15) +/* The Host uses this in used->flags to advise the Guest: don't kick me when * you add a buffer. It's unreliable, so it's simply an optimization. Guest * will still kick if it's out of buffers. */ @@ -51,6 +55,17 @@* optimization. */ #define VRING_AVAIL_F_NO_INTERRUPT 1 +/* Enable events. */ +#define VRING_EVENT_F_ENABLE 0x0 +/* Disable events. */ +#define VRING_EVENT_F_DISABLE 0x1 +/* +* Enable events for a specific descriptor +* (as specified by Descriptor Ring Change Event Offset/Wrap Counter). +* Only valid if VIRTIO_RING_F_EVENT_IDX has been negotiated. +*/ +#define VRING_EVENT_F_DESC 0x2 +/* We support indirect buffer descriptors */ #define VIRTIO_RING_F_INDIRECT_DESC 28@@ -169,4 +184,32 @@ static inline int vring_need_event(uint16_t event_idx, uint16_t new_idx, uint16_return (uint16_t )(new_idx - event_idx -1 ) < (uint16_t )(new_idx - old); } +struct vring_packed_desc_event { +/* Descriptor Ring Change Event Offset/Wrap Counter. */ +__virtio16 off_wrap; +/* Descriptor Ring Change Event Flags. */ +__virtio16 flags; +}; +
include/standard-headers/linux/virtio_ring.h static inline int vring_need_event(uint16_t event_idx, uint16_t new_idx, uint16_ · line 203
Quoted 10 lines
include/standard-headers/linux/virtio_ring.h +10
-0+struct vring_packed_desc { +/* Buffer Address. */ +__virtio64 addr; +/* Buffer Length. */ +__virtio32 len; +/* Buffer ID. */ +__virtio16 id; +/* The flags depending on descriptor type. */ +__virtio16 flags; +};
This seems strange. I don't think this needs __virtio variants - they are unconditionally LE.
include/standard-headers/linux/virtio_ring.h static inline int vring_need_event(uint16_t event_idx, uint16_t new_idx, uint16_ · line 214
Quoted 11 lines
include/standard-headers/linux/virtio_ring.h +11
-0+ +struct vring_packed { +unsigned int num; + +struct vring_packed_desc *desc; + +struct vring_packed_desc_event *driver; + +struct vring_packed_desc_event *device; +}; +
I don't know why we have this in the UAPI. I think it's a good idea to drop it, in Linux as well.
Quoted 3 lines
include/standard-headers/linux/virtio_ring.h +0
-0#endif /* _LINUX_VIRTIO_RING_H */ Signature · 1 line
1.8.3.1
PATCH v2 2/15 virtio: redefine structure & memory cache for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
Redefine packed ring structure according to Qemu nomenclature, field data(wrap counter, etc) are introduced also.
--- hw/virtio/virtio.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
hw/virtio/virtio.c +16 -0
@@ -39,6 +39,13 @@ typedef struct VRingDescuint16_t next; } VRingDesc; +typedef struct VRingPackedDesc { +uint64_t addr; +uint32_t len; +uint16_t id; +uint16_t flags; +}VRingPackedDesc ; + typedef struct VRingAvail { uint16_t flags;
@@ -77,6 +84,11 @@ typedef struct VRingVRingMemoryRegionCaches *caches; } VRing; +typedef struct VRingPackedDescEvent { +uint16_t off_wrap; +uint16_t flags; +}VRingPackedDescEvent ; +struct VirtQueue {VRing vring;
@@ -87,7 +99,11 @@ struct VirtQueue/* Last avail_idx read from VQ. */ uint16_t shadow_avail_idx; +bool event_wrap_counter; +bool avail_wrap_counter; +uint16_t used_idx; +bool used_wrap_counter;/* Last used index value we have signalled on */ uint16_t signalled_used;
Signature · 1 line
1.8.3.1
PATCH v2 3/15 virtio: expand offset calculation for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
Expand 1.0 to 1.1 by adding offset calculation accordingly.
--- hw/virtio/virtio.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)
hw/virtio/virtio.c +18 -4
@@ -2336,14 +2336,28 @@ hwaddr virtio_queue_get_desc_size(VirtIODevice *vdev, int n)hwaddr virtio_queue_get_avail_size (VirtIODevice *vdev,int n) { -return offsetof(VRingAvail , ring) + -sizeof (uint16_t ) * vdev->vq[n].vring.num; +int s; + +if (virtio_vdev_has_feature (vdev,VIRTIO_F_RING_PACKED )) { +return sizeof (struct VRingPackedDescEvent ); + }else { + s =virtio_vdev_has_feature (vdev,VIRTIO_RING_F_EVENT_IDX ) ?2 :0 ; +return offsetof(VRingAvail , ring) + +sizeof (uint16_t ) * vdev->vq[n].vring.num + s; + } }hwaddr virtio_queue_get_used_size (VirtIODevice *vdev,int n) { -return offsetof(VRingUsed , ring) + -sizeof (VRingUsedElem) * vdev->vq[n].vring.num; +int s; + +if (virtio_vdev_has_feature (vdev,VIRTIO_F_RING_PACKED )) { +return sizeof (struct VRingPackedDescEvent ); + }else { + s =virtio_vdev_has_feature (vdev,VIRTIO_RING_F_EVENT_IDX ) ?2 :0 ; +return offsetof(VRingUsed , ring) + +sizeof (VRingUsedElem) * vdev->vq[n].vring.num + s; + } }uint16_t virtio_queue_get_last_avail_idx (VirtIODevice *vdev,int n)
Signature · 1 line
1.8.3.1
PATCH v2 4/15 virtio: add memory region init for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
--- hw/virtio/virtio.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
hw/virtio/virtio.c +6 -6
@@ -156,10 +156,8 @@ static void virtio_init_region_cache(VirtIODevice *vdev, int n)VRingMemoryRegionCaches *old = vq->vring.caches;VRingMemoryRegionCaches *new =NULL ;hwaddr addr, size; -int event_size;int64_t len; - - event_size =virtio_vdev_has_feature (vq->vdev,VIRTIO_RING_F_EVENT_IDX ) ?2 :0 ; +bool attr; addr = vq->vring.desc;if (!addr) {
@@ -167,14 +165,16 @@ static void virtio_init_region_cache(VirtIODevice *vdev, int n) } new =g_new0 (VRingMemoryRegionCaches,1 ); size =virtio_queue_get_desc_size (vdev, n); + attr =virtio_vdev_has_feature (vq->vdev,VIRTIO_F_RING_PACKED ) ? + true : false; len =address_space_cache_init (&new->desc, vdev->dma_as, - addr, size, false); + addr, size, attr);if (len < size) {virtio_error (vdev,"Cannot map desc" ); goto err_desc; } - size =virtio_queue_get_used_size (vdev, n) + event_size; + size =virtio_queue_get_used_size (vdev, n); len =address_space_cache_init (&new->used, vdev->dma_as, vq->vring.used, size, true);if (len < size) {
@@ -182,7 +182,7 @@ static void virtio_init_region_cache(VirtIODevice *vdev, int n) goto err_used; } - size =virtio_queue_get_avail_size (vdev, n) + event_size; + size =virtio_queue_get_avail_size (vdev, n); len =address_space_cache_init (&new->avail, vdev->dma_as, vq->vring.avail, size, false);if (len < size) {
Signature · 1 line
1.8.3.1
PATCH v2 5/15 virtio: init wrap counter for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
--- hw/virtio/virtio.c | 3 +++ 1 file changed, 3 insertions(+)
hw/virtio/virtio.c +3 -0
@@ -1239,6 +1239,9 @@ void virtio_reset(void *opaque) vdev->vq[i].last_avail_idx =0 ; vdev->vq[i].shadow_avail_idx =0 ; vdev->vq[i].used_idx =0 ; + vdev->vq[i].avail_wrap_counter = true; + vdev->vq[i].event_wrap_counter = true; + vdev->vq[i].used_wrap_counter = true;virtio_queue_set_vector (vdev, i,VIRTIO_NO_VECTOR ); vdev->vq[i].signalled_used =0 ; vdev->vq[i].signalled_used_valid = false;
Signature · 1 line
1.8.3.1
PATCH v2 6/15 virtio: init and desc empty check for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
ring check and other basical helpers for packed ring.
--- hw/virtio/virtio.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-)
hw/virtio/virtio.c +58 -1
@@ -24,6 +24,9 @@#include "hw/virtio/virtio-access.h" #include "sysemu/dma.h" +#define AVAIL_DESC_PACKED (b) ((b) << 7) +#define USED_DESC_PACKED (b) ((b) << 15) +/* * The alignment to use between consumer and producer parts of vring. * x86 pagesize again. This is the default, used by transports like PCI
@@ -369,6 +372,25 @@ int virtio_queue_ready(VirtQueue *vq)return vq->vring.avail !=0 ; } +static void vring_packed_desc_read_flags (VirtIODevice *vdev, +VRingPackedDesc *desc,MemoryRegionCache *cache,int i) +{ +address_space_read_cached (cache, + i *sizeof (VRingPackedDesc) + offsetof(VRingPackedDesc , flags), + &desc->flags,sizeof (desc->flags)); +virtio_tswap16s (vdev, &desc->flags); +} + +static inline bool is_desc_avail (struct VRingPackedDesc *desc, +bool wrap_counter) +{ +bool avail, used; + + avail = !!(desc->flags &AVAIL_DESC_PACKED (1 )); + used = !!(desc->flags &USED_DESC_PACKED (1 )); +return (avail != used) && (avail == wrap_counter); +} +/* Fetch avail_idx from VQ memory only when we really need to know if * guest has added some buffers. * Called within rcu_read_lock(). */
@@ -389,7 +411,7 @@ static int virtio_queue_empty_rcu(VirtQueue *vq)return vring_avail_idx (vq) == vq->last_avail_idx; } -int virtio_queue_empty (VirtQueue *vq) +static int virtio_queue_split_empty (VirtQueue *vq) {bool empty;
@@ -411,6 +433,41 @@ int virtio_queue_empty(VirtQueue *vq)return empty; } +static int virtio_queue_packed_empty_rcu (VirtQueue *vq) +{ +struct VRingPackedDesc desc; +VRingMemoryRegionCaches *cache; + +if (unlikely (!vq->vring.desc)) { +return 1 ; + } + + cache =vring_get_region_caches (vq); +vring_packed_desc_read_flags (vq->vdev, &desc, &cache->desc, + vq->last_avail_idx); + +return !is_desc_avail (&desc, vq->avail_wrap_counter); +} + +static int virtio_queue_packed_empty (VirtQueue *vq) +{ +bool empty; + +rcu_read_lock (); + empty =virtio_queue_packed_empty_rcu (vq); +rcu_read_unlock (); +return empty; +} + +int virtio_queue_empty (VirtQueue *vq) +{ +if (virtio_vdev_has_feature (vq->vdev,VIRTIO_F_RING_PACKED )) { +return virtio_queue_packed_empty (vq); + }else { +return virtio_queue_split_empty (vq); + } +} +static void virtqueue_unmap_sg (VirtQueue *vq,const VirtQueueElement *elem,unsigned int len) {
Signature · 1 line
1.8.3.1
PATCH v2 7/15 virtio: get avail bytes check for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
--- hw/virtio/virtio.c | 180 +++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 167 insertions(+), 13 deletions(-)
hw/virtio/virtio.c +167 -13
@@ -372,6 +372,17 @@ int virtio_queue_ready(VirtQueue *vq)return vq->vring.avail !=0 ; } +static void vring_packed_desc_read (VirtIODevice *vdev,VRingPackedDesc *desc, +MemoryRegionCache *cache,int i) +{ +address_space_read_cached (cache, i *sizeof (VRingPackedDesc), + desc,sizeof (VRingPackedDesc)); +virtio_tswap16s (vdev, &desc->flags); +virtio_tswap64s (vdev, &desc->addr); +virtio_tswap32s (vdev, &desc->len); +virtio_tswap16s (vdev, &desc->id); +} +static void vring_packed_desc_read_flags (VirtIODevice *vdev,VRingPackedDesc *desc,MemoryRegionCache *cache,int i) {
@@ -671,9 +682,9 @@ static int virtqueue_read_next_desc(VirtIODevice *vdev, VRingDesc *desc,return VIRTQUEUE_READ_DESC_MORE ; } -void virtqueue_get_avail_bytes (VirtQueue *vq,unsigned int *in_bytes, -unsigned int *out_bytes, -unsigned max_in_bytes,unsigned max_out_bytes) +static void virtqueue_split_get_avail_bytes (VirtQueue *vq, +unsigned int *in_bytes,unsigned int *out_bytes, +unsigned max_in_bytes,unsigned max_out_bytes) {VirtIODevice *vdev = vq->vdev;unsigned int max, idx;
@@ -683,16 +694,6 @@ void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,int64_t len =0 ;int rc; -if (unlikely (!vq->vring.desc)) { -if (in_bytes) { - *in_bytes =0 ; - } -if (out_bytes) { - *out_bytes =0 ; - } -return ; - } -rcu_read_lock (); idx = vq->last_avail_idx; total_bufs = in_total = out_total =0 ;
@@ -796,6 +797,159 @@ err: goto done; } +static void virtqueue_packed_get_avail_bytes (VirtQueue *vq, +unsigned int *in_bytes,unsigned int *out_bytes, +unsigned max_in_bytes,unsigned max_out_bytes) +{ +VirtIODevice *vdev = vq->vdev; +unsigned int max, idx; +unsigned int total_bufs, in_total, out_total; +MemoryRegionCache *desc_cache; +VRingMemoryRegionCaches *caches; +MemoryRegionCache indirect_desc_cache =MEMORY_REGION_CACHE_INVALID ; +int64_t len =0 ; +VRingPackedDesc desc; +bool wrap_counter; + +rcu_read_lock (); + idx = vq->last_avail_idx; + wrap_counter = vq->avail_wrap_counter; + total_bufs = in_total = out_total =0 ; + + max = vq->vring.num; + caches =vring_get_region_caches (vq); +if (caches->desc.len < max *sizeof (VRingPackedDesc)) { +virtio_error (vdev,"Cannot map descriptor ring" ); + goto err; + } + + desc_cache = &caches->desc; +vring_packed_desc_read_flags (vdev, &desc, desc_cache, idx); +while (is_desc_avail (&desc, wrap_counter)) { +unsigned int num_bufs; +unsigned int i =0 ; + + num_bufs = total_bufs; + +/* Make sure all the fields have been exposed. */ +smp_rmb (); +vring_packed_desc_read (vdev, &desc, desc_cache, idx); + +if (desc.flags &VRING_DESC_F_INDIRECT ) { +if (desc.len %sizeof (VRingPackedDesc)) { +virtio_error (vdev,"Invalid size for indirect buffer table" ); + goto err; + } + +/* If we've got too many, that implies a descriptor loop. */ +if (num_bufs >= max) { +virtio_error (vdev,"Looped descriptor" ); + goto err; + } + +/* loop over the indirect descriptor table */ + len =address_space_cache_init (&indirect_desc_cache, + vdev->dma_as, + desc.addr, desc.len, false); + desc_cache = &indirect_desc_cache; +if (len < desc.len) { +virtio_error (vdev,"Cannot map indirect buffer" ); + goto err; + } + + max = desc.len /sizeof (VRingPackedDesc); + num_bufs = i =0 ; +vring_packed_desc_read (vdev, &desc, desc_cache, i); + } + +do { +/* If we've got too many, that implies a descriptor loop. */ +if (++num_bufs > max) { +virtio_error (vdev,"Looped descriptor" ); + goto err; + } + +if (desc.flags &VRING_DESC_F_WRITE ) { + in_total += desc.len; + }else { + out_total += desc.len; + } +if (in_total >= max_in_bytes && out_total >= max_out_bytes) { + goto done; + } + +if (desc_cache == &indirect_desc_cache) { +vring_packed_desc_read (vdev, &desc, desc_cache, i); + }else { +if (++idx >= vq->vring.num) { + idx -= vq->vring.num; + wrap_counter = !wrap_counter; + } +vring_packed_desc_read (vdev, &desc, desc_cache, idx); + } +/* Make sure we see the flags */ +smp_rmb (); + }while (desc.flags &VRING_DESC_F_NEXT ); + +if (desc_cache == &indirect_desc_cache) { +address_space_cache_destroy (&indirect_desc_cache); + total_bufs++; +/* We missed one step on for indirect desc */ + idx++; +if (++idx >= vq->vring.num) { + idx -= vq->vring.num; + wrap_counter = !wrap_counter; + } + }else { + total_bufs = num_bufs; + } + + desc_cache = &caches->desc; +vring_packed_desc_read_flags (vdev, &desc, desc_cache, idx); + } + +/* Record the index and wrap counter for a kick we want */ + vq->shadow_avail_idx = idx; + vq->event_wrap_counter = wrap_counter; +done: +address_space_cache_destroy (&indirect_desc_cache); +if (in_bytes) { + *in_bytes = in_total; + } +if (out_bytes) { + *out_bytes = out_total; + } +rcu_read_unlock (); +return ; + +err: + in_total = out_total =0 ; + goto done; +} + +void virtqueue_get_avail_bytes (VirtQueue *vq,unsigned int *in_bytes, +unsigned int *out_bytes, +unsigned max_in_bytes,unsigned max_out_bytes) +{ +if (unlikely (!vq->vring.desc)) { +if (in_bytes) { + *in_bytes =0 ; + } +if (out_bytes) { + *out_bytes =0 ; + } +return ; + } + +if (virtio_vdev_has_feature (vq->vdev,VIRTIO_F_RING_PACKED )) { +virtqueue_packed_get_avail_bytes (vq, in_bytes, out_bytes, + max_in_bytes, max_out_bytes); + }else { +virtqueue_split_get_avail_bytes (vq, in_bytes, out_bytes, + max_in_bytes, max_out_bytes); + } +} +int virtqueue_avail_bytes (VirtQueue *vq,unsigned int in_bytes,unsigned int out_bytes) {
Signature · 1 line
1.8.3.1
PATCH v2 8/15 virtio: fill/flush/pop for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
--- hw/virtio/virtio.c | 295 ++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 278 insertions(+), 17 deletions(-)
hw/virtio/virtio.c +278 -17
@@ -383,6 +383,25 @@ static void vring_packed_desc_read(VirtIODevice *vdev, VRingPackedDesc *desc,virtio_tswap16s (vdev, &desc->id); } +static void vring_packed_desc_write_data (VirtIODevice *vdev, +VRingPackedDesc *desc,MemoryRegionCache *cache,int i) +{ +virtio_tswap32s (vdev, &desc->len); +virtio_tswap16s (vdev, &desc->id); +address_space_write_cached (cache, + i *sizeof (VRingPackedDesc) + offsetof(VRingPackedDesc , id), + &desc->id,sizeof (desc->id)); +address_space_cache_invalidate (cache, + i *sizeof (VRingPackedDesc) + offsetof(VRingPackedDesc , id), +sizeof (desc->id)); +address_space_write_cached (cache, + i *sizeof (VRingPackedDesc) + offsetof(VRingPackedDesc , len), + &desc->len,sizeof (desc->len)); +address_space_cache_invalidate (cache, + i *sizeof (VRingPackedDesc) + offsetof(VRingPackedDesc , len), +sizeof (desc->len)); +} +static void vring_packed_desc_read_flags (VirtIODevice *vdev,VRingPackedDesc *desc,MemoryRegionCache *cache,int i) {
@@ -392,6 +411,18 @@ static void vring_packed_desc_read_flags(VirtIODevice *vdev,virtio_tswap16s (vdev, &desc->flags); } +static void vring_packed_desc_write_flags (VirtIODevice *vdev, +VRingPackedDesc *desc,MemoryRegionCache *cache,int i) +{ +virtio_tswap16s (vdev, &desc->flags); +address_space_write_cached (cache, + i *sizeof (VRingPackedDesc) + offsetof(VRingPackedDesc , flags), + &desc->flags,sizeof (desc->flags)); +address_space_cache_invalidate (cache, + i *sizeof (VRingPackedDesc) + offsetof(VRingPackedDesc , flags), +sizeof (desc->flags)); +} +static inline bool is_desc_avail (struct VRingPackedDesc *desc,bool wrap_counter) {
@@ -558,19 +589,11 @@ bool virtqueue_rewind(VirtQueue *vq, unsigned int num) }/* Called within rcu_read_lock(). */ -void virtqueue_fill (VirtQueue *vq,const VirtQueueElement *elem, +static void virtqueue_split_fill (VirtQueue *vq,const VirtQueueElement *elem,unsigned int len,unsigned int idx) {VRingUsedElem uelem; -trace_virtqueue_fill (vq, elem, len, idx); - -virtqueue_unmap_sg (vq, elem, len); - -if (unlikely (vq->vdev->broken)) { -return ; - } -if (unlikely (!vq->vring.used)) {return ; }
@@ -582,16 +605,72 @@ void virtqueue_fill(VirtQueue *vq, const VirtQueueElement *elem,vring_used_write (vq, &uelem, idx); } -/* Called within rcu_read_lock(). */ -void virtqueue_flush (VirtQueue *vq,unsigned int count) +static void virtqueue_packed_fill (VirtQueue *vq,const VirtQueueElement *elem, +unsigned int len,unsigned int idx) { -uint16_t old, new; +uint16_t head; +VRingMemoryRegionCaches *caches; +VRingPackedDesc desc = { + .flags =0 , + }; +bool wrap_counter = vq->used_wrap_counter; + +if (unlikely (!vq->vring.desc)) { +return ; + } + + caches =vring_get_region_caches (vq); + desc.id = elem->index; + desc.len = len; + + head = vq->used_idx + idx; +if (head >= vq->vring.num) { + head -= vq->vring.num; + wrap_counter ^=1 ; + } +if (wrap_counter) { + desc.flags |=VRING_DESC_F_AVAIL ; + desc.flags |=VRING_DESC_F_USED ; + }else { + desc.flags &= ~VRING_DESC_F_AVAIL ; + desc.flags &= ~VRING_DESC_F_USED ; + } + +vring_packed_desc_write_data (vq->vdev, &desc, &caches->desc, head); +if (idx ==0 ) { +/* +* Make sure descriptor id and len is written before +* flags for the first used buffer. +*/ +smp_wmb (); + } + +vring_packed_desc_write_flags (vq->vdev, &desc, &caches->desc, head); +} + +void virtqueue_fill (VirtQueue *vq,const VirtQueueElement *elem, +unsigned int len,unsigned int idx) +{ +trace_virtqueue_fill (vq, elem, len, idx); + +virtqueue_unmap_sg (vq, elem, len);if (unlikely (vq->vdev->broken)) { - vq->inuse -= count;return ; } +if (virtio_vdev_has_feature (vq->vdev,VIRTIO_F_RING_PACKED )) { +virtqueue_packed_fill (vq, elem, len, idx); + }else { +virtqueue_split_fill (vq, elem, len, idx); + } +} + +/* Called within rcu_read_lock(). */ +static void virtqueue_split_flush (VirtQueue *vq,unsigned int count) +{ +uint16_t old, new; +if (unlikely (!vq->vring.used)) {return ; }
@@ -607,6 +686,31 @@ void virtqueue_flush(VirtQueue *vq, unsigned int count) vq->signalled_used_valid = false; } +static void virtqueue_packed_flush (VirtQueue *vq,unsigned int count) +{ +if (unlikely (!vq->vring.desc)) { +return ; + } + + vq->inuse -= count; + vq->used_idx = vq->last_avail_idx; + vq->used_wrap_counter = vq->avail_wrap_counter; +} + +void virtqueue_flush (VirtQueue *vq,unsigned int count) +{ +if (unlikely (vq->vdev->broken)) { + vq->inuse -= count; +return ; + } + +if (virtio_vdev_has_feature (vq->vdev,VIRTIO_F_RING_PACKED )) { +virtqueue_packed_flush (vq, count); + }else { +virtqueue_split_flush (vq, count); + } +} +void virtqueue_push (VirtQueue *vq,const VirtQueueElement *elem,unsigned int len) {
@@ -1074,7 +1178,7 @@ static void *virtqueue_alloc_element(size_t sz, unsigned out_num, unsigned in_nureturn elem; } -void *virtqueue_pop (VirtQueue *vq,size_t sz) +static void *virtqueue_split_pop (VirtQueue *vq,size_t sz) {unsigned int i, head, max;VRingMemoryRegionCaches *caches;
@@ -1089,9 +1193,6 @@ void *virtqueue_pop(VirtQueue *vq, size_t sz)VRingDesc desc;int rc; -if (unlikely (vdev->broken)) { -return NULL ; - }rcu_read_lock ();if (virtio_queue_empty_rcu (vq)) { goto done;
@@ -1209,6 +1310,166 @@ err_undo_map: goto done; } +static void *virtqueue_packed_pop (VirtQueue *vq,size_t sz) +{ +unsigned int i, head, max; +VRingMemoryRegionCaches *caches; +MemoryRegionCache indirect_desc_cache =MEMORY_REGION_CACHE_INVALID ; +MemoryRegionCache *cache; +int64_t len; +VirtIODevice *vdev = vq->vdev; +VirtQueueElement *elem =NULL ; +unsigned out_num, in_num, elem_entries; +hwaddr addr[VIRTQUEUE_MAX_SIZE ]; +struct iovec iov[VIRTQUEUE_MAX_SIZE ]; +VRingPackedDesc desc; +uint16_t id; + +rcu_read_lock (); +if (virtio_queue_packed_empty_rcu (vq)) { + goto done; + } + +/* When we start there are none of either input nor output. */ + out_num = in_num = elem_entries =0 ; + + max = vq->vring.num; + +if (vq->inuse >= vq->vring.num) { +virtio_error (vdev,"Virtqueue size exceeded" ); + goto done; + } + + head = vq->last_avail_idx; + i = head; + + caches =vring_get_region_caches (vq); + cache = &caches->desc; + +/* +* Empty check has been done at the beginning, so it is an available +* entry already, make sure all fields has been exposed by guest +*/ +smp_rmb (); +vring_packed_desc_read (vdev, &desc, cache, i); + + id = desc.id; +if (desc.flags &VRING_DESC_F_INDIRECT ) { + +if (desc.len %sizeof (VRingPackedDesc)) { +virtio_error (vdev,"Invalid size for indirect buffer table" ); + goto done; + } + +/* loop over the indirect descriptor table */ + len =address_space_cache_init (&indirect_desc_cache, vdev->dma_as, + desc.addr, desc.len, false); + cache = &indirect_desc_cache; +if (len < desc.len) { +virtio_error (vdev,"Cannot map indirect buffer" ); + goto done; + } + + max = desc.len /sizeof (VRingPackedDesc); + i =0 ; +vring_packed_desc_read (vdev, &desc, cache, i); +/* Make sure we see all the fields*/ +smp_rmb (); + } + +/* Collect all the descriptors */ +while (1 ) { +bool map_ok; + +if (desc.flags &VRING_DESC_F_WRITE ) { + map_ok =virtqueue_map_desc (vdev, &in_num, addr + out_num, + iov + out_num, +VIRTQUEUE_MAX_SIZE - out_num, true, + desc.addr, desc.len); + }else { +if (in_num) { +virtio_error (vdev,"Incorrect order for descriptors" ); + goto err_undo_map; + } + map_ok =virtqueue_map_desc (vdev, &out_num, addr, iov, +VIRTQUEUE_MAX_SIZE , false, + desc.addr, desc.len); + } +if (!map_ok) { + goto err_undo_map; + } + +/* If we've got too many, that implies a descriptor loop. */ +if (++elem_entries > max) { +virtio_error (vdev,"Looped descriptor" ); + goto err_undo_map; + } + +if (++i >= vq->vring.num) { + i -= vq->vring.num; + } + +if (cache == &indirect_desc_cache) { +if (i == max) { +break ; + } +vring_packed_desc_read (vq->vdev, &desc, cache, i); + }else if (desc.flags &VRING_DESC_F_NEXT ) { +vring_packed_desc_read (vq->vdev, &desc, cache, i); + }else { +break ; + } + } + +/* Now copy what we have collected and mapped */ + elem =virtqueue_alloc_element (sz, out_num, in_num); + elem->index = id; +for (i =0 ; i < out_num; i++) { + elem->out_addr[i] = addr[i]; + elem->out_sg[i] = iov[i]; + } +for (i =0 ; i < in_num; i++) { + elem->in_addr[i] = addr[head + out_num + i]; + elem->in_sg[i] = iov[out_num + i]; + } + + vq->last_avail_idx += (cache == &indirect_desc_cache) ? +1 : elem_entries; +if (vq->last_avail_idx >= vq->vring.num) { + vq->last_avail_idx -= vq->vring.num; + vq->avail_wrap_counter = !vq->avail_wrap_counter; + } + vq->inuse++; + + vq->shadow_avail_idx = vq->last_avail_idx; + vq->event_wrap_counter = vq->avail_wrap_counter; + +trace_virtqueue_pop (vq, elem, elem->in_num, elem->out_num); +done: +address_space_cache_destroy (&indirect_desc_cache); +rcu_read_unlock (); + +return elem; + +err_undo_map: +virtqueue_undo_map_desc (out_num, in_num, iov); +g_free (elem); + goto done; +} + +void *virtqueue_pop (VirtQueue *vq,size_t sz) +{ +if (unlikely (vq->vdev->broken)) { +return NULL ; + } + +if (virtio_vdev_has_feature (vq->vdev,VIRTIO_F_RING_PACKED )) { +return virtqueue_packed_pop (vq, sz); + }else { +return virtqueue_split_pop (vq, sz); + } +} +/* virtqueue_drop_all: * @vq: The #VirtQueue * Drops all queued buffers and indicates them to the guest
Signature · 1 line
1.8.3.1
PATCH v2 9/15 virtio: event suppression support for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
--- hw/virtio/virtio.c | 121 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 118 insertions(+), 3 deletions(-)
hw/virtio/virtio.c +118 -3
@@ -238,6 +238,30 @@ static void vring_desc_read(VirtIODevice *vdev, VRingDesc *desc,virtio_tswap16s (vdev, &desc->next); } +static void vring_packed_event_read (VirtIODevice *vdev, +MemoryRegionCache *cache,VRingPackedDescEvent *e) +{ +address_space_read_cached (cache,0 , e,sizeof (*e)); +virtio_tswap16s (vdev, &e->off_wrap); +virtio_tswap16s (vdev, &e->flags); +} + +static void vring_packed_off_wrap_write (VirtIODevice *vdev, +MemoryRegionCache *cache,uint16_t off_wrap) +{ +virtio_tswap16s (vdev, &off_wrap); +address_space_write_cached (cache,0 , &off_wrap,sizeof (off_wrap)); +address_space_cache_invalidate (cache,0 ,sizeof (off_wrap)); +} + +static void vring_packed_flags_write (VirtIODevice *vdev, +MemoryRegionCache *cache,uint16_t flags) +{ +virtio_tswap16s (vdev, &flags); +address_space_write_cached (cache,sizeof (uint16_t ), &flags,sizeof (flags)); +address_space_cache_invalidate (cache,sizeof (uint16_t ),sizeof (flags)); +} +static VRingMemoryRegionCaches *vring_get_region_caches (struct VirtQueue *vq) {VRingMemoryRegionCaches *caches =atomic_rcu_read (&vq->vring.caches);
@@ -344,7 +368,7 @@ static inline void vring_set_avail_event(VirtQueue *vq, uint16_t val)address_space_cache_invalidate (&caches->used, pa,sizeof (val)); } -void virtio_queue_set_notification (VirtQueue *vq,int enable) +static void virtio_queue_set_notification_split (VirtQueue *vq,int enable) { vq->notification = enable;
@@ -367,6 +391,51 @@ void virtio_queue_set_notification(VirtQueue *vq, int enable)rcu_read_unlock (); } +static void virtio_queue_set_notification_packed (VirtQueue *vq,int enable) +{ +VRingPackedDescEvent e; +VRingMemoryRegionCaches *caches; + +rcu_read_lock (); + caches =vring_get_region_caches (vq); +vring_packed_event_read (vq->vdev, &caches->used, &e); + +if (!enable) { + e.flags =RING_EVENT_FLAGS_DISABLE ; + goto out; + } + + e.flags =RING_EVENT_FLAGS_ENABLE ; +if (virtio_vdev_has_feature (vq->vdev,VIRTIO_RING_F_EVENT_IDX )) { +uint16_t off_wrap = vq->shadow_avail_idx | vq->event_wrap_counter <<15 ; + +vring_packed_off_wrap_write (vq->vdev, &caches->used, off_wrap); +/* Make sure off_wrap is wrote before flags */ +smp_wmb (); + + e.flags =RING_EVENT_FLAGS_DESC ; + } + +out: +vring_packed_flags_write (vq->vdev, &caches->used, e.flags); +rcu_read_unlock (); +} + +void virtio_queue_set_notification (VirtQueue *vq,int enable) +{ + vq->notification = enable; + +if (!vq->vring.desc) { +return ; + } + +if (virtio_vdev_has_feature (vq->vdev,VIRTIO_F_RING_PACKED )) { +virtio_queue_set_notification_packed (vq, enable); + }else { +virtio_queue_set_notification_split (vq, enable); + } +} +int virtio_queue_ready (VirtQueue *vq) {return vq->vring.avail !=0 ;
@@ -2118,8 +2187,7 @@ static void virtio_set_isr(VirtIODevice *vdev, int value) } } -/* Called within rcu_read_lock(). */ -static bool virtio_should_notify (VirtIODevice *vdev,VirtQueue *vq) +static bool virtio_split_should_notify (VirtIODevice *vdev,VirtQueue *vq) {uint16_t old, new;bool v;
@@ -2142,6 +2210,53 @@ static bool virtio_should_notify(VirtIODevice *vdev, VirtQueue *vq)return !v ||vring_need_event (vring_get_used_event (vq), new, old); } +static bool vring_packed_need_event (VirtQueue *vq,bool wrap, +uint16_t off_wrap,uint16_t new,uint16_t old) +{ +int off = off_wrap & ~(1 <<15 ); + +if (wrap != off_wrap >>15 ) { + off -= vq->vring.num; + } + +return vring_need_event (off, new, old); +} + +static bool virtio_packed_should_notify (VirtIODevice *vdev,VirtQueue *vq) +{ +VRingPackedDescEvent e; +uint16_t old, new; +bool v; +VRingMemoryRegionCaches *caches; + + caches =vring_get_region_caches (vq); +vring_packed_event_read (vdev, &caches->avail, &e); + + old = vq->signalled_used; + new = vq->signalled_used = vq->used_idx; + v = vq->signalled_used_valid; + vq->signalled_used_valid = true; + +if (e.flags ==RING_EVENT_FLAGS_DISABLE ) { +return false; + }else if (e.flags ==RING_EVENT_FLAGS_ENABLE ) { +return true; + } + +return !v ||vring_packed_need_event (vq, + vq->used_wrap_counter, e.off_wrap, new, old); +} + +/* Called within rcu_read_lock(). */ +static bool virtio_should_notify (VirtIODevice *vdev,VirtQueue *vq) +{ +if (virtio_vdev_has_feature (vdev,VIRTIO_F_RING_PACKED )) { +return virtio_packed_should_notify (vdev, vq); + }else { +return virtio_split_should_notify (vdev, vq); + } +} +void virtio_notify_irqfd (VirtIODevice *vdev,VirtQueue *vq) {bool should_notify;
Signature · 1 line
1.8.3.1
PATCH v2 10/15 virtio-net: fill head desc after done all in a chain
In reply to [email protected]From: Wei Xu <[email protected]>
With the support of marking a descriptor used/unused in 'flags' field for 1.1, the current way of filling a chained descriptors does not work since driver side may get the wrong 'num_buffer' information in case of the head descriptor has been filled in while the subsequent ones are still in processing in device side.
This patch fills the head one after done all the others one.
--- hw/net/virtio-net.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +10 -1
@@ -1198,6 +1198,8 @@ static ssize_t virtio_net_receive_rcu(NetClientState *nc, const uint8_t *buf,struct virtio_net_hdr_mrg_rxbuf mhdr;unsigned mhdr_cnt =0 ;size_t offset, i, guest_offset; +VirtQueueElement head; +int head_len =0 ;if (!virtio_net_can_receive (nc)) {return -1 ;
@@ -1275,7 +1277,13 @@ static ssize_t virtio_net_receive_rcu(NetClientState *nc, const uint8_t *buf, }/* signal other side */ -virtqueue_fill (q->rx_vq, elem, total, i++); +if (i ==0 ) { + head_len = total; + head = *elem; + }else { +virtqueue_fill (q->rx_vq, elem, len, i); + } + i++;g_free (elem); }
@@ -1286,6 +1294,7 @@ static ssize_t virtio_net_receive_rcu(NetClientState *nc, const uint8_t *buf, &mhdr.num_buffers, sizeof mhdr.num_buffers); } +virtqueue_fill (q->rx_vq, &head, head_len,0 );virtqueue_flush (q->rx_vq, i);virtio_notify (vdev, q->rx_vq);
Signature · 1 line
1.8.3.1
PATCH v2 11/15 virtio: add userspace migration for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
--- hw/virtio/virtio.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
hw/virtio/virtio.c +39 -0
@@ -2346,6 +2346,13 @@ static bool virtio_virtqueue_needed(void *opaque)return virtio_host_has_feature (vdev,VIRTIO_F_VERSION_1 ); } +static bool virtio_packed_virtqueue_needed (void *opaque) +{ +VirtIODevice *vdev = opaque; + +return virtio_host_has_feature (vdev,VIRTIO_F_RING_PACKED ); +} +static bool virtio_ringsize_needed (void *opaque) {VirtIODevice *vdev = opaque;
@@ -2387,6 +2394,21 @@ static const VMStateDescription vmstate_virtqueue = { } }; +static const VMStateDescription vmstate_packed_virtqueue = { + .name ="packed_virtqueue_state" , + .version_id =1 , + .minimum_version_id =1 , + .fields = (VMStateField []) { + VMSTATE_BOOL(avail_wrap_counter, struct VirtQueue), + VMSTATE_BOOL(event_wrap_counter, struct VirtQueue), + VMSTATE_BOOL(used_wrap_counter, struct VirtQueue), + VMSTATE_UINT16(used_idx, struct VirtQueue), + VMSTATE_UINT16(shadow_avail_idx, struct VirtQueue), + VMSTATE_UINT32(inuse, struct VirtQueue), +VMSTATE_END_OF_LIST () + } +}; +static const VMStateDescription vmstate_virtio_virtqueues = { .name ="virtio/virtqueues" , .version_id =1 ,
@@ -2399,6 +2421,18 @@ static const VMStateDescription vmstate_virtio_virtqueues = { } }; +static const VMStateDescription vmstate_virtio_packed_virtqueues = { + .name ="virtio/packed_virtqueues" , + .version_id =1 , + .minimum_version_id =1 , + .needed = &virtio_packed_virtqueue_needed, + .fields = (VMStateField []) { + VMSTATE_STRUCT_VARRAY_POINTER_KNOWN(vq, struct VirtIODevice, +VIRTIO_QUEUE_MAX ,0 , vmstate_packed_virtqueue, VirtQueue), +VMSTATE_END_OF_LIST () + } +}; +static const VMStateDescription vmstate_ringsize = { .name ="ringsize_state" , .version_id =1 ,
@@ -2516,6 +2550,7 @@ static const VMStateDescription vmstate_virtio = { &vmstate_virtio_device_endian, &vmstate_virtio_64bit_features, &vmstate_virtio_virtqueues, + &vmstate_virtio_packed_virtqueues, &vmstate_virtio_ringsize, &vmstate_virtio_broken, &vmstate_virtio_extra_state,
@@ -2791,6 +2826,10 @@ int virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)virtio_queue_update_rings (vdev, i); } +if (virtio_vdev_has_feature (vdev,VIRTIO_F_RING_PACKED )) { +continue ; + } + nheads =vring_avail_idx (&vdev->vq[i]) - vdev->vq[i].last_avail_idx;/* Check it isn't doing strange things with descriptor numbers. */ if (nheads > vdev->vq[i].vring.num) {
Signature · 1 line
1.8.3.1
PATCH v2 12/15 virtio: add vhost-net migration for packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
--- hw/virtio/virtio.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-)
hw/virtio/virtio.c +24 -3
@@ -2991,17 +2991,34 @@ hwaddr virtio_queue_get_used_size(VirtIODevice *vdev, int n)uint16_t virtio_queue_get_last_avail_idx (VirtIODevice *vdev,int n) { -return vdev->vq[n].last_avail_idx; +uint16_t idx; + +if (virtio_vdev_has_feature (vdev,VIRTIO_F_RING_PACKED )) { + idx = vdev->vq[n].last_avail_idx; + idx |= ((int )vdev->vq[n].avail_wrap_counter) <<15 ; + }else { + idx = (int )vdev->vq[n].last_avail_idx; + } +return idx; }void virtio_queue_set_last_avail_idx (VirtIODevice *vdev,int n,uint16_t idx) { - vdev->vq[n].last_avail_idx = idx; - vdev->vq[n].shadow_avail_idx = idx; +if (virtio_vdev_has_feature (vdev,VIRTIO_F_RING_PACKED )) { + vdev->vq[n].last_avail_idx = idx &0x7fff ; + vdev->vq[n].avail_wrap_counter = !!(idx &0x8000 ); + }else { + vdev->vq[n].last_avail_idx = idx; + vdev->vq[n].shadow_avail_idx = idx; + } }void virtio_queue_restore_last_avail_idx (VirtIODevice *vdev,int n) { +if (virtio_vdev_has_feature (vdev,VIRTIO_F_RING_PACKED )) { +return ; + } +rcu_read_lock ();if (vdev->vq[n].vring.desc) { vdev->vq[n].last_avail_idx =vring_used_idx (&vdev->vq[n]);
@@ -3012,6 +3029,10 @@ void virtio_queue_restore_last_avail_idx(VirtIODevice *vdev, int n)void virtio_queue_update_used_idx (VirtIODevice *vdev,int n) { +if (virtio_vdev_has_feature (vdev,VIRTIO_F_RING_PACKED )) { +return ; + } +rcu_read_lock ();if (vdev->vq[n].vring.desc) { vdev->vq[n].used_idx =vring_used_idx (&vdev->vq[n]);
Signature · 1 line
1.8.3.1
PATCH v2 13/15 virtio: packed ring feature bit for userspace backend
In reply to [email protected]From: Wei Xu <[email protected]>
--- hw/net/vhost_net.c | 1 + 1 file changed, 1 insertion(+)
hw/net/vhost_net.c +1 -0
@@ -78,6 +78,7 @@ static const int user_feature_bits[] = {VIRTIO_NET_F_MRG_RXBUF ,VIRTIO_NET_F_MTU ,VIRTIO_F_IOMMU_PLATFORM , +VIRTIO_F_RING_PACKED ,/* This bit implies RARP isn't sent by QEMU out of band */ VIRTIO_NET_F_GUEST_ANNOUNCE,
Signature · 1 line
1.8.3.1
PATCH v2 14/15 vhost: enable packed ring
In reply to [email protected]From: Wei Xu <[email protected]>
--- hw/net/vhost_net.c | 1 + 1 file changed, 1 insertion(+)
hw/net/vhost_net.c +1 -0
@@ -53,6 +53,7 @@ static const int kernel_feature_bits[] = {VIRTIO_F_VERSION_1 ,VIRTIO_NET_F_MTU ,VIRTIO_F_IOMMU_PLATFORM , +VIRTIO_F_RING_PACKED , VHOST_INVALID_FEATURE_BIT };
Signature · 1 line
1.8.3.1
PATCH v2 15/15 virtio: enable packed ring via a new command line
In reply to [email protected]From: Wei Xu <[email protected]>
--- include/hw/virtio/virtio.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
include/hw/virtio/virtio.h +3 -1
@@ -264,7 +264,9 @@ typedef struct VirtIORNGConf VirtIORNGConf;DEFINE_PROP_BIT64 ("any_layout" , _state, _field, \VIRTIO_F_ANY_LAYOUT , true), \DEFINE_PROP_BIT64 ("iommu_platform" , _state, _field, \ - VIRTIO_F_IOMMU_PLATFORM, false) +VIRTIO_F_IOMMU_PLATFORM , false), \ +DEFINE_PROP_BIT64 ("ring_packed" , _state, _field, \ + VIRTIO_F_RING_PACKED, false)hwaddr virtio_queue_get_desc_addr (VirtIODevice *vdev,int n);hwaddr virtio_queue_get_avail_addr (VirtIODevice *vdev,int n);
Signature · 1 line
1.8.3.1