Skip to content

RFC Net Control VQ support in SVQ

Eugenio Pérez qemu-devel 9 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v9 RFC Latest23 patches 41 repliesIn review
  2. v8 RFC21 patches 29 repliesSuperseded
  3. v7 RFC25 patches 24 repliesSuperseded
Earlier versions · 6
VersionPostedPatchesRepliesStatus
v6 RFC23 patches0 repliesSuperseded
v5 RFC23 patches8 repliesSuperseded
v4 RFC20 patches0 repliesSuperseded
v3 RFC19 patches0 repliesSuperseded
v2 RFC18 patches0 repliesSuperseded
v1 RFC9 patches2 repliesSuperseded

Changes in v9 from the cover letter

* Remove ASID part. Delete x-svq-cvq mode too.
* Move all DMA memory management to net/vhost-vdpa, instead of svq.
* Use of qemu_real_host_page_size.
* Improved doc, general fixes.

Patches in v9 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/23vhost: Return earlier if used buffers overrunhw/virtio/vhost-shadow-virtqueue.c+8 -61 reply
2/23vhost: move descriptor translation to vhost_svq_vring_write_descshw/virtio/vhost-shadow-virtqueue.c+28 -110 replies
3/23vdpa: delay set_vring_ready after DRIVER_OKhw/virtio/vhost-vdpa.c+16 -65 replies
4/23vhost: Get vring base from vq, not svqhw/virtio/vhost-vdpa.c hw/virtio/virtio.c include/hw/virtio/virtio.h+9 -44 replies
5/23vhost: Add ShadowVirtQueueStart operationhw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h hw/virtio/vhost-vdpa.c+40 -30 replies
6/23virtio-net: Expose ctrl virtqueue logichw/net/virtio-net.c include/hw/virtio/virtio-net.h+53 -350 replies
7/23vhost: add vhost_svq_push_elemhw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h+18 -00 replies
8/23vhost: Decouple vhost_svq_add_split from VirtQueueElementhw/virtio/vhost-shadow-virtqueue.c+27 -113 replies
9/23vhost: Add SVQElementhw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h+33 -163 replies
10/23vhost: Reorder vhost_svq_last_desc_of_chainhw/virtio/vhost-shadow-virtqueue.c+10 -100 replies
11/23vhost: Move last chain id to SVQ elementhw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h+11 -61 reply
12/23vhost: Add opaque member to SVQElementhw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h+9 -75 replies
13/23vhost: Add vhost_svq_injecthw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h+36 -03 replies
14/23vhost: add vhost_svq_pollhw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h+52 -32 replies
15/23vhost: Add custom used buffer callbackhw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h+16 -50 replies
16/23vhost: Add svq avail_handler callbackhw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h+23 -10 replies
17/23vhost: add detach SVQ operationhw/virtio/vhost-shadow-virtqueue.c hw/virtio/vhost-shadow-virtqueue.h+21 -10 replies
18/23vdpa: Export vhost_vdpa_dma_map and unmap callshw/virtio/vhost-vdpa.c include/hw/virtio/vhost-vdpa.h+7 -41 reply
19/23vdpa: Extract get features part from vhost_vdpa_get_max_queue_pairsnet/vhost-vdpa.c+20 -101 reply
20/23vdpa: Buffer CVQ support on shadow virtqueuehw/virtio/vhost-vdpa.c include/hw/virtio/vhost-vdpa.h net/vhost-vdpa.c+379 -25 replies
21/23vdpa: Add vhost_vdpa_start_control_svqnet/vhost-vdpa.c+19 -03 replies
22/23vdpa: Inject virtio-net mac address via CVQ at startnet/vhost-vdpa.c+48 -10 replies
23/23vdpa: Add x-svq to NetdevVhostVDPAOptionsnet/vhost-vdpa.c qapi/net.json+77 -44 replies

Reviews trailers per patch

5 of 23 patches have a Reviewed-by or Acked-by.

PatchJason WangMarkus Armbruster
1/23A-b
2/23A-b
3/23
4/23
5/23
6/23
7/23
8/23
9/23
10/23
11/23
12/23
13/23
14/23
15/23
16/23
17/23
18/23A-b
19/23A-b
20/23
21/23
22/23
23/23A-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v9 thread

Cover letter [RFC PATCH v9 00/23] Net Control VQ support in SVQ

Control virtqueue is used by networking device for accepting various commands from the driver. It's a must to support multiqueue and other configurations.

Shadow VirtQueue (SVQ) already makes possible migration of virtqueue states, effectively intercepting them so qemu can track what regions of memory are dirty because device action and needs migration. However, this does not solve networking device state seen by the driver because CVQ messages, like changes on MAC addresses from the driver.

This series uses SVQ infraestructure to intercept networking control messages used by the device. This way, qemu is able to update VirtIONet device model and to migrate it.

To intercept all queues slows device data forwarding, so this is not the final solution. To solve that, only the CVQ must be intercepted all the time. This will be achieved in future revisions using the ASID infraestructure, that allows different translations for different virtqueues.

Another pending item is to move data virtqueues from passthrough mode to SVQ one. To achieve that, a reliable way to obtain the vq state is needed. STOP ioctl will be added for that.

To intercept all the virtqueues and update the qemu nic device model as the guest changes the device state add the cmdline opt x-svq=on:

-netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa0,x-svq=on

Lastly, device state is sent each time qemu starts the device, using SVQ to inject commands through CVQ. This allows the guest to transparently see the same guest-visible state at resume.

First two patches reorder code so it's easier to apply later patches on top of the code base.

Third patch reorders the device ok and the set_vring_enable ioctl sending. This is done so CVQ commands reach the device before the device have the chance to use rx queues, with incorrect data.

Fourth patch replaces the way of getting vq state. Since qemu will be able to inject buffers, device's used_idx is not valid anymore and we must use guest-visible one.

Fifth patch creates the API in SVQ to call when device start. This will allow vhost-vdpa net to inject control commands before the rest of queues start.

Sixth path enables SVQ to return buffers externally. While it's not possible at this point in the series, CVQ will need to return the available buffers

Patches 8-12 enables SVQ to communicate the caller of SVQ context data of the used buffer.

Patch 13 enables vhost-vdpa net to inject buffers to the device. This will be used both to inject the state at the beginning and to decouple guest's CVQ buffers from the ones sent to the device. This brings protections against TOCTOU, avoiding the device and qemu to see different messages. In the future, this may also be used to emulate _F_ANNOUNCE.

The previous patch and patches from 14 to 17 makes SVQ capable of being inspected.

Patches 18 to 20 enable the update of the virtio-net device model for each CVQ message acknoledged by the device.

Patches 21-22 enables the update of the device configuration right at start.

Finally, last commit enables x-svq parameter.

Comments are welcomed.

TODO:
* Review failure paths, some are with TODO notes, other don't.

Changes from rfc v8:
* Remove ASID part. Delete x-svq-cvq mode too.
* Move all DMA memory management to net/vhost-vdpa, instead of svq.
* Use of qemu_real_host_page_size.
* Improved doc, general fixes.

Changes from rfc v7:
* Don't map all guest space in ASID 1 but copy all the buffers. No need for more memory listeners.
* Move net backend start callback to SVQ.
* Wait for device CVQ commands used by the device at SVQ start, avoiding races.
* Changed ioctls, but they're provisional anyway.
* Reorder commits so refactor and code adding ones are closer to usage.
* Usual cleaning: better tracing, doc, patches messages, ...

Changes from rfc v6:
* Fix bad iotlb updates order when batching was enabled
* Add reference counting to iova_tree so cleaning is simpler.

Changes from rfc v5:
* Fixes bad calculus of cvq end group when MQ is not acked by the guest.

Changes from rfc v4:
* Add missing tracing
* Add multiqueue support
* Use already sent version for replacing g_memdup
* Care with memory management

Changes from rfc v3:
* Fix bad returning of descriptors to SVQ list.

Changes from rfc v2:
* Fix use-after-free.

Changes from rfc v1:
* Rebase to latest master.
* Configure ASID instead of assuming cvq asid != data vqs asid.
* Update device model so (MAC) state can be migrated too.

Eugenio Pérez (23):
vhost: Return earlier if used buffers overrun
vhost: move descriptor translation to vhost_svq_vring_write_descs
vdpa: delay set_vring_ready after DRIVER_OK
vhost: Get vring base from vq, not svq
vhost: Add ShadowVirtQueueStart operation
virtio-net: Expose ctrl virtqueue logic
vhost: add vhost_svq_push_elem
vhost: Decouple vhost_svq_add_split from VirtQueueElement
vhost: Add SVQElement
vhost: Reorder vhost_svq_last_desc_of_chain
vhost: Move last chain id to SVQ element
vhost: Add opaque member to SVQElement
vhost: Add vhost_svq_inject
vhost: add vhost_svq_poll
vhost: Add custom used buffer callback
vhost: Add svq avail_handler callback
vhost: add detach SVQ operation
vdpa: Export vhost_vdpa_dma_map and unmap calls
vdpa: Extract get features part from vhost_vdpa_get_max_queue_pairs
vdpa: Buffer CVQ support on shadow virtqueue
vdpa: Add vhost_vdpa_start_control_svq
vdpa: Inject virtio-net mac address via CVQ at start
vdpa: Add x-svq to NetdevVhostVDPAOptions

 qapi/net.json                      |   9 +-
 hw/virtio/vhost-shadow-virtqueue.h |  67 +++-
 include/hw/virtio/vhost-vdpa.h     |   7 +
 include/hw/virtio/virtio-net.h     |   4 +
 include/hw/virtio/virtio.h         |   1 +
 hw/net/virtio-net.c                |  84 +++--
 hw/virtio/vhost-shadow-virtqueue.c | 284 ++++++++++++---
 hw/virtio/vhost-vdpa.c             |  54 ++-
 hw/virtio/virtio.c                 |   5 +
 net/vhost-vdpa.c                   | 541 ++++++++++++++++++++++++++++-
 10 files changed, 935 insertions(+), 121 deletions(-)
Signature · 1 line
2.31.1