vDPA shadow virtqueue
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v3 Latest14 patches 6 repliesIn review
- v214 patches 35 repliesSuperseded
- v131 patches 96 repliesSuperseded
Changes in v3 from the cover letter
* Less assertions and more error handling in iova tree code.
* Better documentation, both fixing errors and making @param: format
* Homogeneize SVQ avail_idx_shadow and shadow_used_idx to make shadow a prefix at both times.
* Fix: Fo not use VirtQueueElement->len field, track separatedly.
* Split vhost_svq_{enable,disable}_notification, so the code looks more like the kernel driver code.
* Small improvements.
v2 link:
https://lore.kernel.org/all/CAJaqyWfXHE0C54R_-OiwJzjC0gPpkE3eX0L8BeeZXGm1ERYPtA@mail.gmail.com/
Patches in v3 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/14 | vhost: Add VhostShadowVirtqueuehw/ hw/ hw/ | +91 | 0 replies | |
| 2/14 | vhost: Add Shadow VirtQueue kick forwarding capabilitieshw/ hw/ hw/ +1 more | +217 | 0 replies | |
| 3/14 | vhost: Add Shadow VirtQueue call forwarding capabilitieshw/ hw/ hw/ | +71 | 0 replies | |
| 4/14 | vhost: Add vhost_svq_valid_features to shadow vqhw/ hw/ hw/ | +61 | 0 replies | |
| 5/14 | virtio: Add vhost_svq_get_vring_addrhw/ hw/ | +38 | 0 replies | |
| 6/14 | vdpa: adapt vhost_ops callbacks to svqhw/ | +41 | 0 replies | |
| 7/14 | vhost: Shadow virtqueue buffers forwardinghw/ hw/ hw/ | +526 | 0 replies | |
| 8/14 | util: Add iova_tree_alloc_mapinclude/ util/ | +153 | 0 replies | |
| 9/14 | vhost: Add VhostIOVATreehw/ hw/ hw/ | +183 | 0 replies | |
| 10/14 | vdpa: Add custom IOTLB translations to SVQhw/ hw/ hw/ +1 more | +181 | 0 replies | |
| 11/14 | vdpa: Adapt vhost_vdpa_get_vring_base to SVQhw/ | +17 | 0 replies | |
| 12/14 | vdpa: Never set log_base addr if SVQ is enabledhw/ | +2 | 0 replies | |
| 13/14 | vdpa: Expose VHOST_F_LOG_ALL on SVQhw/ include/ | +36 | 0 replies | |
| 14/14 | vdpa: Add x-svq to NetdevVhostVDPAOptionsnet/ qapi/ | +44 | 6 replies |
Reviews trailers per patch
2 of 14 patches have a Reviewed-by or Acked-by.
| Patch | Peter Xu | Juan Quintela | John Snow |
|---|---|---|---|
| 1/14 | |||
| 2/14 | |||
| 3/14 | |||
| 4/14 | |||
| 5/14 | |||
| 6/14 | |||
| 7/14 | |||
| 8/14 | R-b | ||
| 9/14 | |||
| 10/14 | |||
| 11/14 | |||
| 12/14 | |||
| 13/14 | |||
| 14/14 | R-b | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.
Cover letter [PATCH v3 00/14] vDPA shadow virtqueue
This series enable shadow virtqueue (SVQ) for vhost-vdpa devices. This is intended as a new method of tracking the memory the devices touch during a migration process: Instead of relay on vhost device's dirty logging capability, SVQ intercepts the VQ dataplane forwarding the descriptors between VM and device. This way qemu is the effective writer of guests memory, like in qemu's virtio device operation.
When SVQ is enabled qemu offers a new virtual address space to the device to read and write into, and it maps new vrings and the guest memory in it. SVQ also intercepts kicks and calls between the device and the guest. Used buffers relay would cause dirty memory being tracked, but at this RFC SVQ is not enabled on migration automatically.
Thanks of being a buffers relay system, SVQ can be used also to communicate devices and drivers with different capabilities, like devices that only supports packed vring and not split and old guest with no driver packed support.
It is based on the ideas of DPDK SW assisted LM, in the series of DPDK's https://patchwork.dpdk.org/cover/48370/ . However, these does not map the shadow vq in guest's VA, but in qemu's.
For qemu to use shadow virtqueues the guest virtio driver must not use features like event_idx.
SVQ needs to be enabled with QMP command:
{ "execute": "x-vhost-set-shadow-vq",
"arguments": { "name": "vhost-vdpa0", "enable": true } }This series includes some patches to delete in the final version that helps with its testing. The first two of the series have been sent sepparately but they haven't been included in qemu main branch.
The two after them adds the feature to stop the device and be able to set and get its status. It's intended to be used with vp_vpda driver in a nested environment, so they are also external to this series. The vp_vdpa driver also need modifications to forward the new status bit, they will be proposed sepparately
Patches 5-12 prepares the SVQ and QMP command to support guest to host notifications forwarding. If the SVQ is enabled with these ones applied and the device supports it, that part can be tested in isolation (for example, with networking), hopping through SVQ.
Same thing is true with patches 13-17, but with device to guest notifications.
Based on them, patches from 18 to 22 implement the actual buffer forwarding, using some features already introduced in previous. However, they will need a host device with no iommu, something that is not available at the moment.
The last part of the series uses properly the host iommu, so the driver can access this new virtual address space created.
Comments are welcome.
TODO:
* Event, indirect, packed, and others features of virtio.
* To sepparate buffers forwarding in its own AIO context, so we can throw more threads to that task and we don't need to stop the main event loop.
* Support multiqueue virtio-net vdpa.
* Proper documentation.
Changes from v2:
* Less assertions and more error handling in iova tree code.
* Better documentation, both fixing errors and making @param: format
* Homogeneize SVQ avail_idx_shadow and shadow_used_idx to make shadow a prefix at both times.
* Fix: Fo not use VirtQueueElement->len field, track separatedly.
* Split vhost_svq_{enable,disable}_notification, so the code looks more like the kernel driver code.
* Small improvements.
v2 link:
https://lore.kernel.org/all/CAJaqyWfXHE0C54R_-OiwJzjC0gPpkE3eX0L8BeeZXGm1ERYPtA@mail.gmail.com/
Changes from v1:
* Feature set at device->SVQ is now the same as SVQ->guest.
* Size of SVQ is not max available device size anymore, but guest's negotiated.
* Add VHOST_FILE_UNBIND kick and call fd treatment.
* Make SVQ a public struct
* Come back to previous approach to iova-tree
* Some assertions are now fail paths. Some errors are now log_guest.
* Only mask _F_LOG feature at vdpa_set_features svq enable path.
* Refactor some errors and messages. Add missing error unwindings.
* Add memory barrier at _F_NO_NOTIFY set.
* Stop checking for features flags out of transport range. v1 link:
https://lore.kernel.org/virtualization/[email protected]/
Changes from v4 RFC:
* Support of allocating / freeing iova ranges in IOVA tree. Extending already present iova-tree for that.
* Proper validation of guest features. Now SVQ can negotiate a different set of features with the device when enabled.
* Support of host notifiers memory regions
* Handling of SVQ full queue in case guest's descriptors span to different memory regions (qemu's VA chunks).
* Flush pending used buffers at end of SVQ operation.
* QMP command now looks by NetClientState name. Other devices will need to implement it's way to enable vdpa.
* Rename QMP command to set, so it looks more like a way of working
* Better use of qemu error system
* Make a few assertions proper error-handling paths.
* Add more documentation
* Less coupling of virtio / vhost, that could cause friction on changes
* Addressed many other small comments and small fixes.
Changes from v3 RFC:
* Move everything to vhost-vdpa backend. A big change, this allowed some cleanup but more code has been added in other places.
* More use of glib utilities, especially to manage memory.
v3 link:
https://lists.nongnu.org/archive/html/qemu-devel/2021-05/msg06032.html
Changes from v2 RFC:
* Adding vhost-vdpa devices support
* Fixed some memory leaks pointed by different comments
v2 link:
https://lists.nongnu.org/archive/html/qemu-devel/2021-03/msg05600.html
Changes from v1 RFC:
* Use QMP instead of migration to start SVQ mode.
* Only accepting IOMMU devices, closer behavior with target devices
(vDPA)
* Fix invalid masking/unmasking of vhost call fd.
* Use of proper methods for synchronization.
* No need to modify VirtIO device code, all of the changes are contained in vhost code.
* Delete superfluous code.
* An intermediate RFC was sent with only the notifications forwarding changes. It can be seen in
https://patchew.org/QEMU/[email protected]/
v1 link:
https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05372.html
Eugenio Pérez (14):
vhost: Add VhostShadowVirtqueue
vhost: Add Shadow VirtQueue kick forwarding capabilities
vhost: Add Shadow VirtQueue call forwarding capabilities
vhost: Add vhost_svq_valid_features to shadow vq
virtio: Add vhost_svq_get_vring_addr
vdpa: adapt vhost_ops callbacks to svq
vhost: Shadow virtqueue buffers forwarding
util: Add iova_tree_alloc_map.
vhost: Add VhostIOVATree
vdpa: Add custom IOTLB translations to SVQ
vdpa: Adapt vhost_vdpa_get_vring_base to SVQ
vdpa: Never set log_base addr if SVQ is enabled
vdpa: Expose VHOST_F_LOG_ALL on SVQ
vdpa: Add x-svq to NetdevVhostVDPAOptions
qapi/net.json | 5 +- hw/virtio/vhost-iova-tree.h | 27 ++ hw/virtio/vhost-shadow-virtqueue.h | 87 ++++ include/hw/virtio/vhost-vdpa.h | 8 + include/qemu/iova-tree.h | 18 + hw/virtio/vhost-iova-tree.c | 155 +++++++ hw/virtio/vhost-shadow-virtqueue.c | 637 +++++++++++++++++++++++++++++ hw/virtio/vhost-vdpa.c | 525 +++++++++++++++++++++++- net/vhost-vdpa.c | 48 ++- util/iova-tree.c | 135 ++++++ hw/virtio/meson.build | 2 +- 11 files changed, 1622 insertions(+), 25 deletions(-) create mode 100644 hw/virtio/vhost-iova-tree.h create mode 100644 hw/virtio/vhost-shadow-virtqueue.h create mode 100644 hw/virtio/vhost-iova-tree.c create mode 100644 hw/virtio/vhost-shadow-virtqueue.c
Signature · 1 line
2.27.0