vDPA shadow virtqueue
Subsystems (2)
Eugenio Pérez
maintainerRFC PATCH v4 4/20 vhost: Make vhost_virtqueue_{start,stop} public
In reply to Eugenio PérezThe easier way to reset vhost behavior after enabling shadow virtqueue is to call vhost_virtqueue_start. Also, vhost_virtqueue_stop provides the simmetrical call to it, and it will be used.
--- include/hw/virtio/vhost.h | 4 ++++ hw/virtio/vhost.c | 12 ++++-------- 2 files changed, 8 insertions(+), 8 deletions(-)
include/hw/virtio/vhost.h +4 -0
@@ -130,6 +130,10 @@ int vhost_net_set_backend(struct vhost_dev *hdev, struct vhost_vring_file *file);int vhost_device_iotlb_miss (struct vhost_dev *dev,uint64_t iova,int write); +int vhost_virtqueue_start (struct vhost_dev *dev,struct VirtIODevice *vdev, +struct vhost_virtqueue *vq,unsigned idx); +void vhost_virtqueue_stop (struct vhost_dev *dev,struct VirtIODevice *vdev, +struct vhost_virtqueue *vq,unsigned idx);int vhost_dev_get_config (struct vhost_dev *hdev,uint8_t *config,uint32_t config_len,Error **errp); int vhost_dev_set_config(struct vhost_dev *dev, const uint8_t *data,
hw/virtio/vhost.c +4 -8
@@ -1049,10 +1049,8 @@ out:return ret; } -static int vhost_virtqueue_start (struct vhost_dev *dev, -struct VirtIODevice *vdev, -struct vhost_virtqueue *vq, -unsigned idx) +int vhost_virtqueue_start (struct vhost_dev *dev,struct VirtIODevice *vdev, +struct vhost_virtqueue *vq,unsigned idx) {BusState *qbus =BUS (qdev_get_parent_bus (DEVICE (vdev)));VirtioBusState *vbus =VIRTIO_BUS (qbus);
@@ -1171,10 +1169,8 @@ fail_alloc_desc:return r; } -static void vhost_virtqueue_stop (struct vhost_dev *dev, -struct VirtIODevice *vdev, -struct vhost_virtqueue *vq, -unsigned idx) +void vhost_virtqueue_stop (struct vhost_dev *dev,struct VirtIODevice *vdev, +struct vhost_virtqueue *vq,unsigned idx) {int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index (dev, idx);struct vhost_vring_state state = {
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 7/20 vdpa: Register vdpa devices in a list
In reply to Eugenio PérezThis way QMP command can iterate through them and find the devices it needs to enable shadow virtqueue in.
--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 5 +++++ 2 files changed, 7 insertions(+)
include/hw/virtio/vhost-vdpa.h +2 -0
@@ -12,6 +12,7 @@ #ifndef HW_VIRTIO_VHOST_VDPA_H #define HW_VIRTIO_VHOST_VDPA_H +#include "qemu/queue.h" #include "hw/virtio/virtio.h" typedef struct VhostVDPAHostNotifier {
@@ -24,6 +25,7 @@ typedef struct vhost_vdpa {uint32_t msg_type;MemoryListener listener;struct vhost_dev *dev; +QLIST_ENTRY (vhost_vdpa) entry;VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ]; } VhostVDPA;
hw/virtio/vhost-vdpa.c +5 -0
@@ -26,6 +26,9 @@#include "qapi/qapi-commands-net.h" #include "qapi/error.h" +static QLIST_HEAD(, vhost_vdpa) vhost_vdpa_devices = +QLIST_HEAD_INITIALIZER (vhost_vdpa_devices); +static bool vhost_vdpa_listener_skipped_section (MemoryRegionSection *section) { return (!memory_region_is_ram(section->mr) &&
@@ -280,6 +283,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; +QLIST_INSERT_HEAD (&vhost_vdpa_devices, v, entry);vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_ACKNOWLEDGE |VIRTIO_CONFIG_S_DRIVER );
@@ -377,6 +381,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev)trace_vhost_vdpa_cleanup (dev, v);vhost_vdpa_host_notifiers_uninit (dev, dev->nvqs);memory_listener_unregister (&v->listener); +QLIST_REMOVE (v, entry); dev->opaque =NULL ;return 0 ;
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 3/20 virtio: Add virtio_queue_is_host_notifier_enabled
In reply to Eugenio PérezThis allows shadow virtqueue code to assert the queue status before making changes.
--- include/hw/virtio/virtio.h | 1 + hw/virtio/virtio.c | 5 +++++ 2 files changed, 6 insertions(+)
include/hw/virtio/virtio.h +1 -0
@@ -318,6 +318,7 @@ void virtio_device_release_ioeventfd(VirtIODevice *vdev);bool virtio_device_ioeventfd_enabled (VirtIODevice *vdev);EventNotifier *virtio_queue_get_host_notifier (VirtQueue *vq);void virtio_queue_set_host_notifier_enabled (VirtQueue *vq,bool enabled); +bool virtio_queue_is_host_notifier_enabled (const VirtQueue *vq);void virtio_queue_host_notifier_read (EventNotifier *n);void virtio_queue_aio_set_host_notifier_handler (VirtQueue *vq,AioContext *ctx,VirtIOHandleAIOOutput handle_output);
hw/virtio/virtio.c +5 -0
@@ -3591,6 +3591,11 @@ EventNotifier *virtio_queue_get_host_notifier(VirtQueue *vq)return &vq->host_notifier; } +bool virtio_queue_is_host_notifier_enabled (const VirtQueue *vq) +{ +return vq->host_notifier_enabled; +} +void virtio_queue_set_host_notifier_enabled (VirtQueue *vq,bool enabled) { vq->host_notifier_enabled = enabled;
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerv4 cover vDPA shadow virtqueue
This series enable shadow virtqueue (SVQ) for vhost-vdpa devices. This is intended as a new method of tracking the memory the devices touch during a migration process: Instead of relay on vhost device's dirty logging capability, SVQ intercepts the VQ dataplane forwarding the descriptors between VM and device. This way qemu is the effective writer of guests memory, like in qemu's virtio device operation.
When SVQ is enabled qemu offers a new vring to the device to read and write into, and also intercepts kicks and calls between the device and the guest. Used buffers relay would cause dirty memory being tracked, but at this RFC SVQ is not enabled on migration automatically.
It is based on the ideas of DPDK SW assisted LM, in the series of DPDK's https://patchwork.dpdk.org/cover/48370/ . However, these does not map the shadow vq in guest's VA, but in qemu's.
For qemu to use shadow virtqueues the guest virtio driver must not use features like event_idx or indirect descriptors. These limitations will be addressed in later series, but they are left out for simplicity at the moment.
SVQ needs to be enabled with QMP command:
{ "execute": "x-vhost-enable-shadow-vq",
"arguments": { "name": "dev0", "enable": true } }This series includes some patches to delete in the final version that helps with its testing. The first two of the series freely implements the feature to stop the device and be able to retrieve its status. It's intended to be used with vp_vpda driver in a nested environment. This driver also need modifications to forward the new status bit.
Patches 2-8 prepares the SVQ and QMP command to support guest to host notifications forwarding. If the SVQ is enabled with these ones applied and the device supports it, that part can be tested in isolation (for example, with networking), hopping through SVQ.
Same thing is true with patches 9-13, but with device to guest notifications.
The rest of the patches implements the actual buffer forwarding.
Comments are welcome.
TODO:
* Event, indirect, packed, and others features of virtio - Waiting for confirmation of the big picture.
* Use already available iova tree to track mappings.
* To sepparate buffers forwarding in its own AIO context, so we can throw more threads to that task and we don't need to stop the main event loop.
* unmap iommu memory. Now the tree can only grow from SVQ enable, but it should be fine as long as not a lot of memory is added to the guest.
* Rebase on top of latest qemu (and, hopefully, on top of multiqueue vdpa).
* Some assertions need to be appropiate error handling paths.
* Proper documentation.
Changes from v3 RFC:
* Move everything to vhost-vdpa backend. A big change, this allowed some cleanup but more code has been added in other places.
* More use of glib utilities, especially to manage memory.
v3 link:
https://lists.nongnu.org/archive/html/qemu-devel/2021-05/msg06032.html
Changes from v2 RFC:
* Adding vhost-vdpa devices support
* Fixed some memory leaks pointed by different comments
v2 link:
https://lists.nongnu.org/archive/html/qemu-devel/2021-03/msg05600.html
Changes from v1 RFC:
* Use QMP instead of migration to start SVQ mode.
* Only accepting IOMMU devices, closer behavior with target devices
(vDPA)
* Fix invalid masking/unmasking of vhost call fd.
* Use of proper methods for synchronization.
* No need to modify VirtIO device code, all of the changes are contained in vhost code.
* Delete superfluous code.
* An intermediate RFC was sent with only the notifications forwarding changes. It can be seen in
https://patchew.org/QEMU/[email protected]/
v1 link:
https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05372.html
Eugenio Pérez (20):
virtio: Add VIRTIO_F_QUEUE_STATE
virtio-net: Honor VIRTIO_CONFIG_S_DEVICE_STOPPED
virtio: Add virtio_queue_is_host_notifier_enabled
vhost: Make vhost_virtqueue_{start,stop} public
vhost: Add x-vhost-enable-shadow-vq qmp
vhost: Add VhostShadowVirtqueue
vdpa: Register vdpa devices in a list
vhost: Route guest->host notification through shadow virtqueue Add vhost_svq_get_svq_call_notifier
Add vhost_svq_set_guest_call_notifier
vdpa: Save call_fd in vhost-vdpa
vhost-vdpa: Take into account SVQ in vhost_vdpa_set_vring_call
vhost: Route host->guest notification through shadow virtqueue
virtio: Add vhost_shadow_vq_get_vring_addr
vdpa: Save host and guest features
vhost: Add vhost_svq_valid_device_features to shadow vq
vhost: Shadow virtqueue buffers forwarding
vhost: Add VhostIOVATree
vhost: Use a tree to store memory mappings
vdpa: Add custom IOTLB translations to SVQ
Eugenio Pérez (20):
virtio: Add VIRTIO_F_QUEUE_STATE
virtio-net: Honor VIRTIO_CONFIG_S_DEVICE_STOPPED
virtio: Add virtio_queue_is_host_notifier_enabled
vhost: Make vhost_virtqueue_{start,stop} public
vhost: Add x-vhost-enable-shadow-vq qmp
vhost: Add VhostShadowVirtqueue
vdpa: Register vdpa devices in a list
vhost: Route guest->host notification through shadow virtqueue
vdpa: Save call_fd in vhost-vdpa
vhost-vdpa: Take into account SVQ in vhost_vdpa_set_vring_call
vhost: Route host->guest notification through shadow virtqueue
virtio: Add vhost_shadow_vq_get_vring_addr
vdpa: Save host and guest features
vhost: Add vhost_svq_valid_device_features to shadow vq
vhost: Shadow virtqueue buffers forwarding
vhost: Check for device VRING_USED_F_NO_NOTIFY at shadow virtqueue
kick
vhost: Use VRING_AVAIL_F_NO_INTERRUPT at device call on shadow
virtqueue
vhost: Add VhostIOVATree
vhost: Use a tree to store memory mappings
vdpa: Add custom IOTLB translations to SVQ
qapi/net.json | 23 + hw/virtio/vhost-iova-tree.h | 40 ++ hw/virtio/vhost-shadow-virtqueue.h | 37 ++ hw/virtio/virtio-pci.h | 1 + include/hw/virtio/vhost-vdpa.h | 13 + include/hw/virtio/vhost.h | 4 + include/hw/virtio/virtio.h | 5 +- .../standard-headers/linux/virtio_config.h | 5 + include/standard-headers/linux/virtio_pci.h | 2 + hw/net/virtio-net.c | 6 +- hw/virtio/vhost-iova-tree.c | 230 +++++++ hw/virtio/vhost-shadow-virtqueue.c | 619 ++++++++++++++++++ hw/virtio/vhost-vdpa.c | 412 +++++++++++- hw/virtio/vhost.c | 12 +- hw/virtio/virtio-pci.c | 16 +- hw/virtio/virtio.c | 5 + hw/virtio/meson.build | 2 +- hw/virtio/trace-events | 1 + 18 files changed, 1413 insertions(+), 20 deletions(-) create mode 100644 hw/virtio/vhost-iova-tree.h create mode 100644 hw/virtio/vhost-shadow-virtqueue.h create mode 100644 hw/virtio/vhost-iova-tree.c create mode 100644 hw/virtio/vhost-shadow-virtqueue.c
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 5/20 vhost: Add x-vhost-enable-shadow-vq qmp
Command to enable shadow virtqueue.
--- qapi/net.json | 23 +++++++++++++++++++++++ hw/virtio/vhost-vdpa.c | 8 ++++++++ 2 files changed, 31 insertions(+)
qapi/net.json +23 -0
@@ -79,6 +79,29 @@ { 'command': 'netdev_del', 'data': {'id': 'str'}, 'allow-preconfig': true } +## +# @x-vhost-enable-shadow-vq: +# +# Use vhost shadow virtqueue. +# +# @name: the device name of the VirtIO device +# +# @enable: true to use the alternate shadow VQ notifications +# +# Returns: Always error, since SVQ is not implemented at the moment. +# +# Since: 6.2 +# +# Example: +# +# -> { "execute": "x-vhost-enable-shadow-vq", +# "arguments": { "name": "virtio-net", "enable": false } } +# +## +{ 'command': 'x-vhost-enable-shadow-vq', + 'data': {'name': 'str', 'enable': 'bool'}, + 'if': 'defined(CONFIG_VHOST_KERNEL)' } + ## # @NetLegacyNicOptions: #
hw/virtio/vhost-vdpa.c +8 -0
@@ -23,6 +23,8 @@#include "cpu.h" #include "trace.h" #include "qemu-common.h" +#include "qapi/qapi-commands-net.h" +#include "qapi/error.h" static bool vhost_vdpa_listener_skipped_section (MemoryRegionSection *section) {
@@ -656,6 +658,12 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } + +void qmp_x_vhost_enable_shadow_vq (const char *name,bool enable,Error **errp) +{ +error_setg (errp,"Shadow virtqueue still not implemented" ); +} +const VhostOps vdpa_ops = { .backend_type =VHOST_BACKEND_TYPE_VDPA , .vhost_backend_init = vhost_vdpa_init,
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 9/20 vdpa: Save call_fd in vhost-vdpa
In reply to Eugenio PérezWe need to know it to switch to Shadow VirtQueue.
--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 5 +++++ 2 files changed, 7 insertions(+)
include/hw/virtio/vhost-vdpa.h +2 -0
@@ -30,6 +30,8 @@ typedef struct vhost_vdpa {GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry; +/* File descriptor the device uses to call VM/SVQ */ +int call_fd[VIRTIO_QUEUE_MAX ];VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ]; } VhostVDPA;
hw/virtio/vhost-vdpa.c +5 -0
@@ -652,7 +652,12 @@ static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev,static int vhost_vdpa_set_vring_call (struct vhost_dev *dev,struct vhost_vring_file *file) { +struct vhost_vdpa *v = dev->opaque; +int vdpa_idx =vhost_vdpa_get_vq_index (dev, file->index); +trace_vhost_vdpa_set_vring_call (dev, file->index, file->fd); + + v->call_fd[vdpa_idx] = file->fd;return vhost_vdpa_call (dev,VHOST_SET_VRING_CALL , file); }
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 1/20 virtio: Add VIRTIO_F_QUEUE_STATE
In reply to Eugenio Pérezvirtio: Add VIRTIO_F_QUEUE_STATE
Implementation of RFC of device state capability:
https://lists.oasis-open.org/archives/virtio-comment/202012/msg00005.html
With this capability, vdpa device can reset it's index so it can start consuming from guest after disabling shadow virtqueue (SVQ), with state not 0.
The use case is to test SVQ with virtio-pci vdpa (vp_vdpa) with nested virtualization. Spawning a L0 qemu with a virtio-net device, use vp_vdpa driver to handle it in the guest, and then spawn a L1 qemu using that vdpa device. When L1 qemu calls device to set a new state though vdpa ioctl, vp_vdpa should set each queue state though virtio VIRTIO_PCI_COMMON_Q_AVAIL_STATE.
Since this is only for testing vhost-vdpa, it's added here before of proposing to kernel code. No effort is done for checking that device can actually change its state, its layout, or if the device even supports to change state at all. These will be added in the future.
Also, a modified version of vp_vdpa that allows to set these in PCI config is needed.
TODO: Check for feature enabled and split in virtio pci config
--- hw/virtio/virtio-pci.h | 1 + include/hw/virtio/virtio.h | 4 +++- include/standard-headers/linux/virtio_config.h | 3 +++ include/standard-headers/linux/virtio_pci.h | 2 ++ hw/virtio/virtio-pci.c | 9 +++++++++ 5 files changed, 18 insertions(+), 1 deletion(-)
hw/virtio/virtio-pci.h +1 -0
@@ -120,6 +120,7 @@ typedef struct VirtIOPCIQueue {uint32_t desc[2 ];uint32_t avail[2 ];uint32_t used[2 ]; +uint16_t state; } VirtIOPCIQueue;struct VirtIOPCIProxy {
include/hw/virtio/virtio.h +3 -1
@@ -289,7 +289,9 @@ typedef struct VirtIORNGConf VirtIORNGConf;DEFINE_PROP_BIT64 ("iommu_platform" , _state, _field, \VIRTIO_F_IOMMU_PLATFORM , false), \DEFINE_PROP_BIT64 ("packed" , _state, _field, \ - VIRTIO_F_RING_PACKED, false) +VIRTIO_F_RING_PACKED , false), \ +DEFINE_PROP_BIT64 ("save_restore_q_state" , _state, _field, \ + VIRTIO_F_QUEUE_STATE, true)hwaddr virtio_queue_get_desc_addr (VirtIODevice *vdev,int n);bool virtio_queue_enabled_legacy (VirtIODevice *vdev,int n);
include/standard-headers/linux/virtio_config.h +3 -0
@@ -90,4 +90,7 @@* Does the device support Single Root I/O Virtualization? */ #define VIRTIO_F_SR_IOV 37 + +/* Device support save and restore virtqueue state */ +#define VIRTIO_F_QUEUE_STATE 40#endif /* _LINUX_VIRTIO_CONFIG_H */
include/standard-headers/linux/virtio_pci.h +2 -0
@@ -164,6 +164,7 @@ struct virtio_pci_common_cfg {uint32_t queue_avail_hi;/* read-write */ uint32_t queue_used_lo;/* read-write */ uint32_t queue_used_hi;/* read-write */ +uint16_t queue_avail_state;/* read-write */ };/* Fields in VIRTIO_PCI_CAP_PCI_CFG: */
@@ -202,6 +203,7 @@ struct virtio_pci_cfg_cap {#define VIRTIO_PCI_COMMON_Q_AVAILHI 44#define VIRTIO_PCI_COMMON_Q_USEDLO 48#define VIRTIO_PCI_COMMON_Q_USEDHI 52 +#define VIRTIO_PCI_COMMON_Q_AVAIL_STATE 56#endif /* VIRTIO_PCI_NO_MODERN */
hw/virtio/virtio-pci.c +9 -0
@@ -1216,6 +1216,9 @@ static uint64_t virtio_pci_common_read(void *opaque, hwaddr addr,case VIRTIO_PCI_COMMON_Q_USEDHI : val = proxy->vqs[vdev->queue_sel].used[1 ];break ; +case VIRTIO_PCI_COMMON_Q_AVAIL_STATE : + val =virtio_queue_get_last_avail_idx (vdev, vdev->queue_sel); +break ;default : val =0 ; }
@@ -1298,6 +1301,8 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr, proxy->vqs[vdev->queue_sel].avail[0 ], ((uint64_t )proxy->vqs[vdev->queue_sel].used[1 ]) <<32 | proxy->vqs[vdev->queue_sel].used[0]); +virtio_queue_set_last_avail_idx (vdev, vdev->queue_sel, + proxy->vqs[vdev->queue_sel].state); proxy->vqs[vdev->queue_sel].enabled =1 ; }else {virtio_error (vdev,"wrong value for queue_enable %" PRIx64, val);
@@ -1321,6 +1326,9 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,case VIRTIO_PCI_COMMON_Q_USEDHI : proxy->vqs[vdev->queue_sel].used[1 ] = val;break ; +case VIRTIO_PCI_COMMON_Q_AVAIL_STATE : + proxy->vqs[vdev->queue_sel].state = val; +break ;default :break ; }
@@ -1909,6 +1917,7 @@ static void virtio_pci_reset(DeviceState *qdev) proxy->vqs[i].desc[0 ] = proxy->vqs[i].desc[1 ] =0 ; proxy->vqs[i].avail[0 ] = proxy->vqs[i].avail[1 ] =0 ; proxy->vqs[i].used[0 ] = proxy->vqs[i].used[1 ] =0 ; + proxy->vqs[i].state =0 ; }if (pci_is_express (dev)) {
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 2/20 virtio-net: Honor VIRTIO_CONFIG_S_DEVICE_STOPPED
In reply to Eugenio PérezSo the guest can stop and start net device. It freely implements the RFC
https://lists.oasis-open.org/archives/virtio-comment/202012/msg00027.html
To stop (as "pause") the device is required to migrate status and vring addresses between device and SVQ. Once the device is stopped, the driver can request avail_idx, so it can be assigned to SVQ.
This is a WIP commit: as with VIRTIO_F_QUEUE_STATE, is introduced in virtio_config.h before of even proposing for the kernel, with no feature flag, and, with no checking in the device. It also needs a modified vp_vdpa driver that supports to set and retrieve status.
For virtio-net with qemu device there is no need to restore avail
state: Since every tx and rx operation is entirely done in BQL regarding virtio, it would be enough with restore last_avail_idx with used_idx. Doing this way test the vq state part of the rest of the series.
--- include/standard-headers/linux/virtio_config.h | 2 ++ hw/net/virtio-net.c | 6 ++++-- hw/virtio/virtio-pci.c | 7 +++++-- 3 files changed, 11 insertions(+), 4 deletions(-)
include/standard-headers/linux/virtio_config.h +2 -0
@@ -40,6 +40,8 @@#define VIRTIO_CONFIG_S_DRIVER_OK 4/* Driver has finished configuring features */ #define VIRTIO_CONFIG_S_FEATURES_OK 8 +/* Device is stopped */ +#define VIRTIO_CONFIG_S_DEVICE_STOPPED 32/* Device entered invalid state, driver must reset it */ #define VIRTIO_CONFIG_S_NEEDS_RESET 0x40/* We've given up on this device. */
hw/net/virtio-net.c +4 -2
@@ -198,6 +198,7 @@ static bool virtio_net_started(VirtIONet *n, uint8_t status) {VirtIODevice *vdev =VIRTIO_DEVICE (n);return (status &VIRTIO_CONFIG_S_DRIVER_OK ) && + (!(status &VIRTIO_CONFIG_S_DEVICE_STOPPED )) && (n->status &VIRTIO_NET_S_LINK_UP ) && vdev->vm_running; }
@@ -385,7 +386,7 @@ static void virtio_net_set_status(struct VirtIODevice *vdev, uint8_t status)qemu_flush_queued_packets (ncs); } -if (!q->tx_waiting) { +if (!q->tx_waiting && !(status &VIRTIO_CONFIG_S_DEVICE_STOPPED )) {continue ; }
@@ -1503,7 +1504,8 @@ static bool virtio_net_can_receive(NetClientState *nc) }if (!virtio_queue_ready (q->rx_vq) || - !(vdev->status &VIRTIO_CONFIG_S_DRIVER_OK )) { + !(vdev->status &VIRTIO_CONFIG_S_DRIVER_OK ) || + vdev->status ==VIRTIO_CONFIG_S_DEVICE_STOPPED ) {return false; }
hw/virtio/virtio-pci.c +5 -2
@@ -326,13 +326,15 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) }break ;case VIRTIO_PCI_STATUS : -if (!(val &VIRTIO_CONFIG_S_DRIVER_OK )) { +if (!(val &VIRTIO_CONFIG_S_DRIVER_OK ) || + val &VIRTIO_CONFIG_S_DEVICE_STOPPED ) {virtio_pci_stop_ioeventfd (proxy); }virtio_set_status (vdev, val &0xFF ); -if (val &VIRTIO_CONFIG_S_DRIVER_OK ) { +if (val &VIRTIO_CONFIG_S_DRIVER_OK && + !(val &VIRTIO_CONFIG_S_DEVICE_STOPPED )) {virtio_pci_start_ioeventfd (proxy); }
@@ -1303,6 +1305,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr, proxy->vqs[vdev->queue_sel].used[0]);virtio_queue_set_last_avail_idx (vdev, vdev->queue_sel, proxy->vqs[vdev->queue_sel].state); +virtio_queue_update_used_idx (vdev, vdev->queue_sel); proxy->vqs[vdev->queue_sel].enabled =1 ; }else {virtio_error (vdev,"wrong value for queue_enable %" PRIx64, val);
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 6/20 vhost: Add VhostShadowVirtqueue
In reply to Eugenio PérezVhost shadow virtqueue (SVQ) is an intermediate jump for virtqueue notifications and buffers, allowing qemu to track them. While qemu is forwarding the buffers and virtqueue changes, is able to commit the memory it's being dirtied, the same way regular qemu's VirtIO devices do.
This commit only exposes basic SVQ allocation and free, so changes regarding different aspects of SVQ (notifications forwarding, buffer forwarding, starting/stopping) are more isolated and easier to bisect.
--- hw/virtio/vhost-shadow-virtqueue.h | 21 ++++++++++ hw/virtio/vhost-shadow-virtqueue.c | 64 ++++++++++++++++++++++++++++++ hw/virtio/meson.build | 2 +- 3 files changed, 86 insertions(+), 1 deletion(-) create mode 100644 hw/virtio/vhost-shadow-virtqueue.h create mode 100644 hw/virtio/vhost-shadow-virtqueue.c
hw/virtio/vhost-shadow-virtqueue.h new file +21 -0
@@ -0,0 +1,21 @@ +/* +* vhost shadow virtqueue +* +* SPDX-FileCopyrightText: Red Hat, Inc. 2021 +* SPDX-FileContributor: Author: Eugenio Pérez <[email protected]> +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef VHOST_SHADOW_VIRTQUEUE_H +#define VHOST_SHADOW_VIRTQUEUE_H + +#include "hw/virtio/vhost.h" + +typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ; + +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx); + +void vhost_svq_free (VhostShadowVirtqueue *vq); + +#endif
hw/virtio/vhost-shadow-virtqueue.c new file +64 -0
64 lines not on this page · raw message
hw/virtio/meson.build +1 -1
@@ -11,7 +11,7 @@ softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('vhost-stub.c')) virtio_ss = ss.source_set() virtio_ss.add(files('virtio.c')) -virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c')) +virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c', 'vhost-shadow-virtqueue.c')) virtio_ss.add(when: 'CONFIG_VHOST_USER', if_true: files('vhost-user.c')) virtio_ss.add(when: 'CONFIG_VHOST_VDPA', if_true: files('vhost-vdpa.c')) virtio_ss.add(when: 'CONFIG_VIRTIO_BALLOON', if_true: files('virtio-balloon.c'))
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 10/20 vhost-vdpa: Take into account SVQ in vhost_vdpa_set_vring_call
In reply to Eugenio Pérez--- hw/virtio/vhost-vdpa.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
hw/virtio/vhost-vdpa.c +14 -3
@@ -649,16 +649,27 @@ static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev,return vhost_vdpa_call (dev,VHOST_SET_VRING_KICK , file); } +static int vhost_vdpa_set_vring_dev_call (struct vhost_dev *dev, +struct vhost_vring_file *file) +{ +trace_vhost_vdpa_set_vring_call (dev, file->index, file->fd); +return vhost_vdpa_call (dev,VHOST_SET_VRING_CALL , file); +} +static int vhost_vdpa_set_vring_call (struct vhost_dev *dev,struct vhost_vring_file *file) {struct vhost_vdpa *v = dev->opaque;int vdpa_idx =vhost_vdpa_get_vq_index (dev, file->index); -trace_vhost_vdpa_set_vring_call (dev, file->index, file->fd); - v->call_fd[vdpa_idx] = file->fd; -return vhost_vdpa_call (dev,VHOST_SET_VRING_CALL , file); +if (v->shadow_vqs_enabled) { +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, vdpa_idx); +vhost_svq_set_guest_call_notifier (svq, file->fd); +return 0 ; + }else { +return vhost_vdpa_set_vring_dev_call (dev, file); + } } static int vhost_vdpa_get_features(struct vhost_dev *dev,
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 12/20 virtio: Add vhost_shadow_vq_get_vring_addr
In reply to Eugenio PérezIt reports the shadow virtqueue address from qemu virtual address space
--- hw/virtio/vhost-shadow-virtqueue.h | 4 +++ hw/virtio/vhost-shadow-virtqueue.c | 50 ++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+)
hw/virtio/vhost-shadow-virtqueue.h +4 -0
@@ -16,6 +16,10 @@ typedef struct VhostShadowVirtqueue VhostShadowVirtqueue;EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq);void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); +void vhost_svq_get_vring_addr (const VhostShadowVirtqueue *svq, +struct vhost_vring_addr *addr); +size_t vhost_svq_driver_area_size (const VhostShadowVirtqueue *svq); +size_t vhost_svq_device_area_size (const VhostShadowVirtqueue *svq);bool vhost_svq_start (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq);
hw/virtio/vhost-shadow-virtqueue.c +50 -0
85 lines not on this page · raw message
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 8/20 vhost: Route guest->host notification through shadow virtqueue
In reply to Eugenio PérezShadow virtqueue notifications forwarding is disabled when vhost_dev stops, so code flow follows usual cleanup.
Also, host notifiers must be disabled at SVQ start, and they will not start if SVQ has been enabled when device is stopped. This is trivial to address, but it is left out for simplicity at this moment.
--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.h | 8 ++ include/hw/virtio/vhost-vdpa.h | 4 + hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- 5 files changed, 264 insertions(+), 4 deletions(-)
qapi/net.json +1 -1
@@ -88,7 +88,7 @@ # # @enable: true to use the alternate shadow VQ notifications # -# Returns: Always error, since SVQ is not implemented at the moment. +# Returns: Error if failure, or 'no error' for success. # # Since: 6.2 #
hw/virtio/vhost-shadow-virtqueue.h +8 -0
@@ -14,6 +14,14 @@typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ; +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq); +void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); + +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx);void vhost_svq_free (VhostShadowVirtqueue *vq);
include/hw/virtio/vhost-vdpa.h +4 -0
@@ -12,6 +12,8 @@#ifndef HW_VIRTIO_VHOST_VDPA_H #define HW_VIRTIO_VHOST_VDPA_H +#include <gmodule.h> +#include "qemu/queue.h" #include "hw/virtio/virtio.h"
@@ -24,6 +26,8 @@ typedef struct vhost_vdpa {int device_fd;uint32_t msg_type;MemoryListener listener; +bool shadow_vqs_enabled; +GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry;VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ];
hw/virtio/vhost-shadow-virtqueue.c +137 -1
166 lines not on this page · raw message
hw/virtio/vhost-vdpa.c +114 -2
161 lines not on this page · raw message
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 13/20 vdpa: Save host and guest features
In reply to Eugenio PérezThose are needed for SVQ: Host ones are needed to check if SVQ knows how to talk with the device and for feature negotiation, and guest ones to know if SVQ can talk with it.
--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 31 ++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-)
include/hw/virtio/vhost-vdpa.h +2 -0
@@ -26,6 +26,8 @@ typedef struct vhost_vdpa {int device_fd;uint32_t msg_type;MemoryListener listener; +uint64_t host_features; +uint64_t guest_features;bool shadow_vqs_enabled;GPtrArray *shadow_vqs;struct vhost_dev *dev;
hw/virtio/vhost-vdpa.c +28 -3
@@ -439,10 +439,19 @@ static int vhost_vdpa_set_mem_table(struct vhost_dev *dev,return 0 ; } -static int vhost_vdpa_set_features (struct vhost_dev *dev, -uint64_t features) +/** +* Internal set_features() that follows vhost/VirtIO protocol for that +*/ +static int vhost_vdpa_backend_set_features (struct vhost_dev *dev, +uint64_t features) { +struct vhost_vdpa *v = dev->opaque; +int ret; +if (v->host_features &BIT_ULL (VIRTIO_F_QUEUE_STATE )) { + features |=BIT_ULL (VIRTIO_F_QUEUE_STATE ); + } +trace_vhost_vdpa_set_features (dev, features); ret =vhost_vdpa_call (dev,VHOST_SET_FEATURES , &features);uint8_t status =0 ;
@@ -455,6 +464,17 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,return !(status &VIRTIO_CONFIG_S_FEATURES_OK ); } +/** +* Exposed vhost set features +*/ +static int vhost_vdpa_set_features (struct vhost_dev *dev, +uint64_t features) +{ +struct vhost_vdpa *v = dev->opaque; + v->guest_features = features; +return vhost_vdpa_backend_set_features (dev, features); +} +static int vhost_vdpa_set_backend_cap (struct vhost_dev *dev) {uint64_t features;
@@ -673,12 +693,17 @@ static int vhost_vdpa_set_vring_call(struct vhost_dev *dev, }static int vhost_vdpa_get_features (struct vhost_dev *dev, -uint64_t *features) +uint64_t *features) {int ret; ret =vhost_vdpa_call (dev,VHOST_GET_FEATURES , features);trace_vhost_vdpa_get_features (dev, *features); + +if (ret ==0 ) { +struct vhost_vdpa *v = dev->opaque; + v->host_features = *features; + }return ret; }
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 17/20 vhost: Use VRING_AVAIL_F_NO_INTERRUPT at device call on shadow virtqueue
In reply to Eugenio Pérez--- hw/virtio/vhost-shadow-virtqueue.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
hw/virtio/vhost-shadow-virtqueue.c +23 -1
@@ -60,6 +60,9 @@ typedef struct VhostShadowVirtqueue {/* Next head to consume from device */ uint16_t used_idx; + +/* Cache for the exposed notification flag */ +bool notification; } VhostShadowVirtqueue;/* If the device is using some of these, SVQ cannot communicate */
@@ -105,6 +108,24 @@ bool vhost_svq_valid_device_features(uint64_t *dev_features)return r; } +static void vhost_svq_set_notification (VhostShadowVirtqueue *svq,bool enable) +{ +uint16_t notification_flag; + +if (svq->notification == enable) { +return ; + } + + notification_flag =cpu_to_le16 (VRING_AVAIL_F_NO_INTERRUPT ); + + svq->notification = enable; +if (enable) { + svq->vring.avail->flags &= ~notification_flag; + }else { + svq->vring.avail->flags |= notification_flag; + } +} +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq,const struct iovec *iovec,size_t num,bool more_descs,bool write)
@@ -273,7 +294,7 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n) do { unsigned i = 0; - /* TODO: Use VRING_AVAIL_F_NO_INTERRUPT */ + vhost_svq_set_notification(svq, false); while (true) { g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); if (!elem) {
@@ -286,6 +307,7 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n)virtqueue_flush (vq, i);event_notifier_set (&svq->guest_call_notifier); +vhost_svq_set_notification (svq, true); }while (vhost_svq_more_used (svq)); }
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 11/20 vhost: Route host->guest notification through shadow virtqueue
In reply to Eugenio PérezThis will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of vhost_virtqueue_start can unmasks and discard call events, vhost_virtqueue_start should be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.
--- hw/virtio/vhost-shadow-virtqueue.c | 19 +++++++++++++++ hw/virtio/vhost-vdpa.c | 38 +++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)
hw/virtio/vhost-shadow-virtqueue.c +19 -0
@@ -53,6 +53,22 @@ static void vhost_handle_guest_kick(EventNotifier *n)event_notifier_set (&svq->kick_notifier); } +/* Forward vhost notifications */ +static void vhost_svq_handle_call_no_test (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + call_notifier); + +event_notifier_set (&svq->guest_call_notifier); +} + +static void vhost_svq_handle_call (EventNotifier *n) +{ +if (likely (event_notifier_test_and_clear (n))) { +vhost_svq_handle_call_no_test (n); + } +} +/* * Obtain the SVQ call notifier, where vhost device notifies SVQ that there * exists pending used buffers.
@@ -180,6 +196,8 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) } svq->vq =virtio_get_queue (dev->vdev, vq_idx); +event_notifier_set_handler (&svq->call_notifier, + vhost_svq_handle_call);return g_steal_pointer (&svq); err_init_call_notifier:
@@ -195,6 +213,7 @@ err_init_kick_notifier:void vhost_svq_free (VhostShadowVirtqueue *vq) {event_notifier_cleanup (&vq->kick_notifier); +event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier);g_free (vq); }
hw/virtio/vhost-vdpa.c +37 -1
@@ -712,13 +712,40 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) {struct vhost_vdpa *v = dev->opaque;VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); -return vhost_svq_start (dev, idx, svq); +EventNotifier *vhost_call_notifier =vhost_svq_get_svq_call_notifier (svq); +struct vhost_vring_file vhost_call_file = { + .index = idx + dev->vq_index, + .fd =event_notifier_get_fd (vhost_call_notifier), + }; +int r; +bool b; + +/* Set shadow vq -> guest notifier */ +assert (v->call_fd[idx]); +vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]); + + b =vhost_svq_start (dev, idx, svq); +if (unlikely (!b)) { +return false; + } + +/* Set device -> SVQ notifier */ + r =vhost_vdpa_set_vring_dev_call (dev, &vhost_call_file); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_call for shadow vq failed" ); +return false; + } + +/* Check for pending calls */ +event_notifier_set (vhost_call_notifier); +return true; }static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev;unsigned n; +int r;if (enable == v->shadow_vqs_enabled) {return hdev->nvqs;
@@ -752,9 +779,18 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)if (!enable) {/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { +struct vhost_vring_file file = { + .index =vhost_vdpa_get_vq_index (hdev, n), + .fd = v->call_fd[n], + }; + + r =vhost_vdpa_set_vring_call (hdev, &file); +assert (r ==0 ); +unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n);VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n);vhost_svq_stop (hdev, n, svq); +/* TODO: This can unmask or override call fd! */ vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); }
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 14/20 vhost: Add vhost_svq_valid_device_features to shadow vq
In reply to Eugenio PérezThis allows it to test if the guest has aknowledge an invalid transport feature for SVQ. This will include packed vq layout, invalid descriptors or event idx at the moment we start forwarding buffers.
We don't check for device features here since they will be re-negotiated again. This allows SVQ to both use more advanced features of the device when they are available and the guest is not capable of run them, and to make SVQ compatible with future transport features.
--- hw/virtio/vhost-shadow-virtqueue.h | 2 ++ hw/virtio/vhost-shadow-virtqueue.c | 6 ++++++ 2 files changed, 8 insertions(+)
hw/virtio/vhost-shadow-virtqueue.h +2 -0
@@ -14,6 +14,8 @@typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ; +bool vhost_svq_valid_device_features (uint64_t *features); +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq);void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); void vhost_svq_get_vring_addr(const VhostShadowVirtqueue *svq,
hw/virtio/vhost-shadow-virtqueue.c +6 -0
@@ -46,6 +46,12 @@ typedef struct VhostShadowVirtqueue {VirtIODevice *vdev; } VhostShadowVirtqueue; +/* If the device is using some of these, SVQ cannot communicate */ +bool vhost_svq_valid_device_features (uint64_t *dev_features) +{ +return true; +} +/* Forward guest notifications */ static void vhost_handle_guest_kick (EventNotifier *n) {
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 19/20 vhost: Use a tree to store memory mappings
In reply to Eugenio PérezTrack memory translations of devices with IOMMU (all vhost-vdpa devices at the moment). It does not work if device has restrictions in its iova range at the moment.
Updates to tree are protected by BQL, each one always run from main event loop context.
--- include/hw/virtio/vhost-vdpa.h | 3 ++ hw/virtio/vhost-vdpa.c | 59 ++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+)
include/hw/virtio/vhost-vdpa.h +3 -0
@@ -15,6 +15,7 @@#include <gmodule.h> #include "qemu/queue.h" +#include "hw/virtio/vhost-iova-tree.h" #include "hw/virtio/virtio.h" typedef struct VhostVDPAHostNotifier {
@@ -29,6 +30,8 @@ typedef struct vhost_vdpa {uint64_t host_features;uint64_t guest_features;bool shadow_vqs_enabled; +/* IOVA mapping used by Shadow Virtqueue */ +VhostIOVATree *iova_map;GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry;
hw/virtio/vhost-vdpa.c +59 -0
101 lines not on this page · raw message
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 16/20 vhost: Check for device VRING_USED_F_NO_NOTIFY at shadow virtqueue kick
In reply to Eugenio Pérez--- hw/virtio/vhost-shadow-virtqueue.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
hw/virtio/vhost-shadow-virtqueue.c +10 -1
@@ -173,6 +173,15 @@ static void vhost_svq_add(VhostShadowVirtqueue *svq, VirtQueueElement *elem) svq->ring_id_maps[qemu_head] = elem; } +static void vhost_svq_kick (VhostShadowVirtqueue *svq) +{ +/* Make sure we are reading updated device flag */ +smp_mb (); +if (!(svq->vring.used->flags &VRING_USED_F_NO_NOTIFY )) { +event_notifier_set (&svq->kick_notifier); + } +} +/* Handle guest->device notifications */ static void vhost_handle_guest_kick (EventNotifier *n) {
@@ -197,7 +206,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) }vhost_svq_add (svq, elem); -event_notifier_set (&svq->kick_notifier); +vhost_svq_kick (svq); }virtio_queue_set_notification (svq->vq, true);
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 15/20 vhost: Shadow virtqueue buffers forwarding
In reply to Eugenio PérezInitial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device.
For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series.
It reuses the VirtQueue code for the device part. The driver part is based on Linux's virtio_ring driver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones.
SVQ uses VIRTIO_CONFIG_S_DEVICE_STOPPED to pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter.
This status bit VIRTIO_CONFIG_S_DEVICE_STOPPED is currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits.
Removal of _S_DEVICE_STOPPED bit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device.
--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-)
qapi/net.json +1 -1
@@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding path # # Returns: Error if failure, or 'no error' for success. #
hw/virtio/vhost-shadow-virtqueue.c +231 -6
298 lines not on this page · raw message
hw/virtio/vhost-vdpa.c +105 -4
184 lines not on this page · raw message
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 20/20 vdpa: Add custom IOTLB translations to SVQ
In reply to Eugenio PérezUse translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)
hw/virtio/vhost-shadow-virtqueue.h +3 -1
@@ -11,6 +11,7 @@#define VHOST_SHADOW_VIRTQUEUE_H #include "hw/virtio/vhost.h" +#include "hw/virtio/vhost-iova-tree.h" typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ;
@@ -28,7 +29,8 @@ bool vhost_svq_start(struct vhost_dev *dev, unsigned idx,void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq); -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map);void vhost_svq_free (VhostShadowVirtqueue *vq);
hw/virtio/vhost-shadow-virtqueue.c +109 -21
234 lines not on this page · raw message
hw/virtio/vhost-vdpa.c +39 -1
@@ -176,6 +176,18 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); +if (v->shadow_vqs_enabled) { +VhostDMAMap mem_region = { + .translated_addr = vaddr, + .size =int128_get64 (llsize) -1 , + .perm =IOMMU_ACCESS_FLAG (true, section->readonly), + }; + +int r =vhost_iova_tree_alloc (v->iova_map, &mem_region); +assert (r ==VHOST_DMA_MAP_OK ); + + iova = mem_region.iova; + } ret =vhost_vdpa_dma_map (v, iova,int128_get64 (llsize), vaddr, section->readonly);
@@ -754,6 +766,23 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +static int vhost_vdpa_get_iova_range (struct vhost_dev *dev, +hwaddr *first,hwaddr *last) +{ +int ret; +struct vhost_vdpa_iova_range range; + + ret =vhost_vdpa_call (dev,VHOST_VDPA_GET_IOVA_RANGE , &range); +if (ret !=0 ) { +return ret; + } + + *first = range.first; + *last = range.last; +trace_vhost_vdpa_get_iova_range (dev, *first, *last); +return ret; +} +/** * Maps QEMU vaddr memory to device in a suitable way for shadow virtqueue: * - It always reference qemu memory address, not guest's memory.
@@ -881,6 +910,7 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx)static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev; +hwaddr iova_first, iova_last;unsigned n;int r;
@@ -894,7 +924,7 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* Allocate resources */ assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) { -VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n, v->iova_map);if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ;
@@ -903,6 +933,8 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } + r =vhost_vdpa_get_iova_range (hdev, &iova_first, &iova_last); +assert (r ==0 ); r =vhost_vdpa_vring_pause (hdev);assert (r ==0 );
@@ -913,6 +945,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } +memory_listener_unregister (&v->listener); +if (vhost_vdpa_dma_unmap (v, iova_first, + (iova_last - iova_first) &TARGET_PAGE_MASK )) { +error_report ("Fail to invalidate device iotlb" ); + } +/* Reset device so it can be configured */ r =vhost_vdpa_dev_start (hdev, false);assert (r ==0 );
hw/virtio/trace-events +1 -0
@@ -52,6 +52,7 @@ vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 0x%"PRIx64 vhost_vdpa_set_owner(void *dev) "dev: %p" vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64 +vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p first: 0x%"PRIx64" last: 0x%"PRIx64 # virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned out_num) "elem %p size %zd in_num %u out_num %u"
Signature · 1 line
2.27.0
Eugenio Pérez
maintainerRFC PATCH v4 18/20 vhost: Add VhostIOVATree
In reply to Eugenio PérezThis tree is able to look for a translated address from an IOVA address.
At first glance is similar to util/iova-tree. However, SVQ working on devices with limited IOVA space need more capabilities, like allocating IOVA chunks or perform reverse translations (qemu addresses to iova).
The allocation capability, as "assign a free IOVA address to this chunk of memory in qemu's address space" allows shadow virtqueue to create a new address space that is not restricted by guest's addressable one, so we can allocate shadow vqs vrings outside of its reachability, nor qemu's one. At the moment, the allocation is just done growing, not allowing deletion.
A different name could be used, but ordered searchable array is a little bit long though.
It duplicates the array so it can search efficiently both directions, and it will signal overlap if iova or the translated address is present in it's each array.
Use of array will be changed to util-iova-tree in future series.
--- hw/virtio/vhost-iova-tree.h | 40 +++++++ hw/virtio/vhost-iova-tree.c | 230 ++++++++++++++++++++++++++++++++++++ hw/virtio/meson.build | 2 +- 3 files changed, 271 insertions(+), 1 deletion(-) create mode 100644 hw/virtio/vhost-iova-tree.h create mode 100644 hw/virtio/vhost-iova-tree.c
hw/virtio/vhost-iova-tree.h new file +40 -0
@@ -0,0 +1,40 @@ +/* +* vhost software live migration ring +* +* SPDX-FileCopyrightText: Red Hat, Inc. 2021 +* SPDX-FileContributor: Author: Eugenio Pérez <[email protected]> +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef HW_VIRTIO_VHOST_IOVA_TREE_H +#define HW_VIRTIO_VHOST_IOVA_TREE_H + +#include "exec/memory.h" + +typedef struct VhostDMAMap { +void *translated_addr; +hwaddr iova; +hwaddr size;/* Inclusive */ +IOMMUAccessFlags perm; +}VhostDMAMap ; + +typedef enum VhostDMAMapNewRC { +VHOST_DMA_MAP_NO_SPACE =-3 , +VHOST_DMA_MAP_OVERLAP =-2 , +VHOST_DMA_MAP_INVALID =-1 , +VHOST_DMA_MAP_OK =0 , +}VhostDMAMapNewRC ; + +typedef struct VhostIOVATree VhostIOVATree ; + +VhostIOVATree *vhost_iova_tree_new (void ); +void vhost_iova_tree_unref (VhostIOVATree *iova_rm); +G_DEFINE_AUTOPTR_CLEANUP_FUNC (VhostIOVATree, vhost_iova_tree_unref); + +const VhostDMAMap *vhost_iova_tree_find_iova (const VhostIOVATree *iova_rm, +const VhostDMAMap *map); +VhostDMAMapNewRC vhost_iova_tree_alloc (VhostIOVATree *iova_rm, +VhostDMAMap *map); + +#endif
hw/virtio/vhost-iova-tree.c new file +230 -0
230 lines not on this page · raw message
hw/virtio/meson.build +1 -1
@@ -11,7 +11,7 @@ softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('vhost-stub.c')) virtio_ss = ss.source_set() virtio_ss.add(files('virtio.c')) -virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c', 'vhost-shadow-virtqueue.c')) +virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c', 'vhost-shadow-virtqueue.c', 'vhost-iova-tree.c')) virtio_ss.add(when: 'CONFIG_VHOST_USER', if_true: files('vhost-user.c')) virtio_ss.add(when: 'CONFIG_VHOST_VDPA', if_true: files('vhost-vdpa.c')) virtio_ss.add(when: 'CONFIG_VIRTIO_BALLOON', if_true: files('virtio-balloon.c'))
Signature · 1 line
2.27.0
Quoted 37 lines · Eugenio Pérez
This series enable shadow virtqueue (SVQ) for vhost-vdpa devices. This is intended as a new method of tracking the memory the devices touch during a migration process: Instead of relay on vhost device's dirty logging capability, SVQ intercepts the VQ dataplane forwarding the descriptors between VM and device. This way qemu is the effective writer of guests memory, like in qemu's virtio device operation.
When SVQ is enabled qemu offers a new vring to the device to read and write into, and also intercepts kicks and calls between the device and the guest. Used buffers relay would cause dirty memory being tracked, but at this RFC SVQ is not enabled on migration automatically.
It is based on the ideas of DPDK SW assisted LM, in the series of DPDK's https://patchwork.dpdk.org/cover/48370/ . However, these does not map the shadow vq in guest's VA, but in qemu's.
For qemu to use shadow virtqueues the guest virtio driver must not use features like
event_idxor indirect descriptors. These limitations will be addressed in later series, but they are left out for simplicity at the moment.SVQ needs to be enabled with QMP command:
{ "execute": "x-vhost-enable-shadow-vq",
"arguments": { "name": "dev0", "enable": true } }This series includes some patches to delete in the final version that helps with its testing. The first two of the series freely implements the feature to stop the device and be able to retrieve its status. It's intended to be used with
vp_vpdadriver in a nested environment. This driver also need modifications to forward the new status bit.Patches 2-8 prepares the SVQ and QMP command to support guest to host notifications forwarding. If the SVQ is enabled with these ones applied and the device supports it, that part can be tested in isolation (for example, with networking), hopping through SVQ.
Quoted 6 lines
Same thing is true with patches 9-13, but with device to guest notifications.
The rest of the patches implements the actual buffer forwarding.
Comments are welcome.
Hi Eugenio:
It would be helpful to have a public git repo for us to ease the review.
Thanks
Quoted 109 lines
TODO:
* Event, indirect, packed, and others features of virtio - Waiting for confirmation of the big picture.
* Use already available iova tree to track mappings.
* To sepparate buffers forwarding in its own AIO context, so we can throw more threads to that task and we don't need to stop the main event loop.
* unmap iommu memory. Now the tree can only grow from SVQ enable, but it should be fine as long as not a lot of memory is added to the guest.
* Rebase on top of latest qemu (and, hopefully, on top of multiqueue vdpa).
* Some assertions need to be appropiate error handling paths.
* Proper documentation.Changes from v3 RFC:
* Move everything to vhost-vdpa backend. A big change, this allowed some cleanup but more code has been added in other places.
* More use of glib utilities, especially to manage memory.
v3 link:
https://lists.nongnu.org/archive/html/qemu-devel/2021-05/msg06032.htmlChanges from v2 RFC:
* Adding vhost-vdpa devices support
* Fixed some memory leaks pointed by different comments
v2 link:
https://lists.nongnu.org/archive/html/qemu-devel/2021-03/msg05600.htmlChanges from v1 RFC:
* Use QMP instead of migration to start SVQ mode.
* Only accepting IOMMU devices, closer behavior with target devices(vDPA)* Fix invalid masking/unmasking of vhost call fd.
* Use of proper methods for synchronization.
* No need to modify VirtIO device code, all of the changes are contained in vhost code.
* Delete superfluous code.
* An intermediate RFC was sent with only the notifications forwarding changes. It can be seen in
https://patchew.org/QEMU/[email protected]/
v1 link:
https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05372.htmlEugenio Pérez (20):
virtio: AddVIRTIO_F_QUEUE_STATE
virtio-net: HonorVIRTIO_CONFIG_S_DEVICE_STOPPED
virtio: Addvirtio_queue_is_host_notifier_enabled
vhost: Make vhost_virtqueue_{start,stop} public
vhost: Add x-vhost-enable-shadow-vq qmp
vhost: Add VhostShadowVirtqueue
vdpa: Register vdpa devices in a list
vhost: Routeguest->hostnotification through shadow virtqueue Addvhost_svq_get_svq_call_notifier
Addvhost_svq_set_guest_call_notifier
vdpa: Savecall_fdin vhost-vdpa
vhost-vdpa: Take into account SVQ invhost_vdpa_set_vring_call
vhost: Routehost->guestnotification through shadow virtqueue
virtio: Addvhost_shadow_vq_get_vring_addr
vdpa: Save host and guest features
vhost: Addvhost_svq_valid_device_featuresto shadow vq
vhost: Shadow virtqueue buffers forwarding
vhost: Add VhostIOVATree
vhost: Use a tree to store memory mappings
vdpa: Add custom IOTLB translations to SVQEugenio Pérez (20):
virtio: AddVIRTIO_F_QUEUE_STATE
virtio-net: HonorVIRTIO_CONFIG_S_DEVICE_STOPPED
virtio: Addvirtio_queue_is_host_notifier_enabled
vhost: Make vhost_virtqueue_{start,stop} public
vhost: Add x-vhost-enable-shadow-vq qmp
vhost: Add VhostShadowVirtqueue
vdpa: Register vdpa devices in a list
vhost: Routeguest->hostnotification through shadow virtqueue
vdpa: Savecall_fdin vhost-vdpa
vhost-vdpa: Take into account SVQ invhost_vdpa_set_vring_call
vhost: Routehost->guestnotification through shadow virtqueue
virtio: Addvhost_shadow_vq_get_vring_addr
vdpa: Save host and guest features
vhost: Addvhost_svq_valid_device_featuresto shadow vq
vhost: Shadow virtqueue buffers forwarding
vhost: Check for deviceVRING_USED_F_NO_NOTIFYat shadow virtqueue
kick
vhost: UseVRING_AVAIL_F_NO_INTERRUPTat device call on shadow
virtqueue
vhost: Add VhostIOVATree
vhost: Use a tree to store memory mappings
vdpa: Add custom IOTLB translations to SVQqapi/net.json | 23 + hw/virtio/vhost-iova-tree.h | 40 ++ hw/virtio/vhost-shadow-virtqueue.h | 37 ++ hw/virtio/virtio-pci.h | 1 + include/hw/virtio/vhost-vdpa.h | 13 + include/hw/virtio/vhost.h | 4 + include/hw/virtio/virtio.h | 5 +- .../standard-headers/linux/virtio_config.h | 5 + include/standard-headers/linux/virtio_pci.h | 2 + hw/net/virtio-net.c | 6 +- hw/virtio/vhost-iova-tree.c | 230 +++++++ hw/virtio/vhost-shadow-virtqueue.c | 619 ++++++++++++++++++ hw/virtio/vhost-vdpa.c | 412 +++++++++++- hw/virtio/vhost.c | 12 +- hw/virtio/virtio-pci.c | 16 +- hw/virtio/virtio.c | 5 + hw/virtio/meson.build | 2 +- hw/virtio/trace-events | 1 + 18 files changed, 1413 insertions(+), 20 deletions(-)
Quoted 5 lines
create mode 100644
hw/virtio/vhost-iova-tree.hcreate mode 100644hw/virtio/vhost-shadow-virtqueue.hcreate mode 100644hw/virtio/vhost-iova-tree.ccreate mode 100644hw/virtio/vhost-shadow-virtqueue.c
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 48 lines · Jason Wang
Quoted 37 lines · Eugenio Pérez
This series enable shadow virtqueue (SVQ) for vhost-vdpa devices. This is intended as a new method of tracking the memory the devices touch during a migration process: Instead of relay on vhost device's dirty logging capability, SVQ intercepts the VQ dataplane forwarding the descriptors between VM and device. This way qemu is the effective writer of guests memory, like in qemu's virtio device operation.
When SVQ is enabled qemu offers a new vring to the device to read and write into, and also intercepts kicks and calls between the device and the guest. Used buffers relay would cause dirty memory being tracked, but at this RFC SVQ is not enabled on migration automatically.
It is based on the ideas of DPDK SW assisted LM, in the series of DPDK's https://patchwork.dpdk.org/cover/48370/ . However, these does not map the shadow vq in guest's VA, but in qemu's.
For qemu to use shadow virtqueues the guest virtio driver must not use features like
event_idxor indirect descriptors. These limitations will be addressed in later series, but they are left out for simplicity at the moment.SVQ needs to be enabled with QMP command:
{ "execute": "x-vhost-enable-shadow-vq",
"arguments": { "name": "dev0", "enable": true } }This series includes some patches to delete in the final version that helps with its testing. The first two of the series freely implements the feature to stop the device and be able to retrieve its status. It's intended to be used with
vp_vpdadriver in a nested environment. This driver also need modifications to forward the new status bit.Patches 2-8 prepares the SVQ and QMP command to support guest to host notifications forwarding. If the SVQ is enabled with these ones applied and the device supports it, that part can be tested in isolation (for example, with networking), hopping through SVQ.
Quoted 6 lines
Same thing is true with patches 9-13, but with device to guest notifications.
The rest of the patches implements the actual buffer forwarding.
Comments are welcome.
Quoted 7 lines
Hi Eugenio:
It would be helpful to have a public git repo for us to ease the review.
Thanks
Btw, we also need to measure the performance impact of the shadow virtqueue.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 05/20] vhost: Add x-vhost-enable-shadow-vq qmp
In reply to Eugenio PérezQuoted 13 lines · Eugenio Pérez
Command to enable shadow virtqueue.
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 23 +++++++++++++++++++++++ hw/virtio/vhost-vdpa.c | 8 ++++++++ 2 files changed, 31 insertions(+)qapi/net.json +0
-0
qapi/net.json line 87
Quoted 9 lines
qapi/net.json +6
-0@@ -79,6 +79,29 @@ { 'command': 'netdev_del', 'data': {'id': 'str'}, 'allow-preconfig': true } +## +# @x-vhost-enable-shadow-vq: +# +# Use vhost shadow virtqueue. +# +# @name: the device name of the VirtIO device
Is this a qdev ID? A network client name?
Quoted 7 lines
qapi/net.json +7
-0+# +# @enable: true to use the alternate shadow VQ notifications +# +# Returns: Always error, since SVQ is not implemented at the moment. +# +# Since: 6.2 +#
qapi/net.json line 104
Quoted 10 lines
qapi/net.json +10
-0+# Example: +# +# -> { "execute": "x-vhost-enable-shadow-vq", +# "arguments": { "name": "virtio-net", "enable": false } } +# +## +{ 'command': 'x-vhost-enable-shadow-vq', + 'data': {'name': 'str', 'enable': 'bool'}, + 'if': 'defined(CONFIG_VHOST_KERNEL)' } +
Adding an command just for controlling a flag in some object is fine for quick experiments. As a permanent interface, it's problematic: one command per flag would result in way too many commands. Better: one command to control a set of related properties.
I hesitate to suggest qom-set, because qom-set is not introspectable. Recurring complaint about QOM: poor integration with QAPI/QMP.
Naming nitpick: since the command can both enable and disable, I'd call it -set-vq instead of -enable-vq.
Quoted 29 lines
qapi/net.json +0
-0## # @NetLegacyNicOptions: #hw/virtio/vhost-vdpa.c +8
-0@@ -23,6 +23,8 @@#include "cpu.h" #include "trace.h" #include "qemu-common.h" +#include "qapi/qapi-commands-net.h" +#include "qapi/error.h" static bool vhost_vdpa_listener_skipped_section (MemoryRegionSection *section) {@@ -656,6 +658,12 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } + +void qmp_x_vhost_enable_shadow_vq (const char *name,bool enable,Error **errp) +{ +error_setg (errp,"Shadow virtqueue still not implemented" ); +} +const VhostOps vdpa_ops = { .backend_type =VHOST_BACKEND_TYPE_VDPA , .vhost_backend_init = vhost_vdpa_init,
Re: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue
In reply to Eugenio PérezQuoted 21 lines · Eugenio Pérez
Shadow virtqueue notifications forwarding is disabled when
vhost_devstops, so code flow follows usual cleanup.Also, host notifiers must be disabled at SVQ start, and they will not start if SVQ has been enabled when device is stopped. This is trivial to address, but it is left out for simplicity at this moment.
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.h | 8 ++ include/hw/virtio/vhost-vdpa.h | 4 + hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- 5 files changed, 264 insertions(+), 4 deletions(-)qapi/net.json +0
-0
qapi/net.json line 91
Quoted 5 lines
qapi/net.json +1
-1@@ -88,7 +88,7 @@ # # @enable: true to use the alternate shadow VQ notifications # -# Returns: Always error, since SVQ is not implemented at the moment. +# Returns: Error if failure, or 'no error' for success.
Delete the whole line, please.
Quoted 3 lines
qapi/net.json +0
-0# # Since: 6.2 #
[...]
Re: [RFC PATCH v4 15/20] vhost: Shadow virtqueue buffers forwarding
In reply to Eugenio PérezQuoted 41 lines · Eugenio Pérez
Initial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device.
For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series.
It reuses the VirtQueue code for the device part. The driver part is based on Linux's
virtio_ringdriver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones.SVQ uses
VIRTIO_CONFIG_S_DEVICE_STOPPEDto pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter.This status bit
VIRTIO_CONFIG_S_DEVICE_STOPPEDis currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits.Removal of
_S_DEVICE_STOPPEDbit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device.Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-)qapi/net.json +0
-0
qapi/net.json line 89
Quoted 5 lines
qapi/net.json +1
-1@@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding path
Uh, why does the flag change meaning half-way through this series?
Quoted 3 lines
qapi/net.json +0
-0# # Returns: Error if failure, or 'no error' for success. #
[...]
Eugenio Perez Martin
maintainerQuoted 50 lines · Jason Wang
Quoted 48 lines · Jason Wang
Quoted 43 lines · Eugenio Pérez
This series enable shadow virtqueue (SVQ) for vhost-vdpa devices. This is intended as a new method of tracking the memory the devices touch during a migration process: Instead of relay on vhost device's dirty logging capability, SVQ intercepts the VQ dataplane forwarding the descriptors between VM and device. This way qemu is the effective writer of guests memory, like in qemu's virtio device operation.
When SVQ is enabled qemu offers a new vring to the device to read and write into, and also intercepts kicks and calls between the device and the guest. Used buffers relay would cause dirty memory being tracked, but at this RFC SVQ is not enabled on migration automatically.
It is based on the ideas of DPDK SW assisted LM, in the series of DPDK's https://patchwork.dpdk.org/cover/48370/ . However, these does not map the shadow vq in guest's VA, but in qemu's.
For qemu to use shadow virtqueues the guest virtio driver must not use features like
event_idxor indirect descriptors. These limitations will be addressed in later series, but they are left out for simplicity at the moment.SVQ needs to be enabled with QMP command:
{ "execute": "x-vhost-enable-shadow-vq",
"arguments": { "name": "dev0", "enable": true } }This series includes some patches to delete in the final version that helps with its testing. The first two of the series freely implements the feature to stop the device and be able to retrieve its status. It's intended to be used with
vp_vpdadriver in a nested environment. This driver also need modifications to forward the new status bit.Patches 2-8 prepares the SVQ and QMP command to support guest to host notifications forwarding. If the SVQ is enabled with these ones applied and the device supports it, that part can be tested in isolation (for example, with networking), hopping through SVQ.
Same thing is true with patches 9-13, but with device to guest notifications.
The rest of the patches implements the actual buffer forwarding.
Comments are welcome.
Quoted 7 lines
Quoted 7 lines
Hi Eugenio:
It would be helpful to have a public git repo for us to ease the review.
Thanks
Hi Jason,
I just pushed this tag to
https://github.com/eugpermar/qemu/tree/vdpa_sw_live_migration.d/vdpa-v4
,
but let me know if you find another way more convenient.
Thanks!
Quoted 3 lines
Btw, we also need to measure the performance impact of the shadow virtqueue.
I will measure it in subsequent series, since I'm still making some changes. At the moment I'm also testing with nested virtualization that can affect it.
However we need to take into account that this series still has a lot of room for improvement. I would say that packed vq and isolating code in its own aio context could give a noticeable boost on the numbers.
Thanks!
Quoted 2 lines
Thanks
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 05/20] vhost: Add x-vhost-enable-shadow-vq qmp
In reply to Markus ArmbrusterQuoted 19 lines · Markus Armbruster
Quoted 16 lines · Eugenio Pérez
Command to enable shadow virtqueue.
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 23 +++++++++++++++++++++++ hw/virtio/vhost-vdpa.c | 8 ++++++++ 2 files changed, 31 insertions(+)qapi/net.json +0
-0@@ -79,6 +79,29 @@ { 'command': 'netdev_del', 'data': {'id': 'str'}, 'allow-preconfig': true }
Quoted 9 lines
qapi/net.json line 87
Quoted 6 lines
qapi/net.json +6
-0+## +# @x-vhost-enable-shadow-vq: +# +# Use vhost shadow virtqueue. +# +# @name: the device name of the VirtIO deviceIs this a qdev ID? A network client name?
At this moment is the virtio device name, the one specified at the call of "virtio_init". But this should change, maybe the qdev id or something that can be provided by the command line fits better here.
Quoted 23 lines
Quoted 7 lines
qapi/net.json +7
-0+# +# @enable: true to use the alternate shadow VQ notifications +# +# Returns: Always error, since SVQ is not implemented at the moment. +# +# Since: 6.2 +#qapi/net.json line 104
Quoted 10 lines
qapi/net.json +10
-0+# Example: +# +# -> { "execute": "x-vhost-enable-shadow-vq", +# "arguments": { "name": "virtio-net", "enable": false } } +# +## +{ 'command': 'x-vhost-enable-shadow-vq', + 'data': {'name': 'str', 'enable': 'bool'}, + 'if': 'defined(CONFIG_VHOST_KERNEL)' } +Adding an command just for controlling a flag in some object is fine for quick experiments. As a permanent interface, it's problematic: one command per flag would result in way too many commands. Better: one command to control a set of related properties.
Quoted 3 lines
I hesitate to suggest qom-set, because qom-set is not introspectable. Recurring complaint about QOM: poor integration with QAPI/QMP.
I will take it into account, but it's only temporary, that's why it has the x- prefix. It's not like event_idx or other device feature
flags: Every vDPA device can potentially use SVQ datapath in a transparent way, neither the vDPA device nor the guest know that qemu supports it.
Ideally, this mode will kick in at the migration time automatically, no need to perform more actions.
Quoted 3 lines
Naming nitpick: since the command can both enable and disable, I'd call it -set-vq instead of -enable-vq.
Got it, I will replace it.
Thanks!
Quoted 30 lines
Quoted 29 lines
qapi/net.json +0
-0## # @NetLegacyNicOptions: #hw/virtio/vhost-vdpa.c +8
-0@@ -23,6 +23,8 @@#include "cpu.h" #include "trace.h" #include "qemu-common.h" +#include "qapi/qapi-commands-net.h" +#include "qapi/error.h" static bool vhost_vdpa_listener_skipped_section (MemoryRegionSection *section) {@@ -656,6 +658,12 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } + +void qmp_x_vhost_enable_shadow_vq (const char *name,bool enable,Error **errp) +{ +error_setg (errp,"Shadow virtqueue still not implemented" ); +} +const VhostOps vdpa_ops = { .backend_type =VHOST_BACKEND_TYPE_VDPA , .vhost_backend_init = vhost_vdpa_init,
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue
In reply to Markus ArmbrusterQuoted 19 lines · Markus Armbruster
Quoted 16 lines · Eugenio Pérez
Shadow virtqueue notifications forwarding is disabled when
vhost_devstops, so code flow follows usual cleanup.Also, host notifiers must be disabled at SVQ start, and they will not start if SVQ has been enabled when device is stopped. This is trivial to address, but it is left out for simplicity at this moment.
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.h | 8 ++ include/hw/virtio/vhost-vdpa.h | 4 + hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- 5 files changed, 264 insertions(+), 4 deletions(-)
Quoted 13 lines
qapi/net.json line 91
Quoted 10 lines
qapi/net.json +1
-1@@ -88,7 +88,7 @@ # # @enable: true to use the alternate shadow VQ notifications # -# Returns: Always error, since SVQ is not implemented at the moment. +# Returns: Error if failure, or 'no error' for success.Delete the whole line, please.
I will do it.
Thanks!
Quoted 5 lines
Quoted 3 lines
qapi/net.json +0
-0# # Since: 6.2 #[...]
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 15/20] vhost: Shadow virtqueue buffers forwarding
In reply to Markus ArmbrusterQuoted 39 lines · Markus Armbruster
Quoted 36 lines · Eugenio Pérez
Initial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device.
For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series.
It reuses the VirtQueue code for the device part. The driver part is based on Linux's
virtio_ringdriver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones.SVQ uses
VIRTIO_CONFIG_S_DEVICE_STOPPEDto pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter.This status bit
VIRTIO_CONFIG_S_DEVICE_STOPPEDis currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits.Removal of
_S_DEVICE_STOPPEDbit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device.Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-)
Quoted 13 lines
qapi/net.json line 89
Quoted 10 lines
qapi/net.json +1
-1@@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding pathUh, why does the flag change meaning half-way through this series?
Before this patch, the SVQ mode just makes an extra hop for notifications. Guest ones are now received by qemu via ioeventfd, and qemu forwards them to the device using a different eventfd. The reverse is also true: the device ones will be received by qemu by device call fd, and then qemu will forward them to the guest using a different irqfd.
This intermediate step is not very useful by itself, but helps for checking that that part of the communication works fine, with no need for shadow virtqueue to understand vring format. Doing that way also produces smaller patches.
So it makes sense to me to tell what QMP command does exactly at every point of the series. However I can directly document it as "use the alternate shadow VQ buffers forwarding path" from the beginning.
Does this make sense, or will it be better to write the final intention of the command?
Thanks!
Quoted 6 lines
Quoted 3 lines
qapi/net.json +0
-0# # Returns: Error if failure, or 'no error' for success. #[...]
Re: [RFC PATCH v4 05/20] vhost: Add x-vhost-enable-shadow-vq qmp
In reply to Eugenio Perez MartinQuoted 29 lines · Eugenio Perez Martin
Quoted 19 lines · Markus Armbruster
Quoted 16 lines · Eugenio Pérez
Command to enable shadow virtqueue.
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 23 +++++++++++++++++++++++ hw/virtio/vhost-vdpa.c | 8 ++++++++ 2 files changed, 31 insertions(+)qapi/net.json +0
-0@@ -79,6 +79,29 @@ { 'command': 'netdev_del', 'data': {'id': 'str'}, 'allow-preconfig': true }Quoted 8 lines
qapi/net.json line 87
Quoted 6 lines
qapi/net.json +6
-0+## +# @x-vhost-enable-shadow-vq: +# +# Use vhost shadow virtqueue. +# +# @name: the device name of the VirtIO deviceIs this a qdev ID? A network client name?
Quoted 3 lines
At this moment is the virtio device name, the one specified at the call of "
virtio_init". But this should change, maybe the qdev id or something that can be provided by the command line fits better here.
To refer to a device backend, we commonly use a backend-specific ID. For network backends, that's NetClientState member name.
To refer to a device frontend, we commonly use a qdev ID or a QOM path.
[...]
Re: [RFC PATCH v4 15/20] vhost: Shadow virtqueue buffers forwarding
In reply to Eugenio Perez MartinQuoted 66 lines · Eugenio Perez Martin
Quoted 39 lines · Markus Armbruster
Quoted 36 lines · Eugenio Pérez
Initial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device.
For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series.
It reuses the VirtQueue code for the device part. The driver part is based on Linux's
virtio_ringdriver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones.SVQ uses
VIRTIO_CONFIG_S_DEVICE_STOPPEDto pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter.This status bit
VIRTIO_CONFIG_S_DEVICE_STOPPEDis currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits.Removal of
_S_DEVICE_STOPPEDbit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device.Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-)Quoted 13 lines
qapi/net.json line 89
Quoted 10 lines
qapi/net.json +1
-1@@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding pathUh, why does the flag change meaning half-way through this series?
Before this patch, the SVQ mode just makes an extra hop for notifications. Guest ones are now received by qemu via ioeventfd, and qemu forwards them to the device using a different eventfd. The reverse is also true: the device ones will be received by qemu by device call fd, and then qemu will forward them to the guest using a different irqfd.
This intermediate step is not very useful by itself, but helps for checking that that part of the communication works fine, with no need for shadow virtqueue to understand vring format. Doing that way also produces smaller patches.
Quoted 8 lines
So it makes sense to me to tell what QMP command does exactly at every point of the series. However I can directly document it as "use the alternate shadow VQ buffers forwarding path" from the beginning.
Does this make sense, or will it be better to write the final intention of the command?
Thanks!
Working your explanation into commit messages and possibly comments should do.
Re: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue
In reply to Eugenio PérezQuoted 4 lines · Eugenio Pérez
Shadow virtqueue notifications forwarding is disabled when
vhost_devstops, so code flow follows usual cleanup.Also, host notifiers must be disabled at SVQ start,
Any reason for this?
Quoted 3 lines
and they will not
start if SVQ has been enabled when device is stopped. This is trivial to address, but it is left out for simplicity at this moment.
It looks to me this patch also contains the following logics
1) codes to enable svq
2) codes to let svq to be enabled from QMP.
I think they need to be split out, we may endup with the following series of patches
1) svq skeleton with enable/disable
2) route host notifier to svq
3) route guest notifier to svq
4) codes to enable svq
5) enable svq via QMP
Quoted 28 lines
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.h | 8 ++ include/hw/virtio/vhost-vdpa.h | 4 + hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- 5 files changed, 264 insertions(+), 4 deletions(-)qapi/net.json +1
-1@@ -88,7 +88,7 @@ # # @enable: true to use the alternate shadow VQ notifications # -# Returns: Always error, since SVQ is not implemented at the moment. +# Returns: Error if failure, or 'no error' for success. # # Since: 6.2 #hw/virtio/vhost-shadow-virtqueue.h +0
-0
hw/virtio/vhost-shadow-virtqueue.h line 17
Quoted 4 lines
hw/virtio/vhost-shadow-virtqueue.h +1
-0@@ -14,6 +14,14 @@typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ; +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq);
Let's move this function to another patch since it's unrelated to the guest->host routing.
Quoted 57 lines
hw/virtio/vhost-shadow-virtqueue.h +7
-0+void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); + +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx);void vhost_svq_free (VhostShadowVirtqueue *vq);include/hw/virtio/vhost-vdpa.h +4
-0@@ -12,6 +12,8 @@#ifndef HW_VIRTIO_VHOST_VDPA_H #define HW_VIRTIO_VHOST_VDPA_H +#include <gmodule.h> +#include "qemu/queue.h" #include "hw/virtio/virtio.h" @@ -24,6 +26,8 @@ typedef struct vhost_vdpa {int device_fd;uint32_t msg_type;MemoryListener listener; +bool shadow_vqs_enabled; +GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry;VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ];hw/virtio/vhost-shadow-virtqueue.c +7
-1@@ -9,9 +9,12 @@#include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" +#include "hw/virtio/vhost.h" + +#include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" -#include "qemu/event_notifier.h" +#include "qemu/main-loop.h" /* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue {@@ -19,14 +22,146 @@ typedef struct VhostShadowVirtqueue {EventNotifier kick_notifier;/* Shadow call notifier, sent to vhost */ EventNotifier call_notifier; + +/* +* Borrowed virtqueue's guest to host notifier.
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 37
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+* To borrow it in this event notifier allows to register on the event +* loop and access the associated shadow virtqueue easily. If we use the +* VirtQueue, we don't have an easy way to retrieve it. +* +* So shadow virtqueue must not clean it, or we would lose VirtQueue one. +*/ +EventNotifier host_notifier; + +/* Guest's call notifier, where SVQ calls guest. */ +EventNotifier guest_call_notifier;
To be consistent, let's simply use "guest_notifier" here.
Quoted 4 lines
hw/virtio/vhost-shadow-virtqueue.c +3
-0+ +/* Virtio queue shadowing */ +VirtQueue *vq; } VhostShadowVirtqueue;
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 51
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +9
-0+/* Forward guest notifications */ +static void vhost_handle_guest_kick (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + host_notifier); + +if (unlikely (!event_notifier_test_and_clear (n))) { +return ; + }
Is there a chance that we may stop the processing of available buffers during the svq enabling? There could be no kick from the guest in this case.
Quoted 34 lines
hw/virtio/vhost-shadow-virtqueue.c +34
-0+ +event_notifier_set (&svq->kick_notifier); +} + +/* +* Obtain the SVQ call notifier, where vhost device notifies SVQ that there +* exists pending used buffers. +* +* @svq Shadow Virtqueue +*/ +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq) +{ +return &svq->call_notifier; +} + +/* +* Set the call notifier for the SVQ to call the guest +* +* @svq Shadow virtqueue +* @call_fd call notifier +* +* Called on BQL context. +*/ +void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd) +{ +event_notifier_init_fd (&svq->guest_call_notifier, call_fd); +} + +/* +* Restore the vhost guest to host notifier, i.e., disables svq effect. +*/ +static int vhost_svq_restore_vdev_host_notifier (struct vhost_dev *dev, +unsigned vhost_index, +VhostShadowVirtqueue *svq)
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 95
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = vhost_index, + .fd =event_notifier_get_fd (vq_host_notifier), + }; +int r; + +/* Restore vhost kick */ + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);
And remap the notification area if necessary.
Quoted 17 lines
hw/virtio/vhost-shadow-virtqueue.c +17
-0+return r ? -errno :0 ; +} + +/* +* Start shadow virtqueue operation. +* @dev vhost device +* @hidx vhost virtqueue index +* @svq Shadow Virtqueue +*/ +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = dev->vhost_ops->vhost_get_vq_index (dev, dev->vq_index + idx), + .fd =event_notifier_get_fd (&svq->kick_notifier), + };
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 122
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+int r; + +/* Check that notifications are still going directly to vhost dev */ +assert (virtio_queue_is_host_notifier_enabled (svq->vq)); + +/* +* event_notifier_set_handler already checks for guest's notifications if +* they arrive in the switch, so there is no need to explicitely check for +* them. +*/
If this is true, shouldn't we call vhost_set_vring_kick() before the event_notifier_set_handler()?
Btw, I think we should update the fd if set_vring_kick() was called after this function?
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 127
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+event_notifier_init_fd (&svq->host_notifier, +event_notifier_get_fd (vq_host_notifier)); +event_notifier_set_handler (&svq->host_notifier, vhost_handle_guest_kick); + + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);
And we need to stop the notification area mmap.
Quoted 112 lines
hw/virtio/vhost-shadow-virtqueue.c +32
-0+if (unlikely (r !=0 )) { +error_report ("Couldn't set kick fd: %s" ,strerror (errno)); + goto err_set_vring_kick; + } + +return true; + +err_set_vring_kick: +event_notifier_set_handler (&svq->host_notifier,NULL ); + +return false; +} + +/* +* Stop shadow virtqueue operation. +* @dev vhost device +* @idx vhost queue index +* @svq Shadow Virtqueue +*/ +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +int r =vhost_svq_restore_vdev_host_notifier (dev, idx, svq); +if (unlikely (r <0 )) { +error_report ("Couldn't restore vq kick fd: %s" ,strerror (-r)); + } + +event_notifier_set_handler (&svq->host_notifier,NULL ); +} +/* * Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) { +int vq_idx = dev->vq_index + idx; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);int r;@@ -44,6 +179,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) goto err_init_call_notifier; } + svq->vq =virtio_get_queue (dev->vdev, vq_idx);return g_steal_pointer (&svq); err_init_call_notifier:hw/virtio/vhost-vdpa.c +24
-0@@ -17,6 +17,7 @@#include "hw/virtio/vhost.h" #include "hw/virtio/vhost-backend.h" #include "hw/virtio/virtio-net.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" @@ -272,6 +273,16 @@ static void vhost_vdpa_add_status(struct vhost_dev *dev, uint8_t status)vhost_vdpa_call (dev,VHOST_VDPA_SET_STATUS , &s); } +/** +* Adaptor function to free shadow virtqueue through gpointer +* +* @svq The Shadow Virtqueue +*/ +static void vhost_psvq_free (gpointer svq) +{ +vhost_svq_free (svq); +} +static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp) {struct vhost_vdpa *v;@@ -283,6 +294,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; + v->shadow_vqs =g_ptr_array_new_full (dev->nvqs, vhost_psvq_free);QLIST_INSERT_HEAD (&vhost_vdpa_devices, v, entry); vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |@@ -373,6 +385,17 @@ err:return ; } +static void vhost_vdpa_svq_cleanup (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +size_t idx; + +for (idx =0 ; idx < v->shadow_vqs->len; ++idx) { +vhost_svq_stop (dev, idx,g_ptr_array_index (v->shadow_vqs, idx)); + } +g_ptr_array_free (v->shadow_vqs, true); +} +static int vhost_vdpa_cleanup (struct vhost_dev *dev) {struct vhost_vdpa *v;@@ -381,6 +404,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev)trace_vhost_vdpa_cleanup (dev, v);vhost_vdpa_host_notifiers_uninit (dev, dev->nvqs);memory_listener_unregister (&v->listener); +vhost_vdpa_svq_cleanup (dev);QLIST_REMOVE (v, entry); dev->opaque =NULL ;
hw/virtio/vhost-vdpa.c static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) · line 586
Quoted 7 lines
hw/virtio/vhost-vdpa.c +3
-1@@ -557,7 +581,9 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started)if (started) {uint8_t status =0 ;memory_listener_register (&v->listener, &address_space_memory); -vhost_vdpa_host_notifiers_init (dev); +if (!v->shadow_vqs_enabled) { +vhost_vdpa_host_notifiers_init (dev); + }
This looks like a trick, why not check and setup shadow_vqs inside:
1) vhost_vdpa_host_notifiers_init()
and
2) vhost_vdpa_set_vring_kick()
Quoted 37 lines
hw/virtio/vhost-vdpa.c +30
-0vhost_vdpa_set_vring_ready (dev);vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_DRIVER_OK );vhost_vdpa_call (dev,VHOST_VDPA_GET_STATUS , &status);@@ -663,10 +689,96 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +/* +* Start shadow virtqueue. +*/ +static bool vhost_vdpa_svq_start_vq (struct vhost_dev *dev,unsigned idx) +{ +struct vhost_vdpa *v = dev->opaque; +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); +return vhost_svq_start (dev, idx, svq); +} + +static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) +{ +struct vhost_dev *hdev = v->dev; +unsigned n; + +if (enable == v->shadow_vqs_enabled) { +return hdev->nvqs; + } + +if (enable) { +/* Allocate resources */ +assert (v->shadow_vqs->len ==0 ); +for (n =0 ; n < hdev->nvqs; ++n) { +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +bool ok; + +if (unlikely (!svq)) { +g_ptr_array_set_size (v->shadow_vqs,0 ); +return 0 ; + }
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 731
Quoted 10 lines
hw/virtio/vhost-vdpa.c +10
-0+g_ptr_array_add (v->shadow_vqs, svq); + + ok =vhost_vdpa_svq_start_vq (hdev, n); +if (unlikely (!ok)) { +/* Free still not started svqs */ +g_ptr_array_set_size (v->shadow_vqs, n); + enable = false; +break ; + } + }
Since there's almost no logic could be shared between enable and disable. Let's split those logic out into dedicated functions where the codes looks more easy to be reviewed (e.g have a better error handling etc).
Quoted 22 lines
hw/virtio/vhost-vdpa.c +19
-0+ } + + v->shadow_vqs_enabled = enable; + +if (!enable) { +/* Disable all queues or clean up failed start */ +for (n =0 ; n < v->shadow_vqs->len; ++n) { +unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n); +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n); +vhost_svq_stop (hdev, n, svq); +vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); + } + +/* Resources cleanup */ +g_ptr_array_set_size (v->shadow_vqs,0 ); + } + +return n; +}void qmp_x_vhost_enable_shadow_vq (const char *name,bool enable,Error **errp) {
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 762
Quoted 10 lines
hw/virtio/vhost-vdpa.c +9
-1-error_setg (errp,"Shadow virtqueue still not implemented" ); +struct vhost_vdpa *v; +const char *err_cause =NULL ; +bool r; + + QLIST_FOREACH(v, &vhost_vdpa_devices, entry) { +if (v->dev->vdev &&0 ==strcmp (v->dev->vdev->name, name)) { +break ; + } + }
I think you can iterate the NetClientStates to ge tthe vhost-vdpa backends.
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 768
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+ +if (!v) { + err_cause ="Device not found" ; + goto err; + }else if (v->notifier[0 ].addr) { + err_cause ="Device has host notifiers enabled" ;
I don't get this.
Btw this function should be implemented in an independent patch after svq is fully functional.
Thanks
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+ goto err; + } + + r =vhost_vdpa_enable_svq (v, enable); +if (unlikely (!r)) { + err_cause ="Error enabling (see monitor)" ;
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 784
Quoted 10 lines
hw/virtio/vhost-vdpa.c +7
-0+ goto err; + } + +err: +if (err_cause) { +error_setg (errp,"Can't enable shadow vq on %s: %s" , name, err_cause); + } }const VhostOps vdpa_ops = {
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 09/20] vdpa: Save call_fd in vhost-vdpa
In reply to Eugenio PérezQuoted 13 lines · Eugenio Pérez
We need to know it to switch to Shadow VirtQueue.
Signed-off-by Eugenio Pérez <[email protected]>--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 5 +++++ 2 files changed, 7 insertions(+)include/hw/virtio/vhost-vdpa.h +0
-0
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 34
Quoted 5 lines
include/hw/virtio/vhost-vdpa.h +2
-0@@ -30,6 +30,8 @@ typedef struct vhost_vdpa {GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry; +/* File descriptor the device uses to call VM/SVQ */ +int call_fd[VIRTIO_QUEUE_MAX ];
Any reason we don't do this for kick_fd or why virtio_queue_get_guest_notifier() can't work here? Need a comment or commit log.
I think we need to have a consistent way to handle both kick and call fd.
Thanks
Quoted 10 lines
include/hw/virtio/vhost-vdpa.h +0
-0VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ]; } VhostVDPA;hw/virtio/vhost-vdpa.c +0
-0@@ -652,7 +652,12 @@ static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev,static int vhost_vdpa_set_vring_call (struct vhost_dev *dev,struct vhost_vring_file *file)
hw/virtio/vhost-vdpa.c static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev, · line 663
Quoted 10 lines
hw/virtio/vhost-vdpa.c +5
-0{ +struct vhost_vdpa *v = dev->opaque; +int vdpa_idx =vhost_vdpa_get_vq_index (dev, file->index); +trace_vhost_vdpa_set_vring_call (dev, file->index, file->fd); + + v->call_fd[vdpa_idx] = file->fd;return vhost_vdpa_call (dev,VHOST_SET_VRING_CALL , file); }
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 10/20] vhost-vdpa: Take into account SVQ in vhost_vdpa_set_vring_call
In reply to Eugenio PérezQuoted 27 lines · Eugenio Pérez
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)hw/virtio/vhost-vdpa.c +7
-1@@ -649,16 +649,27 @@ static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev,return vhost_vdpa_call (dev,VHOST_SET_VRING_KICK , file); } +static int vhost_vdpa_set_vring_dev_call (struct vhost_dev *dev, +struct vhost_vring_file *file) +{ +trace_vhost_vdpa_set_vring_call (dev, file->index, file->fd); +return vhost_vdpa_call (dev,VHOST_SET_VRING_CALL , file); +} +static int vhost_vdpa_set_vring_call (struct vhost_dev *dev,struct vhost_vring_file *file) {struct vhost_vdpa *v = dev->opaque;int vdpa_idx =vhost_vdpa_get_vq_index (dev, file->index); -trace_vhost_vdpa_set_vring_call (dev, file->index, file->fd);
hw/virtio/vhost-vdpa.c static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev, · line 672
Quoted 10 lines
hw/virtio/vhost-vdpa.c +7
-2- v->call_fd[vdpa_idx] = file->fd; -return vhost_vdpa_call (dev,VHOST_SET_VRING_CALL , file); +if (v->shadow_vqs_enabled) { +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, vdpa_idx); +vhost_svq_set_guest_call_notifier (svq, file->fd); +return 0 ; + }else { +return vhost_vdpa_set_vring_dev_call (dev, file); + }
I feel like we should do the same for kick fd.
Thanks
hw/virtio/vhost-vdpa.c static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev, · line 675
Quoted 3 lines
hw/virtio/vhost-vdpa.c +0
-0} static int vhost_vdpa_get_features(struct vhost_dev *dev,
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue
In reply to Eugenio PérezQuoted 72 lines · Eugenio Pérez
This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of
vhost_virtqueue_startcan unmasks and discard call events,vhost_virtqueue_startshould be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.c | 19 +++++++++++++++ hw/virtio/vhost-vdpa.c | 38 +++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +19
-0@@ -53,6 +53,22 @@ static void vhost_handle_guest_kick(EventNotifier *n)event_notifier_set (&svq->kick_notifier); } +/* Forward vhost notifications */ +static void vhost_svq_handle_call_no_test (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + call_notifier); + +event_notifier_set (&svq->guest_call_notifier); +} + +static void vhost_svq_handle_call (EventNotifier *n) +{ +if (likely (event_notifier_test_and_clear (n))) { +vhost_svq_handle_call_no_test (n); + } +} +/* * Obtain the SVQ call notifier, where vhost device notifies SVQ that there * exists pending used buffers. @@ -180,6 +196,8 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) } svq->vq =virtio_get_queue (dev->vdev, vq_idx); +event_notifier_set_handler (&svq->call_notifier, + vhost_svq_handle_call);return g_steal_pointer (&svq); err_init_call_notifier:@@ -195,6 +213,7 @@ err_init_kick_notifier:void vhost_svq_free (VhostShadowVirtqueue *vq) {event_notifier_cleanup (&vq->kick_notifier); +event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier);g_free (vq); }hw/virtio/vhost-vdpa.c +0
-1@@ -712,13 +712,40 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) {struct vhost_vdpa *v = dev->opaque;VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); -return vhost_svq_start (dev, idx, svq);
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 724
Quoted 10 lines
hw/virtio/vhost-vdpa.c +10
-0+EventNotifier *vhost_call_notifier =vhost_svq_get_svq_call_notifier (svq); +struct vhost_vring_file vhost_call_file = { + .index = idx + dev->vq_index, + .fd =event_notifier_get_fd (vhost_call_notifier), + }; +int r; +bool b; + +/* Set shadow vq -> guest notifier */ +assert (v->call_fd[idx]);
We need aovid the asser() here. On which case we can hit this?
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 737
Quoted 13 lines
hw/virtio/vhost-vdpa.c +13
-0+vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]); + + b =vhost_svq_start (dev, idx, svq); +if (unlikely (!b)) { +return false; + } + +/* Set device -> SVQ notifier */ + r =vhost_vdpa_set_vring_dev_call (dev, &vhost_call_file); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_call for shadow vq failed" ); +return false; + }
Similar to kick, do we need to set_vring_call() before vhost_svq_start()?
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 740
Quoted 3 lines
hw/virtio/vhost-vdpa.c +3
-0+ +/* Check for pending calls */ +event_notifier_set (vhost_call_notifier);
Interesting, can this result spurious interrupt?
Quoted 17 lines
hw/virtio/vhost-vdpa.c +4
-0+return true; }static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev;unsigned n; +int r;if (enable == v->shadow_vqs_enabled) {return hdev->nvqs;@@ -752,9 +779,18 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)if (!enable) {/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { +struct vhost_vring_file file = { + .index = vhost_vdpa_get_vq_index(hdev, n),
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 793
Quoted 10 lines
hw/virtio/vhost-vdpa.c +7
-0+ .fd = v->call_fd[n], + }; + + r =vhost_vdpa_set_vring_call (hdev, &file); +assert (r ==0 ); +unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n);VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n);vhost_svq_stop (hdev, n, svq); +/* TODO: This can unmask or override call fd! */
I don't get this comment. Does this mean the current code can't work with mask_notifiers? If yes, this is something we need to fix.
Thanks
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 796
Quoted 3 lines
hw/virtio/vhost-vdpa.c +0
-0vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); }
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue
In reply to Eugenio PérezQuoted 12 lines · Eugenio Pérez
This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of
vhost_virtqueue_startcan unmasks and discard call events,vhost_virtqueue_startshould be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.
Btw, the log looks not clear. I guess this patch goes for method 3. If yes, we need explain it and why.
Thanks
Quoted 2 lines
Signed-off-by Eugenio Pérez<[email protected]>
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 12/20] virtio: Add vhost_shadow_vq_get_vring_addr
In reply to Eugenio PérezQuoted 1 line · Eugenio Pérez
It reports the shadow virtqueue address from qemu virtual address space
I think both the title and commit log needs to more tweaks. Looking at the codes, what id does is actually introduce vring into svq.
Quoted 60 lines
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +++ hw/virtio/vhost-shadow-virtqueue.c | 50 ++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+)hw/virtio/vhost-shadow-virtqueue.h +4
-0@@ -16,6 +16,10 @@ typedef struct VhostShadowVirtqueue VhostShadowVirtqueue;EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq);void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); +void vhost_svq_get_vring_addr (const VhostShadowVirtqueue *svq, +struct vhost_vring_addr *addr); +size_t vhost_svq_driver_area_size (const VhostShadowVirtqueue *svq); +size_t vhost_svq_device_area_size (const VhostShadowVirtqueue *svq);bool vhost_svq_start (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq);hw/virtio/vhost-shadow-virtqueue.c +16
-0@@ -18,6 +18,9 @@/* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { + /* Shadow vring */ + struct vring vring; + /* Shadow kick notifier, sent to vhost */ EventNotifier kick_notifier; /* Shadow call notifier, sent to vhost */@@ -38,6 +41,9 @@ typedef struct VhostShadowVirtqueue {/* Virtio queue shadowing */ VirtQueue *vq; + +/* Virtio device */ +VirtIODevice *vdev; } VhostShadowVirtqueue;/* Forward guest notifications */ @@ -93,6 +99,35 @@ void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call_fd)event_notifier_init_fd (&svq->guest_call_notifier, call_fd); } +/* +* Get the shadow vq vring address. +* @svq Shadow virtqueue +* @addr Destination to store address +*/ +void vhost_svq_get_vring_addr (const VhostShadowVirtqueue *svq, +struct vhost_vring_addr *addr) +{ + addr->desc_user_addr = (uint64_t )svq->vring.desc; + addr->avail_user_addr = (uint64_t )svq->vring.avail;
hw/virtio/vhost-shadow-virtqueue.c void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call_fd) · line 121
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+ addr->used_user_addr = (uint64_t )svq->vring.used; +} + +size_t vhost_svq_driver_area_size (const VhostShadowVirtqueue *svq) +{ +uint16_t vq_idx =virtio_get_queue_index (svq->vq); +size_t desc_size =virtio_queue_get_desc_size (svq->vdev, vq_idx); +size_t avail_size =virtio_queue_get_avail_size (svq->vdev, vq_idx); + +return ROUND_UP (desc_size + avail_size, qemu_real_host_page_size);
Is this round up required by the spec?
Quoted 24 lines
hw/virtio/vhost-shadow-virtqueue.c +13
-0+} + +size_t vhost_svq_device_area_size (const VhostShadowVirtqueue *svq) +{ +uint16_t vq_idx =virtio_get_queue_index (svq->vq); +size_t used_size =virtio_queue_get_used_size (svq->vdev, vq_idx); +return ROUND_UP (used_size, qemu_real_host_page_size); +} +/* * Restore the vhost guest to host notifier, i.e., disables svq effect. */ @@ -178,6 +213,10 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) {int vq_idx = dev->vq_index + idx; +unsigned num =virtio_queue_get_num (dev->vdev, vq_idx); +size_t desc_size =virtio_queue_get_desc_size (dev->vdev, vq_idx); +size_t driver_size; +size_t device_size; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);int r;
hw/virtio/vhost-shadow-virtqueue.c VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) · line 244
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +7
-0@@ -196,6 +235,15 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) } svq->vq =virtio_get_queue (dev->vdev, vq_idx); + svq->vdev = dev->vdev; + driver_size =vhost_svq_driver_area_size (svq); + device_size =vhost_svq_device_area_size (svq); + svq->vring.num = num; + svq->vring.desc =qemu_memalign (qemu_real_host_page_size, driver_size); + svq->vring.avail = (void *)((char *)svq->vring.desc + desc_size); +memset (svq->vring.desc,0 , driver_size);
Any reason for using the contiguous area for both desc and avail?
Thanks
hw/virtio/vhost-shadow-virtqueue.c void vhost_svq_free(VhostShadowVirtqueue *vq) · line 269
Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c +4
-0+ svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size); +memset (svq->vring.used,0 , device_size);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);@@ -215,5 +263,7 @@ void vhost_svq_free(VhostShadowVirtqueue *vq)event_notifier_cleanup (&vq->kick_notifier);event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier); +qemu_vfree (vq->vring.desc); +qemu_vfree (vq->vring.used);g_free (vq); }
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 13/20] vdpa: Save host and guest features
In reply to Eugenio PérezQuoted 15 lines · Eugenio Pérez
Those are needed for SVQ: Host ones are needed to check if SVQ knows how to talk with the device and for feature negotiation, and guest ones to know if SVQ can talk with it.
Signed-off-by Eugenio Pérez <[email protected]>--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 31 ++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-)include/hw/virtio/vhost-vdpa.h +0
-0
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 30
Quoted 5 lines
include/hw/virtio/vhost-vdpa.h +2
-0@@ -26,6 +26,8 @@ typedef struct vhost_vdpa {int device_fd;uint32_t msg_type;MemoryListener listener; +uint64_t host_features; +uint64_t guest_features;
Any reason that we can't use the features stored in VirtioDevice?
Thanks
Quoted 56 lines
include/hw/virtio/vhost-vdpa.h +0
-0bool shadow_vqs_enabled;GPtrArray *shadow_vqs;struct vhost_dev *dev;hw/virtio/vhost-vdpa.c +23
-3@@ -439,10 +439,19 @@ static int vhost_vdpa_set_mem_table(struct vhost_dev *dev,return 0 ; } -static int vhost_vdpa_set_features (struct vhost_dev *dev, -uint64_t features) +/** +* Internal set_features() that follows vhost/VirtIO protocol for that +*/ +static int vhost_vdpa_backend_set_features (struct vhost_dev *dev, +uint64_t features) { +struct vhost_vdpa *v = dev->opaque; +int ret; +if (v->host_features &BIT_ULL (VIRTIO_F_QUEUE_STATE )) { + features |=BIT_ULL (VIRTIO_F_QUEUE_STATE ); + } +trace_vhost_vdpa_set_features (dev, features); ret =vhost_vdpa_call (dev,VHOST_SET_FEATURES , &features);uint8_t status =0 ;@@ -455,6 +464,17 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,return !(status &VIRTIO_CONFIG_S_FEATURES_OK ); } +/** +* Exposed vhost set features +*/ +static int vhost_vdpa_set_features (struct vhost_dev *dev, +uint64_t features) +{ +struct vhost_vdpa *v = dev->opaque; + v->guest_features = features; +return vhost_vdpa_backend_set_features (dev, features); +} +static int vhost_vdpa_set_backend_cap (struct vhost_dev *dev) {uint64_t features;@@ -673,12 +693,17 @@ static int vhost_vdpa_set_vring_call(struct vhost_dev *dev, }static int vhost_vdpa_get_features (struct vhost_dev *dev, -uint64_t *features) +uint64_t *features) {int ret;
hw/virtio/vhost-vdpa.c static int vhost_vdpa_set_vring_call(struct vhost_dev *dev, · line 709
Quoted 10 lines
hw/virtio/vhost-vdpa.c +5
-0ret =vhost_vdpa_call (dev,VHOST_GET_FEATURES , features);trace_vhost_vdpa_get_features (dev, *features); + +if (ret ==0 ) { +struct vhost_vdpa *v = dev->opaque; + v->host_features = *features; + }return ret; }
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 15/20] vhost: Shadow virtqueue buffers forwarding
In reply to Eugenio PérezQuoted 70 lines · Eugenio Pérez
Initial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device.
For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series.
It reuses the VirtQueue code for the device part. The driver part is based on Linux's
virtio_ringdriver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones.SVQ uses
VIRTIO_CONFIG_S_DEVICE_STOPPEDto pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter.This status bit
VIRTIO_CONFIG_S_DEVICE_STOPPEDis currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits.Removal of
_S_DEVICE_STOPPEDbit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device.Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-)qapi/net.json +1
-1@@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding path # # Returns: Error if failure, or 'no error' for success. #hw/virtio/vhost-shadow-virtqueue.c +6
-0@@ -10,6 +10,7 @@#include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" +#include "hw/virtio/virtio-access.h" #include "standard-headers/linux/vhost_types.h" @@ -44,15 +45,135 @@ typedef struct VhostShadowVirtqueue {/* Virtio device */ VirtIODevice *vdev; + +/* Map for returning guest's descriptors */ +VirtQueueElement **ring_id_maps; + +/* Next head to expose to device */
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 62
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+uint16_t avail_idx_shadow; + +/* Next free descriptor */ +uint16_t free_head; + +/* Last seen used idx */ +uint16_t shadow_used_idx; + +/* Next head to consume from device */ +uint16_t used_idx;
Let's use "last_used_idx" as kernel driver did.
Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +2
-1} VhostShadowVirtqueue;/* If the device is using some of these, SVQ cannot communicate */ bool vhost_svq_valid_device_features (uint64_t *dev_features) { -return true; +uint64_t b; +bool r = true;
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 79
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+ +for (b =VIRTIO_TRANSPORT_F_START ; b <=VIRTIO_TRANSPORT_F_END ; ++b) { +switch (b) { +case VIRTIO_F_NOTIFY_ON_EMPTY : +case VIRTIO_F_ANY_LAYOUT : +/* SVQ is fine with this feature */ +continue ; + +case VIRTIO_F_ACCESS_PLATFORM : +/* SVQ needs this feature disabled. Can't continue */
So code can explain itself, need a comment to explain why.
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 87
Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-0+if (*dev_features &BIT_ULL (b)) { +clear_bit (b, dev_features); + r = false; + } +break ; + +case VIRTIO_F_VERSION_1 : +/* SVQ needs this feature, so can't continue */
A comment to explain why SVQ needs this feature.
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c +9
-0+if (!(*dev_features &BIT_ULL (b))) { +set_bit (b, dev_features); + r = false; + } +continue ; + +default : +/* +* SVQ must disable this feature, let's hope the device is fine
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 106
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+* without it. +*/ +if (*dev_features &BIT_ULL (b)) { +clear_bit (b, dev_features); + } + } + } + +return r; +}
Let's move this to patch 14.
Quoted 75 lines
hw/virtio/vhost-shadow-virtqueue.c +68
-1+ +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq, +const struct iovec *iovec, +size_t num,bool more_descs,bool write) +{ +uint16_t i = svq->free_head, last = svq->free_head; +unsigned n; +uint16_t flags = write ?cpu_to_le16 (VRING_DESC_F_WRITE ) :0 ; +vring_desc_t *descs = svq->vring.desc; + +if (num ==0 ) { +return ; + } + +for (n =0 ; n < num; n++) { +if (more_descs || (n +1 < num)) { + descs[i].flags = flags |cpu_to_le16 (VRING_DESC_F_NEXT ); + }else { + descs[i].flags = flags; + } + descs[i].addr =cpu_to_le64 ((hwaddr )iovec[n].iov_base); + descs[i].len =cpu_to_le32 (iovec[n].iov_len); + + last = i; + i =cpu_to_le16 (descs[i].next); + } + + svq->free_head =le16_to_cpu (descs[last].next); +} + +static unsigned vhost_svq_add_split (VhostShadowVirtqueue *svq, +VirtQueueElement *elem) +{ +int head; +unsigned avail_idx; +vring_avail_t *avail = svq->vring.avail; + + head = svq->free_head; + +/* We need some descriptors here */ +assert (elem->out_num || elem->in_num); + +vhost_vring_write_descs (svq, elem->out_sg, elem->out_num, + elem->in_num >0 , false); +vhost_vring_write_descs (svq, elem->in_sg, elem->in_num, false, true); + +/* +* Put entry in available array (but don't update avail->idx until they +* do sync). +*/ + avail_idx = svq->avail_idx_shadow & (svq->vring.num -1 ); + avail->ring[avail_idx] =cpu_to_le16 (head); + svq->avail_idx_shadow++; + +/* Update avail index after the descriptor is wrote */ +smp_wmb (); + avail->idx =cpu_to_le16 (svq->avail_idx_shadow); + +return head; + } -/* Forward guest notifications */ +static void vhost_svq_add (VhostShadowVirtqueue *svq,VirtQueueElement *elem) +{ +unsigned qemu_head =vhost_svq_add_split (svq, elem); + + svq->ring_id_maps[qemu_head] = elem; +} + +/* Handle guest->device notifications */ static void vhost_handle_guest_kick (EventNotifier *n) { VhostShadowVirtqueue *svq = container_of(n, VhostShadowVirtqueue,
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 191
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-1@@ -62,7 +183,74 @@ static void vhost_handle_guest_kick(EventNotifier *n)return ; } -event_notifier_set (&svq->kick_notifier); +/* Make available as many buffers as possible */ + do { + if (virtio_queue_get_notification(svq->vq)) { + /* No more notifications until process all available */ + virtio_queue_set_notification(svq->vq, false); + }
This can be done outside the loop.
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 203
Quoted 12 lines
hw/virtio/vhost-shadow-virtqueue.c +12
-0+ +while (true) { +VirtQueueElement *elem =virtqueue_pop (svq->vq,sizeof (*elem)); +if (!elem) { +break ; + } + +vhost_svq_add (svq, elem); +event_notifier_set (&svq->kick_notifier); + } + +virtio_queue_set_notification (svq->vq, true);
I think this can be moved to the end of this function.
Btw, we probably need a quota to make sure the svq is not hogging the main event loop.
Similar issue could be found in both virtio-net TX (using timer or bh) and TAP (a quota).
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 214
Quoted 11 lines
hw/virtio/vhost-shadow-virtqueue.c +11
-0+ }while (!virtio_queue_empty (svq->vq)); +} + +static bool vhost_svq_more_used (VhostShadowVirtqueue *svq) +{ +if (svq->used_idx != svq->shadow_used_idx) { +return true; + } + +/* Get used idx must not be reordered */ +smp_rmb ();
Interesting, we don't do this for kernel drivers. It would be helpful to explain it more clear by "X must be done before Y".
Quoted 52 lines
hw/virtio/vhost-shadow-virtqueue.c +43
-2+ svq->shadow_used_idx =cpu_to_le16 (svq->vring.used->idx); + +return svq->used_idx != svq->shadow_used_idx; +} + +static VirtQueueElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) +{ +vring_desc_t *descs = svq->vring.desc; +const vring_used_t *used = svq->vring.used; +vring_used_elem_t used_elem; +uint16_t last_used; + +if (!vhost_svq_more_used (svq)) { +return NULL ; + } + + last_used = svq->used_idx & (svq->vring.num -1 ); + used_elem.id =le32_to_cpu (used->ring[last_used].id); + used_elem.len =le32_to_cpu (used->ring[last_used].len); + + svq->used_idx++; +if (unlikely (used_elem.id >= svq->vring.num)) { +error_report ("Device %s says index %u is used" , svq->vdev->name, + used_elem.id); +return NULL ; + } + +if (unlikely (!svq->ring_id_maps[used_elem.id])) { +error_report ( +"Device %s says index %u is used, but it was not available" , + svq->vdev->name, used_elem.id); +return NULL ; + } + + descs[used_elem.id].next = svq->free_head; + svq->free_head = used_elem.id; + + svq->ring_id_maps[used_elem.id]->len = used_elem.len; +return g_steal_pointer (&svq->ring_id_maps[used_elem.id]); }/* Forward vhost notifications */ @@ -70,8 +258,26 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n) {VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, call_notifier); - -event_notifier_set (&svq->guest_call_notifier); +VirtQueue *vq = svq->vq; + +/* Make as many buffers as possible used. */ + do {
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_handle_call_no_test(EventNotifier *n) · line 274
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+unsigned i =0 ; + +/* TODO: Use VRING_AVAIL_F_NO_INTERRUPT */ +while (true) { + g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); +if (!elem) { +break ; + } + +assert (i < svq->vring.num);
Let's return error instead of using the assert.
Quoted 21 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-0+virtqueue_fill (vq, elem, elem->len, i++); + } + +virtqueue_flush (vq, i); +event_notifier_set (&svq->guest_call_notifier); + }while (vhost_svq_more_used (svq)); }static void vhost_svq_handle_call (EventNotifier *n)@@ -204,12 +410,25 @@ err_set_vring_kick:void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq) { +int i;int r =vhost_svq_restore_vdev_host_notifier (dev, idx, svq); +if (unlikely (r <0 )) {error_report ("Couldn't restore vq kick fd: %s" ,strerror (-r)); }event_notifier_set_handler (&svq->host_notifier,NULL );
hw/virtio/vhost-shadow-virtqueue.c err_set_vring_kick: · line 430
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+ +for (i =0 ; i < svq->vring.num; ++i) { + g_autofree VirtQueueElement *elem = svq->ring_id_maps[i]; +/* +* Although the doc says we must unpop in order, it's ok to unpop +* everything. +*/ +if (elem) { +virtqueue_unpop (svq->vq, elem, elem->len); + }
Will this result some of the "pending" buffers to be submitted multiple times? If yes, should we wait for all the buffers used instead of doing the unpop here?
Quoted 76 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-1+ } }/* @@ -224,7 +443,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)size_t driver_size;size_t device_size; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1); -int r; +int r, i; r =event_notifier_init (&svq->kick_notifier,0 );if (r !=0 ) {@@ -250,6 +469,11 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)memset (svq->vring.desc,0 , driver_size); svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size);memset (svq->vring.used,0 , device_size); +for (i =0 ; i < num -1 ; i++) { + svq->vring.desc[i].next =cpu_to_le16 (i +1 ); + } + + svq->ring_id_maps = g_new0(VirtQueueElement *, num);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);@@ -269,6 +493,7 @@ void vhost_svq_free(VhostShadowVirtqueue *vq)event_notifier_cleanup (&vq->kick_notifier);event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier); +g_free (vq->ring_id_maps);qemu_vfree (vq->vring.desc);qemu_vfree (vq->vring.used);g_free (vq);hw/virtio/vhost-vdpa.c +23
-0@@ -19,6 +19,7 @@#include "hw/virtio/virtio-net.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" #include "cpu.h" @@ -475,6 +476,28 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,return vhost_vdpa_backend_set_features (dev, features); } +/** +* Restore guest features to vdpa device +*/ +static int vhost_vdpa_set_guest_features (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +return vhost_vdpa_backend_set_features (dev, v->guest_features); +} + +/** +* Set shadow virtqueue supported features +*/ +static int vhost_vdpa_set_svq_features (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +uint64_t features = v->host_features; +bool b =vhost_svq_valid_device_features (&features); +assert (b); + +return vhost_vdpa_backend_set_features (dev, features); +} +static int vhost_vdpa_set_backend_cap (struct vhost_dev *dev) {uint64_t features;@@ -730,6 +753,19 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true;
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 763
Quoted 10 lines
hw/virtio/vhost-vdpa.c +8
-0} +static int vhost_vdpa_vring_pause (struct vhost_dev *dev) +{ +int r; +uint8_t status; + +vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_DEVICE_STOPPED ); + do { + r = vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);
I guess we'd better add some sleep here.
Quoted 75 lines
hw/virtio/vhost-vdpa.c +45
-3+ }while (r ==0 && !(status &VIRTIO_CONFIG_S_DEVICE_STOPPED )); + +return 0 ; +} +/* * Start shadow virtqueue. */ @@ -742,9 +778,29 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) .index = idx + dev->vq_index, .fd = event_notifier_get_fd(vhost_call_notifier), }; +struct vhost_vring_addr addr = { + .index = idx + dev->vq_index, + }; +struct vhost_vring_state num = { + .index = idx + dev->vq_index, + .num =virtio_queue_get_num (dev->vdev, idx), + };int r;bool b; +vhost_svq_get_vring_addr (svq, &addr); + r =vhost_vdpa_set_vring_addr (dev, &addr); +if (unlikely (r)) { +error_report ("vhost_set_vring_addr for shadow vq failed" ); +return false; + } + + r =vhost_vdpa_set_vring_num (dev, &num); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_num for shadow vq failed" ); +return false; + } +/* Set shadow vq -> guest notifier */ assert (v->call_fd[idx]);vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]);@@ -781,15 +837,32 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) {VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); -bool ok; -if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ; }g_ptr_array_add (v->shadow_vqs, svq); + } + } - ok =vhost_vdpa_svq_start_vq (hdev, n); + r =vhost_vdpa_vring_pause (hdev); +assert (r ==0 ); + +if (enable) { +for (n =0 ; n < v->shadow_vqs->len; ++n) { +/* Obtain Virtqueue state */ +vhost_virtqueue_stop (hdev, hdev->vdev, &hdev->vqs[n], n); + } + } + +/* Reset device so it can be configured */ + r =vhost_vdpa_dev_start (hdev, false); +assert (r ==0 ); + +if (enable) { +int r; +for (n =0 ; n < v->shadow_vqs->len; ++n) { +bool ok =vhost_vdpa_svq_start_vq (hdev, n);if (unlikely (!ok)) {/* Free still not started svqs */ g_ptr_array_set_size (v->shadow_vqs, n);
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 874
Quoted 5 lines
hw/virtio/vhost-vdpa.c +2
-0@@ -797,11 +870,19 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)break ; } } + +/* Need to ack features to set state in vp_vdpa devices */
vhost_vdpa actually?
Quoted 37 lines
hw/virtio/vhost-vdpa.c +12
-0+ r =vhost_vdpa_set_svq_features (hdev); +if (unlikely (r)) { + enable = false; + } } v->shadow_vqs_enabled = enable;if (!enable) { +vhost_vdpa_set_guest_features (hdev); +/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { struct vhost_vring_file file = {@@ -818,7 +899,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* TODO: This can unmask or override call fd! */ vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); } + } + r =vhost_vdpa_dev_start (hdev, true); +assert (r ==0 ); + +if (!enable) {/* Resources cleanup */ g_ptr_array_set_size (v->shadow_vqs,0 ); }@@ -831,6 +917,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp)struct vhost_vdpa *v;const char *err_cause =NULL ;bool r; +uint64_t svq_features; QLIST_FOREACH(v, &vhost_vdpa_devices, entry) {if (v->dev->vdev &&0 ==strcmp (v->dev->vdev->name, name)) {@@ -846,6 +933,20 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) goto err;
hw/virtio/vhost-vdpa.c void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) · line 943
Quoted 10 lines
hw/virtio/vhost-vdpa.c +8
-0} + svq_features = v->host_features; +if (!vhost_svq_valid_device_features (&svq_features)) { +error_setg (errp, +"Can't enable shadow vq on %s: Unexpected feature flags (%lx-%lx)" , + name, v->host_features, svq_features); +return ; + }else { +/* TODO: Check for virtio_vdpa + IOMMU & modern device */
I guess you mean "vhost_vdpa" here. For IOMMU, I guess you mean "vIOMMU" actually?
Thanks
Quoted 10 lines
hw/virtio/vhost-vdpa.c +6
-0+ } + +if (err_cause) { + goto err; + } + r =vhost_vdpa_enable_svq (v, enable);if (unlikely (!r)) { err_cause ="Error enabling (see monitor)" ;
hw/virtio/vhost-vdpa.c void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) · line 960
Quoted 8 lines
hw/virtio/vhost-vdpa.c +1
-1@@ -853,7 +954,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) } err: -if (err_cause) { +if (errp ==NULL && err_cause) {error_setg (errp,"Can't enable shadow vq on %s: %s" , name, err_cause); } }
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 16/20] vhost: Check for device VRING_USED_F_NO_NOTIFY at shadow virtqueue kick
In reply to Eugenio PérezQuoted 10 lines · Eugenio Pérez
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +0
-0
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_add(VhostShadowVirtqueue *svq, VirtQueueElement *elem) · line 178
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +3
-0@@ -173,6 +173,15 @@ static void vhost_svq_add(VhostShadowVirtqueue *svq, VirtQueueElement *elem) svq->ring_id_maps[qemu_head] = elem; } +static void vhost_svq_kick (VhostShadowVirtqueue *svq) +{ +/* Make sure we are reading updated device flag */
I guess this would be better:
/* We need to expose available array entries before checking used
* flags. */
(Borrowed from kernel codes).
Thanks
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+smp_mb (); +if (!(svq->vring.used->flags &VRING_USED_F_NO_NOTIFY )) { +event_notifier_set (&svq->kick_notifier); + } +} +/* Handle guest->device notifications */ static void vhost_handle_guest_kick (EventNotifier *n) {
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 212
Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +1
-1@@ -197,7 +206,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) }vhost_svq_add (svq, elem); -event_notifier_set (&svq->kick_notifier); +vhost_svq_kick (svq); }virtio_queue_set_notification (svq->vq, true);
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 17/20] vhost: Use VRING_AVAIL_F_NO_INTERRUPT at device call on shadow virtqueue
In reply to Eugenio PérezQuoted 1 line · Eugenio Pérez
Signed-off-by Eugenio Pérez <[email protected]>
Commit log please.
Thanks
Quoted 53 lines
--- hw/virtio/vhost-shadow-virtqueue.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +22
-1@@ -60,6 +60,9 @@ typedef struct VhostShadowVirtqueue {/* Next head to consume from device */ uint16_t used_idx; + +/* Cache for the exposed notification flag */ +bool notification; } VhostShadowVirtqueue;/* If the device is using some of these, SVQ cannot communicate */ @@ -105,6 +108,24 @@ bool vhost_svq_valid_device_features(uint64_t *dev_features)return r; } +static void vhost_svq_set_notification (VhostShadowVirtqueue *svq,bool enable) +{ +uint16_t notification_flag; + +if (svq->notification == enable) { +return ; + } + + notification_flag =cpu_to_le16 (VRING_AVAIL_F_NO_INTERRUPT ); + + svq->notification = enable; +if (enable) { + svq->vring.avail->flags &= ~notification_flag; + }else { + svq->vring.avail->flags |= notification_flag; + } +} +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq,const struct iovec *iovec,size_t num,bool more_descs,bool write)@@ -273,7 +294,7 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n) do { unsigned i = 0; - /* TODO: Use VRING_AVAIL_F_NO_INTERRUPT */ + vhost_svq_set_notification(svq, false); while (true) { g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); if (!elem) {
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_handle_call_no_test(EventNotifier *n) · line 313
Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.c +1
-0@@ -286,6 +307,7 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n)virtqueue_flush (vq, i);event_notifier_set (&svq->guest_call_notifier); +vhost_svq_set_notification (svq, true); }while (vhost_svq_more_used (svq)); }
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
In reply to Eugenio PérezQuoted 11 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Quoted 7 lines
Signed-off-by Eugenio Pérez<[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)
Think hard about the whole logic. This is safe since qemu memory map will fail if guest submits a invalidate IOVA.
Then I wonder if we do something much more simpler:
1) Using qemu VA as IOVA but only maps the VA that belongs to guest
2) Then we don't need any IOVA tree here, what we need is to just map vring and use qemu VA without any translation
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue
In reply to Jason WangQuoted 11 lines · Jason Wang
Quoted 4 lines · Eugenio Pérez
Shadow virtqueue notifications forwarding is disabled when
vhost_devstops, so code flow follows usual cleanup.Also, host notifiers must be disabled at SVQ start,
Any reason for this?
It will be addressed in a later series, sorry.
Quoted 6 lines
Quoted 4 lines
and they will not
start if SVQ has been enabled when device is stopped. This is trivial to address, but it is left out for simplicity at this moment.
Quoted 7 lines
It looks to me this patch also contains the following logics
1) codes to enable svq
2) codes to let svq to be enabled from QMP.
I think they need to be split out,
I agree that we can split this more, with the code that belongs to SVQ and the code that belongs to vhost-vdpa. it will be addressed in future series.
Quoted 3 lines
we may endup with the following
series of patches
With "series of patches" do you mean to send every step in a separated series? There are odds of having the need of modifying code already sent & merged with later ones. If you confirm to me that it is fine, I can do it that way for sure.
Quoted 6 lines
1) svq skeleton with enable/disable
2) route host notifier to svq
3) route guest notifier to svq
4) codes to enable svq
5) enable svq via QMP
I'm totally fine with that, but there is code that is never called if the qmp command is not added. The compiler complains about static functions that are not called, making impossible things like bisecting through these commits, unless I use attribute((unused)) or similar. Or have I missed something?
We could do that way with the code that belongs to SVQ though, since all of it is declared in headers. But to delay the "enable svq via qmp" to the last one makes debugging harder, as we cannot just enable notifications forwarding with no buffers forwarding.
If I introduce a change in the notifications code, I can simply go to these commits and enable SVQ for notifications. This way I can have an idea of what part is failing. A similar logic can be applied to other devices than vp_vdpa. We would lose it if we
Quoted 30 lines
Quoted 30 lines
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.h | 8 ++ include/hw/virtio/vhost-vdpa.h | 4 + hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- 5 files changed, 264 insertions(+), 4 deletions(-)qapi/net.json +1
-1@@ -88,7 +88,7 @@ # # @enable: true to use the alternate shadow VQ notifications # -# Returns: Always error, since SVQ is not implemented at the moment. +# Returns: Error if failure, or 'no error' for success. # # Since: 6.2 #hw/virtio/vhost-shadow-virtqueue.h +0
-0@@ -14,6 +14,14 @@
Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.h line 17
Quoted 3 lines
hw/virtio/vhost-shadow-virtqueue.h +1
-0typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ; +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq);Let's move this function to another patch since it's unrelated to the
guest->hostrouting.
Right, I missed it while squashing commits and at later reviews.
Quoted 63 lines
Quoted 63 lines
hw/virtio/vhost-shadow-virtqueue.h +7
-0+void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); + +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx);void vhost_svq_free (VhostShadowVirtqueue *vq);include/hw/virtio/vhost-vdpa.h +4
-0@@ -12,6 +12,8 @@#ifndef HW_VIRTIO_VHOST_VDPA_H #define HW_VIRTIO_VHOST_VDPA_H +#include <gmodule.h> +#include "qemu/queue.h" #include "hw/virtio/virtio.h" @@ -24,6 +26,8 @@ typedef struct vhost_vdpa {int device_fd;uint32_t msg_type;MemoryListener listener; +bool shadow_vqs_enabled; +GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry;VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ];hw/virtio/vhost-shadow-virtqueue.c +12
-1@@ -9,9 +9,12 @@#include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" +#include "hw/virtio/vhost.h" + +#include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" -#include "qemu/event_notifier.h" +#include "qemu/main-loop.h" /* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue {@@ -19,14 +22,146 @@ typedef struct VhostShadowVirtqueue {EventNotifier kick_notifier;/* Shadow call notifier, sent to vhost */ EventNotifier call_notifier; + +/* +* Borrowed virtqueue's guest to host notifier. +* To borrow it in this event notifier allows to register on the event +* loop and access the associated shadow virtqueue easily. If we use the +* VirtQueue, we don't have an easy way to retrieve it. +* +* So shadow virtqueue must not clean it, or we would lose VirtQueue one.
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 37
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+*/ +EventNotifier host_notifier; + +/* Guest's call notifier, where SVQ calls guest. */ +EventNotifier guest_call_notifier;To be consistent, let's simply use "
guest_notifier" here.
It could be confused when the series adds a guest -> qemu kick notifier then. Actually, I think it would be better to rename host_notifier to something like host_svq_notifier. Or host_call and guest_call, since "notifier" is already in the type, making the name to be a little bit "Hungarian notation".
Quoted 6 lines
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +3
-0+ +/* Virtio queue shadowing */ +VirtQueue *vq; } VhostShadowVirtqueue;
Quoted 14 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 51
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c +9
-0+/* Forward guest notifications */ +static void vhost_handle_guest_kick (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + host_notifier); + +if (unlikely (!event_notifier_test_and_clear (n))) { +return ; + }Is there a chance that we may stop the processing of available buffers during the svq enabling? There could be no kick from the guest in this case.
Actually, yes, I think you are right. The guest kick eventfd could have been consumed by vhost but there may be still pending buffers.
I think it would be better to check for available buffers first, then clear the notifier unconditionally, and then re-check and process them if any [1].
However, this problem arises later in the series: At this moment the device is not reset and guest's host notifier is not replaced, so either vhost/device receives the kick, or SVQ does and forwards it.
Does it make sense to you?
Quoted 40 lines
Quoted 40 lines
hw/virtio/vhost-shadow-virtqueue.c +39
-0+ +event_notifier_set (&svq->kick_notifier); +} + +/* +* Obtain the SVQ call notifier, where vhost device notifies SVQ that there +* exists pending used buffers. +* +* @svq Shadow Virtqueue +*/ +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq) +{ +return &svq->call_notifier; +} + +/* +* Set the call notifier for the SVQ to call the guest +* +* @svq Shadow virtqueue +* @call_fd call notifier +* +* Called on BQL context. +*/ +void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd) +{ +event_notifier_init_fd (&svq->guest_call_notifier, call_fd); +} + +/* +* Restore the vhost guest to host notifier, i.e., disables svq effect. +*/ +static int vhost_svq_restore_vdev_host_notifier (struct vhost_dev *dev, +unsigned vhost_index, +VhostShadowVirtqueue *svq) +{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = vhost_index, + .fd =event_notifier_get_fd (vq_host_notifier),
Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 95
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+ }; +int r; + +/* Restore vhost kick */ + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);And remap the notification area if necessary.
Totally right, that step is missed in this series.
However, remapping guest host notifier memory region has no advantages over using ioeventfd to perform guest -> SVQ notifications, doesn't it? By both methods flow needs to go through the hypervisor kernel.
Quoted 25 lines
Quoted 25 lines
hw/virtio/vhost-shadow-virtqueue.c +23
-0+return r ? -errno :0 ; +} + +/* +* Start shadow virtqueue operation. +* @dev vhost device +* @hidx vhost virtqueue index +* @svq Shadow Virtqueue +*/ +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = dev->vhost_ops->vhost_get_vq_index (dev, dev->vq_index + idx), + .fd =event_notifier_get_fd (&svq->kick_notifier), + }; +int r; + +/* Check that notifications are still going directly to vhost dev */ +assert (virtio_queue_is_host_notifier_enabled (svq->vq)); + +/*
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 122
Quoted 4 lines
hw/virtio/vhost-shadow-virtqueue.c +4
-0+* event_notifier_set_handler already checks for guest's notifications if +* they arrive in the switch, so there is no need to explicitely check for +* them. +*/ If this is true, shouldn't we call
vhost_set_vring_kick()before theevent_notifier_set_handler()?
Not at this point of the series, but it could be another solution when we need to reset the device and we are unsure if all buffers have been read. But I think I prefer the solution exposed in [1] and to explicitly call vhost_handle_guest_kick here. Do you think differently?
Quoted 3 lines
Btw, I think we should update the fd if
set_vring_kick()was called after this function?
Kind of. This is currently bad in the code, but...
Backend callbacks vhost_ops->vhost_set_vring_kick and vhost_ops->vhost_set_vring_addr are only called at vhost_virtqueue_start. And they are always called with known data already stored in VirtQueue.
To avoid storing more state in vhost_vdpa, I think that we should avoid duplicating them, but ignore new kick_fd or address in SVQ mode, and retrieve them again at the moment the device is (re)started in SVQ mode. Qemu already avoids things like allowing the guest to set addresses at random time, using the VirtIOPCIProxy to store them.
I also see how duplicating that status could protect vdpa SVQ code against future changes to vhost code, but that would make this series bigger and more complex with no need at this moment in my opinion.
Do you agree?
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 127
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+event_notifier_init_fd (&svq->host_notifier, +event_notifier_get_fd (vq_host_notifier)); +event_notifier_set_handler (&svq->host_notifier, vhost_handle_guest_kick); + + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);And we need to stop the notification area mmap.
Right.
Quoted 124 lines
Quoted 113 lines
hw/virtio/vhost-shadow-virtqueue.c +32
-0+if (unlikely (r !=0 )) { +error_report ("Couldn't set kick fd: %s" ,strerror (errno)); + goto err_set_vring_kick; + } + +return true; + +err_set_vring_kick: +event_notifier_set_handler (&svq->host_notifier,NULL ); + +return false; +} + +/* +* Stop shadow virtqueue operation. +* @dev vhost device +* @idx vhost queue index +* @svq Shadow Virtqueue +*/ +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +int r =vhost_svq_restore_vdev_host_notifier (dev, idx, svq); +if (unlikely (r <0 )) { +error_report ("Couldn't restore vq kick fd: %s" ,strerror (-r)); + } + +event_notifier_set_handler (&svq->host_notifier,NULL ); +} +/* * Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) { +int vq_idx = dev->vq_index + idx; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);int r;@@ -44,6 +179,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) goto err_init_call_notifier; } + svq->vq =virtio_get_queue (dev->vdev, vq_idx);return g_steal_pointer (&svq); err_init_call_notifier:hw/virtio/vhost-vdpa.c +24
-0@@ -17,6 +17,7 @@#include "hw/virtio/vhost.h" #include "hw/virtio/vhost-backend.h" #include "hw/virtio/virtio-net.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" @@ -272,6 +273,16 @@ static void vhost_vdpa_add_status(struct vhost_dev *dev, uint8_t status)vhost_vdpa_call (dev,VHOST_VDPA_SET_STATUS , &s); } +/** +* Adaptor function to free shadow virtqueue through gpointer +* +* @svq The Shadow Virtqueue +*/ +static void vhost_psvq_free (gpointer svq) +{ +vhost_svq_free (svq); +} +static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp) {struct vhost_vdpa *v;@@ -283,6 +294,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; + v->shadow_vqs =g_ptr_array_new_full (dev->nvqs, vhost_psvq_free);QLIST_INSERT_HEAD (&vhost_vdpa_devices, v, entry); vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |@@ -373,6 +385,17 @@ err:return ; } +static void vhost_vdpa_svq_cleanup (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +size_t idx; + +for (idx =0 ; idx < v->shadow_vqs->len; ++idx) { +vhost_svq_stop (dev, idx,g_ptr_array_index (v->shadow_vqs, idx)); + } +g_ptr_array_free (v->shadow_vqs, true); +} +static int vhost_vdpa_cleanup (struct vhost_dev *dev) {struct vhost_vdpa *v;@@ -381,6 +404,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev)trace_vhost_vdpa_cleanup (dev, v);vhost_vdpa_host_notifiers_uninit (dev, dev->nvqs);memory_listener_unregister (&v->listener); +vhost_vdpa_svq_cleanup (dev);QLIST_REMOVE (v, entry); dev->opaque =NULL ;hw/virtio/vhost-vdpa.c static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) · line 586
Quoted 7 lines
hw/virtio/vhost-vdpa.c +3
-1@@ -557,7 +581,9 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started)if (started) {uint8_t status =0 ;memory_listener_register (&v->listener, &address_space_memory); -vhost_vdpa_host_notifiers_init (dev); +if (!v->shadow_vqs_enabled) { +vhost_vdpa_host_notifiers_init (dev); + }This looks like a trick, why not check and setup
shadow_vqsinside:
Quoted 6 lines
1)
vhost_vdpa_host_notifiers_init()and
2)
vhost_vdpa_set_vring_kick()
Ok I will move the checks there.
Quoted 37 lines
Quoted 37 lines
hw/virtio/vhost-vdpa.c +29
-0vhost_vdpa_set_vring_ready (dev);vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_DRIVER_OK );vhost_vdpa_call (dev,VHOST_VDPA_GET_STATUS , &status);@@ -663,10 +689,96 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +/* +* Start shadow virtqueue. +*/ +static bool vhost_vdpa_svq_start_vq (struct vhost_dev *dev,unsigned idx) +{ +struct vhost_vdpa *v = dev->opaque; +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); +return vhost_svq_start (dev, idx, svq); +} + +static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) +{ +struct vhost_dev *hdev = v->dev; +unsigned n; + +if (enable == v->shadow_vqs_enabled) { +return hdev->nvqs; + } + +if (enable) { +/* Allocate resources */ +assert (v->shadow_vqs->len ==0 ); +for (n =0 ; n < hdev->nvqs; ++n) { +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +bool ok; + +if (unlikely (!svq)) { +g_ptr_array_set_size (v->shadow_vqs,0 ); +return 0 ;
Quoted 8 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 728
Quoted 8 lines
hw/virtio/vhost-vdpa.c +8
-0+ } +g_ptr_array_add (v->shadow_vqs, svq); + + ok =vhost_vdpa_svq_start_vq (hdev, n); +if (unlikely (!ok)) { +/* Free still not started svqs */ +g_ptr_array_set_size (v->shadow_vqs, n); + enable = false;
[2]
Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 731
Quoted 3 lines
hw/virtio/vhost-vdpa.c +3
-0+break ; + } + }Since there's almost no logic could be shared between enable and disable. Let's split those logic out into dedicated functions where the codes looks more easy to be reviewed (e.g have a better error handling etc).
Maybe it could be more clear in the code, but the reused logic is the disabling of SVQ and the fallback in case it cannot be enabled with [2]. But I'm not against splitting in two different functions if it makes review easier.
Quoted 21 lines
Quoted 21 lines
hw/virtio/vhost-vdpa.c +19
-0+ } + + v->shadow_vqs_enabled = enable; + +if (!enable) { +/* Disable all queues or clean up failed start */ +for (n =0 ; n < v->shadow_vqs->len; ++n) { +unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n); +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n); +vhost_svq_stop (hdev, n, svq); +vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); + } + +/* Resources cleanup */ +g_ptr_array_set_size (v->shadow_vqs,0 ); + } + +return n; +}
Quoted 16 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 762
Quoted 12 lines
hw/virtio/vhost-vdpa.c +9
-1void qmp_x_vhost_enable_shadow_vq (const char *name,bool enable,Error **errp) { -error_setg (errp,"Shadow virtqueue still not implemented" ); +struct vhost_vdpa *v; +const char *err_cause =NULL ; +bool r; + + QLIST_FOREACH(v, &vhost_vdpa_devices, entry) { +if (v->dev->vdev &&0 ==strcmp (v->dev->vdev->name, name)) { +break ; + } + }I think you can iterate the NetClientStates to ge tthe vhost-vdpa backends.
Right, I missed it.
Quoted 11 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 768
Quoted 7 lines
hw/virtio/vhost-vdpa.c +6
-0+ +if (!v) { + err_cause ="Device not found" ; + goto err; + }else if (v->notifier[0 ].addr) { + err_cause ="Device has host notifiers enabled" ;I don't get this.
At this moment of the series you can enable guest -> SVQ -> 'vdpa device' if the device is not using the host notifiers memory region. The right solution is to disable it for the guest, and to handle it in SVQ. Otherwise, guest kick will bypass SVQ and
It can be done in the same patch, or at least to disable (as unmap) them at this moment and handle them in a posterior patch. but for prototyping the solution I just ignored it in this series. It will be handled some way or another in the next one. I prefer the last one, to handle in a different patch, but let me know if you think it is better otherwise.
Quoted 3 lines
Btw this function should be implemented in an independent patch after svq is fully functional.
(Reasons for that are already commented at the top of this mail :) ).
Thanks!
Quoted 20 lines
Thanks
Quoted 16 lines
hw/virtio/vhost-vdpa.c +13
-0+ goto err; + } + + r =vhost_vdpa_enable_svq (v, enable); +if (unlikely (!r)) { + err_cause ="Error enabling (see monitor)" ; + goto err; + } + +err: +if (err_cause) { +error_setg (errp,"Can't enable shadow vq on %s: %s" , name, err_cause); + } }const VhostOps vdpa_ops = {
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 05/20] vhost: Add x-vhost-enable-shadow-vq qmp
In reply to Markus ArmbrusterQuoted 32 lines · Markus Armbruster
Quoted 29 lines · Eugenio Perez Martin
Quoted 27 lines · Markus Armbruster
Quoted 13 lines · Eugenio Pérez
Command to enable shadow virtqueue.
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 23 +++++++++++++++++++++++ hw/virtio/vhost-vdpa.c | 8 ++++++++ 2 files changed, 31 insertions(+)qapi/net.json +0
-0qapi/net.json line 87
Quoted 9 lines
qapi/net.json +6
-0@@ -79,6 +79,29 @@ { 'command': 'netdev_del', 'data': {'id': 'str'}, 'allow-preconfig': true } +## +# @x-vhost-enable-shadow-vq: +# +# Use vhost shadow virtqueue. +# +# @name: the device name of the VirtIO deviceIs this a qdev ID? A network client name?
Quoted 7 lines
Quoted 3 lines
At this moment is the virtio device name, the one specified at the call of "
virtio_init". But this should change, maybe the qdev id or something that can be provided by the command line fits better here.To refer to a device backend, we commonly use a backend-specific ID. For network backends, that's NetClientState member name.
Ok so I will use the NetClientState member name, it fits way better here than the virtio device name.
Thanks!
Quoted 4 lines
To refer to a device frontend, we commonly use a qdev ID or a QOM path.
[...]
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 09/20] vdpa: Save call_fd in vhost-vdpa
In reply to Jason WangQuoted 27 lines · Jason Wang
Quoted 13 lines · Eugenio Pérez
We need to know it to switch to Shadow VirtQueue.
Signed-off-by Eugenio Pérez <[email protected]>--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 5 +++++ 2 files changed, 7 insertions(+)include/hw/virtio/vhost-vdpa.h +0
-0include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 34
Quoted 5 lines
include/hw/virtio/vhost-vdpa.h +2
-0@@ -30,6 +30,8 @@ typedef struct vhost_vdpa {GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry; +/* File descriptor the device uses to call VM/SVQ */ +int call_fd[VIRTIO_QUEUE_MAX ];Any reason we don't do this for
kick_fdor whyvirtio_queue_get_guest_notifier()can't work here? Need a comment or commit log.
Quoted 4 lines
I think we need to have a consistent way to handle both kick and call fd.
Thanks
The reasons for it have been given in answers to patch 08/20, since both have converged to it somehow. Please let me know if you think otherwise and this needs to be continued here.
Thanks!
Quoted 22 lines
Quoted 21 lines
include/hw/virtio/vhost-vdpa.h +0
-0VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ]; } VhostVDPA;hw/virtio/vhost-vdpa.c +5
-0@@ -652,7 +652,12 @@ static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev,static int vhost_vdpa_set_vring_call (struct vhost_dev *dev,struct vhost_vring_file *file) { +struct vhost_vdpa *v = dev->opaque; +int vdpa_idx =vhost_vdpa_get_vq_index (dev, file->index); +trace_vhost_vdpa_set_vring_call (dev, file->index, file->fd); + + v->call_fd[vdpa_idx] = file->fd;return vhost_vdpa_call (dev,VHOST_SET_VRING_CALL , file); }
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 10/20] vhost-vdpa: Take into account SVQ in vhost_vdpa_set_vring_call
In reply to Jason WangQuoted 44 lines · Jason Wang
Quoted 27 lines · Eugenio Pérez
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)hw/virtio/vhost-vdpa.c +7
-1@@ -649,16 +649,27 @@ static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev,return vhost_vdpa_call (dev,VHOST_SET_VRING_KICK , file); } +static int vhost_vdpa_set_vring_dev_call (struct vhost_dev *dev, +struct vhost_vring_file *file) +{ +trace_vhost_vdpa_set_vring_call (dev, file->index, file->fd); +return vhost_vdpa_call (dev,VHOST_SET_VRING_CALL , file); +} +static int vhost_vdpa_set_vring_call (struct vhost_dev *dev,struct vhost_vring_file *file) {struct vhost_vdpa *v = dev->opaque;int vdpa_idx =vhost_vdpa_get_vq_index (dev, file->index); -trace_vhost_vdpa_set_vring_call (dev, file->index, file->fd);hw/virtio/vhost-vdpa.c static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev, · line 672
Quoted 10 lines
hw/virtio/vhost-vdpa.c +7
-2- v->call_fd[vdpa_idx] = file->fd; -return vhost_vdpa_call (dev,VHOST_SET_VRING_CALL , file); +if (v->shadow_vqs_enabled) { +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, vdpa_idx); +vhost_svq_set_guest_call_notifier (svq, file->fd); +return 0 ; + }else { +return vhost_vdpa_set_vring_dev_call (dev, file); + }I feel like we should do the same for kick fd.
Quoted 2 lines
Thanks
I think this also has been answered on 08/20, but feel free to tell me otherwise if I missed something.
Thanks!
Quoted 5 lines
Quoted 4 lines
hw/virtio/vhost-vdpa.c +0
-0} static int vhost_vdpa_get_features(struct vhost_dev *dev,
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 12/20] virtio: Add vhost_shadow_vq_get_vring_addr
In reply to Jason WangQuoted 9 lines · Jason Wang
Quoted 1 line · Eugenio Pérez
It reports the shadow virtqueue address from qemu virtual address space
I think both the title and commit log needs to more tweaks. Looking at the codes, what id does is actually introduce vring into svq.
Right, this commit evolved a little bit providing more functionality and it is not reflected in the commit message. I will expand it.
Quoted 66 lines
Quoted 66 lines
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +++ hw/virtio/vhost-shadow-virtqueue.c | 50 ++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+)hw/virtio/vhost-shadow-virtqueue.h +4
-0@@ -16,6 +16,10 @@ typedef struct VhostShadowVirtqueue VhostShadowVirtqueue;EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq);void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); +void vhost_svq_get_vring_addr (const VhostShadowVirtqueue *svq, +struct vhost_vring_addr *addr); +size_t vhost_svq_driver_area_size (const VhostShadowVirtqueue *svq); +size_t vhost_svq_device_area_size (const VhostShadowVirtqueue *svq);bool vhost_svq_start (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq);hw/virtio/vhost-shadow-virtqueue.c +21
-0@@ -18,6 +18,9 @@/* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { + /* Shadow vring */ + struct vring vring; + /* Shadow kick notifier, sent to vhost */ EventNotifier kick_notifier; /* Shadow call notifier, sent to vhost */@@ -38,6 +41,9 @@ typedef struct VhostShadowVirtqueue {/* Virtio queue shadowing */ VirtQueue *vq; + +/* Virtio device */ +VirtIODevice *vdev; } VhostShadowVirtqueue;/* Forward guest notifications */ @@ -93,6 +99,35 @@ void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call_fd)event_notifier_init_fd (&svq->guest_call_notifier, call_fd); } +/* +* Get the shadow vq vring address. +* @svq Shadow virtqueue +* @addr Destination to store address +*/ +void vhost_svq_get_vring_addr (const VhostShadowVirtqueue *svq, +struct vhost_vring_addr *addr) +{ + addr->desc_user_addr = (uint64_t )svq->vring.desc; + addr->avail_user_addr = (uint64_t )svq->vring.avail; + addr->used_user_addr = (uint64_t )svq->vring.used; +} + +size_t vhost_svq_driver_area_size (const VhostShadowVirtqueue *svq) +{
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call_fd) · line 121
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+uint16_t vq_idx =virtio_get_queue_index (svq->vq); +size_t desc_size =virtio_queue_get_desc_size (svq->vdev, vq_idx); +size_t avail_size =virtio_queue_get_avail_size (svq->vdev, vq_idx); + +return ROUND_UP (desc_size + avail_size, qemu_real_host_page_size);Is this round up required by the spec?
No, I was trying to avoid that more qemu data get exposed to the device because of mapping at page granularity, in case data gets allocated after some region. I will expand with a comment, but if there are other ways to achieve or it is not needed please let me know!
Quoted 27 lines
Quoted 27 lines
hw/virtio/vhost-shadow-virtqueue.c +13
-0+} + +size_t vhost_svq_device_area_size (const VhostShadowVirtqueue *svq) +{ +uint16_t vq_idx =virtio_get_queue_index (svq->vq); +size_t used_size =virtio_queue_get_used_size (svq->vdev, vq_idx); +return ROUND_UP (used_size, qemu_real_host_page_size); +} +/* * Restore the vhost guest to host notifier, i.e., disables svq effect. */ @@ -178,6 +213,10 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) {int vq_idx = dev->vq_index + idx; +unsigned num =virtio_queue_get_num (dev->vdev, vq_idx); +size_t desc_size =virtio_queue_get_desc_size (dev->vdev, vq_idx); +size_t driver_size; +size_t device_size; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);int r;@@ -196,6 +235,15 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) }
Quoted 12 lines
hw/virtio/vhost-shadow-virtqueue.c VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) · line 244
Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +7
-0svq->vq =virtio_get_queue (dev->vdev, vq_idx); + svq->vdev = dev->vdev; + driver_size =vhost_svq_driver_area_size (svq); + device_size =vhost_svq_device_area_size (svq); + svq->vring.num = num; + svq->vring.desc =qemu_memalign (qemu_real_host_page_size, driver_size); + svq->vring.avail = (void *)((char *)svq->vring.desc + desc_size); +memset (svq->vring.desc,0 , driver_size);Any reason for using the contiguous area for both desc and avail?
No special reason, it can be splitted but if we maintain the page-width padding it could save memory, iotlb entries, etc. Not like it's going to be a big difference but still.
Thanks!
Quoted 17 lines
Thanks
Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c +4
-0+ svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size); +memset (svq->vring.used,0 , device_size);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);@@ -215,5 +263,7 @@ void vhost_svq_free(VhostShadowVirtqueue *vq)event_notifier_cleanup (&vq->kick_notifier);event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier); +qemu_vfree (vq->vring.desc); +qemu_vfree (vq->vring.used);g_free (vq); }
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 13/20] vdpa: Save host and guest features
In reply to Jason WangQuoted 27 lines · Jason Wang
Quoted 15 lines · Eugenio Pérez
Those are needed for SVQ: Host ones are needed to check if SVQ knows how to talk with the device and for feature negotiation, and guest ones to know if SVQ can talk with it.
Signed-off-by Eugenio Pérez <[email protected]>--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 31 ++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-)include/hw/virtio/vhost-vdpa.h +0
-0include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 30
Quoted 5 lines
include/hw/virtio/vhost-vdpa.h +2
-0@@ -26,6 +26,8 @@ typedef struct vhost_vdpa {int device_fd;uint32_t msg_type;MemoryListener listener; +uint64_t host_features; +uint64_t guest_features;Any reason that we can't use the features stored in VirtioDevice?
Quoted 2 lines
Thanks
It was easier to handle the non standard _F_STOP feature flag in vhost-vdpa but I think we can use VirtIODevice flags for the next series.
Thanks!
Quoted 68 lines
Quoted 67 lines
include/hw/virtio/vhost-vdpa.h +0
-0bool shadow_vqs_enabled;GPtrArray *shadow_vqs;struct vhost_dev *dev;hw/virtio/vhost-vdpa.c +28
-3@@ -439,10 +439,19 @@ static int vhost_vdpa_set_mem_table(struct vhost_dev *dev,return 0 ; } -static int vhost_vdpa_set_features (struct vhost_dev *dev, -uint64_t features) +/** +* Internal set_features() that follows vhost/VirtIO protocol for that +*/ +static int vhost_vdpa_backend_set_features (struct vhost_dev *dev, +uint64_t features) { +struct vhost_vdpa *v = dev->opaque; +int ret; +if (v->host_features &BIT_ULL (VIRTIO_F_QUEUE_STATE )) { + features |=BIT_ULL (VIRTIO_F_QUEUE_STATE ); + } +trace_vhost_vdpa_set_features (dev, features); ret =vhost_vdpa_call (dev,VHOST_SET_FEATURES , &features);uint8_t status =0 ;@@ -455,6 +464,17 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,return !(status &VIRTIO_CONFIG_S_FEATURES_OK ); } +/** +* Exposed vhost set features +*/ +static int vhost_vdpa_set_features (struct vhost_dev *dev, +uint64_t features) +{ +struct vhost_vdpa *v = dev->opaque; + v->guest_features = features; +return vhost_vdpa_backend_set_features (dev, features); +} +static int vhost_vdpa_set_backend_cap (struct vhost_dev *dev) {uint64_t features;@@ -673,12 +693,17 @@ static int vhost_vdpa_set_vring_call(struct vhost_dev *dev, }static int vhost_vdpa_get_features (struct vhost_dev *dev, -uint64_t *features) +uint64_t *features) {int ret; ret =vhost_vdpa_call (dev,VHOST_GET_FEATURES , features);trace_vhost_vdpa_get_features (dev, *features); + +if (ret ==0 ) { +struct vhost_vdpa *v = dev->opaque; + v->host_features = *features; + }return ret; }
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 15/20] vhost: Shadow virtqueue buffers forwarding
In reply to Markus ArmbrusterQuoted 73 lines · Markus Armbruster
Quoted 70 lines · Eugenio Perez Martin
Quoted 39 lines · Markus Armbruster
Quoted 36 lines · Eugenio Pérez
Initial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device.
For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series.
It reuses the VirtQueue code for the device part. The driver part is based on Linux's
virtio_ringdriver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones.SVQ uses
VIRTIO_CONFIG_S_DEVICE_STOPPEDto pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter.This status bit
VIRTIO_CONFIG_S_DEVICE_STOPPEDis currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits.Removal of
_S_DEVICE_STOPPEDbit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device.Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-)Quoted 13 lines
qapi/net.json line 89
Quoted 10 lines
qapi/net.json +1
-1@@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding pathUh, why does the flag change meaning half-way through this series?
Before this patch, the SVQ mode just makes an extra hop for notifications. Guest ones are now received by qemu via ioeventfd, and qemu forwards them to the device using a different eventfd. The reverse is also true: the device ones will be received by qemu by device call fd, and then qemu will forward them to the guest using a different irqfd.
This intermediate step is not very useful by itself, but helps for checking that that part of the communication works fine, with no need for shadow virtqueue to understand vring format. Doing that way also produces smaller patches.
So it makes sense to me to tell what QMP command does exactly at every point of the series. However I can directly document it as "use the alternate shadow VQ buffers forwarding path" from the beginning.
Quoted 8 lines
Quoted 4 lines
Does this make sense, or will it be better to write the final intention of the command?
Thanks!
Working your explanation into commit messages and possibly comments should do.
Got it, I will include them in both. Thanks!
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue
In reply to Jason WangQuoted 17 lines · Jason Wang
Quoted 12 lines · Eugenio Pérez
This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of
vhost_virtqueue_startcan unmasks and discard call events,vhost_virtqueue_startshould be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.
Quoted 5 lines
Btw, the log looks not clear. I guess this patch goes for method 3. If yes, we need explain it and why.
Thanks
Sorry about being unclear. This commit log (and code) just exposes the problem and the solutions I came up with but does nothing to solve it. I'm actually going for method 3 for the next series but I'm open to doing it differently.
Quoted 4 lines
Quoted 3 lines
Signed-off-by Eugenio Pérez<[email protected]>
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue
In reply to Jason WangQuoted 80 lines · Jason Wang
Quoted 77 lines · Eugenio Pérez
This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of
vhost_virtqueue_startcan unmasks and discard call events,vhost_virtqueue_startshould be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.c | 19 +++++++++++++++ hw/virtio/vhost-vdpa.c | 38 +++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +19
-0@@ -53,6 +53,22 @@ static void vhost_handle_guest_kick(EventNotifier *n)event_notifier_set (&svq->kick_notifier); } +/* Forward vhost notifications */ +static void vhost_svq_handle_call_no_test (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + call_notifier); + +event_notifier_set (&svq->guest_call_notifier); +} + +static void vhost_svq_handle_call (EventNotifier *n) +{ +if (likely (event_notifier_test_and_clear (n))) { +vhost_svq_handle_call_no_test (n); + } +} +/* * Obtain the SVQ call notifier, where vhost device notifies SVQ that there * exists pending used buffers. @@ -180,6 +196,8 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) } svq->vq =virtio_get_queue (dev->vdev, vq_idx); +event_notifier_set_handler (&svq->call_notifier, + vhost_svq_handle_call);return g_steal_pointer (&svq); err_init_call_notifier:@@ -195,6 +213,7 @@ err_init_kick_notifier:void vhost_svq_free (VhostShadowVirtqueue *vq) {event_notifier_cleanup (&vq->kick_notifier); +event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier);g_free (vq); }hw/virtio/vhost-vdpa.c +5
-1@@ -712,13 +712,40 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) {struct vhost_vdpa *v = dev->opaque;VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); -return vhost_svq_start (dev, idx, svq); +EventNotifier *vhost_call_notifier =vhost_svq_get_svq_call_notifier (svq); +struct vhost_vring_file vhost_call_file = { + .index = idx + dev->vq_index, + .fd =event_notifier_get_fd (vhost_call_notifier), + };
Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 724
Quoted 5 lines
hw/virtio/vhost-vdpa.c +5
-0+int r; +bool b; + +/* Set shadow vq -> guest notifier */ +assert (v->call_fd[idx]);We need aovid the
asser()here. On which case we can hit this?
I would say that there is no way we can actually hit it, so let's remove it.
Quoted 18 lines
Quoted 4 lines
hw/virtio/vhost-vdpa.c +3
-0+vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]); + + b =vhost_svq_start (dev, idx, svq);hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 737
Quoted 10 lines
hw/virtio/vhost-vdpa.c +10
-0+if (unlikely (!b)) { +return false; + } + +/* Set device -> SVQ notifier */ + r =vhost_vdpa_set_vring_dev_call (dev, &vhost_call_file); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_call for shadow vq failed" ); +return false; + }Similar to kick, do we need to
set_vring_call()beforevhost_svq_start()?
It should not matter at this moment because the device should not be started at this point and device calls should not run vhost_svq_handle_call until BQL is released.
The "logic" of doing it after is to make clear that svq must be fully initialized before processing device calls, even in the case that we extract SVQ in its own iothread or similar. But this could be done before vhost_svq_start for sure.
Quoted 8 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 740
Quoted 4 lines
hw/virtio/vhost-vdpa.c +3
-0+ +/* Check for pending calls */ +event_notifier_set (vhost_call_notifier);Interesting, can this result spurious interrupt?
This actually "queues" a vhost_svq_handle_call after the BQL release, where the device should be fully reset. In that regard, if there are no used descriptors there will not be an irq raised to the guest. Does that answer the question? Or have I missed something?
Quoted 24 lines
Quoted 24 lines
hw/virtio/vhost-vdpa.c +10
-0+return true; }static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev;unsigned n; +int r;if (enable == v->shadow_vqs_enabled) {return hdev->nvqs;@@ -752,9 +779,18 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)if (!enable) {/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { +struct vhost_vring_file file = { + .index =vhost_vdpa_get_vq_index (hdev, n), + .fd = v->call_fd[n], + }; + + r =vhost_vdpa_set_vring_call (hdev, &file); +assert (r ==0 ); +
Quoted 9 lines
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 793
Quoted 4 lines
hw/virtio/vhost-vdpa.c +1
-0unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n);VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n);vhost_svq_stop (hdev, n, svq); +/* TODO: This can unmask or override call fd! */ I don't get this comment. Does this mean the current code can't work with
mask_notifiers? If yes, this is something we need to fix.
Yes, but it will be addressed in the next series. I should have explained it bette here, sorry :).
Thanks!
Quoted 7 lines
Thanks
Quoted 3 lines
hw/virtio/vhost-vdpa.c +0
-0vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); }
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 15/20] vhost: Shadow virtqueue buffers forwarding
In reply to Jason WangQuoted 78 lines · Jason Wang
Quoted 75 lines · Eugenio Pérez
Initial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device.
For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series.
It reuses the VirtQueue code for the device part. The driver part is based on Linux's
virtio_ringdriver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones.SVQ uses
VIRTIO_CONFIG_S_DEVICE_STOPPEDto pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter.This status bit
VIRTIO_CONFIG_S_DEVICE_STOPPEDis currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits.Removal of
_S_DEVICE_STOPPEDbit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device.Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-)qapi/net.json +1
-1@@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding path # # Returns: Error if failure, or 'no error' for success. #hw/virtio/vhost-shadow-virtqueue.c +11
-0@@ -10,6 +10,7 @@#include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" +#include "hw/virtio/virtio-access.h" #include "standard-headers/linux/vhost_types.h" @@ -44,15 +45,135 @@ typedef struct VhostShadowVirtqueue {/* Virtio device */ VirtIODevice *vdev; + +/* Map for returning guest's descriptors */ +VirtQueueElement **ring_id_maps; + +/* Next head to expose to device */ +uint16_t avail_idx_shadow; + +/* Next free descriptor */ +uint16_t free_head; +
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 62
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+/* Last seen used idx */ +uint16_t shadow_used_idx; + +/* Next head to consume from device */ +uint16_t used_idx;Let's use "
last_used_idx" as kernel driver did.
Ok I will change it.
Quoted 14 lines
Quoted 14 lines
hw/virtio/vhost-shadow-virtqueue.c +7
-1} VhostShadowVirtqueue;/* If the device is using some of these, SVQ cannot communicate */ bool vhost_svq_valid_device_features (uint64_t *dev_features) { -return true; +uint64_t b; +bool r = true; + +for (b =VIRTIO_TRANSPORT_F_START ; b <=VIRTIO_TRANSPORT_F_END ; ++b) { +switch (b) { +case VIRTIO_F_NOTIFY_ON_EMPTY : +case VIRTIO_F_ANY_LAYOUT :
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 79
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+/* SVQ is fine with this feature */ +continue ; + +case VIRTIO_F_ACCESS_PLATFORM : +/* SVQ needs this feature disabled. Can't continue */ So code can explain itself, need a comment to explain why.
Do you mean that it *doesn't* need a comment to explain why? In that case I will delete them.
Quoted 11 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 87
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-0+if (*dev_features &BIT_ULL (b)) { +clear_bit (b, dev_features); + r = false; + } +break ; + +case VIRTIO_F_VERSION_1 : +/* SVQ needs this feature, so can't continue */
Quoted 2 lines
A comment to explain why SVQ needs this feature.
Sure I will add it.
Quoted 15 lines
Quoted 15 lines
hw/virtio/vhost-shadow-virtqueue.c +14
-0+if (!(*dev_features &BIT_ULL (b))) { +set_bit (b, dev_features); + r = false; + } +continue ; + +default : +/* +* SVQ must disable this feature, let's hope the device is fine +* without it. +*/ +if (*dev_features &BIT_ULL (b)) { +clear_bit (b, dev_features); + }
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 106
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+ } + } + +return r; +}Let's move this to patch 14.
I can move it down to 14/20, but then it is not really accurate, since notifications forwarding can work with all feature sets. Not like we are introducing a regression, but still.
I can always explain that in the patch message though, would that be ok?
Quoted 79 lines
Quoted 79 lines
hw/virtio/vhost-shadow-virtqueue.c +68
-1+ +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq, +const struct iovec *iovec, +size_t num,bool more_descs,bool write) +{ +uint16_t i = svq->free_head, last = svq->free_head; +unsigned n; +uint16_t flags = write ?cpu_to_le16 (VRING_DESC_F_WRITE ) :0 ; +vring_desc_t *descs = svq->vring.desc; + +if (num ==0 ) { +return ; + } + +for (n =0 ; n < num; n++) { +if (more_descs || (n +1 < num)) { + descs[i].flags = flags |cpu_to_le16 (VRING_DESC_F_NEXT ); + }else { + descs[i].flags = flags; + } + descs[i].addr =cpu_to_le64 ((hwaddr )iovec[n].iov_base); + descs[i].len =cpu_to_le32 (iovec[n].iov_len); + + last = i; + i =cpu_to_le16 (descs[i].next); + } + + svq->free_head =le16_to_cpu (descs[last].next); +} + +static unsigned vhost_svq_add_split (VhostShadowVirtqueue *svq, +VirtQueueElement *elem) +{ +int head; +unsigned avail_idx; +vring_avail_t *avail = svq->vring.avail; + + head = svq->free_head; + +/* We need some descriptors here */ +assert (elem->out_num || elem->in_num); + +vhost_vring_write_descs (svq, elem->out_sg, elem->out_num, + elem->in_num >0 , false); +vhost_vring_write_descs (svq, elem->in_sg, elem->in_num, false, true); + +/* +* Put entry in available array (but don't update avail->idx until they +* do sync). +*/ + avail_idx = svq->avail_idx_shadow & (svq->vring.num -1 ); + avail->ring[avail_idx] =cpu_to_le16 (head); + svq->avail_idx_shadow++; + +/* Update avail index after the descriptor is wrote */ +smp_wmb (); + avail->idx =cpu_to_le16 (svq->avail_idx_shadow); + +return head; + } -/* Forward guest notifications */ +static void vhost_svq_add (VhostShadowVirtqueue *svq,VirtQueueElement *elem) +{ +unsigned qemu_head =vhost_svq_add_split (svq, elem); + + svq->ring_id_maps[qemu_head] = elem; +} + +/* Handle guest->device notifications */ static void vhost_handle_guest_kick (EventNotifier *n) { VhostShadowVirtqueue *svq = container_of(n, VhostShadowVirtqueue,@@ -62,7 +183,74 @@ static void vhost_handle_guest_kick(EventNotifier *n)return ; }
Quoted 11 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 191
Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-1-event_notifier_set (&svq->kick_notifier); +/* Make available as many buffers as possible */ + do { + if (virtio_queue_get_notification(svq->vq)) { + /* No more notifications until process all available */ + virtio_queue_set_notification(svq->vq, false); + }This can be done outside the loop.
I think it cannot. The intention of doing this way is that we check for new available buffers *also after* enabling notifications, so we don't miss any of them. It is more or less copied from virtio_blk_handle_vq, which also needs to run to completion.
If we need to loop again because there are more available buffers, we want to disable notifications again. Or am I missing something?
Quoted 17 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 203
Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c +12
-0+ +while (true) { +VirtQueueElement *elem =virtqueue_pop (svq->vq,sizeof (*elem)); +if (!elem) { +break ; + } + +vhost_svq_add (svq, elem); +event_notifier_set (&svq->kick_notifier); + } + +virtio_queue_set_notification (svq->vq, true);I think this can be moved to the end of this function.
(Same as previous answer)
Quoted 6 lines
Btw, we probably need a quota to make sure the svq is not hogging the main event loop.
Similar issue could be found in both virtio-net TX (using timer or bh) and TAP (a quota).
I think that virtqueue size is the natural limit to that: since we are not making any buffers used in the loop, there is no way that it runs more than virtqueue size times. If it does because of an evil/bogus guest, virtqueue_pop raises the message "Virtqueue size exceeded" and returns NULL, effectively breaking the loop.
Virtio-net tx functions mark each buffer right after making them available and use it, so they can hog BQL. But my understanding is that is not possible in the SVQ case.
I can add a comment in the code to make it clearer though.
Quoted 17 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 214
Quoted 12 lines
hw/virtio/vhost-shadow-virtqueue.c +11
-0+ }while (!virtio_queue_empty (svq->vq)); +} + +static bool vhost_svq_more_used (VhostShadowVirtqueue *svq) +{ +if (svq->used_idx != svq->shadow_used_idx) { +return true; + } + +/* Get used idx must not be reordered */ +smp_rmb ();Interesting, we don't do this for kernel drivers. It would be helpful to explain it more clear by "X must be done before Y".
I think this got reordered, it's supposed to be *after* get the used idx, so it matches the one in the kernel with the comment "Only get used array entries after they have been exposed by host.".
I will change it for the next series.
Quoted 58 lines
Quoted 58 lines
hw/virtio/vhost-shadow-virtqueue.c +48
-2+ svq->shadow_used_idx =cpu_to_le16 (svq->vring.used->idx); + +return svq->used_idx != svq->shadow_used_idx; +} + +static VirtQueueElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) +{ +vring_desc_t *descs = svq->vring.desc; +const vring_used_t *used = svq->vring.used; +vring_used_elem_t used_elem; +uint16_t last_used; + +if (!vhost_svq_more_used (svq)) { +return NULL ; + } + + last_used = svq->used_idx & (svq->vring.num -1 ); + used_elem.id =le32_to_cpu (used->ring[last_used].id); + used_elem.len =le32_to_cpu (used->ring[last_used].len); + + svq->used_idx++; +if (unlikely (used_elem.id >= svq->vring.num)) { +error_report ("Device %s says index %u is used" , svq->vdev->name, + used_elem.id); +return NULL ; + } + +if (unlikely (!svq->ring_id_maps[used_elem.id])) { +error_report ( +"Device %s says index %u is used, but it was not available" , + svq->vdev->name, used_elem.id); +return NULL ; + } + + descs[used_elem.id].next = svq->free_head; + svq->free_head = used_elem.id; + + svq->ring_id_maps[used_elem.id]->len = used_elem.len; +return g_steal_pointer (&svq->ring_id_maps[used_elem.id]); }/* Forward vhost notifications */ @@ -70,8 +258,26 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n) {VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, call_notifier); - -event_notifier_set (&svq->guest_call_notifier); +VirtQueue *vq = svq->vq; + +/* Make as many buffers as possible used. */ + do { + unsigned i = 0; + + /* TODO: Use VRING_AVAIL_F_NO_INTERRUPT */ + while (true) { + g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq);
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_handle_call_no_test(EventNotifier *n) · line 274
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+if (!elem) { +break ; + } + +assert (i < svq->vring.num);Let's return error instead of using the assert.
Actually this is a condition that we should never meet: In the case of ring overrun, device would try to set used a descriptor that is either
Quoted 1 line
vring size *or* should try to overrun some of the already used ones.
In both cases, elem should be NULL and the loop should break.
So this is a safety net protecting from both, if we have an i > svq->vring.num means we are not processing used buffers well anymore, and (moreover) this is happening after making used all descriptors.
Taking that into account, should we delete it?
Quoted 21 lines
Quoted 21 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-0+virtqueue_fill (vq, elem, elem->len, i++); + } + +virtqueue_flush (vq, i); +event_notifier_set (&svq->guest_call_notifier); + }while (vhost_svq_more_used (svq)); }static void vhost_svq_handle_call (EventNotifier *n)@@ -204,12 +410,25 @@ err_set_vring_kick:void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq) { +int i;int r =vhost_svq_restore_vdev_host_notifier (dev, idx, svq); +if (unlikely (r <0 )) {error_report ("Couldn't restore vq kick fd: %s" ,strerror (-r)); }
Quoted 17 lines
hw/virtio/vhost-shadow-virtqueue.c err_set_vring_kick: · line 430
Quoted 11 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0event_notifier_set_handler (&svq->host_notifier,NULL ); + +for (i =0 ; i < svq->vring.num; ++i) { + g_autofree VirtQueueElement *elem = svq->ring_id_maps[i]; +/* +* Although the doc says we must unpop in order, it's ok to unpop +* everything. +*/ +if (elem) { +virtqueue_unpop (svq->vq, elem, elem->len); + }Will this result some of the "pending" buffers to be submitted multiple times? If yes, should we wait for all the buffers used instead of doing the unpop here?
Do you mean to call virtqueue_unpop with the same elem (or elem.id) multiple times? That should never happen, because elem.id should be the position in the ring_id_maps. Also, unpop() should just unmap the element and never sync again.
Maybe it is way clearer to call virtqueue_detach_element here directly.
Quoted 79 lines
Quoted 79 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-1+ } }/* @@ -224,7 +443,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)size_t driver_size;size_t device_size; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1); -int r; +int r, i; r =event_notifier_init (&svq->kick_notifier,0 );if (r !=0 ) {@@ -250,6 +469,11 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)memset (svq->vring.desc,0 , driver_size); svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size);memset (svq->vring.used,0 , device_size); +for (i =0 ; i < num -1 ; i++) { + svq->vring.desc[i].next =cpu_to_le16 (i +1 ); + } + + svq->ring_id_maps = g_new0(VirtQueueElement *, num);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);@@ -269,6 +493,7 @@ void vhost_svq_free(VhostShadowVirtqueue *vq)event_notifier_cleanup (&vq->kick_notifier);event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier); +g_free (vq->ring_id_maps);qemu_vfree (vq->vring.desc);qemu_vfree (vq->vring.used);g_free (vq);hw/virtio/vhost-vdpa.c +23
-0@@ -19,6 +19,7 @@#include "hw/virtio/virtio-net.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" #include "cpu.h" @@ -475,6 +476,28 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,return vhost_vdpa_backend_set_features (dev, features); } +/** +* Restore guest features to vdpa device +*/ +static int vhost_vdpa_set_guest_features (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +return vhost_vdpa_backend_set_features (dev, v->guest_features); +} + +/** +* Set shadow virtqueue supported features +*/ +static int vhost_vdpa_set_svq_features (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +uint64_t features = v->host_features; +bool b =vhost_svq_valid_device_features (&features); +assert (b); + +return vhost_vdpa_backend_set_features (dev, features); +} +static int vhost_vdpa_set_backend_cap (struct vhost_dev *dev) {uint64_t features;@@ -730,6 +753,19 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; }
Quoted 12 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 763
Quoted 8 lines
hw/virtio/vhost-vdpa.c +8
-0+static int vhost_vdpa_vring_pause (struct vhost_dev *dev) +{ +int r; +uint8_t status; + +vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_DEVICE_STOPPED ); + do { + r = vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);I guess we'd better add some sleep here.
If the final version still contains the call, I will add the sleep. At the moment I think it's better if we stop the device by a vdpa ioctl.
Quoted 76 lines
Quoted 76 lines
hw/virtio/vhost-vdpa.c +45
-3+ }while (r ==0 && !(status &VIRTIO_CONFIG_S_DEVICE_STOPPED )); + +return 0 ; +} +/* * Start shadow virtqueue. */ @@ -742,9 +778,29 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) .index = idx + dev->vq_index, .fd = event_notifier_get_fd(vhost_call_notifier), }; +struct vhost_vring_addr addr = { + .index = idx + dev->vq_index, + }; +struct vhost_vring_state num = { + .index = idx + dev->vq_index, + .num =virtio_queue_get_num (dev->vdev, idx), + };int r;bool b; +vhost_svq_get_vring_addr (svq, &addr); + r =vhost_vdpa_set_vring_addr (dev, &addr); +if (unlikely (r)) { +error_report ("vhost_set_vring_addr for shadow vq failed" ); +return false; + } + + r =vhost_vdpa_set_vring_num (dev, &num); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_num for shadow vq failed" ); +return false; + } +/* Set shadow vq -> guest notifier */ assert (v->call_fd[idx]);vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]);@@ -781,15 +837,32 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) {VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); -bool ok; -if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ; }g_ptr_array_add (v->shadow_vqs, svq); + } + } - ok =vhost_vdpa_svq_start_vq (hdev, n); + r =vhost_vdpa_vring_pause (hdev); +assert (r ==0 ); + +if (enable) { +for (n =0 ; n < v->shadow_vqs->len; ++n) { +/* Obtain Virtqueue state */ +vhost_virtqueue_stop (hdev, hdev->vdev, &hdev->vqs[n], n); + } + } + +/* Reset device so it can be configured */ + r =vhost_vdpa_dev_start (hdev, false); +assert (r ==0 ); + +if (enable) { +int r; +for (n =0 ; n < v->shadow_vqs->len; ++n) { +bool ok =vhost_vdpa_svq_start_vq (hdev, n);if (unlikely (!ok)) {/* Free still not started svqs */ g_ptr_array_set_size (v->shadow_vqs, n);@@ -797,11 +870,19 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)
Quoted 9 lines
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 874
Quoted 5 lines
hw/virtio/vhost-vdpa.c +2
-0break ; } } + +/* Need to ack features to set state in vp_vdpa devices */
vhost_vdpaactually?
Yes, what a mistake!
Quoted 40 lines
Quoted 40 lines
hw/virtio/vhost-vdpa.c +12
-0+ r =vhost_vdpa_set_svq_features (hdev); +if (unlikely (r)) { + enable = false; + } } v->shadow_vqs_enabled = enable;if (!enable) { +vhost_vdpa_set_guest_features (hdev); +/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { struct vhost_vring_file file = {@@ -818,7 +899,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* TODO: This can unmask or override call fd! */ vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); } + } + r =vhost_vdpa_dev_start (hdev, true); +assert (r ==0 ); + +if (!enable) {/* Resources cleanup */ g_ptr_array_set_size (v->shadow_vqs,0 ); }@@ -831,6 +917,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp)struct vhost_vdpa *v;const char *err_cause =NULL ;bool r; +uint64_t svq_features; QLIST_FOREACH(v, &vhost_vdpa_devices, entry) {if (v->dev->vdev &&0 ==strcmp (v->dev->vdev->name, name)) {@@ -846,6 +933,20 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) goto err; }
Quoted 11 lines
hw/virtio/vhost-vdpa.c void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) · line 943
Quoted 8 lines
hw/virtio/vhost-vdpa.c +8
-0+ svq_features = v->host_features; +if (!vhost_svq_valid_device_features (&svq_features)) { +error_setg (errp, +"Can't enable shadow vq on %s: Unexpected feature flags (%lx-%lx)" , + name, v->host_features, svq_features); +return ; + }else { +/* TODO: Check for virtio_vdpa + IOMMU & modern device */ I guess you mean "
vhost_vdpa" here.
Yes, a similar mistake in less than 50 lines :).
Quoted 3 lines
For IOMMU, I guess you mean "vIOMMU"
actually?
This comment is out of date and inherited from the vhost version, where only the IOMMU version was developed, so it will be deleted in the next series. I think it makes little sense to check vIOMMU if we stick with vDPA since it still does not support it, but we could make the check here for sure.
Thanks!
Quoted 22 lines
Thanks
Quoted 18 lines
hw/virtio/vhost-vdpa.c +7
-1+ } + +if (err_cause) { + goto err; + } + r =vhost_vdpa_enable_svq (v, enable);if (unlikely (!r)) { err_cause ="Error enabling (see monitor)" ;@@ -853,7 +954,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) } err: -if (err_cause) { +if (errp ==NULL && err_cause) {error_setg (errp,"Can't enable shadow vq on %s: %s" , name, err_cause); } }
Re: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue
In reply to Eugenio Perez MartinQuoted 27 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 4 lines · Eugenio Pérez
Shadow virtqueue notifications forwarding is disabled when
vhost_devstops, so code flow follows usual cleanup.Also, host notifiers must be disabled at SVQ start,
Any reason for this?
It will be addressed in a later series, sorry.
Quoted 11 lines
Quoted 3 lines
and they will not
start if SVQ has been enabled when device is stopped. This is trivial to address, but it is left out for simplicity at this moment.It looks to me this patch also contains the following logics
1) codes to enable svq
2) codes to let svq to be enabled from QMP.
I think they need to be split out,
I agree that we can split this more, with the code that belongs to SVQ and the code that belongs to vhost-vdpa. it will be addressed in future series.
Quoted 7 lines
Quoted 3 lines
we may endup with the following
series of patchesWith "series of patches" do you mean to send every step in a separated series? There are odds of having the need of modifying code already sent & merged with later ones. If you confirm to me that it is fine, I can do it that way for sure.
Sorry for being unclear. I meant it's a sub-series actually of the series.
Quoted 7 lines
Quoted 7 lines
1) svq skeleton with enable/disable
2) route host notifier to svq
3) route guest notifier to svq
4) codes to enable svq
5) enable svq via QMP
Quoted 5 lines
I'm totally fine with that, but there is code that is never called if the qmp command is not added. The compiler complains about static functions that are not called, making impossible things like bisecting through these commits, unless I use attribute((unused)) or similar. Or have I missed something?
You're right, then I think we can then:
1) svq skeleton with enable/disable via QMP
2) route host notifier to svq
3) route guest notifier to svq
Quoted 5 lines
We could do that way with the code that belongs to SVQ though, since all of it is declared in headers. But to delay the "enable svq via qmp" to the last one makes debugging harder, as we cannot just enable notifications forwarding with no buffers forwarding.
Yes.
Quoted 5 lines
If I introduce a change in the notifications code, I can simply go to these commits and enable SVQ for notifications. This way I can have an idea of what part is failing. A similar logic can be applied to other devices than
vp_vdpa.
vhost-vdpa actually?
Quoted 109 lines
We would lose it if we
Quoted 28 lines
Quoted 28 lines
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.h | 8 ++ include/hw/virtio/vhost-vdpa.h | 4 + hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- 5 files changed, 264 insertions(+), 4 deletions(-)qapi/net.json +1
-1@@ -88,7 +88,7 @@ # # @enable: true to use the alternate shadow VQ notifications # -# Returns: Always error, since SVQ is not implemented at the moment. +# Returns: Error if failure, or 'no error' for success. # # Since: 6.2 #hw/virtio/vhost-shadow-virtqueue.h +0
-0@@ -14,6 +14,14 @@Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.h line 17
Quoted 3 lines
hw/virtio/vhost-shadow-virtqueue.h +1
-0typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ; +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq);Let's move this function to another patch since it's unrelated to the
guest->hostrouting.Right, I missed it while squashing commits and at later reviews.
Quoted 61 lines
Quoted 61 lines
hw/virtio/vhost-shadow-virtqueue.h +7
-0+void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); + +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx);void vhost_svq_free (VhostShadowVirtqueue *vq);include/hw/virtio/vhost-vdpa.h +4
-0@@ -12,6 +12,8 @@#ifndef HW_VIRTIO_VHOST_VDPA_H #define HW_VIRTIO_VHOST_VDPA_H +#include <gmodule.h> +#include "qemu/queue.h" #include "hw/virtio/virtio.h" @@ -24,6 +26,8 @@ typedef struct vhost_vdpa {int device_fd;uint32_t msg_type;MemoryListener listener; +bool shadow_vqs_enabled; +GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry;VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ];hw/virtio/vhost-shadow-virtqueue.c +11
-1@@ -9,9 +9,12 @@#include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" +#include "hw/virtio/vhost.h" + +#include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" -#include "qemu/event_notifier.h" +#include "qemu/main-loop.h" /* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue {@@ -19,14 +22,146 @@ typedef struct VhostShadowVirtqueue {EventNotifier kick_notifier;/* Shadow call notifier, sent to vhost */ EventNotifier call_notifier; + +/* +* Borrowed virtqueue's guest to host notifier. +* To borrow it in this event notifier allows to register on the event +* loop and access the associated shadow virtqueue easily. If we use the +* VirtQueue, we don't have an easy way to retrieve it. +* Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 37
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+* So shadow virtqueue must not clean it, or we would lose VirtQueue one. +*/ +EventNotifier host_notifier; + +/* Guest's call notifier, where SVQ calls guest. */ +EventNotifier guest_call_notifier;To be consistent, let's simply use "
guest_notifier" here.
Quoted 5 lines
It could be confused when the series adds a guest -> qemu kick notifier then. Actually, I think it would be better to rename
host_notifierto something likehost_svq_notifier. Orhost_callandguest_call, since "notifier" is already in the type, making the name to be a little bit "Hungarian notation".
I think that's fine, just need to make sure we have a consistent name for SVQ notifier.
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +3
-0+ +/* Virtio queue shadowing */ +VirtQueue *vq; } VhostShadowVirtqueue;Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 51
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c +9
-0+/* Forward guest notifications */ +static void vhost_handle_guest_kick (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + host_notifier); + +if (unlikely (!event_notifier_test_and_clear (n))) { +return ; + }Is there a chance that we may stop the processing of available buffers during the svq enabling? There could be no kick from the guest in this case.
Quoted 6 lines
Actually, yes, I think you are right. The guest kick eventfd could have been consumed by vhost but there may be still pending buffers.
I think it would be better to check for available buffers first, then clear the notifier unconditionally, and then re-check and process them if any [1].
Looks like I can't find "[1]" anywhere.
Quoted 6 lines
However, this problem arises later in the series: At this moment the device is not reset and guest's host notifier is not replaced, so either vhost/device receives the kick, or SVQ does and forwards it.
Does it make sense to you?
Kind of, so I think we can:
1) As you said, always check available buffers when switching to SVQ
2) alwasy kick the vhost when switching back to vhost
Quoted 47 lines
Quoted 39 lines
Quoted 39 lines
hw/virtio/vhost-shadow-virtqueue.c +38
-0+ +event_notifier_set (&svq->kick_notifier); +} + +/* +* Obtain the SVQ call notifier, where vhost device notifies SVQ that there +* exists pending used buffers. +* +* @svq Shadow Virtqueue +*/ +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq) +{ +return &svq->call_notifier; +} + +/* +* Set the call notifier for the SVQ to call the guest +* +* @svq Shadow virtqueue +* @call_fd call notifier +* +* Called on BQL context. +*/ +void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd) +{ +event_notifier_init_fd (&svq->guest_call_notifier, call_fd); +} + +/* +* Restore the vhost guest to host notifier, i.e., disables svq effect. +*/ +static int vhost_svq_restore_vdev_host_notifier (struct vhost_dev *dev, +unsigned vhost_index, +VhostShadowVirtqueue *svq) +{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = vhost_index,Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 95
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+ .fd = event_notifier_get_fd(vq_host_notifier), + }; +int r; + +/* Restore vhost kick */ + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);And remap the notification area if necessary.
Quoted 5 lines
Totally right, that step is missed in this series.
However, remapping guest host notifier memory region has no advantages over using ioeventfd to perform guest -> SVQ notifications, doesn't it? By both methods flow needs to go through the hypervisor kernel.
To be clear, I meant restore the notification area mapping from guest to device directly. For SVQ, you are right, there's no much value for bothering notification area map. (Or we can do it in the future).
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
hw/virtio/vhost-shadow-virtqueue.c +22
-0+return r ? -errno :0 ; +} + +/* +* Start shadow virtqueue operation. +* @dev vhost device +* @hidx vhost virtqueue index +* @svq Shadow Virtqueue +*/ +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = dev->vhost_ops->vhost_get_vq_index (dev, dev->vq_index + idx), + .fd =event_notifier_get_fd (&svq->kick_notifier), + }; +int r; + +/* Check that notifications are still going directly to vhost dev */ +assert (virtio_queue_is_host_notifier_enabled (svq->vq)); +Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 122
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+/* +* event_notifier_set_handler already checks for guest's notifications if +* they arrive in the switch, so there is no need to explicitely check for +* them. +*/ If this is true, shouldn't we call
vhost_set_vring_kick()before theevent_notifier_set_handler()?
Quoted 5 lines
Not at this point of the series, but it could be another solution when we need to reset the device and we are unsure if all buffers have been read. But I think I prefer the solution exposed in [1] and to explicitly call
vhost_handle_guest_kickhere. Do you think differently?
I actually mean if we can end up with this situation since SVQ take over the host notifier before set_vring_kick().
1) guest kick vq, vhost is running
2) qemu take over the host notifier
3) guest kick vq
4) qemu route host notifier to SVQ
Then the vq will be handled by both SVQ and vhost?
Quoted 10 lines
Quoted 4 lines
Btw, I think we should update the fd if
set_vring_kick()was called after this function?Kind of. This is currently bad in the code, but...
Backend callbacks
vhost_ops->vhost_set_vring_kickandvhost_ops->vhost_set_vring_addrare only called atvhost_virtqueue_start. And they are always called with known data already stored in VirtQueue.
This is true for now, but I'd suggest to not depend on this since it:
1) it might be changed in the future
2) we're working at vhost layer and expose API to virtio device, the code should be robust to handle set_vring_kick() at any time
3) I think we've already handled similar situation of set_vring_call, so let's be consistent
Quoted 7 lines
To avoid storing more state in
vhost_vdpa, I think that we should avoid duplicating them, but ignore newkick_fdor address in SVQ mode, and retrieve them again at the moment the device is (re)started in SVQ mode. Qemu already avoids things like allowing the guest to set addresses at random time, using the VirtIOPCIProxy to store them.
Quoted 5 lines
I also see how duplicating that status could protect vdpa SVQ code against future changes to vhost code, but that would make this series bigger and more complex with no need at this moment in my opinion.
Do you agree?
Somehow, but consider we can handle set_vring_call(), let's at last make set_vring_kick() more robust.
Quoted 191 lines
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 127
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+event_notifier_init_fd (&svq->host_notifier, +event_notifier_get_fd (vq_host_notifier)); +event_notifier_set_handler (&svq->host_notifier, vhost_handle_guest_kick); + + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);And we need to stop the notification area mmap.
Right.
Quoted 122 lines
Quoted 112 lines
hw/virtio/vhost-shadow-virtqueue.c +32
-0+if (unlikely (r !=0 )) { +error_report ("Couldn't set kick fd: %s" ,strerror (errno)); + goto err_set_vring_kick; + } + +return true; + +err_set_vring_kick: +event_notifier_set_handler (&svq->host_notifier,NULL ); + +return false; +} + +/* +* Stop shadow virtqueue operation. +* @dev vhost device +* @idx vhost queue index +* @svq Shadow Virtqueue +*/ +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +int r =vhost_svq_restore_vdev_host_notifier (dev, idx, svq); +if (unlikely (r <0 )) { +error_report ("Couldn't restore vq kick fd: %s" ,strerror (-r)); + } + +event_notifier_set_handler (&svq->host_notifier,NULL ); +} +/* * Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) { +int vq_idx = dev->vq_index + idx; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);int r;@@ -44,6 +179,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) goto err_init_call_notifier; } + svq->vq =virtio_get_queue (dev->vdev, vq_idx);return g_steal_pointer (&svq); err_init_call_notifier:hw/virtio/vhost-vdpa.c +24
-0@@ -17,6 +17,7 @@#include "hw/virtio/vhost.h" #include "hw/virtio/vhost-backend.h" #include "hw/virtio/virtio-net.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" @@ -272,6 +273,16 @@ static void vhost_vdpa_add_status(struct vhost_dev *dev, uint8_t status)vhost_vdpa_call (dev,VHOST_VDPA_SET_STATUS , &s); } +/** +* Adaptor function to free shadow virtqueue through gpointer +* +* @svq The Shadow Virtqueue +*/ +static void vhost_psvq_free (gpointer svq) +{ +vhost_svq_free (svq); +} +static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp) {struct vhost_vdpa *v;@@ -283,6 +294,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; + v->shadow_vqs =g_ptr_array_new_full (dev->nvqs, vhost_psvq_free);QLIST_INSERT_HEAD (&vhost_vdpa_devices, v, entry); vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |@@ -373,6 +385,17 @@ err:return ; } +static void vhost_vdpa_svq_cleanup (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +size_t idx; + +for (idx =0 ; idx < v->shadow_vqs->len; ++idx) { +vhost_svq_stop (dev, idx,g_ptr_array_index (v->shadow_vqs, idx)); + } +g_ptr_array_free (v->shadow_vqs, true); +} +static int vhost_vdpa_cleanup (struct vhost_dev *dev) {struct vhost_vdpa *v;@@ -381,6 +404,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev)trace_vhost_vdpa_cleanup (dev, v);vhost_vdpa_host_notifiers_uninit (dev, dev->nvqs);memory_listener_unregister (&v->listener); +vhost_vdpa_svq_cleanup (dev);QLIST_REMOVE (v, entry); dev->opaque =NULL ;hw/virtio/vhost-vdpa.c static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) · line 586
Quoted 7 lines
hw/virtio/vhost-vdpa.c +3
-1@@ -557,7 +581,9 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started)if (started) {uint8_t status =0 ;memory_listener_register (&v->listener, &address_space_memory); -vhost_vdpa_host_notifiers_init (dev); +if (!v->shadow_vqs_enabled) { +vhost_vdpa_host_notifiers_init (dev); + }This looks like a trick, why not check and setup
shadow_vqsinside:Quoted 6 lines
1)
vhost_vdpa_host_notifiers_init()and
2)
vhost_vdpa_set_vring_kick()Ok I will move the checks there.
Quoted 36 lines
Quoted 36 lines
hw/virtio/vhost-vdpa.c +29
-0vhost_vdpa_set_vring_ready (dev);vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_DRIVER_OK );vhost_vdpa_call (dev,VHOST_VDPA_GET_STATUS , &status);@@ -663,10 +689,96 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +/* +* Start shadow virtqueue. +*/ +static bool vhost_vdpa_svq_start_vq (struct vhost_dev *dev,unsigned idx) +{ +struct vhost_vdpa *v = dev->opaque; +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); +return vhost_svq_start (dev, idx, svq); +} + +static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) +{ +struct vhost_dev *hdev = v->dev; +unsigned n; + +if (enable == v->shadow_vqs_enabled) { +return hdev->nvqs; + } + +if (enable) { +/* Allocate resources */ +assert (v->shadow_vqs->len ==0 ); +for (n =0 ; n < hdev->nvqs; ++n) { +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +bool ok; + +if (unlikely (!svq)) { +g_ptr_array_set_size (v->shadow_vqs,0 ); +return 0 ;Quoted 8 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 728
Quoted 8 lines
hw/virtio/vhost-vdpa.c +8
-0+ } +g_ptr_array_add (v->shadow_vqs, svq); + + ok =vhost_vdpa_svq_start_vq (hdev, n); +if (unlikely (!ok)) { +/* Free still not started svqs */ +g_ptr_array_set_size (v->shadow_vqs, n); + enable = false;[2]
Quoted 4 lines
Quoted 3 lines
hw/virtio/vhost-vdpa.c +3
-0+break ; + } + }
Quoted 8 lines
Quoted 4 lines
Since there's almost no logic could be shared between enable and disable. Let's split those logic out into dedicated functions where the codes looks more easy to be reviewed (e.g have a better error handling etc).
Maybe it could be more clear in the code, but the reused logic is the disabling of SVQ and the fallback in case it cannot be enabled with [2]. But I'm not against splitting in two different functions if it makes review easier.
Ok.
Quoted 52 lines
Quoted 21 lines
Quoted 21 lines
hw/virtio/vhost-vdpa.c +19
-0+ } + + v->shadow_vqs_enabled = enable; + +if (!enable) { +/* Disable all queues or clean up failed start */ +for (n =0 ; n < v->shadow_vqs->len; ++n) { +unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n); +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n); +vhost_svq_stop (hdev, n, svq); +vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); + } + +/* Resources cleanup */ +g_ptr_array_set_size (v->shadow_vqs,0 ); + } + +return n; +}Quoted 15 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 762
Quoted 12 lines
hw/virtio/vhost-vdpa.c +9
-1void qmp_x_vhost_enable_shadow_vq (const char *name,bool enable,Error **errp) { -error_setg (errp,"Shadow virtqueue still not implemented" ); +struct vhost_vdpa *v; +const char *err_cause =NULL ; +bool r; + + QLIST_FOREACH(v, &vhost_vdpa_devices, entry) { +if (v->dev->vdev &&0 ==strcmp (v->dev->vdev->name, name)) { +break ; + } + }I think you can iterate the NetClientStates to ge tthe vhost-vdpa backends.
Right, I missed it.
Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 768
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+ +if (!v) { + err_cause ="Device not found" ; + goto err; + }else if (v->notifier[0 ].addr) { + err_cause ="Device has host notifiers enabled" ;I don't get this.
At this moment of the series you can enable guest -> SVQ -> 'vdpa device' if the device is not using the host notifiers memory region. The right solution is to disable it for the guest, and to handle it in SVQ. Otherwise, guest kick will bypass SVQ and
Quoted 6 lines
It can be done in the same patch, or at least to disable (as unmap) them at this moment and handle them in a posterior patch. but for prototyping the solution I just ignored it in this series. It will be handled some way or another in the next one. I prefer the last one, to handle in a different patch, but let me know if you think it is better otherwise.
Aha, I see. But I think we need to that in this patch otherwise the we can route host notifier to SVQ.
Thanks
Quoted 26 lines
Quoted 4 lines
Btw this function should be implemented in an independent patch after svq is fully functional.
(Reasons for that are already commented at the top of this mail :) ).
Thanks!
Quoted 18 lines
Thanks
Quoted 15 lines
hw/virtio/vhost-vdpa.c +13
-0+ goto err; + } + + r =vhost_vdpa_enable_svq (v, enable); +if (unlikely (!r)) { + err_cause ="Error enabling (see monitor)" ; + goto err; + } + +err: +if (err_cause) { +error_setg (errp,"Can't enable shadow vq on %s: %s" , name, err_cause); + } }
Quoted 1 line
Quoted 1 line
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 784
Quoted 1 line
hw/virtio/vhost-vdpa.c +0
-0const VhostOps vdpa_ops = {
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 15/20] vhost: Shadow virtqueue buffers forwarding
In reply to Eugenio Perez MartinQuoted 109 lines · Eugenio Perez Martin
Quoted 76 lines · Jason Wang
Quoted 74 lines · Eugenio Pérez
Initial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device.
For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series.
It reuses the VirtQueue code for the device part. The driver part is based on Linux's
virtio_ringdriver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones.SVQ uses
VIRTIO_CONFIG_S_DEVICE_STOPPEDto pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter.This status bit
VIRTIO_CONFIG_S_DEVICE_STOPPEDis currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits.Removal of
_S_DEVICE_STOPPEDbit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device.Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-)qapi/net.json +1
-1@@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding path # # Returns: Error if failure, or 'no error' for success. #hw/virtio/vhost-shadow-virtqueue.c +10
-0@@ -10,6 +10,7 @@#include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" +#include "hw/virtio/virtio-access.h" #include "standard-headers/linux/vhost_types.h" @@ -44,15 +45,135 @@ typedef struct VhostShadowVirtqueue {/* Virtio device */ VirtIODevice *vdev; + +/* Map for returning guest's descriptors */ +VirtQueueElement **ring_id_maps; + +/* Next head to expose to device */ +uint16_t avail_idx_shadow; + +/* Next free descriptor */ +uint16_t free_head;Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 62
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+ +/* Last seen used idx */ +uint16_t shadow_used_idx; + +/* Next head to consume from device */ +uint16_t used_idx;Let's use "
last_used_idx" as kernel driver did.Ok I will change it.
Quoted 12 lines
Quoted 12 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-1} VhostShadowVirtqueue;/* If the device is using some of these, SVQ cannot communicate */ bool vhost_svq_valid_device_features (uint64_t *dev_features) { -return true; +uint64_t b; +bool r = true; + +for (b =VIRTIO_TRANSPORT_F_START ; b <=VIRTIO_TRANSPORT_F_END ; ++b) { +switch (b) { +case VIRTIO_F_NOTIFY_ON_EMPTY :Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 79
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+case VIRTIO_F_ANY_LAYOUT : +/* SVQ is fine with this feature */ +continue ; + +case VIRTIO_F_ACCESS_PLATFORM : +/* SVQ needs this feature disabled. Can't continue */ So code can explain itself, need a comment to explain why.
Quoted 2 lines
Do you mean that it *doesn't* need a comment to explain why? In that case I will delete them.
I meant the comment is duplicated with the code. If you wish, you can explain why ACCESS_PLATFORM needs to be disabled.
Quoted 36 lines
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 87
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-0+if (*dev_features &BIT_ULL (b)) { +clear_bit (b, dev_features); + r = false; + } +break ; + +case VIRTIO_F_VERSION_1 : +/* SVQ needs this feature, so can't continue */ Quoted 2 lines
A comment to explain why SVQ needs this feature.
Sure I will add it.
Quoted 13 lines
Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c +13
-0+if (!(*dev_features &BIT_ULL (b))) { +set_bit (b, dev_features); + r = false; + } +continue ; + +default : +/* +* SVQ must disable this feature, let's hope the device is fine +* without it. +*/ +if (*dev_features &BIT_ULL (b)) { +clear_bit (b, dev_features);Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 106
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+ } + } + } + +return r; +}Let's move this to patch 14.
Quoted 5 lines
I can move it down to 14/20, but then it is not really accurate, since notifications forwarding can work with all feature sets. Not like we are introducing a regression, but still.
I can always explain that in the patch message though, would that be ok?
I'm afraid this will break bisection. E.g for patch 14, it works for any features but for patch 15 it doesn't.
Quoted 89 lines
Quoted 79 lines
Quoted 79 lines
hw/virtio/vhost-shadow-virtqueue.c +68
-1+ +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq, +const struct iovec *iovec, +size_t num,bool more_descs,bool write) +{ +uint16_t i = svq->free_head, last = svq->free_head; +unsigned n; +uint16_t flags = write ?cpu_to_le16 (VRING_DESC_F_WRITE ) :0 ; +vring_desc_t *descs = svq->vring.desc; + +if (num ==0 ) { +return ; + } + +for (n =0 ; n < num; n++) { +if (more_descs || (n +1 < num)) { + descs[i].flags = flags |cpu_to_le16 (VRING_DESC_F_NEXT ); + }else { + descs[i].flags = flags; + } + descs[i].addr =cpu_to_le64 ((hwaddr )iovec[n].iov_base); + descs[i].len =cpu_to_le32 (iovec[n].iov_len); + + last = i; + i =cpu_to_le16 (descs[i].next); + } + + svq->free_head =le16_to_cpu (descs[last].next); +} + +static unsigned vhost_svq_add_split (VhostShadowVirtqueue *svq, +VirtQueueElement *elem) +{ +int head; +unsigned avail_idx; +vring_avail_t *avail = svq->vring.avail; + + head = svq->free_head; + +/* We need some descriptors here */ +assert (elem->out_num || elem->in_num); + +vhost_vring_write_descs (svq, elem->out_sg, elem->out_num, + elem->in_num >0 , false); +vhost_vring_write_descs (svq, elem->in_sg, elem->in_num, false, true); + +/* +* Put entry in available array (but don't update avail->idx until they +* do sync). +*/ + avail_idx = svq->avail_idx_shadow & (svq->vring.num -1 ); + avail->ring[avail_idx] =cpu_to_le16 (head); + svq->avail_idx_shadow++; + +/* Update avail index after the descriptor is wrote */ +smp_wmb (); + avail->idx =cpu_to_le16 (svq->avail_idx_shadow); + +return head; + } -/* Forward guest notifications */ +static void vhost_svq_add (VhostShadowVirtqueue *svq,VirtQueueElement *elem) +{ +unsigned qemu_head =vhost_svq_add_split (svq, elem); + + svq->ring_id_maps[qemu_head] = elem; +} + +/* Handle guest->device notifications */ static void vhost_handle_guest_kick (EventNotifier *n) { VhostShadowVirtqueue *svq = container_of(n, VhostShadowVirtqueue,@@ -62,7 +183,74 @@ static void vhost_handle_guest_kick(EventNotifier *n)return ; }Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 191
Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-1-event_notifier_set (&svq->kick_notifier); +/* Make available as many buffers as possible */ + do { + if (virtio_queue_get_notification(svq->vq)) { + /* No more notifications until process all available */ + virtio_queue_set_notification(svq->vq, false); + }This can be done outside the loop.
Quoted 7 lines
I think it cannot. The intention of doing this way is that we check for new available buffers *also after* enabling notifications, so we don't miss any of them. It is more or less copied from
virtio_blk_handle_vq, which also needs to run to completion.If we need to loop again because there are more available buffers, we want to disable notifications again. Or am I missing something?
I think you're right.
Quoted 30 lines
Quoted 16 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 203
Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c +12
-0+ +while (true) { +VirtQueueElement *elem =virtqueue_pop (svq->vq,sizeof (*elem)); +if (!elem) { +break ; + } + +vhost_svq_add (svq, elem); +event_notifier_set (&svq->kick_notifier); + } + +virtio_queue_set_notification (svq->vq, true);I think this can be moved to the end of this function.
(Same as previous answer)
Quoted 6 lines
Btw, we probably need a quota to make sure the svq is not hogging the main event loop.
Similar issue could be found in both virtio-net TX (using timer or bh) and TAP (a quota).
I think that virtqueue size is the natural limit to that: since we are not making any buffers used in the loop, there is no way that it runs more than virtqueue size times. If it does because of an evil/bogus guest,
virtqueue_popraises the message "Virtqueue size exceeded" and returns NULL, effectively breaking the loop.
Quoted 3 lines
Virtio-net tx functions mark each buffer right after making them available and use it, so they can hog BQL. But my understanding is that is not possible in the SVQ case.
Right.
Quoted 2 lines
I can add a comment in the code to make it clearer though.
Yes, please.
Quoted 16 lines
Quoted 16 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 214
Quoted 12 lines
hw/virtio/vhost-shadow-virtqueue.c +11
-0+ }while (!virtio_queue_empty (svq->vq)); +} + +static bool vhost_svq_more_used (VhostShadowVirtqueue *svq) +{ +if (svq->used_idx != svq->shadow_used_idx) { +return true; + } + +/* Get used idx must not be reordered */ +smp_rmb ();Interesting, we don't do this for kernel drivers. It would be helpful to explain it more clear by "X must be done before Y".
Quoted 3 lines
I think this got reordered, it's supposed to be *after* get the used idx, so it matches the one in the kernel with the comment "Only get used array entries after they have been exposed by host.".
Right.
Quoted 2 lines
I will change it for the next series.
Ok.
Quoted 68 lines
Quoted 57 lines
Quoted 57 lines
hw/virtio/vhost-shadow-virtqueue.c +47
-2+ svq->shadow_used_idx =cpu_to_le16 (svq->vring.used->idx); + +return svq->used_idx != svq->shadow_used_idx; +} + +static VirtQueueElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) +{ +vring_desc_t *descs = svq->vring.desc; +const vring_used_t *used = svq->vring.used; +vring_used_elem_t used_elem; +uint16_t last_used; + +if (!vhost_svq_more_used (svq)) { +return NULL ; + } + + last_used = svq->used_idx & (svq->vring.num -1 ); + used_elem.id =le32_to_cpu (used->ring[last_used].id); + used_elem.len =le32_to_cpu (used->ring[last_used].len); + + svq->used_idx++; +if (unlikely (used_elem.id >= svq->vring.num)) { +error_report ("Device %s says index %u is used" , svq->vdev->name, + used_elem.id); +return NULL ; + } + +if (unlikely (!svq->ring_id_maps[used_elem.id])) { +error_report ( +"Device %s says index %u is used, but it was not available" , + svq->vdev->name, used_elem.id); +return NULL ; + } + + descs[used_elem.id].next = svq->free_head; + svq->free_head = used_elem.id; + + svq->ring_id_maps[used_elem.id]->len = used_elem.len; +return g_steal_pointer (&svq->ring_id_maps[used_elem.id]); }/* Forward vhost notifications */ @@ -70,8 +258,26 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n) {VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, call_notifier); - -event_notifier_set (&svq->guest_call_notifier); +VirtQueue *vq = svq->vq; + +/* Make as many buffers as possible used. */ + do { + unsigned i = 0; + + /* TODO: Use VRING_AVAIL_F_NO_INTERRUPT */ + while (true) {Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_handle_call_no_test(EventNotifier *n) · line 274
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+ g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); +if (!elem) { +break ; + } + +assert (i < svq->vring.num);Let's return error instead of using the assert.
Actually this is a condition that we should never meet: In the case of ring overrun, device would try to set used a descriptor that is either
Quoted 8 lines
Quoted 1 line
vring size *or* should try to overrun some of the already used ones.
In both cases, elem should be NULL and the loop should break.
So this is a safety net protecting from both, if we have an i >
svq->vring.nummeans we are not processing used buffers well anymore, and (moreover) this is happening after making used all descriptors.Taking that into account, should we delete it?
Maybe a warning instead.
Quoted 37 lines
Quoted 21 lines
Quoted 21 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-0+virtqueue_fill (vq, elem, elem->len, i++); + } + +virtqueue_flush (vq, i); +event_notifier_set (&svq->guest_call_notifier); + }while (vhost_svq_more_used (svq)); }static void vhost_svq_handle_call (EventNotifier *n)@@ -204,12 +410,25 @@ err_set_vring_kick:void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq) { +int i;int r =vhost_svq_restore_vdev_host_notifier (dev, idx, svq); +if (unlikely (r <0 )) {error_report ("Couldn't restore vq kick fd: %s" ,strerror (-r)); }Quoted 16 lines
hw/virtio/vhost-shadow-virtqueue.c err_set_vring_kick: · line 430
Quoted 11 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0event_notifier_set_handler (&svq->host_notifier,NULL ); + +for (i =0 ; i < svq->vring.num; ++i) { + g_autofree VirtQueueElement *elem = svq->ring_id_maps[i]; +/* +* Although the doc says we must unpop in order, it's ok to unpop +* everything. +*/ +if (elem) { +virtqueue_unpop (svq->vq, elem, elem->len); + }Will this result some of the "pending" buffers to be submitted multiple times? If yes, should we wait for all the buffers used instead of doing the unpop here?
Quoted 6 lines
Do you mean to call
virtqueue_unpopwith the same elem (or elem.id) multiple times? That should never happen, because elem.id should be the position in thering_id_maps. Also,unpop()should just unmap the element and never sync again.Maybe it is way clearer to call
virtqueue_detach_elementhere directly.
I meant basically for the buffers that were consumed by the device but not made used. In this case if we unpop here. It will be processed by the device later via vhost-vdpa again?
This is probably fine for net but I'm not sure it works for other devices. Another way is to wait until all the consumed buffer used.
Quoted 90 lines
Quoted 79 lines
Quoted 79 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-1+ } }/* @@ -224,7 +443,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)size_t driver_size;size_t device_size; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1); -int r; +int r, i; r =event_notifier_init (&svq->kick_notifier,0 );if (r !=0 ) {@@ -250,6 +469,11 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)memset (svq->vring.desc,0 , driver_size); svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size);memset (svq->vring.used,0 , device_size); +for (i =0 ; i < num -1 ; i++) { + svq->vring.desc[i].next =cpu_to_le16 (i +1 ); + } + + svq->ring_id_maps = g_new0(VirtQueueElement *, num);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);@@ -269,6 +493,7 @@ void vhost_svq_free(VhostShadowVirtqueue *vq)event_notifier_cleanup (&vq->kick_notifier);event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier); +g_free (vq->ring_id_maps);qemu_vfree (vq->vring.desc);qemu_vfree (vq->vring.used);g_free (vq);hw/virtio/vhost-vdpa.c +23
-0@@ -19,6 +19,7 @@#include "hw/virtio/virtio-net.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" #include "cpu.h" @@ -475,6 +476,28 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,return vhost_vdpa_backend_set_features (dev, features); } +/** +* Restore guest features to vdpa device +*/ +static int vhost_vdpa_set_guest_features (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +return vhost_vdpa_backend_set_features (dev, v->guest_features); +} + +/** +* Set shadow virtqueue supported features +*/ +static int vhost_vdpa_set_svq_features (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +uint64_t features = v->host_features; +bool b =vhost_svq_valid_device_features (&features); +assert (b); + +return vhost_vdpa_backend_set_features (dev, features); +} +static int vhost_vdpa_set_backend_cap (struct vhost_dev *dev) {uint64_t features;@@ -730,6 +753,19 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; }Quoted 11 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 763
Quoted 8 lines
hw/virtio/vhost-vdpa.c +8
-0+static int vhost_vdpa_vring_pause (struct vhost_dev *dev) +{ +int r; +uint8_t status; + +vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_DEVICE_STOPPED ); + do { + r = vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);I guess we'd better add some sleep here.
Quoted 2 lines
If the final version still contains the call, I will add the sleep. At the moment I think it's better if we stop the device by a vdpa ioctl.
Ok, so the idea is to sleep in the ioctl?
Quoted 137 lines
Quoted 75 lines
Quoted 75 lines
hw/virtio/vhost-vdpa.c +45
-3+ }while (r ==0 && !(status &VIRTIO_CONFIG_S_DEVICE_STOPPED )); + +return 0 ; +} +/* * Start shadow virtqueue. */ @@ -742,9 +778,29 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) .index = idx + dev->vq_index, .fd = event_notifier_get_fd(vhost_call_notifier), }; +struct vhost_vring_addr addr = { + .index = idx + dev->vq_index, + }; +struct vhost_vring_state num = { + .index = idx + dev->vq_index, + .num =virtio_queue_get_num (dev->vdev, idx), + };int r;bool b; +vhost_svq_get_vring_addr (svq, &addr); + r =vhost_vdpa_set_vring_addr (dev, &addr); +if (unlikely (r)) { +error_report ("vhost_set_vring_addr for shadow vq failed" ); +return false; + } + + r =vhost_vdpa_set_vring_num (dev, &num); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_num for shadow vq failed" ); +return false; + } +/* Set shadow vq -> guest notifier */ assert (v->call_fd[idx]);vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]);@@ -781,15 +837,32 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) {VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); -bool ok; -if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ; }g_ptr_array_add (v->shadow_vqs, svq); + } + } - ok =vhost_vdpa_svq_start_vq (hdev, n); + r =vhost_vdpa_vring_pause (hdev); +assert (r ==0 ); + +if (enable) { +for (n =0 ; n < v->shadow_vqs->len; ++n) { +/* Obtain Virtqueue state */ +vhost_virtqueue_stop (hdev, hdev->vdev, &hdev->vqs[n], n); + } + } + +/* Reset device so it can be configured */ + r =vhost_vdpa_dev_start (hdev, false); +assert (r ==0 ); + +if (enable) { +int r; +for (n =0 ; n < v->shadow_vqs->len; ++n) { +bool ok =vhost_vdpa_svq_start_vq (hdev, n);if (unlikely (!ok)) {/* Free still not started svqs */ g_ptr_array_set_size (v->shadow_vqs, n);Quoted 9 lines
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 874
Quoted 6 lines
hw/virtio/vhost-vdpa.c +2
-0@@ -797,11 +870,19 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)break ; } } + +/* Need to ack features to set state in vp_vdpa devices */
vhost_vdpaactually?Yes, what a mistake!
Quoted 39 lines
Quoted 39 lines
hw/virtio/vhost-vdpa.c +12
-0+ r =vhost_vdpa_set_svq_features (hdev); +if (unlikely (r)) { + enable = false; + } } v->shadow_vqs_enabled = enable;if (!enable) { +vhost_vdpa_set_guest_features (hdev); +/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { struct vhost_vring_file file = {@@ -818,7 +899,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* TODO: This can unmask or override call fd! */ vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); } + } + r =vhost_vdpa_dev_start (hdev, true); +assert (r ==0 ); + +if (!enable) {/* Resources cleanup */ g_ptr_array_set_size (v->shadow_vqs,0 ); }@@ -831,6 +917,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp)struct vhost_vdpa *v;const char *err_cause =NULL ;bool r; +uint64_t svq_features; QLIST_FOREACH(v, &vhost_vdpa_devices, entry) {if (v->dev->vdev &&0 ==strcmp (v->dev->vdev->name, name)) {@@ -846,6 +933,20 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) goto err; }Quoted 10 lines
hw/virtio/vhost-vdpa.c void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) · line 943
Quoted 8 lines
hw/virtio/vhost-vdpa.c +8
-0+ svq_features = v->host_features; +if (!vhost_svq_valid_device_features (&svq_features)) { +error_setg (errp, +"Can't enable shadow vq on %s: Unexpected feature flags (%lx-%lx)" , + name, v->host_features, svq_features); +return ; + }else { +/* TODO: Check for virtio_vdpa + IOMMU & modern device */ I guess you mean "
vhost_vdpa" here.Yes, a similar mistake in less than 50 lines :).
Quoted 8 lines
Quoted 3 lines
For IOMMU, I guess you mean "vIOMMU"
actually?This comment is out of date and inherited from the vhost version, where only the IOMMU version was developed, so it will be deleted in the next series. I think it makes little sense to check vIOMMU if we stick with vDPA since it still does not support it, but we could make the check here for sure.
Right.
Thanks
Quoted 18 lines
Thanks!
Quoted 15 lines
Thanks
Quoted 12 lines
hw/virtio/vhost-vdpa.c +6
-0+ } + +if (err_cause) { + goto err; + } + r =vhost_vdpa_enable_svq (v, enable);if (unlikely (!r)) { err_cause ="Error enabling (see monitor)" ;@@ -853,7 +954,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) }
Quoted 6 lines
Quoted 6 lines
hw/virtio/vhost-vdpa.c void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) · line 960
Quoted 6 lines
hw/virtio/vhost-vdpa.c +1
-1err: -if (err_cause) { +if (errp ==NULL && err_cause) {error_setg (errp,"Can't enable shadow vq on %s: %s" , name, err_cause); } }
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue
In reply to Eugenio Perez MartinQuoted 19 lines · Eugenio Perez Martin
Quoted 18 lines · Jason Wang
Quoted 12 lines · Eugenio Pérez
This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of
vhost_virtqueue_startcan unmasks and discard call events,vhost_virtqueue_startshould be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.Btw, the log looks not clear. I guess this patch goes for method 3. If yes, we need explain it and why.
Quoted 6 lines
Quoted 2 lines
Thanks
Sorry about being unclear. This commit log (and code) just exposes the problem and the solutions I came up with but does nothing to solve it. I'm actually going for method 3 for the next series but I'm open to doing it differently.
That's fine, but need to doc that method 3 is something that is done in the patch.
Thanks
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Signed-off-by Eugenio Pérez<[email protected]>
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue
In reply to Eugenio Perez MartinQuoted 106 lines · Eugenio Perez Martin
Quoted 78 lines · Jason Wang
Quoted 76 lines · Eugenio Pérez
This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of
vhost_virtqueue_startcan unmasks and discard call events,vhost_virtqueue_startshould be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.c | 19 +++++++++++++++ hw/virtio/vhost-vdpa.c | 38 +++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +19
-0@@ -53,6 +53,22 @@ static void vhost_handle_guest_kick(EventNotifier *n)event_notifier_set (&svq->kick_notifier); } +/* Forward vhost notifications */ +static void vhost_svq_handle_call_no_test (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + call_notifier); + +event_notifier_set (&svq->guest_call_notifier); +} + +static void vhost_svq_handle_call (EventNotifier *n) +{ +if (likely (event_notifier_test_and_clear (n))) { +vhost_svq_handle_call_no_test (n); + } +} +/* * Obtain the SVQ call notifier, where vhost device notifies SVQ that there * exists pending used buffers. @@ -180,6 +196,8 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) } svq->vq =virtio_get_queue (dev->vdev, vq_idx); +event_notifier_set_handler (&svq->call_notifier, + vhost_svq_handle_call);return g_steal_pointer (&svq); err_init_call_notifier:@@ -195,6 +213,7 @@ err_init_kick_notifier:void vhost_svq_free (VhostShadowVirtqueue *vq) {event_notifier_cleanup (&vq->kick_notifier); +event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier);g_free (vq); }hw/virtio/vhost-vdpa.c +4
-1@@ -712,13 +712,40 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) {struct vhost_vdpa *v = dev->opaque;VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); -return vhost_svq_start (dev, idx, svq); +EventNotifier *vhost_call_notifier =vhost_svq_get_svq_call_notifier (svq); +struct vhost_vring_file vhost_call_file = { + .index = idx + dev->vq_index, + .fd =event_notifier_get_fd (vhost_call_notifier),Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 724
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+ }; +int r; +bool b; + +/* Set shadow vq -> guest notifier */ +assert (v->call_fd[idx]);We need aovid the
asser()here. On which case we can hit this?I would say that there is no way we can actually hit it, so let's remove it.
Quoted 7 lines
Quoted 7 lines
hw/virtio/vhost-vdpa.c +7
-0+vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]); + + b =vhost_svq_start (dev, idx, svq); +if (unlikely (!b)) { +return false; + } +Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 737
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+/* Set device -> SVQ notifier */ + r =vhost_vdpa_set_vring_dev_call (dev, &vhost_call_file); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_call for shadow vq failed" ); +return false; + }Similar to kick, do we need to
set_vring_call()beforevhost_svq_start()?
Quoted 3 lines
It should not matter at this moment because the device should not be started at this point and device calls should not run
vhost_svq_handle_calluntil BQL is released.
Yes, we stop virtqueue before.
Quoted 10 lines
The "logic" of doing it after is to make clear that svq must be fully initialized before processing device calls, even in the case that we extract SVQ in its own iothread or similar. But this could be done before
vhost_svq_startfor sure.Quoted 4 lines
Quoted 3 lines
hw/virtio/vhost-vdpa.c +3
-0+ +/* Check for pending calls */ +event_notifier_set (vhost_call_notifier);
Quoted 6 lines
Quoted 2 lines
Interesting, can this result spurious interrupt?
This actually "queues" a
vhost_svq_handle_callafter the BQL release, where the device should be fully reset. In that regard, if there are no used descriptors there will not be an irq raised to the guest. Does that answer the question? Or have I missed something?
Yes, please explain this in the comment.
Quoted 32 lines
Quoted 23 lines
Quoted 23 lines
hw/virtio/vhost-vdpa.c +9
-0+return true; }static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev;unsigned n; +int r;if (enable == v->shadow_vqs_enabled) {return hdev->nvqs;@@ -752,9 +779,18 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)if (!enable) {/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { +struct vhost_vring_file file = { + .index =vhost_vdpa_get_vq_index (hdev, n), + .fd = v->call_fd[n], + }; + + r =vhost_vdpa_set_vring_call (hdev, &file); +assert (r ==0 );Quoted 9 lines
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 793
Quoted 5 lines
hw/virtio/vhost-vdpa.c +2
-0+unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n);VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n);vhost_svq_stop (hdev, n, svq); +/* TODO: This can unmask or override call fd! */ I don't get this comment. Does this mean the current code can't work with
mask_notifiers? If yes, this is something we need to fix.
Quoted 2 lines
Yes, but it will be addressed in the next series. I should have explained it bette here, sorry :).
Ok.
Thanks
Quoted 9 lines
Thanks!
Quoted 6 lines
Thanks
Quoted 3 lines
hw/virtio/vhost-vdpa.c +0
-0vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); }
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 16/20] vhost: Check for device VRING_USED_F_NO_NOTIFY at shadow virtqueue kick
In reply to Jason WangQuoted 23 lines · Jason Wang
Quoted 10 lines · Eugenio Pérez
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +0
-0hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_add(VhostShadowVirtqueue *svq, VirtQueueElement *elem) · line 178
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +3
-0@@ -173,6 +173,15 @@ static void vhost_svq_add(VhostShadowVirtqueue *svq, VirtQueueElement *elem) svq->ring_id_maps[qemu_head] = elem; } +static void vhost_svq_kick (VhostShadowVirtqueue *svq) +{ +/* Make sure we are reading updated device flag */ I guess this would be better:
Quoted 7 lines
/* We need to expose available array entries before checking used
* flags. */(Borrowed from kernel codes).
Thanks
Right, I will replace it, thanks!
Quoted 20 lines
Quoted 19 lines
hw/virtio/vhost-shadow-virtqueue.c +7
-1+smp_mb (); +if (!(svq->vring.used->flags &VRING_USED_F_NO_NOTIFY )) { +event_notifier_set (&svq->kick_notifier); + } +} +/* Handle guest->device notifications */ static void vhost_handle_guest_kick (EventNotifier *n) {@@ -197,7 +206,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) }vhost_svq_add (svq, elem); -event_notifier_set (&svq->kick_notifier); +vhost_svq_kick (svq); }virtio_queue_set_notification (svq->vq, true);
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 17/20] vhost: Use VRING_AVAIL_F_NO_INTERRUPT at device call on shadow virtqueue
In reply to Jason WangQuoted 10 lines · Jason Wang
Quoted 1 line · Eugenio Pérez
Signed-off-by Eugenio Pérez <[email protected]>Commit log please.
Thanks
Sorry, this was another commit that was intended to be squashed.
I think I squashed two other (previous) commits by mistake in the rebase editor, that's why you detected unrelated changes and mixed SVQ & vhost-vdpa changes. I will take more care for the next series I send.
Thanks!
Quoted 62 lines
Quoted 61 lines
--- hw/virtio/vhost-shadow-virtqueue.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +23
-1@@ -60,6 +60,9 @@ typedef struct VhostShadowVirtqueue {/* Next head to consume from device */ uint16_t used_idx; + +/* Cache for the exposed notification flag */ +bool notification; } VhostShadowVirtqueue;/* If the device is using some of these, SVQ cannot communicate */ @@ -105,6 +108,24 @@ bool vhost_svq_valid_device_features(uint64_t *dev_features)return r; } +static void vhost_svq_set_notification (VhostShadowVirtqueue *svq,bool enable) +{ +uint16_t notification_flag; + +if (svq->notification == enable) { +return ; + } + + notification_flag =cpu_to_le16 (VRING_AVAIL_F_NO_INTERRUPT ); + + svq->notification = enable; +if (enable) { + svq->vring.avail->flags &= ~notification_flag; + }else { + svq->vring.avail->flags |= notification_flag; + } +} +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq,const struct iovec *iovec,size_t num,bool more_descs,bool write)@@ -273,7 +294,7 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n) do { unsigned i = 0; - /* TODO: Use VRING_AVAIL_F_NO_INTERRUPT */ + vhost_svq_set_notification(svq, false); while (true) { g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); if (!elem) {@@ -286,6 +307,7 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n)virtqueue_flush (vq, i);event_notifier_set (&svq->guest_call_notifier); +vhost_svq_set_notification (svq, true); }while (vhost_svq_more_used (svq)); }
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
In reply to Jason WangQuoted 23 lines · Jason Wang
Quoted 11 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Quoted 7 lines
Signed-off-by Eugenio Pérez<[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)
Quoted 3 lines
Think hard about the whole logic. This is safe since qemu memory map will fail if guest submits a invalidate IOVA.
Can you expand on this? What you mean is that VirtQueue already protects SVQ code if the guest sets an invalid buffer address (GPA), isn't it?
Quoted 6 lines
Then I wonder if we do something much more simpler:
1) Using qemu VA as IOVA but only maps the VA that belongs to guest
2) Then we don't need any IOVA tree here, what we need is to just map vring and use qemu VA without any translation
That would be great, but either qemu's SVQ vring or guest translated buffers address (in qemu VA form) were already in high addresses, outside of the device's iova range (in my test).
I didn't try remapping tricks to make them fit in the range, but I think it does complicate the solution relatively fast if there was already memory in that range owned by qemu before enabling SVQ:
* Guest memory must be contiguous in VA address space, but it "must" support hotplug/unplug (although vDPA currently pins it). Hotplug memory could always overlap with SVQ vring, so we would need to move it.
* Duplicating mapped memory for writing? (Not sure if guest memory is actually movable in qemu).
* Indirect descriptors will need to allocate and free memory more or less frequently, increasing the possibility of overlapping.
If we can move guest memory, however, I can see how we can track it in a tree *but* mark when the tree is 1:1 with qemu's VA, so buffers forwarding does not take the translation penalty. When guest memory cannot be map 1:1, we can resort to tree, and come back to 1:1 translation if the offending tree node(s) get deleted.
However I think this puts the solution a little bit farther than "starting simple" :).
Does it make sense?
Thanks!
Quoted 2 lines
Thanks
Re: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
Quoted 25 lines · Eugenio Perez Martin
Quoted 23 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Signed-off-by Eugenio Pérez<[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)
Quoted 7 lines
Quoted 3 lines
Think hard about the whole logic. This is safe since qemu memory map will fail if guest submits a invalidate IOVA.
Can you expand on this? What you mean is that VirtQueue already protects SVQ code if the guest sets an invalid buffer address (GPA), isn't it?
Yes.
Quoted 11 lines
Quoted 7 lines
Then I wonder if we do something much more simpler:
1) Using qemu VA as IOVA but only maps the VA that belongs to guest
2) Then we don't need any IOVA tree here, what we need is to just map vring and use qemu VA without any translationThat would be great, but either qemu's SVQ vring or guest translated buffers address (in qemu VA form) were already in high addresses, outside of the device's iova range (in my test).
You're right. I miss that and that's why we need e.g iova tree and allocator.
What I proposed only makes sense when shared virtual memory (SVA) is implemented. In the case of SVA, the valid iova range should be the full VA range.
Quoted 5 lines
I didn't try remapping tricks to make them fit in the range, but I think it does complicate the solution relatively fast if there was already memory in that range owned by qemu before enabling SVQ:
Quoted 8 lines
* Guest memory must be contiguous in VA address space, but it "must" support hotplug/unplug (although vDPA currently pins it). Hotplug memory could always overlap with SVQ vring, so we would need to move it.
* Duplicating mapped memory for writing? (Not sure if guest memory is actually movable in qemu).
* Indirect descriptors will need to allocate and free memory more or less frequently, increasing the possibility of overlapping.
I'm not sure I get the problem, but overlapping is not an issue since we're using VA.
Quoted 2 lines
If we can move guest memory,
I'm not sure we can do this or it looks very tricky.
Quoted 10 lines
however, I can see how we can track it in
a tree *but* mark when the tree is 1:1 with qemu's VA, so buffers forwarding does not take the translation penalty. When guest memory cannot be map 1:1, we can resort to tree, and come back to 1:1 translation if the offending tree node(s) get deleted.However I think this puts the solution a little bit farther than "starting simple" :).
Does it make sense?
Yes. So I think I will review the IOVA tree codes and get back to you.
THanks
Quoted 6 lines
Thanks!
Quoted 2 lines
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
In reply to Eugenio Perez MartinQuoted 76 lines · Eugenio Perez Martin
Quoted 70 lines · Jason Wang
Quoted 25 lines · Eugenio Perez Martin
Quoted 23 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Signed-off-by Eugenio Pérez<[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)Quoted 7 lines
Quoted 3 lines
Think hard about the whole logic. This is safe since qemu memory map will fail if guest submits a invalidate IOVA.
Can you expand on this? What you mean is that VirtQueue already protects SVQ code if the guest sets an invalid buffer address (GPA), isn't it?
Yes.
Quoted 11 lines
Quoted 7 lines
Then I wonder if we do something much more simpler:
1) Using qemu VA as IOVA but only maps the VA that belongs to guest
2) Then we don't need any IOVA tree here, what we need is to just map vring and use qemu VA without any translationThat would be great, but either qemu's SVQ vring or guest translated buffers address (in qemu VA form) were already in high addresses, outside of the device's iova range (in my test).
You're right. I miss that and that's why we need e.g iova tree and allocator.
What I proposed only makes sense when shared virtual memory (SVA) is implemented. In the case of SVA, the valid iova range should be the full VA range.
Quoted 5 lines
I didn't try remapping tricks to make them fit in the range, but I think it does complicate the solution relatively fast if there was already memory in that range owned by qemu before enabling SVQ:
Quoted 8 lines
* Guest memory must be contiguous in VA address space, but it "must" support hotplug/unplug (although vDPA currently pins it). Hotplug memory could always overlap with SVQ vring, so we would need to move it.
* Duplicating mapped memory for writing? (Not sure if guest memory is actually movable in qemu).
* Indirect descriptors will need to allocate and free memory more or less frequently, increasing the possibility of overlapping.Quoted 3 lines
I'm not sure I get the problem, but overlapping is not an issue since we're using VA.
Quoted 5 lines
It's basically the same (potential) problem of DPDK's SVQ: IOVA Range goes from 0 to X. That means that both GPA and SVQ must be in IOVA range. As an example, we put GPA at the beginning of the range, that grows upwards when memory is hot plugged, and SVQ vrings that grows downwards when devices are added or set in SVQ mode.
Yes, but this is not the case if we're using VA.
Quoted 12 lines
Even without both space fragmentation problems, we could reach a point where both will take the same address, and we would need to go to the tree.
But since we are able to detect those situations, I can see how we can work in two modes as an optimization: 1:1 when they don't overlap, and fragmented tree where it does. But I don't think it's a good idea to include it from the beginning, and I'm not sure if that is worth it without measuring the tree translation cost first.
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
If we can move guest memory,
I'm not sure we can do this or it looks very tricky.
Just thinking out loud here, but maybe we could map all memory and play with
remap_file_pages[1] a little bit for that.
The problem is that there's no guarantee that it will always succeed. So let's start with the current dedicated IOVA address space. We can do optimization on top anyhow.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
however, I can see how we can track it in
a tree *but* mark when the tree is 1:1 with qemu's VA, so buffers forwarding does not take the translation penalty. When guest memory cannot be map 1:1, we can resort to tree, and come back to 1:1 translation if the offending tree node(s) get deleted.However I think this puts the solution a little bit farther than "starting simple" :).
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Does it make sense?
Yes. So I think I will review the IOVA tree codes and get back to you.
Looking forward to it :).
Thanks
Quoted 15 lines
Thanks!
[1] https://linux.die.net/man/2/remap_file_pages
Quoted 9 lines
THanks
Quoted 6 lines
Thanks!
Quoted 2 lines
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue
In reply to Jason WangQuoted 56 lines · Jason Wang
Quoted 27 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 4 lines · Eugenio Pérez
Shadow virtqueue notifications forwarding is disabled when
vhost_devstops, so code flow follows usual cleanup.Also, host notifiers must be disabled at SVQ start,
Any reason for this?
It will be addressed in a later series, sorry.
Quoted 11 lines
Quoted 3 lines
and they will not
start if SVQ has been enabled when device is stopped. This is trivial to address, but it is left out for simplicity at this moment.It looks to me this patch also contains the following logics
1) codes to enable svq
2) codes to let svq to be enabled from QMP.
I think they need to be split out,
I agree that we can split this more, with the code that belongs to SVQ and the code that belongs to vhost-vdpa. it will be addressed in future series.
Quoted 7 lines
Quoted 3 lines
we may endup with the following
series of patchesWith "series of patches" do you mean to send every step in a separated series? There are odds of having the need of modifying code already sent & merged with later ones. If you confirm to me that it is fine, I can do it that way for sure.
Sorry for being unclear. I meant it's a sub-series actually of the series.
Quoted 7 lines
Quoted 7 lines
1) svq skeleton with enable/disable
2) route host notifier to svq
3) route guest notifier to svq
4) codes to enable svq
5) enable svq via QMPQuoted 5 lines
I'm totally fine with that, but there is code that is never called if the qmp command is not added. The compiler complains about static functions that are not called, making impossible things like bisecting through these commits, unless I use attribute((unused)) or similar. Or have I missed something?
Quoted 6 lines
You're right, then I think we can then:
1) svq skeleton with enable/disable via QMP
2) route host notifier to svq
3) route guest notifier to svq
Got it, I will try to adapt the series.
Quoted 12 lines
Quoted 6 lines
We could do that way with the code that belongs to SVQ though, since all of it is declared in headers. But to delay the "enable svq via qmp" to the last one makes debugging harder, as we cannot just enable notifications forwarding with no buffers forwarding.
Yes.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
If I introduce a change in the notifications code, I can simply go to these commits and enable SVQ for notifications. This way I can have an idea of what part is failing. A similar logic can be applied to other devices than
vp_vdpa.vhost-vdpa actually?
Not this time :).
I was actually talking about other devices with other features or even subtle differences because they being hardware instead of software.
For example, vp_vdpa does not use host notifiers at the moment. If I enable SVQ with only notifications forwarding and not buffer ones, I can bisect pretty fast that there is a problem in the notifications forwarding mechanism. This saved me a few times while writing the series.
There are other ways of reaching the same conclusion of course, and future developments or fixes to this serie will deprecate this method, but I think it is a good idea to let the SVQ enabling early in the serie as a way to test what part of the series is failing.
This is just my opinion of course, and I'm biased by the fact that I'm the one proposing the series. If it makes the reviewing so hard, I'm totally in to delay QMP command to the end of the series.
Quoted 2 lines
Quoted 2 lines
We would lose it if we
Sorry, I think I sent it without a proper review.
Quoted 115 lines
Quoted 108 lines
Quoted 29 lines
Quoted 29 lines
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.h | 8 ++ include/hw/virtio/vhost-vdpa.h | 4 + hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- 5 files changed, 264 insertions(+), 4 deletions(-)qapi/net.json +1
-1@@ -88,7 +88,7 @@ # # @enable: true to use the alternate shadow VQ notifications # -# Returns: Always error, since SVQ is not implemented at the moment. +# Returns: Error if failure, or 'no error' for success. # # Since: 6.2 #hw/virtio/vhost-shadow-virtqueue.h +0
-0@@ -14,6 +14,14 @@Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.h line 17
Quoted 3 lines
hw/virtio/vhost-shadow-virtqueue.h +1
-0typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ; +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq);Let's move this function to another patch since it's unrelated to the
guest->hostrouting.Right, I missed it while squashing commits and at later reviews.
Quoted 61 lines
Quoted 61 lines
hw/virtio/vhost-shadow-virtqueue.h +7
-0+void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); + +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx);void vhost_svq_free (VhostShadowVirtqueue *vq);include/hw/virtio/vhost-vdpa.h +4
-0@@ -12,6 +12,8 @@#ifndef HW_VIRTIO_VHOST_VDPA_H #define HW_VIRTIO_VHOST_VDPA_H +#include <gmodule.h> +#include "qemu/queue.h" #include "hw/virtio/virtio.h" @@ -24,6 +26,8 @@ typedef struct vhost_vdpa {int device_fd;uint32_t msg_type;MemoryListener listener; +bool shadow_vqs_enabled; +GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry;VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ];hw/virtio/vhost-shadow-virtqueue.c +11
-1@@ -9,9 +9,12 @@#include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" +#include "hw/virtio/vhost.h" + +#include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" -#include "qemu/event_notifier.h" +#include "qemu/main-loop.h" /* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue {@@ -19,14 +22,146 @@ typedef struct VhostShadowVirtqueue {EventNotifier kick_notifier;/* Shadow call notifier, sent to vhost */ EventNotifier call_notifier; + +/* +* Borrowed virtqueue's guest to host notifier. +* To borrow it in this event notifier allows to register on the event +* loop and access the associated shadow virtqueue easily. If we use the +* VirtQueue, we don't have an easy way to retrieve it. +* Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 37
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+* So shadow virtqueue must not clean it, or we would lose VirtQueue one. +*/ +EventNotifier host_notifier; + +/* Guest's call notifier, where SVQ calls guest. */ +EventNotifier guest_call_notifier;To be consistent, let's simply use "
guest_notifier" here.Quoted 5 lines
It could be confused when the series adds a guest -> qemu kick notifier then. Actually, I think it would be better to rename
host_notifierto something likehost_svq_notifier. Orhost_callandguest_call, since "notifier" is already in the type, making the name to be a little bit "Hungarian notation".
Quoted 3 lines
I think that's fine, just need to make sure we have a consistent name for SVQ notifier.
Got it.
Quoted 23 lines
Quoted 23 lines
Quoted 7 lines
Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.c +3
-0+ +/* Virtio queue shadowing */ +VirtQueue *vq; } VhostShadowVirtqueue;Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 51
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c +9
-0+/* Forward guest notifications */ +static void vhost_handle_guest_kick (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + host_notifier); + +if (unlikely (!event_notifier_test_and_clear (n))) { +return ; + }Is there a chance that we may stop the processing of available buffers during the svq enabling? There could be no kick from the guest in this case.
Actually, yes, I think you are right. The guest kick eventfd could have been consumed by vhost but there may be still pending buffers.
Quoted 7 lines
Quoted 3 lines
I think it would be better to check for available buffers first, then clear the notifier unconditionally, and then re-check and process them if any [1].
Looks like I can't find "[1]" anywhere.
Sorry, it's at the comment about vhost_svq_start function. It's not isolated, but in the middle of a paragraph, because both comments are
related: The solution to one of them affects the other.
Quoted 6 lines
Quoted 6 lines
However, this problem arises later in the series: At this moment the device is not reset and guest's host notifier is not replaced, so either vhost/device receives the kick, or SVQ does and forwards it.
Quoted 8 lines
Quoted 1 line
Does it make sense to you?
Kind of, so I think we can:
1) As you said, always check available buffers when switching to SVQ
2) alwasy kick the vhost when switching back to vhost
Right. I think I prefer solution 1, but let me delay the choice to compare them properly.
Quoted 50 lines
Quoted 50 lines
Quoted 40 lines
Quoted 40 lines
hw/virtio/vhost-shadow-virtqueue.c +38
-0+ +event_notifier_set (&svq->kick_notifier); +} + +/* +* Obtain the SVQ call notifier, where vhost device notifies SVQ that there +* exists pending used buffers. +* +* @svq Shadow Virtqueue +*/ +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq) +{ +return &svq->call_notifier; +} + +/* +* Set the call notifier for the SVQ to call the guest +* +* @svq Shadow virtqueue +* @call_fd call notifier +* +* Called on BQL context. +*/ +void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd) +{ +event_notifier_init_fd (&svq->guest_call_notifier, call_fd); +} + +/* +* Restore the vhost guest to host notifier, i.e., disables svq effect. +*/ +static int vhost_svq_restore_vdev_host_notifier (struct vhost_dev *dev, +unsigned vhost_index, +VhostShadowVirtqueue *svq) +{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = vhost_index,Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 95
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+ .fd = event_notifier_get_fd(vq_host_notifier), + }; +int r; + +/* Restore vhost kick */ + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);And remap the notification area if necessary.
Totally right, that step is missed in this series.
Quoted 9 lines
Quoted 3 lines
However, remapping guest host notifier memory region has no advantages over using ioeventfd to perform guest -> SVQ notifications, doesn't it? By both methods flow needs to go through the hypervisor kernel.
To be clear, I meant restore the notification area mapping from guest to device directly. For SVQ, you are right, there's no much value for bothering notification area map. (Or we can do it in the future).
Ok, I think I confused the location of your comment so I misunderstood it. You are right, the notification area must be restored here.
Quoted 44 lines
Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
hw/virtio/vhost-shadow-virtqueue.c +22
-0+return r ? -errno :0 ; +} + +/* +* Start shadow virtqueue operation. +* @dev vhost device +* @hidx vhost virtqueue index +* @svq Shadow Virtqueue +*/ +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = dev->vhost_ops->vhost_get_vq_index (dev, dev->vq_index + idx), + .fd =event_notifier_get_fd (&svq->kick_notifier), + }; +int r; + +/* Check that notifications are still going directly to vhost dev */ +assert (virtio_queue_is_host_notifier_enabled (svq->vq)); +Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 122
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+/* +* event_notifier_set_handler already checks for guest's notifications if +* they arrive in the switch, so there is no need to explicitely check for +* them. +*/ If this is true, shouldn't we call
vhost_set_vring_kick()before theevent_notifier_set_handler()?Quoted 5 lines
Not at this point of the series, but it could be another solution when we need to reset the device and we are unsure if all buffers have been read. But I think I prefer the solution exposed in [1] and to explicitly call
vhost_handle_guest_kickhere. Do you think differently?I actually mean if we can end up with this situation since SVQ take over the host notifier before
set_vring_kick().
Quoted 7 lines
1) guest kick vq, vhost is running
2) qemu take over the host notifier
3) guest kick vq
4) qemu route host notifier to SVQThen the vq will be handled by both SVQ and vhost?
It shouldn't because the vhost device must be paused at this moment. What can happen is that vhost swallows that notification but the device does not answer to the kick.
Quoted 13 lines
Quoted 11 lines
Quoted 5 lines
Btw, I think we should update the fd if
set_vring_kick()was called after this function?Kind of. This is currently bad in the code, but...
Backend callbacks
vhost_ops->vhost_set_vring_kickandvhost_ops->vhost_set_vring_addrare only called atvhost_virtqueue_start. And they are always called with known data already stored in VirtQueue.
Quoted 8 lines
This is true for now, but I'd suggest to not depend on this since it:
1) it might be changed in the future
2) we're working at vhost layer and expose API to virtio device, the code should be robust to handleset_vring_kick()at any time
3) I think we've already handled similar situation ofset_vring_call, so let's be consistent
Ok. Do you think the same for addr? Or can we stick to that justification there?
Quoted 283 lines
Quoted 8 lines
To avoid storing more state in
vhost_vdpa, I think that we should avoid duplicating them, but ignore newkick_fdor address in SVQ mode, and retrieve them again at the moment the device is (re)started in SVQ mode. Qemu already avoids things like allowing the guest to set addresses at random time, using the VirtIOPCIProxy to store them.Quoted 5 lines
I also see how duplicating that status could protect vdpa SVQ code against future changes to vhost code, but that would make this series bigger and more complex with no need at this moment in my opinion.
Do you agree?
Somehow, but consider we can handle
set_vring_call(), let's at last makeset_vring_kick()more robust.Quoted 191 lines
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 127
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+event_notifier_init_fd (&svq->host_notifier, +event_notifier_get_fd (vq_host_notifier)); +event_notifier_set_handler (&svq->host_notifier, vhost_handle_guest_kick); + + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);And we need to stop the notification area mmap.
Right.
Quoted 122 lines
Quoted 112 lines
hw/virtio/vhost-shadow-virtqueue.c +32
-0+if (unlikely (r !=0 )) { +error_report ("Couldn't set kick fd: %s" ,strerror (errno)); + goto err_set_vring_kick; + } + +return true; + +err_set_vring_kick: +event_notifier_set_handler (&svq->host_notifier,NULL ); + +return false; +} + +/* +* Stop shadow virtqueue operation. +* @dev vhost device +* @idx vhost queue index +* @svq Shadow Virtqueue +*/ +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +int r =vhost_svq_restore_vdev_host_notifier (dev, idx, svq); +if (unlikely (r <0 )) { +error_report ("Couldn't restore vq kick fd: %s" ,strerror (-r)); + } + +event_notifier_set_handler (&svq->host_notifier,NULL ); +} +/* * Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) { +int vq_idx = dev->vq_index + idx; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);int r;@@ -44,6 +179,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) goto err_init_call_notifier; } + svq->vq =virtio_get_queue (dev->vdev, vq_idx);return g_steal_pointer (&svq); err_init_call_notifier:hw/virtio/vhost-vdpa.c +24
-0@@ -17,6 +17,7 @@#include "hw/virtio/vhost.h" #include "hw/virtio/vhost-backend.h" #include "hw/virtio/virtio-net.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" @@ -272,6 +273,16 @@ static void vhost_vdpa_add_status(struct vhost_dev *dev, uint8_t status)vhost_vdpa_call (dev,VHOST_VDPA_SET_STATUS , &s); } +/** +* Adaptor function to free shadow virtqueue through gpointer +* +* @svq The Shadow Virtqueue +*/ +static void vhost_psvq_free (gpointer svq) +{ +vhost_svq_free (svq); +} +static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp) {struct vhost_vdpa *v;@@ -283,6 +294,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; + v->shadow_vqs =g_ptr_array_new_full (dev->nvqs, vhost_psvq_free);QLIST_INSERT_HEAD (&vhost_vdpa_devices, v, entry); vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |@@ -373,6 +385,17 @@ err:return ; } +static void vhost_vdpa_svq_cleanup (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +size_t idx; + +for (idx =0 ; idx < v->shadow_vqs->len; ++idx) { +vhost_svq_stop (dev, idx,g_ptr_array_index (v->shadow_vqs, idx)); + } +g_ptr_array_free (v->shadow_vqs, true); +} +static int vhost_vdpa_cleanup (struct vhost_dev *dev) {struct vhost_vdpa *v;@@ -381,6 +404,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev)trace_vhost_vdpa_cleanup (dev, v);vhost_vdpa_host_notifiers_uninit (dev, dev->nvqs);memory_listener_unregister (&v->listener); +vhost_vdpa_svq_cleanup (dev);QLIST_REMOVE (v, entry); dev->opaque =NULL ;hw/virtio/vhost-vdpa.c static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) · line 586
Quoted 7 lines
hw/virtio/vhost-vdpa.c +3
-1@@ -557,7 +581,9 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started)if (started) {uint8_t status =0 ;memory_listener_register (&v->listener, &address_space_memory); -vhost_vdpa_host_notifiers_init (dev); +if (!v->shadow_vqs_enabled) { +vhost_vdpa_host_notifiers_init (dev); + }This looks like a trick, why not check and setup
shadow_vqsinside:Quoted 6 lines
1)
vhost_vdpa_host_notifiers_init()and
2)
vhost_vdpa_set_vring_kick()Ok I will move the checks there.
Quoted 36 lines
Quoted 36 lines
hw/virtio/vhost-vdpa.c +29
-0vhost_vdpa_set_vring_ready (dev);vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_DRIVER_OK );vhost_vdpa_call (dev,VHOST_VDPA_GET_STATUS , &status);@@ -663,10 +689,96 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +/* +* Start shadow virtqueue. +*/ +static bool vhost_vdpa_svq_start_vq (struct vhost_dev *dev,unsigned idx) +{ +struct vhost_vdpa *v = dev->opaque; +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); +return vhost_svq_start (dev, idx, svq); +} + +static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) +{ +struct vhost_dev *hdev = v->dev; +unsigned n; + +if (enable == v->shadow_vqs_enabled) { +return hdev->nvqs; + } + +if (enable) { +/* Allocate resources */ +assert (v->shadow_vqs->len ==0 ); +for (n =0 ; n < hdev->nvqs; ++n) { +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +bool ok; + +if (unlikely (!svq)) { +g_ptr_array_set_size (v->shadow_vqs,0 ); +return 0 ;Quoted 8 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 728
Quoted 8 lines
hw/virtio/vhost-vdpa.c +8
-0+ } +g_ptr_array_add (v->shadow_vqs, svq); + + ok =vhost_vdpa_svq_start_vq (hdev, n); +if (unlikely (!ok)) { +/* Free still not started svqs */ +g_ptr_array_set_size (v->shadow_vqs, n); + enable = false;[2]
Quoted 4 lines
Quoted 3 lines
hw/virtio/vhost-vdpa.c +3
-0+break ; + } + }Quoted 8 lines
Quoted 4 lines
Since there's almost no logic could be shared between enable and disable. Let's split those logic out into dedicated functions where the codes looks more easy to be reviewed (e.g have a better error handling etc).
Maybe it could be more clear in the code, but the reused logic is the disabling of SVQ and the fallback in case it cannot be enabled with [2]. But I'm not against splitting in two different functions if it makes review easier.
Ok.
Quoted 52 lines
Quoted 21 lines
Quoted 21 lines
hw/virtio/vhost-vdpa.c +19
-0+ } + + v->shadow_vqs_enabled = enable; + +if (!enable) { +/* Disable all queues or clean up failed start */ +for (n =0 ; n < v->shadow_vqs->len; ++n) { +unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n); +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n); +vhost_svq_stop (hdev, n, svq); +vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); + } + +/* Resources cleanup */ +g_ptr_array_set_size (v->shadow_vqs,0 ); + } + +return n; +}Quoted 15 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 762
Quoted 12 lines
hw/virtio/vhost-vdpa.c +9
-1void qmp_x_vhost_enable_shadow_vq (const char *name,bool enable,Error **errp) { -error_setg (errp,"Shadow virtqueue still not implemented" ); +struct vhost_vdpa *v; +const char *err_cause =NULL ; +bool r; + + QLIST_FOREACH(v, &vhost_vdpa_devices, entry) { +if (v->dev->vdev &&0 ==strcmp (v->dev->vdev->name, name)) { +break ; + } + }I think you can iterate the NetClientStates to ge tthe vhost-vdpa backends.
Right, I missed it.
Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 768
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+ +if (!v) { + err_cause ="Device not found" ; + goto err; + }else if (v->notifier[0 ].addr) { + err_cause ="Device has host notifiers enabled" ;I don't get this.
At this moment of the series you can enable guest -> SVQ -> 'vdpa device' if the device is not using the host notifiers memory region. The right solution is to disable it for the guest, and to handle it in SVQ. Otherwise, guest kick will bypass SVQ and
Quoted 6 lines
It can be done in the same patch, or at least to disable (as unmap) them at this moment and handle them in a posterior patch. but for prototyping the solution I just ignored it in this series. It will be handled some way or another in the next one. I prefer the last one, to handle in a different patch, but let me know if you think it is better otherwise.
Quoted 5 lines
Aha, I see. But I think we need to that in this patch otherwise the we can route host notifier to SVQ.
Thanks
Ok, I will check it.
Thanks!
Quoted 29 lines
Quoted 28 lines
Quoted 5 lines
Btw this function should be implemented in an independent patch after svq is fully functional.
(Reasons for that are already commented at the top of this mail :) ).
Thanks!
Quoted 19 lines
Thanks
Quoted 16 lines
hw/virtio/vhost-vdpa.c +13
-0+ goto err; + } + + r =vhost_vdpa_enable_svq (v, enable); +if (unlikely (!r)) { + err_cause ="Error enabling (see monitor)" ; + goto err; + } + +err: +if (err_cause) { +error_setg (errp,"Can't enable shadow vq on %s: %s" , name, err_cause); + } }const VhostOps vdpa_ops = {
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
In reply to Jason WangQuoted 70 lines · Jason Wang
Quoted 25 lines · Eugenio Perez Martin
Quoted 23 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Signed-off-by Eugenio Pérez<[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)Quoted 7 lines
Quoted 3 lines
Think hard about the whole logic. This is safe since qemu memory map will fail if guest submits a invalidate IOVA.
Can you expand on this? What you mean is that VirtQueue already protects SVQ code if the guest sets an invalid buffer address (GPA), isn't it?
Yes.
Quoted 11 lines
Quoted 7 lines
Then I wonder if we do something much more simpler:
1) Using qemu VA as IOVA but only maps the VA that belongs to guest
2) Then we don't need any IOVA tree here, what we need is to just map vring and use qemu VA without any translationThat would be great, but either qemu's SVQ vring or guest translated buffers address (in qemu VA form) were already in high addresses, outside of the device's iova range (in my test).
You're right. I miss that and that's why we need e.g iova tree and allocator.
What I proposed only makes sense when shared virtual memory (SVA) is implemented. In the case of SVA, the valid iova range should be the full VA range.
Quoted 5 lines
I didn't try remapping tricks to make them fit in the range, but I think it does complicate the solution relatively fast if there was already memory in that range owned by qemu before enabling SVQ:
Quoted 8 lines
* Guest memory must be contiguous in VA address space, but it "must" support hotplug/unplug (although vDPA currently pins it). Hotplug memory could always overlap with SVQ vring, so we would need to move it.
* Duplicating mapped memory for writing? (Not sure if guest memory is actually movable in qemu).
* Indirect descriptors will need to allocate and free memory more or less frequently, increasing the possibility of overlapping.
Quoted 3 lines
I'm not sure I get the problem, but overlapping is not an issue since we're using VA.
It's basically the same (potential) problem of DPDK's SVQ: IOVA Range goes from 0 to X. That means that both GPA and SVQ must be in IOVA range. As an example, we put GPA at the beginning of the range, that grows upwards when memory is hot plugged, and SVQ vrings that grows downwards when devices are added or set in SVQ mode.
Even without both space fragmentation problems, we could reach a point where both will take the same address, and we would need to go to the tree.
But since we are able to detect those situations, I can see how we can work in two modes as an optimization: 1:1 when they don't overlap, and fragmented tree where it does. But I don't think it's a good idea to include it from the beginning, and I'm not sure if that is worth it without measuring the tree translation cost first.
Quoted 5 lines
Quoted 2 lines
If we can move guest memory,
I'm not sure we can do this or it looks very tricky.
Just thinking out loud here, but maybe we could map all memory and play with remap_file_pages [1] a little bit for that.
Quoted 6 lines
Quoted 6 lines
however, I can see how we can track it in
a tree *but* mark when the tree is 1:1 with qemu's VA, so buffers forwarding does not take the translation penalty. When guest memory cannot be map 1:1, we can resort to tree, and come back to 1:1 translation if the offending tree node(s) get deleted.
Quoted 7 lines
Quoted 4 lines
However I think this puts the solution a little bit farther than "starting simple" :).
Does it make sense?
Yes. So I think I will review the IOVA tree codes and get back to you.
Looking forward to it :).
Thanks!
[1] https://linux.die.net/man/2/remap_file_pages
Quoted 9 lines
THanks
Quoted 6 lines
Thanks!
Quoted 2 lines
Thanks
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
Quoted 110 lines · Eugenio Perez Martin
Quoted 70 lines · Jason Wang
Quoted 25 lines · Eugenio Perez Martin
Quoted 23 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Signed-off-by Eugenio Pérez<[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)Quoted 7 lines
Quoted 3 lines
Think hard about the whole logic. This is safe since qemu memory map will fail if guest submits a invalidate IOVA.
Can you expand on this? What you mean is that VirtQueue already protects SVQ code if the guest sets an invalid buffer address (GPA), isn't it?
Yes.
Quoted 11 lines
Quoted 7 lines
Then I wonder if we do something much more simpler:
1) Using qemu VA as IOVA but only maps the VA that belongs to guest
2) Then we don't need any IOVA tree here, what we need is to just map vring and use qemu VA without any translationThat would be great, but either qemu's SVQ vring or guest translated buffers address (in qemu VA form) were already in high addresses, outside of the device's iova range (in my test).
You're right. I miss that and that's why we need e.g iova tree and allocator.
What I proposed only makes sense when shared virtual memory (SVA) is implemented. In the case of SVA, the valid iova range should be the full VA range.
Quoted 5 lines
I didn't try remapping tricks to make them fit in the range, but I think it does complicate the solution relatively fast if there was already memory in that range owned by qemu before enabling SVQ:
Quoted 8 lines
* Guest memory must be contiguous in VA address space, but it "must" support hotplug/unplug (although vDPA currently pins it). Hotplug memory could always overlap with SVQ vring, so we would need to move it.
* Duplicating mapped memory for writing? (Not sure if guest memory is actually movable in qemu).
* Indirect descriptors will need to allocate and free memory more or less frequently, increasing the possibility of overlapping.Quoted 3 lines
I'm not sure I get the problem, but overlapping is not an issue since we're using VA.
It's basically the same (potential) problem of DPDK's SVQ: IOVA Range goes from 0 to X. That means that both GPA and SVQ must be in IOVA range. As an example, we put GPA at the beginning of the range, that grows upwards when memory is hot plugged, and SVQ vrings that grows downwards when devices are added or set in SVQ mode.
Even without both space fragmentation problems, we could reach a point where both will take the same address, and we would need to go to the tree.
But since we are able to detect those situations, I can see how we can work in two modes as an optimization: 1:1 when they don't overlap, and fragmented tree where it does. But I don't think it's a good idea to include it from the beginning, and I'm not sure if that is worth it without measuring the tree translation cost first.
Quoted 5 lines
Quoted 2 lines
If we can move guest memory,
I'm not sure we can do this or it looks very tricky.
Just thinking out loud here, but maybe we could map all memory and play with
remap_file_pages[1] a little bit for that.Quoted 9 lines
Quoted 9 lines
however, I can see how we can track it in
a tree *but* mark when the tree is 1:1 with qemu's VA, so buffers forwarding does not take the translation penalty. When guest memory cannot be map 1:1, we can resort to tree, and come back to 1:1 translation if the offending tree node(s) get deleted.However I think this puts the solution a little bit farther than "starting simple" :).
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Does it make sense?
Yes. So I think I will review the IOVA tree codes and get back to you.
Looking forward to it :).
PS: Actually, they still use the GArray solution as the previous series. I'm currently migrating to use an actual tree and adding allocation features to util/iova-tree, so maybe it is not worth reviewing it at this moment. The public interface to it is the same, but there is little to review there.
Thanks!
Quoted 13 lines
Thanks!
[1] https://linux.die.net/man/2/remap_file_pages
Quoted 9 lines
THanks
Quoted 6 lines
Thanks!
Quoted 2 lines
Thanks
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 15/20] vhost: Shadow virtqueue buffers forwarding
In reply to Jason WangQuoted 112 lines · Jason Wang
Quoted 109 lines · Eugenio Perez Martin
Quoted 76 lines · Jason Wang
Quoted 74 lines · Eugenio Pérez
Initial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device.
For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series.
It reuses the VirtQueue code for the device part. The driver part is based on Linux's
virtio_ringdriver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones.SVQ uses
VIRTIO_CONFIG_S_DEVICE_STOPPEDto pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter.This status bit
VIRTIO_CONFIG_S_DEVICE_STOPPEDis currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits.Removal of
_S_DEVICE_STOPPEDbit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device.Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-)qapi/net.json +1
-1@@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding path # # Returns: Error if failure, or 'no error' for success. #hw/virtio/vhost-shadow-virtqueue.c +10
-0@@ -10,6 +10,7 @@#include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" +#include "hw/virtio/virtio-access.h" #include "standard-headers/linux/vhost_types.h" @@ -44,15 +45,135 @@ typedef struct VhostShadowVirtqueue {/* Virtio device */ VirtIODevice *vdev; + +/* Map for returning guest's descriptors */ +VirtQueueElement **ring_id_maps; + +/* Next head to expose to device */ +uint16_t avail_idx_shadow; + +/* Next free descriptor */ +uint16_t free_head;Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 62
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+ +/* Last seen used idx */ +uint16_t shadow_used_idx; + +/* Next head to consume from device */ +uint16_t used_idx;Let's use "
last_used_idx" as kernel driver did.Ok I will change it.
Quoted 21 lines
Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +2
-1} VhostShadowVirtqueue;/* If the device is using some of these, SVQ cannot communicate */ bool vhost_svq_valid_device_features (uint64_t *dev_features) { -return true; +uint64_t b; +bool r = true;hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 79
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+ +for (b =VIRTIO_TRANSPORT_F_START ; b <=VIRTIO_TRANSPORT_F_END ; ++b) { +switch (b) { +case VIRTIO_F_NOTIFY_ON_EMPTY : +case VIRTIO_F_ANY_LAYOUT : +/* SVQ is fine with this feature */ +continue ; + +case VIRTIO_F_ACCESS_PLATFORM : +/* SVQ needs this feature disabled. Can't continue */ So code can explain itself, need a comment to explain why.
Quoted 7 lines
Quoted 2 lines
Do you mean that it *doesn't* need a comment to explain why? In that case I will delete them.
I meant the comment is duplicated with the code. If you wish, you can explain why
ACCESS_PLATFORMneeds to be disabled.
Got it, I will do something about it.
Quoted 41 lines
Quoted 41 lines
Quoted 11 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 87
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-0+if (*dev_features &BIT_ULL (b)) { +clear_bit (b, dev_features); + r = false; + } +break ; + +case VIRTIO_F_VERSION_1 : +/* SVQ needs this feature, so can't continue */ Quoted 2 lines
A comment to explain why SVQ needs this feature.
Sure I will add it.
Quoted 13 lines
Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c +13
-0+if (!(*dev_features &BIT_ULL (b))) { +set_bit (b, dev_features); + r = false; + } +continue ; + +default : +/* +* SVQ must disable this feature, let's hope the device is fine +* without it. +*/ +if (*dev_features &BIT_ULL (b)) { +clear_bit (b, dev_features);Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 106
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+ } + } + } + +return r; +}Let's move this to patch 14.
I can move it down to 14/20, but then it is not really accurate, since notifications forwarding can work with all feature sets. Not like we are introducing a regression, but still.
Quoted 6 lines
Quoted 1 line
I can always explain that in the patch message though, would that be ok?
I'm afraid this will break bisection. E.g for patch 14, it works for any features but for patch 15 it doesn't.
So no moving them? :).
Quoted 137 lines
Quoted 90 lines
Quoted 80 lines
Quoted 80 lines
hw/virtio/vhost-shadow-virtqueue.c +68
-1+ +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq, +const struct iovec *iovec, +size_t num,bool more_descs,bool write) +{ +uint16_t i = svq->free_head, last = svq->free_head; +unsigned n; +uint16_t flags = write ?cpu_to_le16 (VRING_DESC_F_WRITE ) :0 ; +vring_desc_t *descs = svq->vring.desc; + +if (num ==0 ) { +return ; + } + +for (n =0 ; n < num; n++) { +if (more_descs || (n +1 < num)) { + descs[i].flags = flags |cpu_to_le16 (VRING_DESC_F_NEXT ); + }else { + descs[i].flags = flags; + } + descs[i].addr =cpu_to_le64 ((hwaddr )iovec[n].iov_base); + descs[i].len =cpu_to_le32 (iovec[n].iov_len); + + last = i; + i =cpu_to_le16 (descs[i].next); + } + + svq->free_head =le16_to_cpu (descs[last].next); +} + +static unsigned vhost_svq_add_split (VhostShadowVirtqueue *svq, +VirtQueueElement *elem) +{ +int head; +unsigned avail_idx; +vring_avail_t *avail = svq->vring.avail; + + head = svq->free_head; + +/* We need some descriptors here */ +assert (elem->out_num || elem->in_num); + +vhost_vring_write_descs (svq, elem->out_sg, elem->out_num, + elem->in_num >0 , false); +vhost_vring_write_descs (svq, elem->in_sg, elem->in_num, false, true); + +/* +* Put entry in available array (but don't update avail->idx until they +* do sync). +*/ + avail_idx = svq->avail_idx_shadow & (svq->vring.num -1 ); + avail->ring[avail_idx] =cpu_to_le16 (head); + svq->avail_idx_shadow++; + +/* Update avail index after the descriptor is wrote */ +smp_wmb (); + avail->idx =cpu_to_le16 (svq->avail_idx_shadow); + +return head; + } -/* Forward guest notifications */ +static void vhost_svq_add (VhostShadowVirtqueue *svq,VirtQueueElement *elem) +{ +unsigned qemu_head =vhost_svq_add_split (svq, elem); + + svq->ring_id_maps[qemu_head] = elem; +} + +/* Handle guest->device notifications */ static void vhost_handle_guest_kick (EventNotifier *n) { VhostShadowVirtqueue *svq = container_of(n, VhostShadowVirtqueue,@@ -62,7 +183,74 @@ static void vhost_handle_guest_kick(EventNotifier *n)return ; }Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 191
Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-1-event_notifier_set (&svq->kick_notifier); +/* Make available as many buffers as possible */ + do { + if (virtio_queue_get_notification(svq->vq)) { + /* No more notifications until process all available */ + virtio_queue_set_notification(svq->vq, false); + }This can be done outside the loop.
Quoted 7 lines
I think it cannot. The intention of doing this way is that we check for new available buffers *also after* enabling notifications, so we don't miss any of them. It is more or less copied from
virtio_blk_handle_vq, which also needs to run to completion.If we need to loop again because there are more available buffers, we want to disable notifications again. Or am I missing something?
I think you're right.
Quoted 30 lines
Quoted 16 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 203
Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c +12
-0+ +while (true) { +VirtQueueElement *elem =virtqueue_pop (svq->vq,sizeof (*elem)); +if (!elem) { +break ; + } + +vhost_svq_add (svq, elem); +event_notifier_set (&svq->kick_notifier); + } + +virtio_queue_set_notification (svq->vq, true);I think this can be moved to the end of this function.
(Same as previous answer)
Quoted 6 lines
Btw, we probably need a quota to make sure the svq is not hogging the main event loop.
Similar issue could be found in both virtio-net TX (using timer or bh) and TAP (a quota).
I think that virtqueue size is the natural limit to that: since we are not making any buffers used in the loop, there is no way that it runs more than virtqueue size times. If it does because of an evil/bogus guest,
virtqueue_popraises the message "Virtqueue size exceeded" and returns NULL, effectively breaking the loop.Quoted 3 lines
Virtio-net tx functions mark each buffer right after making them available and use it, so they can hog BQL. But my understanding is that is not possible in the SVQ case.
Quoted 9 lines
Right.
Quoted 2 lines
I can add a comment in the code to make it clearer though.
Yes, please.
Ok I will add the comment.
Quoted 103 lines
Quoted 17 lines
Quoted 17 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_handle_guest_kick(EventNotifier *n) · line 214
Quoted 13 lines
hw/virtio/vhost-shadow-virtqueue.c +11
-0+ }while (!virtio_queue_empty (svq->vq)); +} + +static bool vhost_svq_more_used (VhostShadowVirtqueue *svq) +{ +if (svq->used_idx != svq->shadow_used_idx) { +return true; + } + +/* Get used idx must not be reordered */ +smp_rmb ();Interesting, we don't do this for kernel drivers. It would be helpful to explain it more clear by "X must be done before Y".
Quoted 3 lines
I think this got reordered, it's supposed to be *after* get the used idx, so it matches the one in the kernel with the comment "Only get used array entries after they have been exposed by host.".
Right.
Quoted 2 lines
I will change it for the next series.
Ok.
Quoted 71 lines
Quoted 57 lines
Quoted 57 lines
hw/virtio/vhost-shadow-virtqueue.c +47
-2+ svq->shadow_used_idx =cpu_to_le16 (svq->vring.used->idx); + +return svq->used_idx != svq->shadow_used_idx; +} + +static VirtQueueElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) +{ +vring_desc_t *descs = svq->vring.desc; +const vring_used_t *used = svq->vring.used; +vring_used_elem_t used_elem; +uint16_t last_used; + +if (!vhost_svq_more_used (svq)) { +return NULL ; + } + + last_used = svq->used_idx & (svq->vring.num -1 ); + used_elem.id =le32_to_cpu (used->ring[last_used].id); + used_elem.len =le32_to_cpu (used->ring[last_used].len); + + svq->used_idx++; +if (unlikely (used_elem.id >= svq->vring.num)) { +error_report ("Device %s says index %u is used" , svq->vdev->name, + used_elem.id); +return NULL ; + } + +if (unlikely (!svq->ring_id_maps[used_elem.id])) { +error_report ( +"Device %s says index %u is used, but it was not available" , + svq->vdev->name, used_elem.id); +return NULL ; + } + + descs[used_elem.id].next = svq->free_head; + svq->free_head = used_elem.id; + + svq->ring_id_maps[used_elem.id]->len = used_elem.len; +return g_steal_pointer (&svq->ring_id_maps[used_elem.id]); }/* Forward vhost notifications */ @@ -70,8 +258,26 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n) {VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, call_notifier); - -event_notifier_set (&svq->guest_call_notifier); +VirtQueue *vq = svq->vq; + +/* Make as many buffers as possible used. */ + do { + unsigned i = 0; + + /* TODO: Use VRING_AVAIL_F_NO_INTERRUPT */ + while (true) {Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_handle_call_no_test(EventNotifier *n) · line 274
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +6
-0+ g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); +if (!elem) { +break ; + } + +assert (i < svq->vring.num);Let's return error instead of using the assert.
Actually this is a condition that we should never meet: In the case of ring overrun, device would try to set used a descriptor that is either
Quoted 1 line
vring size *or* should try to overrun some of the already used ones.
In both cases, elem should be NULL and the loop should break.
Quoted 9 lines
Quoted 5 lines
So this is a safety net protecting from both, if we have an i >
svq->vring.nummeans we are not processing used buffers well anymore, and (moreover) this is happening after making used all descriptors.Taking that into account, should we delete it?
Maybe a warning instead.
Ok I will do it.
Quoted 46 lines
Quoted 38 lines
Quoted 22 lines
Quoted 22 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-0+virtqueue_fill (vq, elem, elem->len, i++); + } + +virtqueue_flush (vq, i); +event_notifier_set (&svq->guest_call_notifier); + }while (vhost_svq_more_used (svq)); }static void vhost_svq_handle_call (EventNotifier *n)@@ -204,12 +410,25 @@ err_set_vring_kick:void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq) { +int i;int r =vhost_svq_restore_vdev_host_notifier (dev, idx, svq); +if (unlikely (r <0 )) {error_report ("Couldn't restore vq kick fd: %s" ,strerror (-r)); }Quoted 16 lines
hw/virtio/vhost-shadow-virtqueue.c err_set_vring_kick: · line 430
Quoted 11 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0event_notifier_set_handler (&svq->host_notifier,NULL ); + +for (i =0 ; i < svq->vring.num; ++i) { + g_autofree VirtQueueElement *elem = svq->ring_id_maps[i]; +/* +* Although the doc says we must unpop in order, it's ok to unpop +* everything. +*/ +if (elem) { +virtqueue_unpop (svq->vq, elem, elem->len); + }Will this result some of the "pending" buffers to be submitted multiple times? If yes, should we wait for all the buffers used instead of doing the unpop here?
Quoted 6 lines
Do you mean to call
virtqueue_unpopwith the same elem (or elem.id) multiple times? That should never happen, because elem.id should be the position in thering_id_maps. Also,unpop()should just unmap the element and never sync again.Maybe it is way clearer to call
virtqueue_detach_elementhere directly.
Quoted 7 lines
I meant basically for the buffers that were consumed by the device but not made used. In this case if we unpop here. It will be processed by the device later via vhost-vdpa again?
This is probably fine for net but I'm not sure it works for other devices. Another way is to wait until all the consumed buffer used.
My understanding is that it is the responsibility of the device to pause and mark all of them as used. Without things like INFLIGHT_FD there is no way for qemu to tell the difference between a buffer that should be marked as used but is not, and a buffer that is properly not marked as used. So if the device is not able to do it by itself, it (or vdpa backend device driver) should not offer / fail the pause command.
Similarly, if the device *needs* the inflight_fd mechanism to pause, it should fail the pause if it has not been negotiated.
Quoted 91 lines
Quoted 91 lines
Quoted 91 lines
Quoted 78 lines
hw/virtio/vhost-shadow-virtqueue.c +8
-1+ } }/* @@ -224,7 +443,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)size_t driver_size;size_t device_size; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1); -int r; +int r, i; r =event_notifier_init (&svq->kick_notifier,0 );if (r !=0 ) {@@ -250,6 +469,11 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)memset (svq->vring.desc,0 , driver_size); svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size);memset (svq->vring.used,0 , device_size); +for (i =0 ; i < num -1 ; i++) { + svq->vring.desc[i].next =cpu_to_le16 (i +1 ); + } + + svq->ring_id_maps = g_new0(VirtQueueElement *, num);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);@@ -269,6 +493,7 @@ void vhost_svq_free(VhostShadowVirtqueue *vq)event_notifier_cleanup (&vq->kick_notifier);event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier); +g_free (vq->ring_id_maps);qemu_vfree (vq->vring.desc);qemu_vfree (vq->vring.used);g_free (vq);hw/virtio/vhost-vdpa.c +23
-0@@ -19,6 +19,7 @@#include "hw/virtio/virtio-net.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" #include "cpu.h" @@ -475,6 +476,28 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,return vhost_vdpa_backend_set_features (dev, features); } +/** +* Restore guest features to vdpa device +*/ +static int vhost_vdpa_set_guest_features (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +return vhost_vdpa_backend_set_features (dev, v->guest_features); +} + +/** +* Set shadow virtqueue supported features +*/ +static int vhost_vdpa_set_svq_features (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +uint64_t features = v->host_features; +bool b =vhost_svq_valid_device_features (&features); +assert (b); + +return vhost_vdpa_backend_set_features (dev, features); +} +static int vhost_vdpa_set_backend_cap (struct vhost_dev *dev) {uint64_t features;@@ -730,6 +753,19 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true;hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 763
Quoted 10 lines
hw/virtio/vhost-vdpa.c +8
-0} +static int vhost_vdpa_vring_pause (struct vhost_dev *dev) +{ +int r; +uint8_t status; + +vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_DEVICE_STOPPED ); + do { + r = vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);I guess we'd better add some sleep here.
Quoted 6 lines
Quoted 2 lines
If the final version still contains the call, I will add the sleep. At the moment I think it's better if we stop the device by a vdpa ioctl.
Ok, so the idea is to sleep in the ioctl?
At the moment I would say so: The ioctl can take "as long as it wants". If we decide to support both methods (virtio pause & vdpa pause), then sleep should be added, yes.
If we don't trust the device regarding the time it will take to pause, we could try to move the wait for the pause outside of BQL in the future, so qemu can operate regularly. I think it is not worth it in the first series but it could help the adoption to other stricter or more stateful devices.
Thanks!
Quoted 178 lines
Quoted 138 lines
Quoted 76 lines
Quoted 76 lines
hw/virtio/vhost-vdpa.c +45
-3+ }while (r ==0 && !(status &VIRTIO_CONFIG_S_DEVICE_STOPPED )); + +return 0 ; +} +/* * Start shadow virtqueue. */ @@ -742,9 +778,29 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) .index = idx + dev->vq_index, .fd = event_notifier_get_fd(vhost_call_notifier), }; +struct vhost_vring_addr addr = { + .index = idx + dev->vq_index, + }; +struct vhost_vring_state num = { + .index = idx + dev->vq_index, + .num =virtio_queue_get_num (dev->vdev, idx), + };int r;bool b; +vhost_svq_get_vring_addr (svq, &addr); + r =vhost_vdpa_set_vring_addr (dev, &addr); +if (unlikely (r)) { +error_report ("vhost_set_vring_addr for shadow vq failed" ); +return false; + } + + r =vhost_vdpa_set_vring_num (dev, &num); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_num for shadow vq failed" ); +return false; + } +/* Set shadow vq -> guest notifier */ assert (v->call_fd[idx]);vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]);@@ -781,15 +837,32 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) {VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); -bool ok; -if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ; }g_ptr_array_add (v->shadow_vqs, svq); + } + } - ok =vhost_vdpa_svq_start_vq (hdev, n); + r =vhost_vdpa_vring_pause (hdev); +assert (r ==0 ); + +if (enable) { +for (n =0 ; n < v->shadow_vqs->len; ++n) { +/* Obtain Virtqueue state */ +vhost_virtqueue_stop (hdev, hdev->vdev, &hdev->vqs[n], n); + } + } + +/* Reset device so it can be configured */ + r =vhost_vdpa_dev_start (hdev, false); +assert (r ==0 ); + +if (enable) { +int r; +for (n =0 ; n < v->shadow_vqs->len; ++n) { +bool ok =vhost_vdpa_svq_start_vq (hdev, n);if (unlikely (!ok)) {/* Free still not started svqs */ g_ptr_array_set_size (v->shadow_vqs, n);Quoted 9 lines
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 874
Quoted 6 lines
hw/virtio/vhost-vdpa.c +2
-0@@ -797,11 +870,19 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)break ; } } + +/* Need to ack features to set state in vp_vdpa devices */
vhost_vdpaactually?Yes, what a mistake!
Quoted 39 lines
Quoted 39 lines
hw/virtio/vhost-vdpa.c +12
-0+ r =vhost_vdpa_set_svq_features (hdev); +if (unlikely (r)) { + enable = false; + } } v->shadow_vqs_enabled = enable;if (!enable) { +vhost_vdpa_set_guest_features (hdev); +/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { struct vhost_vring_file file = {@@ -818,7 +899,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* TODO: This can unmask or override call fd! */ vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); } + } + r =vhost_vdpa_dev_start (hdev, true); +assert (r ==0 ); + +if (!enable) {/* Resources cleanup */ g_ptr_array_set_size (v->shadow_vqs,0 ); }@@ -831,6 +917,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp)struct vhost_vdpa *v;const char *err_cause =NULL ;bool r; +uint64_t svq_features; QLIST_FOREACH(v, &vhost_vdpa_devices, entry) {if (v->dev->vdev &&0 ==strcmp (v->dev->vdev->name, name)) {@@ -846,6 +933,20 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) goto err; }Quoted 10 lines
hw/virtio/vhost-vdpa.c void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) · line 943
Quoted 8 lines
hw/virtio/vhost-vdpa.c +8
-0+ svq_features = v->host_features; +if (!vhost_svq_valid_device_features (&svq_features)) { +error_setg (errp, +"Can't enable shadow vq on %s: Unexpected feature flags (%lx-%lx)" , + name, v->host_features, svq_features); +return ; + }else { +/* TODO: Check for virtio_vdpa + IOMMU & modern device */ I guess you mean "
vhost_vdpa" here.Yes, a similar mistake in less than 50 lines :).
Quoted 8 lines
Quoted 3 lines
For IOMMU, I guess you mean "vIOMMU"
actually?This comment is out of date and inherited from the vhost version, where only the IOMMU version was developed, so it will be deleted in the next series. I think it makes little sense to check vIOMMU if we stick with vDPA since it still does not support it, but we could make the check here for sure.
Right.
Thanks
Quoted 24 lines
Thanks!
Quoted 21 lines
Thanks
Quoted 18 lines
hw/virtio/vhost-vdpa.c +7
-1+ } + +if (err_cause) { + goto err; + } + r =vhost_vdpa_enable_svq (v, enable);if (unlikely (!r)) { err_cause ="Error enabling (see monitor)" ;@@ -853,7 +954,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) } err: -if (err_cause) { +if (errp ==NULL && err_cause) {error_setg (errp,"Can't enable shadow vq on %s: %s" , name, err_cause); } }
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue
In reply to Eugenio Perez MartinQuoted 586 lines · Eugenio Perez Martin
Quoted 11 lines · Jason Wang
Quoted 4 lines · Eugenio Pérez
Shadow virtqueue notifications forwarding is disabled when
vhost_devstops, so code flow follows usual cleanup.Also, host notifiers must be disabled at SVQ start,
Any reason for this?
It will be addressed in a later series, sorry.
Quoted 6 lines
Quoted 4 lines
and they will not
start if SVQ has been enabled when device is stopped. This is trivial to address, but it is left out for simplicity at this moment.Quoted 7 lines
It looks to me this patch also contains the following logics
1) codes to enable svq
2) codes to let svq to be enabled from QMP.
I think they need to be split out,
I agree that we can split this more, with the code that belongs to SVQ and the code that belongs to vhost-vdpa. it will be addressed in future series.
Quoted 3 lines
we may endup with the following
series of patchesWith "series of patches" do you mean to send every step in a separated series? There are odds of having the need of modifying code already sent & merged with later ones. If you confirm to me that it is fine, I can do it that way for sure.
Quoted 6 lines
1) svq skeleton with enable/disable
2) route host notifier to svq
3) route guest notifier to svq
4) codes to enable svq
5) enable svq via QMPI'm totally fine with that, but there is code that is never called if the qmp command is not added. The compiler complains about static functions that are not called, making impossible things like bisecting through these commits, unless I use attribute((unused)) or similar. Or have I missed something?
We could do that way with the code that belongs to SVQ though, since all of it is declared in headers. But to delay the "enable svq via qmp" to the last one makes debugging harder, as we cannot just enable notifications forwarding with no buffers forwarding.
If I introduce a change in the notifications code, I can simply go to these commits and enable SVQ for notifications. This way I can have an idea of what part is failing. A similar logic can be applied to other devices than
vp_vdpa. We would lose it if weQuoted 30 lines
Quoted 30 lines
Signed-off-by Eugenio Pérez <[email protected]>--- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.h | 8 ++ include/hw/virtio/vhost-vdpa.h | 4 + hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- 5 files changed, 264 insertions(+), 4 deletions(-)qapi/net.json +1
-1@@ -88,7 +88,7 @@ # # @enable: true to use the alternate shadow VQ notifications # -# Returns: Always error, since SVQ is not implemented at the moment. +# Returns: Error if failure, or 'no error' for success. # # Since: 6.2 #hw/virtio/vhost-shadow-virtqueue.h +0
-0@@ -14,6 +14,14 @@Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.h line 17
Quoted 3 lines
hw/virtio/vhost-shadow-virtqueue.h +1
-0typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ; +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq);Let's move this function to another patch since it's unrelated to the
guest->hostrouting.Right, I missed it while squashing commits and at later reviews.
Quoted 63 lines
Quoted 63 lines
hw/virtio/vhost-shadow-virtqueue.h +7
-0+void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd); + +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx);void vhost_svq_free (VhostShadowVirtqueue *vq);include/hw/virtio/vhost-vdpa.h +4
-0@@ -12,6 +12,8 @@#ifndef HW_VIRTIO_VHOST_VDPA_H #define HW_VIRTIO_VHOST_VDPA_H +#include <gmodule.h> +#include "qemu/queue.h" #include "hw/virtio/virtio.h" @@ -24,6 +26,8 @@ typedef struct vhost_vdpa {int device_fd;uint32_t msg_type;MemoryListener listener; +bool shadow_vqs_enabled; +GPtrArray *shadow_vqs;struct vhost_dev *dev;QLIST_ENTRY (vhost_vdpa) entry;VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX ];hw/virtio/vhost-shadow-virtqueue.c +12
-1@@ -9,9 +9,12 @@#include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" +#include "hw/virtio/vhost.h" + +#include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" -#include "qemu/event_notifier.h" +#include "qemu/main-loop.h" /* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue {@@ -19,14 +22,146 @@ typedef struct VhostShadowVirtqueue {EventNotifier kick_notifier;/* Shadow call notifier, sent to vhost */ EventNotifier call_notifier; + +/* +* Borrowed virtqueue's guest to host notifier. +* To borrow it in this event notifier allows to register on the event +* loop and access the associated shadow virtqueue easily. If we use the +* VirtQueue, we don't have an easy way to retrieve it. +* +* So shadow virtqueue must not clean it, or we would lose VirtQueue one. Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 37
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+*/ +EventNotifier host_notifier; + +/* Guest's call notifier, where SVQ calls guest. */ +EventNotifier guest_call_notifier;To be consistent, let's simply use "
guest_notifier" here.It could be confused when the series adds a guest -> qemu kick notifier then. Actually, I think it would be better to rename
host_notifierto something likehost_svq_notifier. Orhost_callandguest_call, since "notifier" is already in the type, making the name to be a little bit "Hungarian notation".Quoted 6 lines
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +3
-0+ +/* Virtio queue shadowing */ +VirtQueue *vq; } VhostShadowVirtqueue;Quoted 14 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 51
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c +9
-0+/* Forward guest notifications */ +static void vhost_handle_guest_kick (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + host_notifier); + +if (unlikely (!event_notifier_test_and_clear (n))) { +return ; + }Is there a chance that we may stop the processing of available buffers during the svq enabling? There could be no kick from the guest in this case.
Actually, yes, I think you are right. The guest kick eventfd could have been consumed by vhost but there may be still pending buffers.
I think it would be better to check for available buffers first, then clear the notifier unconditionally, and then re-check and process them if any [1].
However, this problem arises later in the series: At this moment the device is not reset and guest's host notifier is not replaced, so either vhost/device receives the kick, or SVQ does and forwards it.
Does it make sense to you?
Quoted 40 lines
Quoted 40 lines
hw/virtio/vhost-shadow-virtqueue.c +39
-0+ +event_notifier_set (&svq->kick_notifier); +} + +/* +* Obtain the SVQ call notifier, where vhost device notifies SVQ that there +* exists pending used buffers. +* +* @svq Shadow Virtqueue +*/ +EventNotifier *vhost_svq_get_svq_call_notifier (VhostShadowVirtqueue *svq) +{ +return &svq->call_notifier; +} + +/* +* Set the call notifier for the SVQ to call the guest +* +* @svq Shadow virtqueue +* @call_fd call notifier +* +* Called on BQL context. +*/ +void vhost_svq_set_guest_call_notifier (VhostShadowVirtqueue *svq,int call_fd) +{ +event_notifier_init_fd (&svq->guest_call_notifier, call_fd); +} + +/* +* Restore the vhost guest to host notifier, i.e., disables svq effect. +*/ +static int vhost_svq_restore_vdev_host_notifier (struct vhost_dev *dev, +unsigned vhost_index, +VhostShadowVirtqueue *svq) +{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = vhost_index, + .fd =event_notifier_get_fd (vq_host_notifier),Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 95
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+ }; +int r; + +/* Restore vhost kick */ + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);And remap the notification area if necessary.
Totally right, that step is missed in this series.
However, remapping guest host notifier memory region has no advantages over using ioeventfd to perform guest -> SVQ notifications, doesn't it? By both methods flow needs to go through the hypervisor kernel.
Quoted 25 lines
Quoted 25 lines
hw/virtio/vhost-shadow-virtqueue.c +23
-0+return r ? -errno :0 ; +} + +/* +* Start shadow virtqueue operation. +* @dev vhost device +* @hidx vhost virtqueue index +* @svq Shadow Virtqueue +*/ +bool vhost_svq_start (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +EventNotifier *vq_host_notifier =virtio_queue_get_host_notifier (svq->vq); +struct vhost_vring_file file = { + .index = dev->vhost_ops->vhost_get_vq_index (dev, dev->vq_index + idx), + .fd =event_notifier_get_fd (&svq->kick_notifier), + }; +int r; + +/* Check that notifications are still going directly to vhost dev */ +assert (virtio_queue_is_host_notifier_enabled (svq->vq)); + +/* Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 122
Quoted 4 lines
hw/virtio/vhost-shadow-virtqueue.c +4
-0+* event_notifier_set_handler already checks for guest's notifications if +* they arrive in the switch, so there is no need to explicitely check for +* them. +*/ If this is true, shouldn't we call
vhost_set_vring_kick()before theevent_notifier_set_handler()?Not at this point of the series, but it could be another solution when we need to reset the device and we are unsure if all buffers have been read. But I think I prefer the solution exposed in [1] and to explicitly call
vhost_handle_guest_kickhere. Do you think differently?Quoted 3 lines
Btw, I think we should update the fd if
set_vring_kick()was called after this function?Kind of. This is currently bad in the code, but...
Backend callbacks
vhost_ops->vhost_set_vring_kickandvhost_ops->vhost_set_vring_addrare only called atvhost_virtqueue_start. And they are always called with known data already stored in VirtQueue.To avoid storing more state in
vhost_vdpa, I think that we should avoid duplicating them, but ignore newkick_fdor address in SVQ mode, and retrieve them again at the moment the device is (re)started in SVQ mode. Qemu already avoids things like allowing the guest to set addresses at random time, using the VirtIOPCIProxy to store them.I also see how duplicating that status could protect vdpa SVQ code against future changes to vhost code, but that would make this series bigger and more complex with no need at this moment in my opinion.
Do you agree?
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c typedef struct VhostShadowVirtqueue { · line 127
Quoted 6 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+event_notifier_init_fd (&svq->host_notifier, +event_notifier_get_fd (vq_host_notifier)); +event_notifier_set_handler (&svq->host_notifier, vhost_handle_guest_kick); + + r = dev->vhost_ops->vhost_set_vring_kick (dev, &file);And we need to stop the notification area mmap.
Right.
Quoted 124 lines
Quoted 113 lines
hw/virtio/vhost-shadow-virtqueue.c +32
-0+if (unlikely (r !=0 )) { +error_report ("Couldn't set kick fd: %s" ,strerror (errno)); + goto err_set_vring_kick; + } + +return true; + +err_set_vring_kick: +event_notifier_set_handler (&svq->host_notifier,NULL ); + +return false; +} + +/* +* Stop shadow virtqueue operation. +* @dev vhost device +* @idx vhost queue index +* @svq Shadow Virtqueue +*/ +void vhost_svq_stop (struct vhost_dev *dev,unsigned idx, +VhostShadowVirtqueue *svq) +{ +int r =vhost_svq_restore_vdev_host_notifier (dev, idx, svq); +if (unlikely (r <0 )) { +error_report ("Couldn't restore vq kick fd: %s" ,strerror (-r)); + } + +event_notifier_set_handler (&svq->host_notifier,NULL ); +} +/* * Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) { +int vq_idx = dev->vq_index + idx; g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);int r;@@ -44,6 +179,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) goto err_init_call_notifier; } + svq->vq =virtio_get_queue (dev->vdev, vq_idx);return g_steal_pointer (&svq); err_init_call_notifier:hw/virtio/vhost-vdpa.c +24
-0@@ -17,6 +17,7 @@#include "hw/virtio/vhost.h" #include "hw/virtio/vhost-backend.h" #include "hw/virtio/virtio-net.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" @@ -272,6 +273,16 @@ static void vhost_vdpa_add_status(struct vhost_dev *dev, uint8_t status)vhost_vdpa_call (dev,VHOST_VDPA_SET_STATUS , &s); } +/** +* Adaptor function to free shadow virtqueue through gpointer +* +* @svq The Shadow Virtqueue +*/ +static void vhost_psvq_free (gpointer svq) +{ +vhost_svq_free (svq); +} +static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp) {struct vhost_vdpa *v;@@ -283,6 +294,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; + v->shadow_vqs =g_ptr_array_new_full (dev->nvqs, vhost_psvq_free);QLIST_INSERT_HEAD (&vhost_vdpa_devices, v, entry); vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |@@ -373,6 +385,17 @@ err:return ; } +static void vhost_vdpa_svq_cleanup (struct vhost_dev *dev) +{ +struct vhost_vdpa *v = dev->opaque; +size_t idx; + +for (idx =0 ; idx < v->shadow_vqs->len; ++idx) { +vhost_svq_stop (dev, idx,g_ptr_array_index (v->shadow_vqs, idx)); + } +g_ptr_array_free (v->shadow_vqs, true); +} +static int vhost_vdpa_cleanup (struct vhost_dev *dev) {struct vhost_vdpa *v;@@ -381,6 +404,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev)trace_vhost_vdpa_cleanup (dev, v);vhost_vdpa_host_notifiers_uninit (dev, dev->nvqs);memory_listener_unregister (&v->listener); +vhost_vdpa_svq_cleanup (dev);QLIST_REMOVE (v, entry); dev->opaque =NULL ;hw/virtio/vhost-vdpa.c static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) · line 586
Quoted 7 lines
hw/virtio/vhost-vdpa.c +3
-1@@ -557,7 +581,9 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started)if (started) {uint8_t status =0 ;memory_listener_register (&v->listener, &address_space_memory); -vhost_vdpa_host_notifiers_init (dev); +if (!v->shadow_vqs_enabled) { +vhost_vdpa_host_notifiers_init (dev); + }This looks like a trick, why not check and setup
shadow_vqsinside:Quoted 6 lines
1)
vhost_vdpa_host_notifiers_init()and
2)
vhost_vdpa_set_vring_kick()Ok I will move the checks there.
Quoted 37 lines
Quoted 37 lines
hw/virtio/vhost-vdpa.c +29
-0vhost_vdpa_set_vring_ready (dev);vhost_vdpa_add_status (dev,VIRTIO_CONFIG_S_DRIVER_OK );vhost_vdpa_call (dev,VHOST_VDPA_GET_STATUS , &status);@@ -663,10 +689,96 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +/* +* Start shadow virtqueue. +*/ +static bool vhost_vdpa_svq_start_vq (struct vhost_dev *dev,unsigned idx) +{ +struct vhost_vdpa *v = dev->opaque; +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); +return vhost_svq_start (dev, idx, svq); +} + +static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) +{ +struct vhost_dev *hdev = v->dev; +unsigned n; + +if (enable == v->shadow_vqs_enabled) { +return hdev->nvqs; + } + +if (enable) { +/* Allocate resources */ +assert (v->shadow_vqs->len ==0 ); +for (n =0 ; n < hdev->nvqs; ++n) { +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +bool ok; + +if (unlikely (!svq)) { +g_ptr_array_set_size (v->shadow_vqs,0 ); +return 0 ;Quoted 8 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 728
Quoted 8 lines
hw/virtio/vhost-vdpa.c +8
-0+ } +g_ptr_array_add (v->shadow_vqs, svq); + + ok =vhost_vdpa_svq_start_vq (hdev, n); +if (unlikely (!ok)) { +/* Free still not started svqs */ +g_ptr_array_set_size (v->shadow_vqs, n); + enable = false;[2]
Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 731
Quoted 3 lines
hw/virtio/vhost-vdpa.c +3
-0+break ; + } + }Since there's almost no logic could be shared between enable and disable. Let's split those logic out into dedicated functions where the codes looks more easy to be reviewed (e.g have a better error handling etc).
Maybe it could be more clear in the code, but the reused logic is the disabling of SVQ and the fallback in case it cannot be enabled with [2]. But I'm not against splitting in two different functions if it makes review easier.
Quoted 21 lines
Quoted 21 lines
hw/virtio/vhost-vdpa.c +19
-0+ } + + v->shadow_vqs_enabled = enable; + +if (!enable) { +/* Disable all queues or clean up failed start */ +for (n =0 ; n < v->shadow_vqs->len; ++n) { +unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n); +VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n); +vhost_svq_stop (hdev, n, svq); +vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); + } + +/* Resources cleanup */ +g_ptr_array_set_size (v->shadow_vqs,0 ); + } + +return n; +}Quoted 16 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 762
Quoted 12 lines
hw/virtio/vhost-vdpa.c +9
-1void qmp_x_vhost_enable_shadow_vq (const char *name,bool enable,Error **errp) { -error_setg (errp,"Shadow virtqueue still not implemented" ); +struct vhost_vdpa *v; +const char *err_cause =NULL ; +bool r; + + QLIST_FOREACH(v, &vhost_vdpa_devices, entry) { +if (v->dev->vdev &&0 ==strcmp (v->dev->vdev->name, name)) { +break ; + } + }I think you can iterate the NetClientStates to ge tthe vhost-vdpa backends.
Quoted 2 lines
Right, I missed it.
Actually, that would always miss other device types like blk (isn't it?).
But using just the name is definitely a bad idea.
Quoted 52 lines
Quoted 11 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) · line 768
Quoted 7 lines
hw/virtio/vhost-vdpa.c +6
-0+ +if (!v) { + err_cause ="Device not found" ; + goto err; + }else if (v->notifier[0 ].addr) { + err_cause ="Device has host notifiers enabled" ;I don't get this.
At this moment of the series you can enable guest -> SVQ -> 'vdpa device' if the device is not using the host notifiers memory region. The right solution is to disable it for the guest, and to handle it in SVQ. Otherwise, guest kick will bypass SVQ and
It can be done in the same patch, or at least to disable (as unmap) them at this moment and handle them in a posterior patch. but for prototyping the solution I just ignored it in this series. It will be handled some way or another in the next one. I prefer the last one, to handle in a different patch, but let me know if you think it is better otherwise.
Quoted 3 lines
Btw this function should be implemented in an independent patch after svq is fully functional.
(Reasons for that are already commented at the top of this mail :) ).
Thanks!
Quoted 20 lines
Thanks
Quoted 16 lines
hw/virtio/vhost-vdpa.c +13
-0+ goto err; + } + + r =vhost_vdpa_enable_svq (v, enable); +if (unlikely (!r)) { + err_cause ="Error enabling (see monitor)" ; + goto err; + } + +err: +if (err_cause) { +error_setg (errp,"Can't enable shadow vq on %s: %s" , name, err_cause); + } }const VhostOps vdpa_ops = {
Re: [RFC PATCH v4 18/20] vhost: Add VhostIOVATree
In reply to Eugenio PérezQuoted 5 lines · Eugenio Pérez
This tree is able to look for a translated address from an IOVA address.
At first glance is similar to util/iova-tree. However, SVQ working on devices with limited IOVA space need more capabilities, like allocating IOVA chunks or perform reverse translations (qemu addresses to iova).
I don't see any reverse translation is used in the shadow code. Or anything I missed?
Quoted 8 lines
The allocation capability, as "assign a free IOVA address to this chunk of memory in qemu's address space" allows shadow virtqueue to create a new address space that is not restricted by guest's addressable one, so we can allocate shadow vqs vrings outside of its reachability, nor qemu's one. At the moment, the allocation is just done growing, not allowing deletion.
Quoted 8 lines
A different name could be used, but ordered searchable array is a little bit long though.
It duplicates the array so it can search efficiently both directions, and it will signal overlap if iova or the translated address is present in it's each array.
Use of array will be changed to util-iova-tree in future series.
Adding Peter.
It looks to me the only thing miseed is the iova allocator. And it looks to me it's better to decouple the allocator from the iova tree.
Then we had:
1) initialize iova range
2) iova = iova_alloc(size)
3) built the iova tree map
4) buffer forwarding
5) iova_free(size)
Quoted 86 lines
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-iova-tree.h | 40 +++++++ hw/virtio/vhost-iova-tree.c | 230 ++++++++++++++++++++++++++++++++++++ hw/virtio/meson.build | 2 +- 3 files changed, 271 insertions(+), 1 deletion(-)create mode 100644
hw/virtio/vhost-iova-tree.h
create mode 100644hw/virtio/vhost-iova-tree.chw/virtio/vhost-iova-tree.h new file +40
-0@@ -0,0 +1,40 @@ +/* +* vhost software live migration ring +* +* SPDX-FileCopyrightText: Red Hat, Inc. 2021 +* SPDX-FileContributor: Author: Eugenio Pérez <[email protected]> +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef HW_VIRTIO_VHOST_IOVA_TREE_H +#define HW_VIRTIO_VHOST_IOVA_TREE_H + +#include "exec/memory.h" + +typedef struct VhostDMAMap { +void *translated_addr; +hwaddr iova; +hwaddr size;/* Inclusive */ +IOMMUAccessFlags perm; +}VhostDMAMap ; + +typedef enum VhostDMAMapNewRC { +VHOST_DMA_MAP_NO_SPACE =-3 , +VHOST_DMA_MAP_OVERLAP =-2 , +VHOST_DMA_MAP_INVALID =-1 , +VHOST_DMA_MAP_OK =0 , +}VhostDMAMapNewRC ; + +typedef struct VhostIOVATree VhostIOVATree ; + +VhostIOVATree *vhost_iova_tree_new (void ); +void vhost_iova_tree_unref (VhostIOVATree *iova_rm); +G_DEFINE_AUTOPTR_CLEANUP_FUNC (VhostIOVATree, vhost_iova_tree_unref); + +const VhostDMAMap *vhost_iova_tree_find_iova (const VhostIOVATree *iova_rm, +const VhostDMAMap *map); +VhostDMAMapNewRC vhost_iova_tree_alloc (VhostIOVATree *iova_rm, +VhostDMAMap *map); + +#endif hw/virtio/vhost-iova-tree.c new file +24
-0@@ -0,0 +1,230 @@ +/* +* vhost software live migration ring +* +* SPDX-FileCopyrightText: Red Hat, Inc. 2021 +* SPDX-FileContributor: Author: Eugenio Pérez <[email protected]> +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "qemu/osdep.h" +#include "vhost-iova-tree.h" + +#define G_ARRAY_NOT_ZERO_TERMINATED false +#define G_ARRAY_NOT_CLEAR_ON_ALLOC false + +#define iova_min qemu_real_host_page_size + +/** +* VhostIOVATree, able to: +* - Translate iova address +* - Reverse translate iova address (from translated to iova) +* - Allocate IOVA regions for translated range (potentially slow operation) +* +* Note that it cannot remove nodes.
hw/virtio/vhost-iova-tree.c line 34
Quoted 10 lines
hw/virtio/vhost-iova-tree.c new file +10
-0+*/ +struct VhostIOVATree { +/* Ordered array of reverse translations, IOVA address to qemu memory. */ +GArray *iova_taddr_map; + +/* +* Ordered array of translations from qemu virtual memory address to iova +*/ +GArray *taddr_iova_map; +};
Any reason for using GArray? Is it faster?
Quoted 79 lines
hw/virtio/vhost-iova-tree.c new file +79
-0+ +/** +* Inserts an element after an existing one in garray. +* +* @array The array +* @prev_elem The previous element of array of NULL if prepending +* @map The DMA map +* +* It provides the aditional advantage of being type safe over +* g_array_insert_val, which accepts a reference pointer instead of a value +* with no complains. +*/ +static void vhost_iova_tree_insert_after (GArray *array, +const VhostDMAMap *prev_elem, +const VhostDMAMap *map) +{ +size_t pos; + +if (!prev_elem) { + pos =0 ; + }else { + pos = prev_elem - &g_array_index (array,typeof (*prev_elem),0 ) +1 ; + } + +g_array_insert_val (array, pos, *map); +} + +static gint vhost_iova_tree_cmp_taddr (gconstpointer a,gconstpointer b) +{ +const VhostDMAMap *m1 = a, *m2 = b; + +if (m1->translated_addr > m2->translated_addr + m2->size) { +return 1 ; + } + +if (m1->translated_addr + m1->size < m2->translated_addr) { +return -1 ; + } + +/* Overlapped */ +return 0 ; +} + +/** +* Find the previous node to a given iova +* +* @array The ascending ordered-by-translated-addr array of VhostDMAMap +* @map The map to insert +* @prev Returned location of the previous map +* +* Return VHOST_DMA_MAP_OK if everything went well, or VHOST_DMA_MAP_OVERLAP if +* it already exists. It is ok to use this function to check if a given range +* exists, but it will use a linear search. +* +* TODO: We can use bsearch to locate the entry if we save the state in the +* needle, knowing that the needle is always the first argument to +* compare_func. +*/ +static VhostDMAMapNewRC vhost_iova_tree_find_prev (const GArray *array, +GCompareFunc compare_func, +const VhostDMAMap *map, +const VhostDMAMap **prev) +{ +size_t i; +int r; + + *prev =NULL ; +for (i =0 ; i < array->len; ++i) { + r =compare_func (map, &g_array_index (array,typeof (*map), i)); +if (r ==0 ) { +return VHOST_DMA_MAP_OVERLAP ; + } +if (r <0 ) { +return VHOST_DMA_MAP_OK ; + } + + *prev = &g_array_index (array,typeof (**prev), i); + } +
hw/virtio/vhost-iova-tree.c line 123
Quoted 10 lines
hw/virtio/vhost-iova-tree.c new file +10
-0+return VHOST_DMA_MAP_OK ; +} + +/** +* Create a new IOVA tree +* +* Returns the new IOVA tree +*/ +VhostIOVATree *vhost_iova_tree_new (void ) +{
So I think it needs to be initialized with the range we get from get_iova_range().
Thanks
Quoted 112 lines
hw/virtio/vhost-iova-tree.c new file +107
-0+VhostIOVATree *tree =g_new (VhostIOVATree,1 ); + tree->iova_taddr_map =g_array_new (G_ARRAY_NOT_ZERO_TERMINATED , +G_ARRAY_NOT_CLEAR_ON_ALLOC , +sizeof (VhostDMAMap)); + tree->taddr_iova_map =g_array_new (G_ARRAY_NOT_ZERO_TERMINATED , +G_ARRAY_NOT_CLEAR_ON_ALLOC , +sizeof (VhostDMAMap)); +return tree; +} + +/** +* Destroy an IOVA tree +* +* @tree The iova tree +*/ +void vhost_iova_tree_unref (VhostIOVATree *tree) +{ +g_array_unref (g_steal_pointer (&tree->iova_taddr_map)); +g_array_unref (g_steal_pointer (&tree->taddr_iova_map)); +} + +/** +* Find the IOVA address stored from a memory address +* +* @tree The iova tree +* @map The map with the memory address +* +* Return the stored mapping, or NULL if not found. +*/ +const VhostDMAMap *vhost_iova_tree_find_iova (const VhostIOVATree *tree, +const VhostDMAMap *map) +{ +/* +* This can be replaced with g_array_binary_search (Since glib 2.62) when +* that version become common enough. +*/ +return bsearch (map, tree->taddr_iova_map->data, tree->taddr_iova_map->len, +sizeof (*map), vhost_iova_tree_cmp_taddr); +} + +static bool vhost_iova_tree_find_iova_hole (const GArray *iova_map, +const VhostDMAMap *map, +const VhostDMAMap **prev_elem) +{ +size_t i; +hwaddr iova = iova_min; + + *prev_elem =NULL ; +for (i =0 ; i < iova_map->len; i++) { +const VhostDMAMap *next = &g_array_index (iova_map,typeof (*next), i); +hwaddr hole_end = next->iova; +if (map->size < hole_end - iova) { +return true; + } + + iova = next->iova + next->size +1 ; + *prev_elem = next; + } + +return ((hwaddr)-1 - iova) > iova_map->len; +} + +/** +* Allocate a new mapping +* +* @tree The iova tree +* @map The iova map +* +* Returns: +* - VHOST_DMA_MAP_OK if the map fits in the container +* - VHOST_DMA_MAP_INVALID if the map does not make sense (like size overflow) +* - VHOST_DMA_MAP_OVERLAP if the tree already contains that map +* - VHOST_DMA_MAP_NO_SPACE if iova_rm cannot allocate more space. +* +* It returns assignated iova in map->iova if return value is VHOST_DMA_MAP_OK. +*/ +VhostDMAMapNewRC vhost_iova_tree_alloc (VhostIOVATree *tree, +VhostDMAMap *map) +{ +const VhostDMAMap *qemu_prev, *iova_prev; +int find_prev_rc; +bool fit; + +if (map->translated_addr + map->size < map->translated_addr || + map->iova + map->size < map->iova || map->perm ==IOMMU_NONE ) { +return VHOST_DMA_MAP_INVALID ; + } + +/* Search for a hole in iova space big enough */ + fit =vhost_iova_tree_find_iova_hole (tree->iova_taddr_map, map, + &iova_prev); +if (!fit) { +return VHOST_DMA_MAP_NO_SPACE ; + } + + map->iova = iova_prev ? (iova_prev->iova + iova_prev->size) +1 : iova_min; + find_prev_rc =vhost_iova_tree_find_prev (tree->taddr_iova_map, + vhost_iova_tree_cmp_taddr, map, + &qemu_prev); +if (find_prev_rc ==VHOST_DMA_MAP_OVERLAP ) { +return VHOST_DMA_MAP_OVERLAP ; + } + +vhost_iova_tree_insert_after (tree->iova_taddr_map, iova_prev, map); +vhost_iova_tree_insert_after (tree->taddr_iova_map, qemu_prev, map); +return VHOST_DMA_MAP_OK ; +}hw/virtio/meson.build +0
-0
hw/virtio/meson.build softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('vhost-stub.c')) · line 17
Quoted 8 lines
hw/virtio/meson.build +1
-1@@ -11,7 +11,7 @@ softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('vhost-stub.c')) virtio_ss = ss.source_set() virtio_ss.add(files('virtio.c')) -virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c', 'vhost-shadow-virtqueue.c')) +virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c', 'vhost-shadow-virtqueue.c', 'vhost-iova-tree.c')) virtio_ss.add(when: 'CONFIG_VHOST_USER', if_true: files('vhost-user.c')) virtio_ss.add(when: 'CONFIG_VHOST_VDPA', if_true: files('vhost-vdpa.c')) virtio_ss.add(when: 'CONFIG_VIRTIO_BALLOON', if_true: files('virtio-balloon.c'))
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue
In reply to Jason WangQuoted 129 lines · Jason Wang
Quoted 106 lines · Eugenio Perez Martin
Quoted 78 lines · Jason Wang
Quoted 76 lines · Eugenio Pérez
This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of
vhost_virtqueue_startcan unmasks and discard call events,vhost_virtqueue_startshould be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.c | 19 +++++++++++++++ hw/virtio/vhost-vdpa.c | 38 +++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +19
-0@@ -53,6 +53,22 @@ static void vhost_handle_guest_kick(EventNotifier *n)event_notifier_set (&svq->kick_notifier); } +/* Forward vhost notifications */ +static void vhost_svq_handle_call_no_test (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + call_notifier); + +event_notifier_set (&svq->guest_call_notifier); +} + +static void vhost_svq_handle_call (EventNotifier *n) +{ +if (likely (event_notifier_test_and_clear (n))) { +vhost_svq_handle_call_no_test (n); + } +} +/* * Obtain the SVQ call notifier, where vhost device notifies SVQ that there * exists pending used buffers. @@ -180,6 +196,8 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) } svq->vq =virtio_get_queue (dev->vdev, vq_idx); +event_notifier_set_handler (&svq->call_notifier, + vhost_svq_handle_call);return g_steal_pointer (&svq); err_init_call_notifier:@@ -195,6 +213,7 @@ err_init_kick_notifier:void vhost_svq_free (VhostShadowVirtqueue *vq) {event_notifier_cleanup (&vq->kick_notifier); +event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier);g_free (vq); }hw/virtio/vhost-vdpa.c +4
-1@@ -712,13 +712,40 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) {struct vhost_vdpa *v = dev->opaque;VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); -return vhost_svq_start (dev, idx, svq); +EventNotifier *vhost_call_notifier =vhost_svq_get_svq_call_notifier (svq); +struct vhost_vring_file vhost_call_file = { + .index = idx + dev->vq_index, + .fd =event_notifier_get_fd (vhost_call_notifier),Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 724
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+ }; +int r; +bool b; + +/* Set shadow vq -> guest notifier */ +assert (v->call_fd[idx]);We need aovid the
asser()here. On which case we can hit this?I would say that there is no way we can actually hit it, so let's remove it.
Quoted 7 lines
Quoted 7 lines
hw/virtio/vhost-vdpa.c +7
-0+vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]); + + b =vhost_svq_start (dev, idx, svq); +if (unlikely (!b)) { +return false; + } +Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 737
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+/* Set device -> SVQ notifier */ + r =vhost_vdpa_set_vring_dev_call (dev, &vhost_call_file); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_call for shadow vq failed" ); +return false; + }Similar to kick, do we need to
set_vring_call()beforevhost_svq_start()?Quoted 3 lines
It should not matter at this moment because the device should not be started at this point and device calls should not run
vhost_svq_handle_calluntil BQL is released.Yes, we stop virtqueue before.
Quoted 12 lines
The "logic" of doing it after is to make clear that svq must be fully initialized before processing device calls, even in the case that we extract SVQ in its own iothread or similar. But this could be done before
vhost_svq_startfor sure.Quoted 6 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 740
Quoted 3 lines
hw/virtio/vhost-vdpa.c +3
-0+ +/* Check for pending calls */ +event_notifier_set (vhost_call_notifier);Interesting, can this result spurious interrupt?
Quoted 8 lines
Quoted 4 lines
This actually "queues" a
vhost_svq_handle_callafter the BQL release, where the device should be fully reset. In that regard, if there are no used descriptors there will not be an irq raised to the guest. Does that answer the question? Or have I missed something?Yes, please explain this in the comment.
I'm reviewing this again, and actually I think I was wrong in solving the issue.
Since at this point the device is being configured, there is no chance that we had a missing call notification here: A previous kick is needed for the device to generate any calls, and these cannot be processed.
What is not solved in this series is that we could have pending used buffers in vdpa device stopping SVQ, but queuing a check for that is not going to solve anything, since SVQ vring would be already destroyed:
* vdpa device marks N > 0 buffers as used, and calls.
* Before processing them, SVQ stop is called. SVQ have not processed these, and cleans them, making this event_notifier_set useless.
So this would require a few changes. Mainly, instead of queueing a check for used, these need to be checked before svq cleaning. After that, obtain the VQ state (is not obtained in the stop at the moment, trusting in guest's used idx) and run a last vhost_svq_handle_call_no_test while the device is paused.
Thanks!
Quoted 52 lines
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
hw/virtio/vhost-vdpa.c +9
-0+return true; }static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev;unsigned n; +int r;if (enable == v->shadow_vqs_enabled) {return hdev->nvqs;@@ -752,9 +779,18 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)if (!enable) {/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { +struct vhost_vring_file file = { + .index =vhost_vdpa_get_vq_index (hdev, n), + .fd = v->call_fd[n], + }; + + r =vhost_vdpa_set_vring_call (hdev, &file); +assert (r ==0 );Quoted 9 lines
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 793
Quoted 5 lines
hw/virtio/vhost-vdpa.c +2
-0+unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n);VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n);vhost_svq_stop (hdev, n, svq); +/* TODO: This can unmask or override call fd! */ I don't get this comment. Does this mean the current code can't work with
mask_notifiers? If yes, this is something we need to fix.Quoted 2 lines
Yes, but it will be addressed in the next series. I should have explained it bette here, sorry :).
Ok.
Thanks
Quoted 9 lines
Thanks!
Quoted 6 lines
Thanks
Quoted 3 lines
hw/virtio/vhost-vdpa.c +0
-0vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); }
Re: [RFC PATCH v4 18/20] vhost: Add VhostIOVATree
In reply to Jason WangQuoted 5 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
This tree is able to look for a translated address from an IOVA address.
Quoted 7 lines
Quoted 3 lines
At first glance is similar to util/iova-tree. However, SVQ working on devices with limited IOVA space need more capabilities, like allocating IOVA chunks or perform reverse translations (qemu addresses to iova).
I don't see any reverse translation is used in the shadow code. Or anything I missed?
Ok, it looks to me that it is used in the iova allocator. But I think it's better to decouple it to an independent allocator instead of vhost iova tree.
Thanks
Re: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
In reply to Eugenio PérezQuoted 129 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)hw/virtio/vhost-shadow-virtqueue.h +3
-1@@ -11,6 +11,7 @@#define VHOST_SHADOW_VIRTQUEUE_H #include "hw/virtio/vhost.h" +#include "hw/virtio/vhost-iova-tree.h" typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ;@@ -28,7 +29,8 @@ bool vhost_svq_start(struct vhost_dev *dev, unsigned idx,void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq); -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map);void vhost_svq_free (VhostShadowVirtqueue *vq);hw/virtio/vhost-shadow-virtqueue.c +44
-8@@ -11,12 +11,19 @@#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" #include "hw/virtio/virtio-access.h" +#include "hw/virtio/vhost-iova-tree.h" #include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" #include "qemu/main-loop.h" +typedef struct SVQElement { +VirtQueueElement elem; +void **in_sg_stash; +void **out_sg_stash; +}SVQElement ; +/* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { /* Shadow vring */@@ -46,8 +53,11 @@ typedef struct VhostShadowVirtqueue {/* Virtio device */ VirtIODevice *vdev; +/* IOVA mapping if used */ +VhostIOVATree *iova_map; +/* Map for returning guest's descriptors */ -VirtQueueElement **ring_id_maps; +SVQElement **ring_id_maps;/* Next head to expose to device */ uint16_t avail_idx_shadow;@@ -79,13 +89,6 @@ bool vhost_svq_valid_device_features(uint64_t *dev_features)continue ;case VIRTIO_F_ACCESS_PLATFORM : -/* SVQ needs this feature disabled. Can't continue */ -if (*dev_features &BIT_ULL (b)) { -clear_bit (b, dev_features); - r = false; - } -break ; -case VIRTIO_F_VERSION_1 :/* SVQ needs this feature, so can't continue */ if (!(*dev_features &BIT_ULL (b))) {@@ -126,6 +129,64 @@ static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) } } +static void vhost_svq_stash_addr (void ***stash,const struct iovec *iov, +size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + + *stash = g_new(void *, num); +for (i =0 ; i < num; ++i) { + (*stash)[i] = iov[i].iov_base; + } +} + +static void vhost_svq_unstash_addr (void **stash,struct iovec *iov,size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + +for (i =0 ; i < num; ++i) { + iov[i].iov_base = stash[i]; + } +g_free (stash); +} + +static void vhost_svq_translate_addr (const VhostShadowVirtqueue *svq, +struct iovec *iovec,size_t num) +{ +size_t i;
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) · line 174
Quoted 10 lines
hw/virtio/vhost-shadow-virtqueue.c +10
-0+ +for (i =0 ; i < num; ++i) { +VhostDMAMap needle = { + .translated_addr = iovec[i].iov_base, + .size = iovec[i].iov_len, + }; +size_t off; + +const VhostDMAMap *map =vhost_iova_tree_find_iova (svq->iova_map, + &needle);
Is it possible that we end up with more than one maps here?
Quoted 30 lines
hw/virtio/vhost-shadow-virtqueue.c +17
-1+/* +* Map cannot be NULL since iova map contains all guest space and +* qemu already has a physical address mapped +*/ +assert (map); + +/* +* Map->iova chunk size is ignored. What to do if descriptor +* (addr, size) does not fit is delegated to the device. +*/ + off = needle.translated_addr - map->translated_addr; + iovec[i].iov_base = (void *)(map->iova + off); + } +} +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq,const struct iovec *iovec,size_t num,bool more_descs,bool write)@@ -156,8 +217,9 @@ static void vhost_vring_write_descs(VhostShadowVirtqueue *svq, }static unsigned vhost_svq_add_split (VhostShadowVirtqueue *svq, -VirtQueueElement *elem) +SVQElement *svq_elem) { +VirtQueueElement *elem = &svq_elem->elem;int head;unsigned avail_idx;vring_avail_t *avail = svq->vring.avail;
hw/virtio/vhost-shadow-virtqueue.c static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, · line 233
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +2
-0@@ -167,6 +229,12 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq,/* We need some descriptors here */ assert (elem->out_num || elem->in_num); +vhost_svq_stash_addr (&svq_elem->in_sg_stash, elem->in_sg, elem->in_num); +vhost_svq_stash_addr (&svq_elem->out_sg_stash, elem->out_sg, elem->out_num);
I wonder if we can solve the trick like stash and unstash with a dedicated sgs in svq_elem, instead of reusing the elem.
Thanks
Quoted 210 lines
hw/virtio/vhost-shadow-virtqueue.c +36
-12+ +vhost_svq_translate_addr (svq, elem->in_sg, elem->in_num); +vhost_svq_translate_addr (svq, elem->out_sg, elem->out_num); +vhost_vring_write_descs (svq, elem->out_sg, elem->out_num, elem->in_num >0 , false);vhost_vring_write_descs (svq, elem->in_sg, elem->in_num, false, true);@@ -187,7 +255,7 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, } -static void vhost_svq_add (VhostShadowVirtqueue *svq,VirtQueueElement *elem) +static void vhost_svq_add (VhostShadowVirtqueue *svq,SVQElement *elem) {unsigned qemu_head =vhost_svq_add_split (svq, elem);@@ -221,7 +289,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) }while (true) { -VirtQueueElement *elem =virtqueue_pop (svq->vq,sizeof (*elem)); +SVQElement *elem =virtqueue_pop (svq->vq,sizeof (*elem));if (!elem) {break ; }@@ -247,7 +315,7 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)return svq->used_idx != svq->shadow_used_idx; } -static VirtQueueElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) +static SVQElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) {vring_desc_t *descs = svq->vring.desc;const vring_used_t *used = svq->vring.used;@@ -279,7 +347,7 @@ static VirtQueueElement *vhost_svq_get_buf(VhostShadowVirtqueue *svq) descs[used_elem.id].next = svq->free_head; svq->free_head = used_elem.id; - svq->ring_id_maps[used_elem.id]->len = used_elem.len; + svq->ring_id_maps[used_elem.id]->elem.len = used_elem.len;return g_steal_pointer (&svq->ring_id_maps[used_elem.id]); }@@ -296,12 +364,19 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n)vhost_svq_set_notification (svq, false);while (true) { - g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); -if (!elem) { + g_autofree SVQElement *svq_elem = vhost_svq_get_buf(svq); +VirtQueueElement *elem; +if (!svq_elem) {break ; }assert (i < svq->vring.num); + elem = &svq_elem->elem; + +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num);virtqueue_fill (vq, elem, elem->len, i++); }@@ -451,14 +526,24 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,event_notifier_set_handler (&svq->host_notifier,NULL );for (i =0 ; i < svq->vring.num; ++i) { - g_autofree VirtQueueElement *elem = svq->ring_id_maps[i]; + g_autofree SVQElement *svq_elem = svq->ring_id_maps[i]; +VirtQueueElement *elem; + +if (!svq_elem) { +continue ; + } + + elem = &svq_elem->elem; +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num); +/* * Although the doc says we must unpop in order, it's ok to unpop * everything. */ -if (elem) { -virtqueue_unpop (svq->vq, elem, elem->len); - } +virtqueue_unpop (svq->vq, elem, elem->len); } }@@ -466,7 +551,8 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,* Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map) {int vq_idx = dev->vq_index + idx;unsigned num =virtio_queue_get_num (dev->vdev, vq_idx);@@ -500,11 +586,13 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)memset (svq->vring.desc,0 , driver_size); svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size);memset (svq->vring.used,0 , device_size); + svq->iova_map = iova_map; +for (i =0 ; i < num -1 ; i++) { svq->vring.desc[i].next =cpu_to_le16 (i +1 ); } - svq->ring_id_maps = g_new0(VirtQueueElement *, num); + svq->ring_id_maps = g_new0(SVQElement *, num);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);hw/virtio/vhost-vdpa.c +39
-1@@ -176,6 +176,18 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); +if (v->shadow_vqs_enabled) { +VhostDMAMap mem_region = { + .translated_addr = vaddr, + .size =int128_get64 (llsize) -1 , + .perm =IOMMU_ACCESS_FLAG (true, section->readonly), + }; + +int r =vhost_iova_tree_alloc (v->iova_map, &mem_region); +assert (r ==VHOST_DMA_MAP_OK ); + + iova = mem_region.iova; + } ret =vhost_vdpa_dma_map (v, iova,int128_get64 (llsize), vaddr, section->readonly);@@ -754,6 +766,23 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +static int vhost_vdpa_get_iova_range (struct vhost_dev *dev, +hwaddr *first,hwaddr *last) +{ +int ret; +struct vhost_vdpa_iova_range range; + + ret =vhost_vdpa_call (dev,VHOST_VDPA_GET_IOVA_RANGE , &range); +if (ret !=0 ) { +return ret; + } + + *first = range.first; + *last = range.last; +trace_vhost_vdpa_get_iova_range (dev, *first, *last); +return ret; +} +/** * Maps QEMU vaddr memory to device in a suitable way for shadow virtqueue: * - It always reference qemu memory address, not guest's memory. @@ -881,6 +910,7 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx)static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev; +hwaddr iova_first, iova_last;unsigned n;int r;@@ -894,7 +924,7 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* Allocate resources */ assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) { -VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n, v->iova_map);if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ;@@ -903,6 +933,8 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } + r =vhost_vdpa_get_iova_range (hdev, &iova_first, &iova_last); +assert (r ==0 ); r =vhost_vdpa_vring_pause (hdev);assert (r ==0 );@@ -913,6 +945,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } +memory_listener_unregister (&v->listener); +if (vhost_vdpa_dma_unmap (v, iova_first, + (iova_last - iova_first) &TARGET_PAGE_MASK )) { +error_report ("Fail to invalidate device iotlb" ); + } +/* Reset device so it can be configured */ r =vhost_vdpa_dev_start (hdev, false);assert (r ==0 );hw/virtio/trace-events +0
-0
hw/virtio/trace-events vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: · line 58
Quoted 7 lines
hw/virtio/trace-events +1
-0@@ -52,6 +52,7 @@ vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 0x%"PRIx64 vhost_vdpa_set_owner(void *dev) "dev: %p" vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64 +vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p first: 0x%"PRIx64" last: 0x%"PRIx64 # virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned out_num) "elem %p size %zd in_num %u out_num %u"
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
Quoted 137 lines · Jason Wang
Quoted 134 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)hw/virtio/vhost-shadow-virtqueue.h +3
-1@@ -11,6 +11,7 @@#define VHOST_SHADOW_VIRTQUEUE_H #include "hw/virtio/vhost.h" +#include "hw/virtio/vhost-iova-tree.h" typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ;@@ -28,7 +29,8 @@ bool vhost_svq_start(struct vhost_dev *dev, unsigned idx,void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq); -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map);void vhost_svq_free (VhostShadowVirtqueue *vq);hw/virtio/vhost-shadow-virtqueue.c +49
-8@@ -11,12 +11,19 @@#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" #include "hw/virtio/virtio-access.h" +#include "hw/virtio/vhost-iova-tree.h" #include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" #include "qemu/main-loop.h" +typedef struct SVQElement { +VirtQueueElement elem; +void **in_sg_stash; +void **out_sg_stash; +}SVQElement ; +/* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { /* Shadow vring */@@ -46,8 +53,11 @@ typedef struct VhostShadowVirtqueue {/* Virtio device */ VirtIODevice *vdev; +/* IOVA mapping if used */ +VhostIOVATree *iova_map; +/* Map for returning guest's descriptors */ -VirtQueueElement **ring_id_maps; +SVQElement **ring_id_maps;/* Next head to expose to device */ uint16_t avail_idx_shadow;@@ -79,13 +89,6 @@ bool vhost_svq_valid_device_features(uint64_t *dev_features)continue ;case VIRTIO_F_ACCESS_PLATFORM : -/* SVQ needs this feature disabled. Can't continue */ -if (*dev_features &BIT_ULL (b)) { -clear_bit (b, dev_features); - r = false; - } -break ; -case VIRTIO_F_VERSION_1 :/* SVQ needs this feature, so can't continue */ if (!(*dev_features &BIT_ULL (b))) {@@ -126,6 +129,64 @@ static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) } } +static void vhost_svq_stash_addr (void ***stash,const struct iovec *iov, +size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + + *stash = g_new(void *, num); +for (i =0 ; i < num; ++i) { + (*stash)[i] = iov[i].iov_base; + } +} + +static void vhost_svq_unstash_addr (void **stash,struct iovec *iov,size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + +for (i =0 ; i < num; ++i) { + iov[i].iov_base = stash[i]; + } +g_free (stash); +} + +static void vhost_svq_translate_addr (const VhostShadowVirtqueue *svq, +struct iovec *iovec,size_t num) +{ +size_t i; + +for (i =0 ; i < num; ++i) { +VhostDMAMap needle = { + .translated_addr = iovec[i].iov_base, + .size = iovec[i].iov_len,
Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) · line 174
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+ }; +size_t off; + +const VhostDMAMap *map =vhost_iova_tree_find_iova (svq->iova_map, + &needle);Is it possible that we end up with more than one maps here?
Actually it is possible, since there is no guarantee that one descriptor (or indirect descriptor) maps exactly to one iov. It could map to many if qemu vaddr is not contiguous but GPA + size is. This is something that must be fixed for the next revision, so thanks for pointing it out!
Taking that into account, the condition that svq vring avail_idx - used_idx was always less or equal than guest's vring avail_idx - used_idx is not true anymore. Checking for that before adding buffers to SVQ is the easy part, but how could we recover in that case?
I think that the easy solution is to check for more available buffers unconditionally at the end of vhost_svq_handle_call, which handles the SVQ used and is supposed to make more room for available buffers. So vhost_handle_guest_kick would not check if eventfd is set or not anymore.
Would that make sense?
Thanks!
Quoted 34 lines
Quoted 34 lines
hw/virtio/vhost-shadow-virtqueue.c +17
-1+/* +* Map cannot be NULL since iova map contains all guest space and +* qemu already has a physical address mapped +*/ +assert (map); + +/* +* Map->iova chunk size is ignored. What to do if descriptor +* (addr, size) does not fit is delegated to the device. +*/ + off = needle.translated_addr - map->translated_addr; + iovec[i].iov_base = (void *)(map->iova + off); + } +} +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq,const struct iovec *iovec,size_t num,bool more_descs,bool write)@@ -156,8 +217,9 @@ static void vhost_vring_write_descs(VhostShadowVirtqueue *svq, }static unsigned vhost_svq_add_split (VhostShadowVirtqueue *svq, -VirtQueueElement *elem) +SVQElement *svq_elem) { +VirtQueueElement *elem = &svq_elem->elem;int head;unsigned avail_idx;vring_avail_t *avail = svq->vring.avail;@@ -167,6 +229,12 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq,/* We need some descriptors here */ assert (elem->out_num || elem->in_num);
Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.c static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, · line 233
Quoted 2 lines
hw/virtio/vhost-shadow-virtqueue.c +2
-0+vhost_svq_stash_addr (&svq_elem->in_sg_stash, elem->in_sg, elem->in_num); +vhost_svq_stash_addr (&svq_elem->out_sg_stash, elem->out_sg, elem->out_num);I wonder if we can solve the trick like stash and unstash with a dedicated sgs in
svq_elem, instead of reusing the elem.
Actually yes, it would be way simpler to use a new sgs array in svq_elem. I will change that.
Thanks!
Quoted 221 lines
Thanks
Quoted 217 lines
hw/virtio/vhost-shadow-virtqueue.c +36
-12+ +vhost_svq_translate_addr (svq, elem->in_sg, elem->in_num); +vhost_svq_translate_addr (svq, elem->out_sg, elem->out_num); +vhost_vring_write_descs (svq, elem->out_sg, elem->out_num, elem->in_num >0 , false);vhost_vring_write_descs (svq, elem->in_sg, elem->in_num, false, true);@@ -187,7 +255,7 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, } -static void vhost_svq_add (VhostShadowVirtqueue *svq,VirtQueueElement *elem) +static void vhost_svq_add (VhostShadowVirtqueue *svq,SVQElement *elem) {unsigned qemu_head =vhost_svq_add_split (svq, elem);@@ -221,7 +289,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) }while (true) { -VirtQueueElement *elem =virtqueue_pop (svq->vq,sizeof (*elem)); +SVQElement *elem =virtqueue_pop (svq->vq,sizeof (*elem));if (!elem) {break ; }@@ -247,7 +315,7 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)return svq->used_idx != svq->shadow_used_idx; } -static VirtQueueElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) +static SVQElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) {vring_desc_t *descs = svq->vring.desc;const vring_used_t *used = svq->vring.used;@@ -279,7 +347,7 @@ static VirtQueueElement *vhost_svq_get_buf(VhostShadowVirtqueue *svq) descs[used_elem.id].next = svq->free_head; svq->free_head = used_elem.id; - svq->ring_id_maps[used_elem.id]->len = used_elem.len; + svq->ring_id_maps[used_elem.id]->elem.len = used_elem.len;return g_steal_pointer (&svq->ring_id_maps[used_elem.id]); }@@ -296,12 +364,19 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n)vhost_svq_set_notification (svq, false);while (true) { - g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); -if (!elem) { + g_autofree SVQElement *svq_elem = vhost_svq_get_buf(svq); +VirtQueueElement *elem; +if (!svq_elem) {break ; }assert (i < svq->vring.num); + elem = &svq_elem->elem; + +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num);virtqueue_fill (vq, elem, elem->len, i++); }@@ -451,14 +526,24 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,event_notifier_set_handler (&svq->host_notifier,NULL );for (i =0 ; i < svq->vring.num; ++i) { - g_autofree VirtQueueElement *elem = svq->ring_id_maps[i]; + g_autofree SVQElement *svq_elem = svq->ring_id_maps[i]; +VirtQueueElement *elem; + +if (!svq_elem) { +continue ; + } + + elem = &svq_elem->elem; +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num); +/* * Although the doc says we must unpop in order, it's ok to unpop * everything. */ -if (elem) { -virtqueue_unpop (svq->vq, elem, elem->len); - } +virtqueue_unpop (svq->vq, elem, elem->len); } }@@ -466,7 +551,8 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,* Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map) {int vq_idx = dev->vq_index + idx;unsigned num =virtio_queue_get_num (dev->vdev, vq_idx);@@ -500,11 +586,13 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)memset (svq->vring.desc,0 , driver_size); svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size);memset (svq->vring.used,0 , device_size); + svq->iova_map = iova_map; +for (i =0 ; i < num -1 ; i++) { svq->vring.desc[i].next =cpu_to_le16 (i +1 ); } - svq->ring_id_maps = g_new0(VirtQueueElement *, num); + svq->ring_id_maps = g_new0(SVQElement *, num);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);hw/virtio/vhost-vdpa.c +39
-1@@ -176,6 +176,18 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); +if (v->shadow_vqs_enabled) { +VhostDMAMap mem_region = { + .translated_addr = vaddr, + .size =int128_get64 (llsize) -1 , + .perm =IOMMU_ACCESS_FLAG (true, section->readonly), + }; + +int r =vhost_iova_tree_alloc (v->iova_map, &mem_region); +assert (r ==VHOST_DMA_MAP_OK ); + + iova = mem_region.iova; + } ret =vhost_vdpa_dma_map (v, iova,int128_get64 (llsize), vaddr, section->readonly);@@ -754,6 +766,23 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +static int vhost_vdpa_get_iova_range (struct vhost_dev *dev, +hwaddr *first,hwaddr *last) +{ +int ret; +struct vhost_vdpa_iova_range range; + + ret =vhost_vdpa_call (dev,VHOST_VDPA_GET_IOVA_RANGE , &range); +if (ret !=0 ) { +return ret; + } + + *first = range.first; + *last = range.last; +trace_vhost_vdpa_get_iova_range (dev, *first, *last); +return ret; +} +/** * Maps QEMU vaddr memory to device in a suitable way for shadow virtqueue: * - It always reference qemu memory address, not guest's memory. @@ -881,6 +910,7 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx)static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev; +hwaddr iova_first, iova_last;unsigned n;int r;@@ -894,7 +924,7 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* Allocate resources */ assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) { -VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n, v->iova_map);if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ;@@ -903,6 +933,8 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } + r =vhost_vdpa_get_iova_range (hdev, &iova_first, &iova_last); +assert (r ==0 ); r =vhost_vdpa_vring_pause (hdev);assert (r ==0 );@@ -913,6 +945,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } +memory_listener_unregister (&v->listener); +if (vhost_vdpa_dma_unmap (v, iova_first, + (iova_last - iova_first) &TARGET_PAGE_MASK )) { +error_report ("Fail to invalidate device iotlb" ); + } +/* Reset device so it can be configured */ r =vhost_vdpa_dev_start (hdev, false);assert (r ==0 );hw/virtio/trace-events +1
-0@@ -52,6 +52,7 @@ vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 0x%"PRIx64 vhost_vdpa_set_owner(void *dev) "dev: %p" vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64 +vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p first: 0x%"PRIx64" last: 0x%"PRIx64 # virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned out_num) "elem %p size %zd in_num %u out_num %u"
Re: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue
In reply to Eugenio Perez MartinQuoted 156 lines · Eugenio Perez Martin
Quoted 129 lines · Jason Wang
Quoted 106 lines · Eugenio Perez Martin
Quoted 78 lines · Jason Wang
Quoted 76 lines · Eugenio Pérez
This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of
vhost_virtqueue_startcan unmasks and discard call events,vhost_virtqueue_startshould be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.c | 19 +++++++++++++++ hw/virtio/vhost-vdpa.c | 38 +++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +19
-0@@ -53,6 +53,22 @@ static void vhost_handle_guest_kick(EventNotifier *n)event_notifier_set (&svq->kick_notifier); } +/* Forward vhost notifications */ +static void vhost_svq_handle_call_no_test (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + call_notifier); + +event_notifier_set (&svq->guest_call_notifier); +} + +static void vhost_svq_handle_call (EventNotifier *n) +{ +if (likely (event_notifier_test_and_clear (n))) { +vhost_svq_handle_call_no_test (n); + } +} +/* * Obtain the SVQ call notifier, where vhost device notifies SVQ that there * exists pending used buffers. @@ -180,6 +196,8 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) } svq->vq =virtio_get_queue (dev->vdev, vq_idx); +event_notifier_set_handler (&svq->call_notifier, + vhost_svq_handle_call);return g_steal_pointer (&svq); err_init_call_notifier:@@ -195,6 +213,7 @@ err_init_kick_notifier:void vhost_svq_free (VhostShadowVirtqueue *vq) {event_notifier_cleanup (&vq->kick_notifier); +event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier);g_free (vq); }hw/virtio/vhost-vdpa.c +4
-1@@ -712,13 +712,40 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) {struct vhost_vdpa *v = dev->opaque;VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); -return vhost_svq_start (dev, idx, svq); +EventNotifier *vhost_call_notifier =vhost_svq_get_svq_call_notifier (svq); +struct vhost_vring_file vhost_call_file = { + .index = idx + dev->vq_index, + .fd =event_notifier_get_fd (vhost_call_notifier),Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 724
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+ }; +int r; +bool b; + +/* Set shadow vq -> guest notifier */ +assert (v->call_fd[idx]);We need aovid the
asser()here. On which case we can hit this?I would say that there is no way we can actually hit it, so let's remove it.
Quoted 7 lines
Quoted 7 lines
hw/virtio/vhost-vdpa.c +7
-0+vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]); + + b =vhost_svq_start (dev, idx, svq); +if (unlikely (!b)) { +return false; + } +Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 737
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+/* Set device -> SVQ notifier */ + r =vhost_vdpa_set_vring_dev_call (dev, &vhost_call_file); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_call for shadow vq failed" ); +return false; + }Similar to kick, do we need to
set_vring_call()beforevhost_svq_start()?Quoted 3 lines
It should not matter at this moment because the device should not be started at this point and device calls should not run
vhost_svq_handle_calluntil BQL is released.Yes, we stop virtqueue before.
Quoted 12 lines
The "logic" of doing it after is to make clear that svq must be fully initialized before processing device calls, even in the case that we extract SVQ in its own iothread or similar. But this could be done before
vhost_svq_startfor sure.Quoted 6 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 740
Quoted 3 lines
hw/virtio/vhost-vdpa.c +3
-0+ +/* Check for pending calls */ +event_notifier_set (vhost_call_notifier);Interesting, can this result spurious interrupt?
Quoted 8 lines
Quoted 4 lines
This actually "queues" a
vhost_svq_handle_callafter the BQL release, where the device should be fully reset. In that regard, if there are no used descriptors there will not be an irq raised to the guest. Does that answer the question? Or have I missed something?Yes, please explain this in the comment.
I'm reviewing this again, and actually I think I was wrong in solving the issue.
Since at this point the device is being configured, there is no chance that we had a missing call notification here: A previous kick is needed for the device to generate any calls, and these cannot be processed.
What is not solved in this series is that we could have pending used buffers in vdpa device stopping SVQ, but queuing a check for that is not going to solve anything, since SVQ vring would be already destroyed:
* vdpa device marks N > 0 buffers as used, and calls.
* Before processing them, SVQ stop is called. SVQ have not processed these, and cleans them, making thisevent_notifier_setuseless.
Quoted 5 lines
So this would require a few changes. Mainly, instead of queueing a check for used, these need to be checked before svq cleaning. After that, obtain the VQ state (is not obtained in the stop at the moment, trusting in guest's used idx) and run a last
vhost_svq_handle_call_no_testwhile the device is paused.
It looks to me what's really important is that SVQ needs to drain/forwared used buffers after vdpa is stopped. Then we should be fine.
Quoted 56 lines
Thanks!
Quoted 52 lines
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
hw/virtio/vhost-vdpa.c +9
-0+return true; }static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev;unsigned n; +int r;if (enable == v->shadow_vqs_enabled) {return hdev->nvqs;@@ -752,9 +779,18 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)if (!enable) {/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { +struct vhost_vring_file file = { + .index =vhost_vdpa_get_vq_index (hdev, n), + .fd = v->call_fd[n], + }; + + r =vhost_vdpa_set_vring_call (hdev, &file); +assert (r ==0 );Quoted 9 lines
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 793
Quoted 5 lines
hw/virtio/vhost-vdpa.c +2
-0+unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n);VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n);vhost_svq_stop (hdev, n, svq); +/* TODO: This can unmask or override call fd! */ I don't get this comment. Does this mean the current code can't work with
mask_notifiers? If yes, this is something we need to fix.Quoted 2 lines
Yes, but it will be addressed in the next series. I should have explained it bette here, sorry :).
Ok.
Thanks
Quoted 9 lines
Thanks!
Quoted 6 lines
Thanks
Quoted 3 lines
hw/virtio/vhost-vdpa.c +0
-0vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); }
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
In reply to Eugenio Perez MartinQuoted 160 lines · Eugenio Perez Martin
Quoted 137 lines · Jason Wang
Quoted 134 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)hw/virtio/vhost-shadow-virtqueue.h +3
-1@@ -11,6 +11,7 @@#define VHOST_SHADOW_VIRTQUEUE_H #include "hw/virtio/vhost.h" +#include "hw/virtio/vhost-iova-tree.h" typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ;@@ -28,7 +29,8 @@ bool vhost_svq_start(struct vhost_dev *dev, unsigned idx,void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq); -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map);void vhost_svq_free (VhostShadowVirtqueue *vq);hw/virtio/vhost-shadow-virtqueue.c +49
-8@@ -11,12 +11,19 @@#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" #include "hw/virtio/virtio-access.h" +#include "hw/virtio/vhost-iova-tree.h" #include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" #include "qemu/main-loop.h" +typedef struct SVQElement { +VirtQueueElement elem; +void **in_sg_stash; +void **out_sg_stash; +}SVQElement ; +/* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { /* Shadow vring */@@ -46,8 +53,11 @@ typedef struct VhostShadowVirtqueue {/* Virtio device */ VirtIODevice *vdev; +/* IOVA mapping if used */ +VhostIOVATree *iova_map; +/* Map for returning guest's descriptors */ -VirtQueueElement **ring_id_maps; +SVQElement **ring_id_maps;/* Next head to expose to device */ uint16_t avail_idx_shadow;@@ -79,13 +89,6 @@ bool vhost_svq_valid_device_features(uint64_t *dev_features)continue ;case VIRTIO_F_ACCESS_PLATFORM : -/* SVQ needs this feature disabled. Can't continue */ -if (*dev_features &BIT_ULL (b)) { -clear_bit (b, dev_features); - r = false; - } -break ; -case VIRTIO_F_VERSION_1 :/* SVQ needs this feature, so can't continue */ if (!(*dev_features &BIT_ULL (b))) {@@ -126,6 +129,64 @@ static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) } } +static void vhost_svq_stash_addr (void ***stash,const struct iovec *iov, +size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + + *stash = g_new(void *, num); +for (i =0 ; i < num; ++i) { + (*stash)[i] = iov[i].iov_base; + } +} + +static void vhost_svq_unstash_addr (void **stash,struct iovec *iov,size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + +for (i =0 ; i < num; ++i) { + iov[i].iov_base = stash[i]; + } +g_free (stash); +} + +static void vhost_svq_translate_addr (const VhostShadowVirtqueue *svq, +struct iovec *iovec,size_t num) +{ +size_t i; + +for (i =0 ; i < num; ++i) { +VhostDMAMap needle = { + .translated_addr = iovec[i].iov_base, + .size = iovec[i].iov_len,Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) · line 174
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+ }; +size_t off; + +const VhostDMAMap *map =vhost_iova_tree_find_iova (svq->iova_map, + &needle);Is it possible that we end up with more than one maps here?
Actually it is possible, since there is no guarantee that one descriptor (or indirect descriptor) maps exactly to one iov. It could map to many if qemu vaddr is not contiguous but GPA + size is. This is something that must be fixed for the next revision, so thanks for pointing it out!
Taking that into account, the condition that svq vring
avail_idx-used_idxwas always less or equal than guest's vringavail_idx-used_idxis not true anymore. Checking for that before adding buffers to SVQ is the easy part, but how could we recover in that case?
Quoted 7 lines
I think that the easy solution is to check for more available buffers unconditionally at the end of
vhost_svq_handle_call, which handles the SVQ used and is supposed to make more room for available buffers. Sovhost_handle_guest_kickwould not check if eventfd is set or not anymore.Would that make sense?
Yes, I think it should work.
Thanks
Quoted 45 lines
Thanks!
Quoted 34 lines
Quoted 34 lines
hw/virtio/vhost-shadow-virtqueue.c +17
-1+/* +* Map cannot be NULL since iova map contains all guest space and +* qemu already has a physical address mapped +*/ +assert (map); + +/* +* Map->iova chunk size is ignored. What to do if descriptor +* (addr, size) does not fit is delegated to the device. +*/ + off = needle.translated_addr - map->translated_addr; + iovec[i].iov_base = (void *)(map->iova + off); + } +} +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq,const struct iovec *iovec,size_t num,bool more_descs,bool write)@@ -156,8 +217,9 @@ static void vhost_vring_write_descs(VhostShadowVirtqueue *svq, }static unsigned vhost_svq_add_split (VhostShadowVirtqueue *svq, -VirtQueueElement *elem) +SVQElement *svq_elem) { +VirtQueueElement *elem = &svq_elem->elem;int head;unsigned avail_idx;vring_avail_t *avail = svq->vring.avail;@@ -167,6 +229,12 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq,/* We need some descriptors here */ assert (elem->out_num || elem->in_num);Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.c static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, · line 233
Quoted 2 lines
hw/virtio/vhost-shadow-virtqueue.c +2
-0+vhost_svq_stash_addr (&svq_elem->in_sg_stash, elem->in_sg, elem->in_num); +vhost_svq_stash_addr (&svq_elem->out_sg_stash, elem->out_sg, elem->out_num);I wonder if we can solve the trick like stash and unstash with a dedicated sgs in
svq_elem, instead of reusing the elem.
Quoted 227 lines
Actually yes, it would be way simpler to use a new sgs array in
svq_elem. I will change that.Thanks!
Quoted 221 lines
Thanks
Quoted 217 lines
hw/virtio/vhost-shadow-virtqueue.c +36
-12+ +vhost_svq_translate_addr (svq, elem->in_sg, elem->in_num); +vhost_svq_translate_addr (svq, elem->out_sg, elem->out_num); +vhost_vring_write_descs (svq, elem->out_sg, elem->out_num, elem->in_num >0 , false);vhost_vring_write_descs (svq, elem->in_sg, elem->in_num, false, true);@@ -187,7 +255,7 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, } -static void vhost_svq_add (VhostShadowVirtqueue *svq,VirtQueueElement *elem) +static void vhost_svq_add (VhostShadowVirtqueue *svq,SVQElement *elem) {unsigned qemu_head =vhost_svq_add_split (svq, elem);@@ -221,7 +289,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) }while (true) { -VirtQueueElement *elem =virtqueue_pop (svq->vq,sizeof (*elem)); +SVQElement *elem =virtqueue_pop (svq->vq,sizeof (*elem));if (!elem) {break ; }@@ -247,7 +315,7 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)return svq->used_idx != svq->shadow_used_idx; } -static VirtQueueElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) +static SVQElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) {vring_desc_t *descs = svq->vring.desc;const vring_used_t *used = svq->vring.used;@@ -279,7 +347,7 @@ static VirtQueueElement *vhost_svq_get_buf(VhostShadowVirtqueue *svq) descs[used_elem.id].next = svq->free_head; svq->free_head = used_elem.id; - svq->ring_id_maps[used_elem.id]->len = used_elem.len; + svq->ring_id_maps[used_elem.id]->elem.len = used_elem.len;return g_steal_pointer (&svq->ring_id_maps[used_elem.id]); }@@ -296,12 +364,19 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n)vhost_svq_set_notification (svq, false);while (true) { - g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); -if (!elem) { + g_autofree SVQElement *svq_elem = vhost_svq_get_buf(svq); +VirtQueueElement *elem; +if (!svq_elem) {break ; }assert (i < svq->vring.num); + elem = &svq_elem->elem; + +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num);virtqueue_fill (vq, elem, elem->len, i++); }@@ -451,14 +526,24 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,event_notifier_set_handler (&svq->host_notifier,NULL );for (i =0 ; i < svq->vring.num; ++i) { - g_autofree VirtQueueElement *elem = svq->ring_id_maps[i]; + g_autofree SVQElement *svq_elem = svq->ring_id_maps[i]; +VirtQueueElement *elem; + +if (!svq_elem) { +continue ; + } + + elem = &svq_elem->elem; +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num); +/* * Although the doc says we must unpop in order, it's ok to unpop * everything. */ -if (elem) { -virtqueue_unpop (svq->vq, elem, elem->len); - } +virtqueue_unpop (svq->vq, elem, elem->len); } }@@ -466,7 +551,8 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,* Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map) {int vq_idx = dev->vq_index + idx;unsigned num =virtio_queue_get_num (dev->vdev, vq_idx);@@ -500,11 +586,13 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)memset (svq->vring.desc,0 , driver_size); svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size);memset (svq->vring.used,0 , device_size); + svq->iova_map = iova_map; +for (i =0 ; i < num -1 ; i++) { svq->vring.desc[i].next =cpu_to_le16 (i +1 ); } - svq->ring_id_maps = g_new0(VirtQueueElement *, num); + svq->ring_id_maps = g_new0(SVQElement *, num);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);hw/virtio/vhost-vdpa.c +39
-1@@ -176,6 +176,18 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); +if (v->shadow_vqs_enabled) { +VhostDMAMap mem_region = { + .translated_addr = vaddr, + .size =int128_get64 (llsize) -1 , + .perm =IOMMU_ACCESS_FLAG (true, section->readonly), + }; + +int r =vhost_iova_tree_alloc (v->iova_map, &mem_region); +assert (r ==VHOST_DMA_MAP_OK ); + + iova = mem_region.iova; + } ret =vhost_vdpa_dma_map (v, iova,int128_get64 (llsize), vaddr, section->readonly);@@ -754,6 +766,23 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +static int vhost_vdpa_get_iova_range (struct vhost_dev *dev, +hwaddr *first,hwaddr *last) +{ +int ret; +struct vhost_vdpa_iova_range range; + + ret =vhost_vdpa_call (dev,VHOST_VDPA_GET_IOVA_RANGE , &range); +if (ret !=0 ) { +return ret; + } + + *first = range.first; + *last = range.last; +trace_vhost_vdpa_get_iova_range (dev, *first, *last); +return ret; +} +/** * Maps QEMU vaddr memory to device in a suitable way for shadow virtqueue: * - It always reference qemu memory address, not guest's memory. @@ -881,6 +910,7 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx)static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev; +hwaddr iova_first, iova_last;unsigned n;int r;@@ -894,7 +924,7 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* Allocate resources */ assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) { -VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n, v->iova_map);if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ;@@ -903,6 +933,8 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } + r =vhost_vdpa_get_iova_range (hdev, &iova_first, &iova_last); +assert (r ==0 ); r =vhost_vdpa_vring_pause (hdev);assert (r ==0 );@@ -913,6 +945,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } +memory_listener_unregister (&v->listener); +if (vhost_vdpa_dma_unmap (v, iova_first, + (iova_last - iova_first) &TARGET_PAGE_MASK )) { +error_report ("Fail to invalidate device iotlb" ); + } +/* Reset device so it can be configured */ r =vhost_vdpa_dev_start (hdev, false);assert (r ==0 );hw/virtio/trace-events +1
-0@@ -52,6 +52,7 @@ vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 0x%"PRIx64 vhost_vdpa_set_owner(void *dev) "dev: %p" vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64 +vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p first: 0x%"PRIx64" last: 0x%"PRIx64 # virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned out_num) "elem %p size %zd in_num %u out_num %u"
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
Quoted 169 lines · Jason Wang
Quoted 166 lines · Eugenio Perez Martin
Quoted 137 lines · Jason Wang
Quoted 134 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)hw/virtio/vhost-shadow-virtqueue.h +3
-1@@ -11,6 +11,7 @@#define VHOST_SHADOW_VIRTQUEUE_H #include "hw/virtio/vhost.h" +#include "hw/virtio/vhost-iova-tree.h" typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ;@@ -28,7 +29,8 @@ bool vhost_svq_start(struct vhost_dev *dev, unsigned idx,void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq); -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map);void vhost_svq_free (VhostShadowVirtqueue *vq);hw/virtio/vhost-shadow-virtqueue.c +49
-8@@ -11,12 +11,19 @@#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" #include "hw/virtio/virtio-access.h" +#include "hw/virtio/vhost-iova-tree.h" #include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" #include "qemu/main-loop.h" +typedef struct SVQElement { +VirtQueueElement elem; +void **in_sg_stash; +void **out_sg_stash; +}SVQElement ; +/* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { /* Shadow vring */@@ -46,8 +53,11 @@ typedef struct VhostShadowVirtqueue {/* Virtio device */ VirtIODevice *vdev; +/* IOVA mapping if used */ +VhostIOVATree *iova_map; +/* Map for returning guest's descriptors */ -VirtQueueElement **ring_id_maps; +SVQElement **ring_id_maps;/* Next head to expose to device */ uint16_t avail_idx_shadow;@@ -79,13 +89,6 @@ bool vhost_svq_valid_device_features(uint64_t *dev_features)continue ;case VIRTIO_F_ACCESS_PLATFORM : -/* SVQ needs this feature disabled. Can't continue */ -if (*dev_features &BIT_ULL (b)) { -clear_bit (b, dev_features); - r = false; - } -break ; -case VIRTIO_F_VERSION_1 :/* SVQ needs this feature, so can't continue */ if (!(*dev_features &BIT_ULL (b))) {@@ -126,6 +129,64 @@ static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) } } +static void vhost_svq_stash_addr (void ***stash,const struct iovec *iov, +size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + + *stash = g_new(void *, num); +for (i =0 ; i < num; ++i) { + (*stash)[i] = iov[i].iov_base; + } +} + +static void vhost_svq_unstash_addr (void **stash,struct iovec *iov,size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + +for (i =0 ; i < num; ++i) { + iov[i].iov_base = stash[i]; + } +g_free (stash); +} + +static void vhost_svq_translate_addr (const VhostShadowVirtqueue *svq, +struct iovec *iovec,size_t num) +{ +size_t i; + +for (i =0 ; i < num; ++i) { +VhostDMAMap needle = { + .translated_addr = iovec[i].iov_base, + .size = iovec[i].iov_len,Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) · line 174
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+ }; +size_t off; + +const VhostDMAMap *map =vhost_iova_tree_find_iova (svq->iova_map, + &needle);Is it possible that we end up with more than one maps here?
Actually it is possible, since there is no guarantee that one descriptor (or indirect descriptor) maps exactly to one iov. It could map to many if qemu vaddr is not contiguous but GPA + size is. This is something that must be fixed for the next revision, so thanks for pointing it out!
Taking that into account, the condition that svq vring
avail_idx-used_idxwas always less or equal than guest's vringavail_idx-used_idxis not true anymore. Checking for that before adding buffers to SVQ is the easy part, but how could we recover in that case?I think that the easy solution is to check for more available buffers unconditionally at the end of
vhost_svq_handle_call, which handles the SVQ used and is supposed to make more room for available buffers. Sovhost_handle_guest_kickwould not check if eventfd is set or not anymore.
Quoted 3 lines
Quoted 1 line
Would that make sense?
Yes, I think it should work.
Btw, I wonder how to handle indirect descriptors. SVQ doesn't use indirect descriptors for now, but it looks like a must otherwise we may end up SVQ is full before VQ.
It looks to me an easy way is to always use indirect descriptors if #sg >= 2?
Thanks
Quoted 275 lines
Thanks
Quoted 272 lines
Thanks!
Quoted 34 lines
Quoted 34 lines
hw/virtio/vhost-shadow-virtqueue.c +17
-1+/* +* Map cannot be NULL since iova map contains all guest space and +* qemu already has a physical address mapped +*/ +assert (map); + +/* +* Map->iova chunk size is ignored. What to do if descriptor +* (addr, size) does not fit is delegated to the device. +*/ + off = needle.translated_addr - map->translated_addr; + iovec[i].iov_base = (void *)(map->iova + off); + } +} +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq,const struct iovec *iovec,size_t num,bool more_descs,bool write)@@ -156,8 +217,9 @@ static void vhost_vring_write_descs(VhostShadowVirtqueue *svq, }static unsigned vhost_svq_add_split (VhostShadowVirtqueue *svq, -VirtQueueElement *elem) +SVQElement *svq_elem) { +VirtQueueElement *elem = &svq_elem->elem;int head;unsigned avail_idx;vring_avail_t *avail = svq->vring.avail;@@ -167,6 +229,12 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq,/* We need some descriptors here */ assert (elem->out_num || elem->in_num);Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.c static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, · line 233
Quoted 2 lines
hw/virtio/vhost-shadow-virtqueue.c +2
-0+vhost_svq_stash_addr (&svq_elem->in_sg_stash, elem->in_sg, elem->in_num); +vhost_svq_stash_addr (&svq_elem->out_sg_stash, elem->out_sg, elem->out_num);I wonder if we can solve the trick like stash and unstash with a dedicated sgs in
svq_elem, instead of reusing the elem.Actually yes, it would be way simpler to use a new sgs array in
svq_elem. I will change that.Thanks!
Quoted 221 lines
Thanks
Quoted 217 lines
hw/virtio/vhost-shadow-virtqueue.c +36
-12+ +vhost_svq_translate_addr (svq, elem->in_sg, elem->in_num); +vhost_svq_translate_addr (svq, elem->out_sg, elem->out_num); +vhost_vring_write_descs (svq, elem->out_sg, elem->out_num, elem->in_num >0 , false);vhost_vring_write_descs (svq, elem->in_sg, elem->in_num, false, true);@@ -187,7 +255,7 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, } -static void vhost_svq_add (VhostShadowVirtqueue *svq,VirtQueueElement *elem) +static void vhost_svq_add (VhostShadowVirtqueue *svq,SVQElement *elem) {unsigned qemu_head =vhost_svq_add_split (svq, elem);@@ -221,7 +289,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) }while (true) { -VirtQueueElement *elem =virtqueue_pop (svq->vq,sizeof (*elem)); +SVQElement *elem =virtqueue_pop (svq->vq,sizeof (*elem));if (!elem) {break ; }@@ -247,7 +315,7 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)return svq->used_idx != svq->shadow_used_idx; } -static VirtQueueElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) +static SVQElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) {vring_desc_t *descs = svq->vring.desc;const vring_used_t *used = svq->vring.used;@@ -279,7 +347,7 @@ static VirtQueueElement *vhost_svq_get_buf(VhostShadowVirtqueue *svq) descs[used_elem.id].next = svq->free_head; svq->free_head = used_elem.id; - svq->ring_id_maps[used_elem.id]->len = used_elem.len; + svq->ring_id_maps[used_elem.id]->elem.len = used_elem.len;return g_steal_pointer (&svq->ring_id_maps[used_elem.id]); }@@ -296,12 +364,19 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n)vhost_svq_set_notification (svq, false);while (true) { - g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); -if (!elem) { + g_autofree SVQElement *svq_elem = vhost_svq_get_buf(svq); +VirtQueueElement *elem; +if (!svq_elem) {break ; }assert (i < svq->vring.num); + elem = &svq_elem->elem; + +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num);virtqueue_fill (vq, elem, elem->len, i++); }@@ -451,14 +526,24 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,event_notifier_set_handler (&svq->host_notifier,NULL );for (i =0 ; i < svq->vring.num; ++i) { - g_autofree VirtQueueElement *elem = svq->ring_id_maps[i]; + g_autofree SVQElement *svq_elem = svq->ring_id_maps[i]; +VirtQueueElement *elem; + +if (!svq_elem) { +continue ; + } + + elem = &svq_elem->elem; +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num); +/* * Although the doc says we must unpop in order, it's ok to unpop * everything. */ -if (elem) { -virtqueue_unpop (svq->vq, elem, elem->len); - } +virtqueue_unpop (svq->vq, elem, elem->len); } }@@ -466,7 +551,8 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,* Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map) {int vq_idx = dev->vq_index + idx;unsigned num =virtio_queue_get_num (dev->vdev, vq_idx);@@ -500,11 +586,13 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)memset (svq->vring.desc,0 , driver_size); svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size);memset (svq->vring.used,0 , device_size); + svq->iova_map = iova_map; +for (i =0 ; i < num -1 ; i++) { svq->vring.desc[i].next =cpu_to_le16 (i +1 ); } - svq->ring_id_maps = g_new0(VirtQueueElement *, num); + svq->ring_id_maps = g_new0(SVQElement *, num);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);hw/virtio/vhost-vdpa.c +39
-1@@ -176,6 +176,18 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); +if (v->shadow_vqs_enabled) { +VhostDMAMap mem_region = { + .translated_addr = vaddr, + .size =int128_get64 (llsize) -1 , + .perm =IOMMU_ACCESS_FLAG (true, section->readonly), + }; + +int r =vhost_iova_tree_alloc (v->iova_map, &mem_region); +assert (r ==VHOST_DMA_MAP_OK ); + + iova = mem_region.iova; + } ret =vhost_vdpa_dma_map (v, iova,int128_get64 (llsize), vaddr, section->readonly);@@ -754,6 +766,23 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +static int vhost_vdpa_get_iova_range (struct vhost_dev *dev, +hwaddr *first,hwaddr *last) +{ +int ret; +struct vhost_vdpa_iova_range range; + + ret =vhost_vdpa_call (dev,VHOST_VDPA_GET_IOVA_RANGE , &range); +if (ret !=0 ) { +return ret; + } + + *first = range.first; + *last = range.last; +trace_vhost_vdpa_get_iova_range (dev, *first, *last); +return ret; +} +/** * Maps QEMU vaddr memory to device in a suitable way for shadow virtqueue: * - It always reference qemu memory address, not guest's memory. @@ -881,6 +910,7 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx)static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev; +hwaddr iova_first, iova_last;unsigned n;int r;@@ -894,7 +924,7 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* Allocate resources */ assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) { -VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n, v->iova_map);if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ;@@ -903,6 +933,8 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } + r =vhost_vdpa_get_iova_range (hdev, &iova_first, &iova_last); +assert (r ==0 ); r =vhost_vdpa_vring_pause (hdev);assert (r ==0 );@@ -913,6 +945,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } +memory_listener_unregister (&v->listener); +if (vhost_vdpa_dma_unmap (v, iova_first, + (iova_last - iova_first) &TARGET_PAGE_MASK )) { +error_report ("Fail to invalidate device iotlb" ); + } +/* Reset device so it can be configured */ r =vhost_vdpa_dev_start (hdev, false);assert (r ==0 );hw/virtio/trace-events +1
-0@@ -52,6 +52,7 @@ vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 0x%"PRIx64 vhost_vdpa_set_owner(void *dev) "dev: %p" vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64 +vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p first: 0x%"PRIx64" last: 0x%"PRIx64 # virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned out_num) "elem %p size %zd in_num %u out_num %u"
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue
In reply to Jason WangQuoted 165 lines · Jason Wang
Quoted 156 lines · Eugenio Perez Martin
Quoted 129 lines · Jason Wang
Quoted 106 lines · Eugenio Perez Martin
Quoted 78 lines · Jason Wang
Quoted 76 lines · Eugenio Pérez
This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed.
Since the use of
vhost_virtqueue_startcan unmasks and discard call events,vhost_virtqueue_startshould be modified in one of these ways:
* Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former.
* Actually store and check if the guest notifier is masked, and do it conditionally.
* Left as it is, and duplicate all the logic in vhost-vdpa.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.c | 19 +++++++++++++++ hw/virtio/vhost-vdpa.c | 38 +++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)hw/virtio/vhost-shadow-virtqueue.c +19
-0@@ -53,6 +53,22 @@ static void vhost_handle_guest_kick(EventNotifier *n)event_notifier_set (&svq->kick_notifier); } +/* Forward vhost notifications */ +static void vhost_svq_handle_call_no_test (EventNotifier *n) +{ +VhostShadowVirtqueue *svq =container_of (n, VhostShadowVirtqueue, + call_notifier); + +event_notifier_set (&svq->guest_call_notifier); +} + +static void vhost_svq_handle_call (EventNotifier *n) +{ +if (likely (event_notifier_test_and_clear (n))) { +vhost_svq_handle_call_no_test (n); + } +} +/* * Obtain the SVQ call notifier, where vhost device notifies SVQ that there * exists pending used buffers. @@ -180,6 +196,8 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) } svq->vq =virtio_get_queue (dev->vdev, vq_idx); +event_notifier_set_handler (&svq->call_notifier, + vhost_svq_handle_call);return g_steal_pointer (&svq); err_init_call_notifier:@@ -195,6 +213,7 @@ err_init_kick_notifier:void vhost_svq_free (VhostShadowVirtqueue *vq) {event_notifier_cleanup (&vq->kick_notifier); +event_notifier_set_handler (&vq->call_notifier,NULL );event_notifier_cleanup (&vq->call_notifier);g_free (vq); }hw/virtio/vhost-vdpa.c +4
-1@@ -712,13 +712,40 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) {struct vhost_vdpa *v = dev->opaque;VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, idx); -return vhost_svq_start (dev, idx, svq); +EventNotifier *vhost_call_notifier =vhost_svq_get_svq_call_notifier (svq); +struct vhost_vring_file vhost_call_file = { + .index = idx + dev->vq_index, + .fd =event_notifier_get_fd (vhost_call_notifier),Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 724
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+ }; +int r; +bool b; + +/* Set shadow vq -> guest notifier */ +assert (v->call_fd[idx]);We need aovid the
asser()here. On which case we can hit this?I would say that there is no way we can actually hit it, so let's remove it.
Quoted 7 lines
Quoted 7 lines
hw/virtio/vhost-vdpa.c +7
-0+vhost_svq_set_guest_call_notifier (svq, v->call_fd[idx]); + + b =vhost_svq_start (dev, idx, svq); +if (unlikely (!b)) { +return false; + } +Quoted 9 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 737
Quoted 6 lines
hw/virtio/vhost-vdpa.c +6
-0+/* Set device -> SVQ notifier */ + r =vhost_vdpa_set_vring_dev_call (dev, &vhost_call_file); +if (unlikely (r)) { +error_report ("vhost_vdpa_set_vring_call for shadow vq failed" ); +return false; + }Similar to kick, do we need to
set_vring_call()beforevhost_svq_start()?Quoted 3 lines
It should not matter at this moment because the device should not be started at this point and device calls should not run
vhost_svq_handle_calluntil BQL is released.Yes, we stop virtqueue before.
Quoted 12 lines
The "logic" of doing it after is to make clear that svq must be fully initialized before processing device calls, even in the case that we extract SVQ in its own iothread or similar. But this could be done before
vhost_svq_startfor sure.Quoted 6 lines
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) · line 740
Quoted 3 lines
hw/virtio/vhost-vdpa.c +3
-0+ +/* Check for pending calls */ +event_notifier_set (vhost_call_notifier);Interesting, can this result spurious interrupt?
Quoted 8 lines
Quoted 4 lines
This actually "queues" a
vhost_svq_handle_callafter the BQL release, where the device should be fully reset. In that regard, if there are no used descriptors there will not be an irq raised to the guest. Does that answer the question? Or have I missed something?Yes, please explain this in the comment.
I'm reviewing this again, and actually I think I was wrong in solving the issue.
Since at this point the device is being configured, there is no chance that we had a missing call notification here: A previous kick is needed for the device to generate any calls, and these cannot be processed.
What is not solved in this series is that we could have pending used buffers in vdpa device stopping SVQ, but queuing a check for that is not going to solve anything, since SVQ vring would be already destroyed:
* vdpa device marks N > 0 buffers as used, and calls.
* Before processing them, SVQ stop is called. SVQ have not processed these, and cleans them, making thisevent_notifier_setuseless.Quoted 5 lines
So this would require a few changes. Mainly, instead of queueing a check for used, these need to be checked before svq cleaning. After that, obtain the VQ state (is not obtained in the stop at the moment, trusting in guest's used idx) and run a last
vhost_svq_handle_call_no_testwhile the device is paused.
Quoted 4 lines
It looks to me what's really important is that SVQ needs to drain/forwared used buffers after vdpa is stopped. Then we should be fine.
Right. I think I picked the wrong place to raise the concern, but the next revision will include the drain of the pending buffers.
Thanks!
Quoted 57 lines
Quoted 56 lines
Thanks!
Quoted 52 lines
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
hw/virtio/vhost-vdpa.c +9
-0+return true; }static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev;unsigned n; +int r;if (enable == v->shadow_vqs_enabled) {return hdev->nvqs;@@ -752,9 +779,18 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)if (!enable) {/* Disable all queues or clean up failed start */ for (n =0 ; n < v->shadow_vqs->len; ++n) { +struct vhost_vring_file file = { + .index =vhost_vdpa_get_vq_index (hdev, n), + .fd = v->call_fd[n], + }; + + r =vhost_vdpa_set_vring_call (hdev, &file); +assert (r ==0 );Quoted 9 lines
hw/virtio/vhost-vdpa.c static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) · line 793
Quoted 5 lines
hw/virtio/vhost-vdpa.c +2
-0+unsigned vq_idx =vhost_vdpa_get_vq_index (hdev, n);VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, n);vhost_svq_stop (hdev, n, svq); +/* TODO: This can unmask or override call fd! */ I don't get this comment. Does this mean the current code can't work with
mask_notifiers? If yes, this is something we need to fix.Quoted 2 lines
Yes, but it will be addressed in the next series. I should have explained it bette here, sorry :).
Ok.
Thanks
Quoted 9 lines
Thanks!
Quoted 6 lines
Thanks
Quoted 3 lines
hw/virtio/vhost-vdpa.c +0
-0vhost_virtqueue_start (hdev, hdev->vdev, &hdev->vqs[n], vq_idx); }
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ
In reply to Jason WangQuoted 171 lines · Jason Wang
Quoted 169 lines · Jason Wang
Quoted 166 lines · Eugenio Perez Martin
Quoted 137 lines · Jason Wang
Quoted 134 lines · Eugenio Pérez
Use translations added in VhostIOVATree in SVQ.
Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone.
TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it.
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-)hw/virtio/vhost-shadow-virtqueue.h +3
-1@@ -11,6 +11,7 @@#define VHOST_SHADOW_VIRTQUEUE_H #include "hw/virtio/vhost.h" +#include "hw/virtio/vhost-iova-tree.h" typedef struct VhostShadowVirtqueue VhostShadowVirtqueue ;@@ -28,7 +29,8 @@ bool vhost_svq_start(struct vhost_dev *dev, unsigned idx,void vhost_svq_stop (struct vhost_dev *dev,unsigned idx,VhostShadowVirtqueue *svq); -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx); +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map);void vhost_svq_free (VhostShadowVirtqueue *vq);hw/virtio/vhost-shadow-virtqueue.c +49
-8@@ -11,12 +11,19 @@#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" #include "hw/virtio/virtio-access.h" +#include "hw/virtio/vhost-iova-tree.h" #include "standard-headers/linux/vhost_types.h" #include "qemu/error-report.h" #include "qemu/main-loop.h" +typedef struct SVQElement { +VirtQueueElement elem; +void **in_sg_stash; +void **out_sg_stash; +}SVQElement ; +/* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { /* Shadow vring */@@ -46,8 +53,11 @@ typedef struct VhostShadowVirtqueue {/* Virtio device */ VirtIODevice *vdev; +/* IOVA mapping if used */ +VhostIOVATree *iova_map; +/* Map for returning guest's descriptors */ -VirtQueueElement **ring_id_maps; +SVQElement **ring_id_maps;/* Next head to expose to device */ uint16_t avail_idx_shadow;@@ -79,13 +89,6 @@ bool vhost_svq_valid_device_features(uint64_t *dev_features)continue ;case VIRTIO_F_ACCESS_PLATFORM : -/* SVQ needs this feature disabled. Can't continue */ -if (*dev_features &BIT_ULL (b)) { -clear_bit (b, dev_features); - r = false; - } -break ; -case VIRTIO_F_VERSION_1 :/* SVQ needs this feature, so can't continue */ if (!(*dev_features &BIT_ULL (b))) {@@ -126,6 +129,64 @@ static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) } } +static void vhost_svq_stash_addr (void ***stash,const struct iovec *iov, +size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + + *stash = g_new(void *, num); +for (i =0 ; i < num; ++i) { + (*stash)[i] = iov[i].iov_base; + } +} + +static void vhost_svq_unstash_addr (void **stash,struct iovec *iov,size_t num) +{ +size_t i; + +if (num ==0 ) { +return ; + } + +for (i =0 ; i < num; ++i) { + iov[i].iov_base = stash[i]; + } +g_free (stash); +} + +static void vhost_svq_translate_addr (const VhostShadowVirtqueue *svq, +struct iovec *iovec,size_t num) +{ +size_t i; + +for (i =0 ; i < num; ++i) { +VhostDMAMap needle = { + .translated_addr = iovec[i].iov_base, + .size = iovec[i].iov_len,Quoted 9 lines
hw/virtio/vhost-shadow-virtqueue.c static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool enable) · line 174
Quoted 5 lines
hw/virtio/vhost-shadow-virtqueue.c +5
-0+ }; +size_t off; + +const VhostDMAMap *map =vhost_iova_tree_find_iova (svq->iova_map, + &needle);Is it possible that we end up with more than one maps here?
Actually it is possible, since there is no guarantee that one descriptor (or indirect descriptor) maps exactly to one iov. It could map to many if qemu vaddr is not contiguous but GPA + size is. This is something that must be fixed for the next revision, so thanks for pointing it out!
Taking that into account, the condition that svq vring
avail_idx-used_idxwas always less or equal than guest's vringavail_idx-used_idxis not true anymore. Checking for that before adding buffers to SVQ is the easy part, but how could we recover in that case?I think that the easy solution is to check for more available buffers unconditionally at the end of
vhost_svq_handle_call, which handles the SVQ used and is supposed to make more room for available buffers. Sovhost_handle_guest_kickwould not check if eventfd is set or not anymore.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Would that make sense?
Yes, I think it should work.
Btw, I wonder how to handle indirect descriptors. SVQ doesn't use indirect descriptors for now, but it looks like a must otherwise we may end up SVQ is full before VQ.
We can get to that situation without indirect too, if a single descriptor maps to more than one sg buffer. The next revision is going to control that too.
Quoted 2 lines
It looks to me an easy way is to always use indirect descriptors if #sg >= 2?
I will use that, but that does not solve the case where a descriptor maps to > 1 different buffers in qemu vaddr. So I think that some check after marking descriptors as used is a must somehow.
Quoted 278 lines
Thanks
Quoted 275 lines
Thanks
Quoted 272 lines
Thanks!
Quoted 34 lines
Quoted 34 lines
hw/virtio/vhost-shadow-virtqueue.c +17
-1+/* +* Map cannot be NULL since iova map contains all guest space and +* qemu already has a physical address mapped +*/ +assert (map); + +/* +* Map->iova chunk size is ignored. What to do if descriptor +* (addr, size) does not fit is delegated to the device. +*/ + off = needle.translated_addr - map->translated_addr; + iovec[i].iov_base = (void *)(map->iova + off); + } +} +static void vhost_vring_write_descs (VhostShadowVirtqueue *svq,const struct iovec *iovec,size_t num,bool more_descs,bool write)@@ -156,8 +217,9 @@ static void vhost_vring_write_descs(VhostShadowVirtqueue *svq, }static unsigned vhost_svq_add_split (VhostShadowVirtqueue *svq, -VirtQueueElement *elem) +SVQElement *svq_elem) { +VirtQueueElement *elem = &svq_elem->elem;int head;unsigned avail_idx;vring_avail_t *avail = svq->vring.avail;@@ -167,6 +229,12 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq,/* We need some descriptors here */ assert (elem->out_num || elem->in_num);Quoted 7 lines
hw/virtio/vhost-shadow-virtqueue.c static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, · line 233
Quoted 2 lines
hw/virtio/vhost-shadow-virtqueue.c +2
-0+vhost_svq_stash_addr (&svq_elem->in_sg_stash, elem->in_sg, elem->in_num); +vhost_svq_stash_addr (&svq_elem->out_sg_stash, elem->out_sg, elem->out_num);I wonder if we can solve the trick like stash and unstash with a dedicated sgs in
svq_elem, instead of reusing the elem.Actually yes, it would be way simpler to use a new sgs array in
svq_elem. I will change that.Thanks!
Quoted 221 lines
Thanks
Quoted 217 lines
hw/virtio/vhost-shadow-virtqueue.c +36
-12+ +vhost_svq_translate_addr (svq, elem->in_sg, elem->in_num); +vhost_svq_translate_addr (svq, elem->out_sg, elem->out_num); +vhost_vring_write_descs (svq, elem->out_sg, elem->out_num, elem->in_num >0 , false);vhost_vring_write_descs (svq, elem->in_sg, elem->in_num, false, true);@@ -187,7 +255,7 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, } -static void vhost_svq_add (VhostShadowVirtqueue *svq,VirtQueueElement *elem) +static void vhost_svq_add (VhostShadowVirtqueue *svq,SVQElement *elem) {unsigned qemu_head =vhost_svq_add_split (svq, elem);@@ -221,7 +289,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) }while (true) { -VirtQueueElement *elem =virtqueue_pop (svq->vq,sizeof (*elem)); +SVQElement *elem =virtqueue_pop (svq->vq,sizeof (*elem));if (!elem) {break ; }@@ -247,7 +315,7 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)return svq->used_idx != svq->shadow_used_idx; } -static VirtQueueElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) +static SVQElement *vhost_svq_get_buf (VhostShadowVirtqueue *svq) {vring_desc_t *descs = svq->vring.desc;const vring_used_t *used = svq->vring.used;@@ -279,7 +347,7 @@ static VirtQueueElement *vhost_svq_get_buf(VhostShadowVirtqueue *svq) descs[used_elem.id].next = svq->free_head; svq->free_head = used_elem.id; - svq->ring_id_maps[used_elem.id]->len = used_elem.len; + svq->ring_id_maps[used_elem.id]->elem.len = used_elem.len;return g_steal_pointer (&svq->ring_id_maps[used_elem.id]); }@@ -296,12 +364,19 @@ static void vhost_svq_handle_call_no_test(EventNotifier *n)vhost_svq_set_notification (svq, false);while (true) { - g_autofree VirtQueueElement *elem = vhost_svq_get_buf(svq); -if (!elem) { + g_autofree SVQElement *svq_elem = vhost_svq_get_buf(svq); +VirtQueueElement *elem; +if (!svq_elem) {break ; }assert (i < svq->vring.num); + elem = &svq_elem->elem; + +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num);virtqueue_fill (vq, elem, elem->len, i++); }@@ -451,14 +526,24 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,event_notifier_set_handler (&svq->host_notifier,NULL );for (i =0 ; i < svq->vring.num; ++i) { - g_autofree VirtQueueElement *elem = svq->ring_id_maps[i]; + g_autofree SVQElement *svq_elem = svq->ring_id_maps[i]; +VirtQueueElement *elem; + +if (!svq_elem) { +continue ; + } + + elem = &svq_elem->elem; +vhost_svq_unstash_addr (svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); +vhost_svq_unstash_addr (svq_elem->out_sg_stash, elem->out_sg, + elem->out_num); +/* * Although the doc says we must unpop in order, it's ok to unpop * everything. */ -if (elem) { -virtqueue_unpop (svq->vq, elem, elem->len); - } +virtqueue_unpop (svq->vq, elem, elem->len); } }@@ -466,7 +551,8 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx,* Creates vhost shadow virtqueue, and instruct vhost device to use the shadow * methods and file descriptors. */ -VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx) +VhostShadowVirtqueue *vhost_svq_new (struct vhost_dev *dev,int idx, +VhostIOVATree *iova_map) {int vq_idx = dev->vq_index + idx;unsigned num =virtio_queue_get_num (dev->vdev, vq_idx);@@ -500,11 +586,13 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx)memset (svq->vring.desc,0 , driver_size); svq->vring.used =qemu_memalign (qemu_real_host_page_size, device_size);memset (svq->vring.used,0 , device_size); + svq->iova_map = iova_map; +for (i =0 ; i < num -1 ; i++) { svq->vring.desc[i].next =cpu_to_le16 (i +1 ); } - svq->ring_id_maps = g_new0(VirtQueueElement *, num); + svq->ring_id_maps = g_new0(SVQElement *, num);event_notifier_set_handler (&svq->call_notifier, vhost_svq_handle_call);return g_steal_pointer (&svq);hw/virtio/vhost-vdpa.c +39
-1@@ -176,6 +176,18 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); +if (v->shadow_vqs_enabled) { +VhostDMAMap mem_region = { + .translated_addr = vaddr, + .size =int128_get64 (llsize) -1 , + .perm =IOMMU_ACCESS_FLAG (true, section->readonly), + }; + +int r =vhost_iova_tree_alloc (v->iova_map, &mem_region); +assert (r ==VHOST_DMA_MAP_OK ); + + iova = mem_region.iova; + } ret =vhost_vdpa_dma_map (v, iova,int128_get64 (llsize), vaddr, section->readonly);@@ -754,6 +766,23 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev)return true; } +static int vhost_vdpa_get_iova_range (struct vhost_dev *dev, +hwaddr *first,hwaddr *last) +{ +int ret; +struct vhost_vdpa_iova_range range; + + ret =vhost_vdpa_call (dev,VHOST_VDPA_GET_IOVA_RANGE , &range); +if (ret !=0 ) { +return ret; + } + + *first = range.first; + *last = range.last; +trace_vhost_vdpa_get_iova_range (dev, *first, *last); +return ret; +} +/** * Maps QEMU vaddr memory to device in a suitable way for shadow virtqueue: * - It always reference qemu memory address, not guest's memory. @@ -881,6 +910,7 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx)static unsigned vhost_vdpa_enable_svq (struct vhost_vdpa *v,bool enable) {struct vhost_dev *hdev = v->dev; +hwaddr iova_first, iova_last;unsigned n;int r;@@ -894,7 +924,7 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable)/* Allocate resources */ assert (v->shadow_vqs->len ==0 );for (n =0 ; n < hdev->nvqs; ++n) { -VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n); +VhostShadowVirtqueue *svq =vhost_svq_new (hdev, n, v->iova_map);if (unlikely (!svq)) {g_ptr_array_set_size (v->shadow_vqs,0 );return 0 ;@@ -903,6 +933,8 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } + r =vhost_vdpa_get_iova_range (hdev, &iova_first, &iova_last); +assert (r ==0 ); r =vhost_vdpa_vring_pause (hdev);assert (r ==0 );@@ -913,6 +945,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) } } +memory_listener_unregister (&v->listener); +if (vhost_vdpa_dma_unmap (v, iova_first, + (iova_last - iova_first) &TARGET_PAGE_MASK )) { +error_report ("Fail to invalidate device iotlb" ); + } +/* Reset device so it can be configured */ r =vhost_vdpa_dev_start (hdev, false);assert (r ==0 );hw/virtio/trace-events +1
-0@@ -52,6 +52,7 @@ vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 0x%"PRIx64 vhost_vdpa_set_owner(void *dev) "dev: %p" vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64 +vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p first: 0x%"PRIx64" last: 0x%"PRIx64 # virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned out_num) "elem %p size %zd in_num %u out_num %u"
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 18/20] vhost: Add VhostIOVATree
In reply to Jason WangQuoted 25 lines · Jason Wang
Quoted 5 lines · Eugenio Pérez
This tree is able to look for a translated address from an IOVA address.
At first glance is similar to util/iova-tree. However, SVQ working on devices with limited IOVA space need more capabilities, like allocating IOVA chunks or perform reverse translations (qemu addresses to iova).
I don't see any reverse translation is used in the shadow code. Or anything I missed?
Quoted 11 lines
The allocation capability, as "assign a free IOVA address to this chunk of memory in qemu's address space" allows shadow virtqueue to create a new address space that is not restricted by guest's addressable one, so we can allocate shadow vqs vrings outside of its reachability, nor qemu's one. At the moment, the allocation is just done growing, not allowing deletion.
A different name could be used, but ordered searchable array is a little bit long though.
Quoted 9 lines
Quoted 5 lines
It duplicates the array so it can search efficiently both directions, and it will signal overlap if iova or the translated address is present in it's each array.
Use of array will be changed to util-iova-tree in future series.
Adding Peter.
Thanks, I missed CC him!
Quoted 11 lines
It looks to me the only thing miseed is the iova allocator. And it looks to me it's better to decouple the allocator from the iova tree.
Then we had:
1) initialize iova range
2) iova =iova_alloc(size)
3) built the iova tree map
4) buffer forwarding
5)iova_free(size)
The next series I will send once I have solved all the comments is done that way, but the allocation is done in iova tree, not outside. Reasons below.
Quoted 92 lines
Quoted 92 lines
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-iova-tree.h | 40 +++++++ hw/virtio/vhost-iova-tree.c | 230 ++++++++++++++++++++++++++++++++++++ hw/virtio/meson.build | 2 +- 3 files changed, 271 insertions(+), 1 deletion(-)create mode 100644
hw/virtio/vhost-iova-tree.h
create mode 100644hw/virtio/vhost-iova-tree.chw/virtio/vhost-iova-tree.h new file +40
-0@@ -0,0 +1,40 @@ +/* +* vhost software live migration ring +* +* SPDX-FileCopyrightText: Red Hat, Inc. 2021 +* SPDX-FileContributor: Author: Eugenio Pérez <[email protected]> +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef HW_VIRTIO_VHOST_IOVA_TREE_H +#define HW_VIRTIO_VHOST_IOVA_TREE_H + +#include "exec/memory.h" + +typedef struct VhostDMAMap { +void *translated_addr; +hwaddr iova; +hwaddr size;/* Inclusive */ +IOMMUAccessFlags perm; +}VhostDMAMap ; + +typedef enum VhostDMAMapNewRC { +VHOST_DMA_MAP_NO_SPACE =-3 , +VHOST_DMA_MAP_OVERLAP =-2 , +VHOST_DMA_MAP_INVALID =-1 , +VHOST_DMA_MAP_OK =0 , +}VhostDMAMapNewRC ; + +typedef struct VhostIOVATree VhostIOVATree ; + +VhostIOVATree *vhost_iova_tree_new (void ); +void vhost_iova_tree_unref (VhostIOVATree *iova_rm); +G_DEFINE_AUTOPTR_CLEANUP_FUNC (VhostIOVATree, vhost_iova_tree_unref); + +const VhostDMAMap *vhost_iova_tree_find_iova (const VhostIOVATree *iova_rm, +const VhostDMAMap *map); +VhostDMAMapNewRC vhost_iova_tree_alloc (VhostIOVATree *iova_rm, +VhostDMAMap *map); + +#endif hw/virtio/vhost-iova-tree.c new file +29
-0@@ -0,0 +1,230 @@ +/* +* vhost software live migration ring +* +* SPDX-FileCopyrightText: Red Hat, Inc. 2021 +* SPDX-FileContributor: Author: Eugenio Pérez <[email protected]> +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "qemu/osdep.h" +#include "vhost-iova-tree.h" + +#define G_ARRAY_NOT_ZERO_TERMINATED false +#define G_ARRAY_NOT_CLEAR_ON_ALLOC false + +#define iova_min qemu_real_host_page_size + +/** +* VhostIOVATree, able to: +* - Translate iova address +* - Reverse translate iova address (from translated to iova) +* - Allocate IOVA regions for translated range (potentially slow operation) +* +* Note that it cannot remove nodes. +*/ +struct VhostIOVATree { +/* Ordered array of reverse translations, IOVA address to qemu memory. */ +GArray *iova_taddr_map; +
Quoted 9 lines
hw/virtio/vhost-iova-tree.c line 34
Quoted 5 lines
hw/virtio/vhost-iova-tree.c new file +5
-0+/* +* Ordered array of translations from qemu virtual memory address to iova +*/ +GArray *taddr_iova_map; +};Any reason for using GArray? Is it faster?
To be honest, I used a GArray mainly for prototyping reasons, because it allowed me to "insert a next element" once I've located either a hole (in case of iova) or an address. To do it in the iova tree required to either add code to iterate elements with arguments or to allocate them. Another possibility is to add yet another structure with "free regions".
I would say that at this moment GArray will be faster than GTree due to GArray ability to bisect it and the better data locality, since we will not be adding and deleting regions frequently during migration. But that could change in case we support viommu platforms, and I didn't measure it.
For the next revision I've added allocation capabilities to iova tree, which I think fits pretty well there, and a separated qemu's vaddr -> iova translation GTree. The allocation still transverse the utils/iova tree linearly, but we can add a more performant allocator in the future easily if needed. It should not affect anyway unless we hotplug memory or similar.
Quoted 86 lines
Quoted 86 lines
hw/virtio/vhost-iova-tree.c new file +85
-0+ +/** +* Inserts an element after an existing one in garray. +* +* @array The array +* @prev_elem The previous element of array of NULL if prepending +* @map The DMA map +* +* It provides the aditional advantage of being type safe over +* g_array_insert_val, which accepts a reference pointer instead of a value +* with no complains. +*/ +static void vhost_iova_tree_insert_after (GArray *array, +const VhostDMAMap *prev_elem, +const VhostDMAMap *map) +{ +size_t pos; + +if (!prev_elem) { + pos =0 ; + }else { + pos = prev_elem - &g_array_index (array,typeof (*prev_elem),0 ) +1 ; + } + +g_array_insert_val (array, pos, *map); +} + +static gint vhost_iova_tree_cmp_taddr (gconstpointer a,gconstpointer b) +{ +const VhostDMAMap *m1 = a, *m2 = b; + +if (m1->translated_addr > m2->translated_addr + m2->size) { +return 1 ; + } + +if (m1->translated_addr + m1->size < m2->translated_addr) { +return -1 ; + } + +/* Overlapped */ +return 0 ; +} + +/** +* Find the previous node to a given iova +* +* @array The ascending ordered-by-translated-addr array of VhostDMAMap +* @map The map to insert +* @prev Returned location of the previous map +* +* Return VHOST_DMA_MAP_OK if everything went well, or VHOST_DMA_MAP_OVERLAP if +* it already exists. It is ok to use this function to check if a given range +* exists, but it will use a linear search. +* +* TODO: We can use bsearch to locate the entry if we save the state in the +* needle, knowing that the needle is always the first argument to +* compare_func. +*/ +static VhostDMAMapNewRC vhost_iova_tree_find_prev (const GArray *array, +GCompareFunc compare_func, +const VhostDMAMap *map, +const VhostDMAMap **prev) +{ +size_t i; +int r; + + *prev =NULL ; +for (i =0 ; i < array->len; ++i) { + r =compare_func (map, &g_array_index (array,typeof (*map), i)); +if (r ==0 ) { +return VHOST_DMA_MAP_OVERLAP ; + } +if (r <0 ) { +return VHOST_DMA_MAP_OK ; + } + + *prev = &g_array_index (array,typeof (**prev), i); + } + +return VHOST_DMA_MAP_OK ; +} + +/** +* Create a new IOVA tree +*
Quoted 9 lines
hw/virtio/vhost-iova-tree.c line 123
Quoted 4 lines
hw/virtio/vhost-iova-tree.c new file +4
-0+* Returns the new IOVA tree +*/ +VhostIOVATree *vhost_iova_tree_new (void ) +{So I think it needs to be initialized with the range we get from
get_iova_range().
Right, it is done that way for the next revision.
Quoted 124 lines
Thanks
Quoted 120 lines
hw/virtio/vhost-iova-tree.c new file +107
-0+VhostIOVATree *tree =g_new (VhostIOVATree,1 ); + tree->iova_taddr_map =g_array_new (G_ARRAY_NOT_ZERO_TERMINATED , +G_ARRAY_NOT_CLEAR_ON_ALLOC , +sizeof (VhostDMAMap)); + tree->taddr_iova_map =g_array_new (G_ARRAY_NOT_ZERO_TERMINATED , +G_ARRAY_NOT_CLEAR_ON_ALLOC , +sizeof (VhostDMAMap)); +return tree; +} + +/** +* Destroy an IOVA tree +* +* @tree The iova tree +*/ +void vhost_iova_tree_unref (VhostIOVATree *tree) +{ +g_array_unref (g_steal_pointer (&tree->iova_taddr_map)); +g_array_unref (g_steal_pointer (&tree->taddr_iova_map)); +} + +/** +* Find the IOVA address stored from a memory address +* +* @tree The iova tree +* @map The map with the memory address +* +* Return the stored mapping, or NULL if not found. +*/ +const VhostDMAMap *vhost_iova_tree_find_iova (const VhostIOVATree *tree, +const VhostDMAMap *map) +{ +/* +* This can be replaced with g_array_binary_search (Since glib 2.62) when +* that version become common enough. +*/ +return bsearch (map, tree->taddr_iova_map->data, tree->taddr_iova_map->len, +sizeof (*map), vhost_iova_tree_cmp_taddr); +} + +static bool vhost_iova_tree_find_iova_hole (const GArray *iova_map, +const VhostDMAMap *map, +const VhostDMAMap **prev_elem) +{ +size_t i; +hwaddr iova = iova_min; + + *prev_elem =NULL ; +for (i =0 ; i < iova_map->len; i++) { +const VhostDMAMap *next = &g_array_index (iova_map,typeof (*next), i); +hwaddr hole_end = next->iova; +if (map->size < hole_end - iova) { +return true; + } + + iova = next->iova + next->size +1 ; + *prev_elem = next; + } + +return ((hwaddr)-1 - iova) > iova_map->len; +} + +/** +* Allocate a new mapping +* +* @tree The iova tree +* @map The iova map +* +* Returns: +* - VHOST_DMA_MAP_OK if the map fits in the container +* - VHOST_DMA_MAP_INVALID if the map does not make sense (like size overflow) +* - VHOST_DMA_MAP_OVERLAP if the tree already contains that map +* - VHOST_DMA_MAP_NO_SPACE if iova_rm cannot allocate more space. +* +* It returns assignated iova in map->iova if return value is VHOST_DMA_MAP_OK. +*/ +VhostDMAMapNewRC vhost_iova_tree_alloc (VhostIOVATree *tree, +VhostDMAMap *map) +{ +const VhostDMAMap *qemu_prev, *iova_prev; +int find_prev_rc; +bool fit; + +if (map->translated_addr + map->size < map->translated_addr || + map->iova + map->size < map->iova || map->perm ==IOMMU_NONE ) { +return VHOST_DMA_MAP_INVALID ; + } + +/* Search for a hole in iova space big enough */ + fit =vhost_iova_tree_find_iova_hole (tree->iova_taddr_map, map, + &iova_prev); +if (!fit) { +return VHOST_DMA_MAP_NO_SPACE ; + } + + map->iova = iova_prev ? (iova_prev->iova + iova_prev->size) +1 : iova_min; + find_prev_rc =vhost_iova_tree_find_prev (tree->taddr_iova_map, + vhost_iova_tree_cmp_taddr, map, + &qemu_prev); +if (find_prev_rc ==VHOST_DMA_MAP_OVERLAP ) { +return VHOST_DMA_MAP_OVERLAP ; + } + +vhost_iova_tree_insert_after (tree->iova_taddr_map, iova_prev, map); +vhost_iova_tree_insert_after (tree->taddr_iova_map, qemu_prev, map); +return VHOST_DMA_MAP_OK ; +}hw/virtio/meson.build +1
-1@@ -11,7 +11,7 @@ softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('vhost-stub.c')) virtio_ss = ss.source_set() virtio_ss.add(files('virtio.c')) -virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c', 'vhost-shadow-virtqueue.c')) +virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c', 'vhost-shadow-virtqueue.c', 'vhost-iova-tree.c')) virtio_ss.add(when: 'CONFIG_VHOST_USER', if_true: files('vhost-user.c')) virtio_ss.add(when: 'CONFIG_VHOST_VDPA', if_true: files('vhost-vdpa.c')) virtio_ss.add(when: 'CONFIG_VIRTIO_BALLOON', if_true: files('virtio-balloon.c'))
Eugenio Perez Martin
maintainerRe: [RFC PATCH v4 18/20] vhost: Add VhostIOVATree
In reply to Jason WangQuoted 12 lines · Jason Wang
Quoted 10 lines · Jason Wang
Quoted 5 lines · Eugenio Pérez
This tree is able to look for a translated address from an IOVA address.
At first glance is similar to util/iova-tree. However, SVQ working on devices with limited IOVA space need more capabilities, like allocating IOVA chunks or perform reverse translations (qemu addresses to iova).
Quoted 7 lines
Quoted 2 lines
I don't see any reverse translation is used in the shadow code. Or anything I missed?
Ok, it looks to me that it is used in the iova allocator. But I think it's better to decouple it to an independent allocator instead of vhost iova tree.
Reverse translation is used every time a buffer is made available, since buffers content are not copied, only the descriptors to SVQ vring.
At this point all the limits are copied to vhost iova tree in the next revision I will send, defined at its creation at vhost_iova_tree_new(). They are outside of util/iova-tree, only sent to the latter at allocation time.
Since vhost_iova_tree has its own vhost_iova_tree_alloc(), that wraps the iova_tree_alloc() [1], limits could be kept in vhost-vdpa and make them an argument of vhost_iova_tree_alloc. But I'm not sure if it's what you are proposing or I'm missing something.
Either way, I think it is harder to talk about this specific case without code, since this one still does not address the limits. Would you prefer me to send another RFC in WIP quality, with *not* all comments addressed? I would say that there is not a lot of pending work to send the next one, but it might be easier for all of us.
Thanks!
[1] This util/iova-tree method will be proposed in the next series, and vhost_iova_tree wraps it since it needs to keep in sync both
trees: iova->qemu vaddr for iova allocation and the reverse one to translate available buffers.
Quoted 2 lines
Thanks