Skip to contentSkip to conversation

Support VIRTIO_F_RING_RESET for virtio-net, vhost-user, vhost-kernel in virtio pci-modern

PATCH In review Kangjie Xu 4 participants 77 messages Aug 16, 2022 – Aug 25, 2022 Networking

lore
Share & tools
lore
author
#15 lore raw

PATCH v2 7/24 virtio-pci: support queue enable

In reply to Kangjie Xu

PCI devices support vq enable.

Based on this function, the driver can re-enable the virtqueue after the virtqueue is reset.

---
 hw/virtio/virtio-pci.c | 1 +
 1 file changed, 1 insertion(+)
hw/virtio/virtio-pci.c +1 -0
index ec8e92052f..3d560e45ad 100644
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
                        proxy->vqs[vdev->queue_sel].avail[0],
                        ((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 |
                        proxy->vqs[vdev->queue_sel].used[0]);
+            virtio_queue_enable(vdev, vdev->queue_sel);
             proxy->vqs[vdev->queue_sel].enabled = 1;
             proxy->vqs[vdev->queue_sel].reset = 0;
         } else {
Signature · 1 line
2.32.0
author
#4 lore raw

PATCH v2 3/24 virtio: introduce virtio_queue_reset()

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

Introduce a new interface function virtio_queue_reset() to implement reset for vq.

Add a new callback to VirtioDeviceClass for queue reset operation for each child device.

---
 hw/virtio/virtio.c         | 11 +++++++++++
 include/hw/virtio/virtio.h |  2 ++
 2 files changed, 13 insertions(+)
hw/virtio/virtio.c +11 -0
index 67d54832a9..0e9d41366f 100644
@@ -2039,6 +2039,17 @@ static void __virtio_queue_reset(VirtIODevice *vdev, uint32_t i)
     virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
 }
 
+void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+
+    if (k->queue_reset) {
+        k->queue_reset(vdev, queue_index);
+    }
+
+    __virtio_queue_reset(vdev, queue_index);
+}
+
 void virtio_reset(void *opaque)
 {
     VirtIODevice *vdev = opaque;
include/hw/virtio/virtio.h +2 -0
index db1c0ddf6b..879394299b 100644
@@ -130,6 +130,7 @@ struct VirtioDeviceClass {
     void (*set_config)(VirtIODevice *vdev, const uint8_t *config);
     void (*reset)(VirtIODevice *vdev);
     void (*set_status)(VirtIODevice *vdev, uint8_t val);
+    void (*queue_reset)(VirtIODevice *vdev, uint32_t queue_index);
     /* For transitional devices, this is a bitmap of features
      * that are only exposed on the legacy interface but not
      * the modern one.
@@ -268,6 +269,7 @@ int virtio_queue_set_host_notifier_mr(VirtIODevice *vdev, int n,
                                       MemoryRegion *mr, bool assign);
 int virtio_set_status(VirtIODevice *vdev, uint8_t val);
 void virtio_reset(void *opaque);
+void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
 void virtio_update_irq(VirtIODevice *vdev);
 int virtio_set_features(VirtIODevice *vdev, uint64_t val);
 
Signature · 1 line
2.32.0
author
#6 lore raw

PATCH v2 4/24 virtio: introduce virtio_queue_enable()

In reply to Kangjie Xu

Introduce the interface queue_enable() in VirtioDeviceClass and the fucntion virtio_queue_enable() in virtio, it can be called when VIRTIO_PCI_COMMON_Q_ENABLE is written and related virtqueue can be started. It only supports the devices of virtio 1 or later. The not-supported devices can only start the virtqueue when DRIVER_OK.

---
 hw/virtio/virtio.c         | 14 ++++++++++++++
 include/hw/virtio/virtio.h |  2 ++
 2 files changed, 16 insertions(+)
hw/virtio/virtio.c +14 -0
index 0e9d41366f..141f18c633 100644
@@ -2050,6 +2050,20 @@ void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
     __virtio_queue_reset(vdev, queue_index);
 }
 
+void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+
+    if (!virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1)) {
+        error_report("queue_enable is only suppported in devices of virtio "
+                     "1.0 or later.");
+    }
+
+    if (k->queue_enable) {
+        k->queue_enable(vdev, queue_index);
+    }
+}
+
 void virtio_reset(void *opaque)
 {
     VirtIODevice *vdev = opaque;
include/hw/virtio/virtio.h +2 -0
index 879394299b..085997d8f3 100644
@@ -131,6 +131,7 @@ struct VirtioDeviceClass {
     void (*reset)(VirtIODevice *vdev);
     void (*set_status)(VirtIODevice *vdev, uint8_t val);
     void (*queue_reset)(VirtIODevice *vdev, uint32_t queue_index);
+    void (*queue_enable)(VirtIODevice *vdev, uint32_t queue_index);
     /* For transitional devices, this is a bitmap of features
      * that are only exposed on the legacy interface but not
      * the modern one.
@@ -270,6 +271,7 @@ int virtio_queue_set_host_notifier_mr(VirtIODevice *vdev, int n,
 int virtio_set_status(VirtIODevice *vdev, uint8_t val);
 void virtio_reset(void *opaque);
 void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
+void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index);
 void virtio_update_irq(VirtIODevice *vdev);
 int virtio_set_features(VirtIODevice *vdev, uint64_t val);
 
Signature · 1 line
2.32.0
author
#41 lore raw

PATCH v2 13/24 docs: vhost-user: add VHOST_USER_RESET_VRING message

In reply to Kangjie Xu

To support the reset operation for an individual virtqueue, we introduce a new message VHOST_USER_RESET_VRING. This message is submitted by the front-end to reset an individual virtqueue to initial states in the back-end. The reply is needed to ensure that the reset operation is complete.

---
 docs/interop/vhost-user.rst | 10 ++++++++++
 1 file changed, 10 insertions(+)
docs/interop/vhost-user.rst +10 -0
index 3f18ab424e..ce7991b9d3 100644
@@ -1422,6 +1422,16 @@ Front-end message types
   query the back-end for its device status as defined in the Virtio
   specification.
 
+``VHOST_USER_RESET_VRING``
+  :id: 41
+  :equivalent ioctl: N/A
+  :request payload: vring state description
+  :reply payload: ``u64``
+
+  When the feature ``VIRTIO_F_RING_RESET`` feature has been successfully
+  negotiated, this message is submitted by the front-end to reset an
+  individual virtqueue to initial states in the back-end. It will ask
+  for a reply to ensure the virtqueue is successfully reset in the back-end.
 
 Back-end message types
 ----------------------
Signature · 1 line
2.32.0
author
#1 lore raw

PATCH v2 1/24 virtio: sync relevant definitions with linux

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

This is updated using scripts/update-linux-headers.sh.

Added VIRTIO_F_RING_RESET, VIRTIO_PCI_COMMON_Q_RESET. It came from here:
https://github.com/oasis-tcs/virtio-spec/issues/124
https://github.com/oasis-tcs/virtio-spec/issues/139

Add VIRTIO_PCI_COMMON_Q_NDATA, which comes from here:
https://github.com/oasis-tcs/virtio-spec/issues/89

---
 include/standard-headers/linux/virtio_config.h | 5 +++++
 include/standard-headers/linux/virtio_pci.h    | 2 ++
 2 files changed, 7 insertions(+)
include/standard-headers/linux/virtio_config.h +5 -0
index 7acd8d4abc..47a7eef5e4 100644
@@ -96,4 +96,9 @@
  * Does the device support Single Root I/O Virtualization?
  */
 #define VIRTIO_F_SR_IOV			37
+
+/*
+ * This feature indicates that the driver can reset a queue individually.
+ */
+#define VIRTIO_F_RING_RESET		40
 #endif /* _LINUX_VIRTIO_CONFIG_H */
include/standard-headers/linux/virtio_pci.h +2 -0
index db7a8e2fcb..be912cfc95 100644
@@ -202,6 +202,8 @@ struct virtio_pci_cfg_cap {
 #define VIRTIO_PCI_COMMON_Q_AVAILHI	44
 #define VIRTIO_PCI_COMMON_Q_USEDLO	48
 #define VIRTIO_PCI_COMMON_Q_USEDHI	52
+#define VIRTIO_PCI_COMMON_Q_NDATA	56
+#define VIRTIO_PCI_COMMON_Q_RESET	58
 
 #endif /* VIRTIO_PCI_NO_MODERN */
 
Signature · 1 line
2.32.0
author
#11 lore raw

PATCH v2 6/24 virtio-pci: support queue reset

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

PCI devices support vq reset.

Based on this function, the driver can adjust the size of the ring, and quickly recycle the buffer in the ring.

The migration of the virtio devices will not happen during a reset operation. This is becuase the global iothread lock is held. Migration thread also needs the lock. As a result, we do not need to migrate the reset state of VirtIOPCIQueue.

---
 hw/virtio/virtio-pci.c         | 19 +++++++++++++++++++
 include/hw/virtio/virtio-pci.h |  1 +
 2 files changed, 20 insertions(+)
hw/virtio/virtio-pci.c +19 -0
index 45327f0b31..ec8e92052f 100644
@@ -1246,6 +1246,9 @@ static uint64_t virtio_pci_common_read(void *opaque, hwaddr addr,
     case VIRTIO_PCI_COMMON_Q_USEDHI:
         val = proxy->vqs[vdev->queue_sel].used[1];
         break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        val = proxy->vqs[vdev->queue_sel].reset;
+        break;
     default:
         val = 0;
     }
@@ -1333,6 +1336,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
                        ((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 |
                        proxy->vqs[vdev->queue_sel].used[0]);
             proxy->vqs[vdev->queue_sel].enabled = 1;
+            proxy->vqs[vdev->queue_sel].reset = 0;
         } else {
             virtio_error(vdev, "wrong value for queue_enable %"PRIx64, val);
         }
@@ -1355,6 +1359,20 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
     case VIRTIO_PCI_COMMON_Q_USEDHI:
         proxy->vqs[vdev->queue_sel].used[1] = val;
         break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        if (val == 1) {
+            /*
+             * With the global iothread lock taken, the migration will not
+             * happen until the virtqueue reset is done.
+             */
+            proxy->vqs[vdev->queue_sel].reset = 1;
+
+            virtio_queue_reset(vdev, vdev->queue_sel);
+
+            proxy->vqs[vdev->queue_sel].reset = 0;
+            proxy->vqs[vdev->queue_sel].enabled = 0;
+        }
+        break;
     default:
         break;
     }
@@ -1950,6 +1968,7 @@ static void virtio_pci_reset(DeviceState *qdev)
 
     for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
         proxy->vqs[i].enabled = 0;
+        proxy->vqs[i].reset = 0;
         proxy->vqs[i].num = 0;
         proxy->vqs[i].desc[0] = proxy->vqs[i].desc[1] = 0;
         proxy->vqs[i].avail[0] = proxy->vqs[i].avail[1] = 0;
include/hw/virtio/virtio-pci.h +1 -0
index 2446dcd9ae..e9290e2b94 100644
@@ -117,6 +117,7 @@ typedef struct VirtIOPCIRegion {
 typedef struct VirtIOPCIQueue {
   uint16_t num;
   bool enabled;
+  bool reset;
   uint32_t desc[2];
   uint32_t avail[2];
   uint32_t used[2];
Signature · 1 line
2.32.0

Kangjie Xu

author
#0 lore raw

cover Support VIRTIO_F_RING_RESET for virtio-net, vhost-user, vhost-kernel in virtio pci-modern

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

    https://github.com/oasis-tcs/virtio-spec/issues/124
    https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of several parts:
1. Patches 1-7 are the basic interfaces for vq reset in virtio and virtio-pci.
2. Patches 8-12 support vq stop and vq restart for vhost-kernel.
3. Patches 13-19 support vq stop and vq restart for vhost-user.
4. Patches 20-22 support vq reset and re-enable for virtio-net.
5. Patches 23-24 enable the vq reset feature for vhost-kernel and vhost-user.

The process of virtqueue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue can be optionally restarted(re-enabled).

Since this patch set involves multiple modules and seems a bit messy, we briefly describe the calling process for different modes below.
virtio-net:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> set enabled, reset status of vq.

vhost-kernel:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_net_set_backend() [vhost]
                -> vhost_dev_virtqueue_stop()
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                -> vhost_net_set_backend()
    -> set enabled, reset status of vq.

vhost-user:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_dev_virtqueue_stop() [vhost]
                    -> vhost_user_reset_vring() [vhost-user]
                        -> send VHOST_USER_RESET_VRING to the device
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                    -> vhost_user_set_single_vring_enable [vhost-user]
                        -> send VHOST_USER_SET_VRING_ENABLE to the device
    -> set enabled, reset status of vq.

Test environment:
Host: 5.19.0-rc3 (With vq reset support)
Qemu: QEMU emulator version 7.0.50
Guest: 5.19.0-rc3 (With vq reset support)
DPDK: 22.07-rc1 (With vq reset support)
Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

The drvier can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

Guest Kernel Patch:

    https://lore.kernel.org/bpf/[email protected]/

DPDK Patch:

    https://github.com/middaywords/dpdk/compare/72206323a5dd3182b13f61b25a64abdddfee595c...eabadfac7953da66bc10ffb8284b490d09bb7ec7

Host Kernel Patch:

    https://github.com/middaywords/linux/commit/19a91e0d7167b2031e46078c6215c213b89cb2c3

Looking forward to your review and comments. Thanks.

Kangjie Xu (19):
virtio: introduce virtio_queue_enable()
virtio: core: vq reset feature negotation support
virtio-pci: support queue enable
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce vhost_dev_virtqueue_stop()
vhost: introduce vhost_dev_virtqueue_restart()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()
docs: vhost-user: add VHOST_USER_RESET_VRING message
vhost-user: introduce vhost_reset_vring() interface
vhost-user: add op to enable or disable a single vring
vhost: vhost-user: update vhost_dev_virtqueue_stop()
vhost: vhost-user: update vhost_dev_virtqueue_restart()
vhost-net: vhost-user: update vhost_net_virtqueue_stop()
vhost-net: vhost-user: update vhost_net_virtqueue_restart()
virtio-net: introduce flush_or_purge_queued_packets()
virtio-net: support queue_enable
vhost: vhost-kernel: enable vq reset feature
vhost: vhost-user: enable vq reset feature

Xuan Zhuo (5):
virtio: sync relevant definitions with linux
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

 docs/interop/vhost-user.rst                   | 10 +++
 hw/core/machine.c                             |  1 +
 hw/net/vhost_net.c                            | 79 +++++++++++++++++++
 hw/net/virtio-net.c                           | 58 ++++++++++++--
 hw/virtio/vhost-user.c                        | 67 ++++++++++++++--
 hw/virtio/vhost.c                             | 79 +++++++++++++++++--
 hw/virtio/virtio-pci.c                        | 20 +++++
 hw/virtio/virtio.c                            | 62 +++++++++++----
 include/hw/virtio/vhost-backend.h             |  6 ++
 include/hw/virtio/vhost.h                     |  5 ++
 include/hw/virtio/virtio-pci.h                |  1 +
 include/hw/virtio/virtio.h                    |  8 +-
 include/net/vhost_net.h                       |  4 +
 .../standard-headers/linux/virtio_config.h    |  5 ++
 include/standard-headers/linux/virtio_pci.h   |  2 +
 15 files changed, 371 insertions(+), 36 deletions(-)
Signature · 1 line
2.32.0
author
#67 lore raw

PATCH v2 23/24 vhost: vhost-kernel: enable vq reset feature

Add virtqueue reset feature for vhost-kernel.

---
 hw/net/vhost_net.c | 1 +
 1 file changed, 1 insertion(+)
hw/net/vhost_net.c +1 -0
index 778081e54a..9ea160e4e9 100644
@@ -46,6 +46,7 @@ static const int kernel_feature_bits[] = {
     VIRTIO_NET_F_MTU,
     VIRTIO_F_IOMMU_PLATFORM,
     VIRTIO_F_RING_PACKED,
+    VIRTIO_F_RING_RESET,
     VIRTIO_NET_F_HASH_REPORT,
     VHOST_INVALID_FEATURE_BIT
 };
Signature · 1 line
2.32.0
author
#22 lore raw

PATCH v2 8/24 vhost: extract the logic of unmapping the vrings and desc

In reply to Kangjie Xu

Introduce vhost_virtqueue_unmap() to ummap the vrings and desc of a virtqueue.

The function will be used later.

---
 hw/virtio/vhost.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
hw/virtio/vhost.c +14 -6
index 0827d631c0..e467dfc7bc 100644
@@ -1197,6 +1197,19 @@ fail_alloc_desc:
     return r;
 }
 
+static void vhost_virtqueue_unmap(struct vhost_dev *dev,
+                                  struct VirtIODevice *vdev,
+                                  struct vhost_virtqueue *vq,
+                                  unsigned idx)
+{
+    vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
+                       1, virtio_queue_get_used_size(vdev, idx));
+    vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
+                       0, virtio_queue_get_avail_size(vdev, idx));
+    vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
+                       0, virtio_queue_get_desc_size(vdev, idx));
+}
+
 static void vhost_virtqueue_stop(struct vhost_dev *dev,
                                     struct VirtIODevice *vdev,
                                     struct vhost_virtqueue *vq,
@@ -1235,12 +1248,7 @@ static void vhost_virtqueue_stop(struct vhost_dev *dev,
                                                 vhost_vq_index);
     }
 
-    vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
-                       1, virtio_queue_get_used_size(vdev, idx));
-    vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
-                       0, virtio_queue_get_avail_size(vdev, idx));
-    vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
-                       0, virtio_queue_get_desc_size(vdev, idx));
+    vhost_virtqueue_unmap(dev, vdev, vq, idx);
 }
 
 static void vhost_eventfd_add(MemoryListener *listener,
Signature · 1 line
2.32.0
author
#8 lore raw

PATCH v2 5/24 virtio: core: vq reset feature negotation support

In reply to Kangjie Xu

A a new command line parameter "queue_reset" is added.

Meanwhile, the vq reset feature is disabled for pre-7.1 machines.

---
 hw/core/machine.c          | 1 +
 include/hw/virtio/virtio.h | 4 +++-
 2 files changed, 4 insertions(+), 1 deletion(-)
hw/core/machine.c +1 -0
index a673302cce..8b22b4647f 100644
@@ -43,6 +43,7 @@
 GlobalProperty hw_compat_7_0[] = {
     { "arm-gicv3-common", "force-8-bit-prio", "on" },
     { "nvme-ns", "eui64-default", "on"},
+    { "virtio-device", "queue_reset", "false" },
 };
 const size_t hw_compat_7_0_len = G_N_ELEMENTS(hw_compat_7_0);
 
include/hw/virtio/virtio.h +3 -1
index 085997d8f3..ed3ecbef80 100644
@@ -295,7 +295,9 @@ typedef struct VirtIORNGConf VirtIORNGConf;
     DEFINE_PROP_BIT64("iommu_platform", _state, _field, \
                       VIRTIO_F_IOMMU_PLATFORM, false), \
     DEFINE_PROP_BIT64("packed", _state, _field, \
-                      VIRTIO_F_RING_PACKED, false)
+                      VIRTIO_F_RING_PACKED, false), \
+    DEFINE_PROP_BIT64("queue_reset", _state, _field, \
+                      VIRTIO_F_RING_RESET, true)
 
 hwaddr virtio_queue_get_desc_addr(VirtIODevice *vdev, int n);
 bool virtio_queue_enabled_legacy(VirtIODevice *vdev, int n);
Signature · 1 line
2.32.0
author
#2 lore raw

PATCH v2 2/24 virtio: introduce __virtio_queue_reset()

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

Separate the logic of vq reset. This logic will be called directly later.

---
 hw/virtio/virtio.c | 37 +++++++++++++++++++++----------------
 1 file changed, 21 insertions(+), 16 deletions(-)
hw/virtio/virtio.c +21 -16
index 5d607aeaa0..67d54832a9 100644
@@ -2019,6 +2019,26 @@ static enum virtio_device_endian virtio_current_cpu_endian(void)
     }
 }
 
+static void __virtio_queue_reset(VirtIODevice *vdev, uint32_t i)
+{
+    vdev->vq[i].vring.desc = 0;
+    vdev->vq[i].vring.avail = 0;
+    vdev->vq[i].vring.used = 0;
+    vdev->vq[i].last_avail_idx = 0;
+    vdev->vq[i].shadow_avail_idx = 0;
+    vdev->vq[i].used_idx = 0;
+    vdev->vq[i].last_avail_wrap_counter = true;
+    vdev->vq[i].shadow_avail_wrap_counter = true;
+    vdev->vq[i].used_wrap_counter = true;
+    virtio_queue_set_vector(vdev, i, VIRTIO_NO_VECTOR);
+    vdev->vq[i].signalled_used = 0;
+    vdev->vq[i].signalled_used_valid = false;
+    vdev->vq[i].notification = true;
+    vdev->vq[i].vring.num = vdev->vq[i].vring.num_default;
+    vdev->vq[i].inuse = 0;
+    virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
+}
+
 void virtio_reset(void *opaque)
 {
     VirtIODevice *vdev = opaque;
@@ -2050,22 +2070,7 @@ void virtio_reset(void *opaque)
     virtio_notify_vector(vdev, vdev->config_vector);
 
     for(i = 0; i < VIRTIO_QUEUE_MAX; i++) {
-        vdev->vq[i].vring.desc = 0;
-        vdev->vq[i].vring.avail = 0;
-        vdev->vq[i].vring.used = 0;
-        vdev->vq[i].last_avail_idx = 0;
-        vdev->vq[i].shadow_avail_idx = 0;
-        vdev->vq[i].used_idx = 0;
-        vdev->vq[i].last_avail_wrap_counter = true;
-        vdev->vq[i].shadow_avail_wrap_counter = true;
-        vdev->vq[i].used_wrap_counter = true;
-        virtio_queue_set_vector(vdev, i, VIRTIO_NO_VECTOR);
-        vdev->vq[i].signalled_used = 0;
-        vdev->vq[i].signalled_used_valid = false;
-        vdev->vq[i].notification = true;
-        vdev->vq[i].vring.num = vdev->vq[i].vring.num_default;
-        vdev->vq[i].inuse = 0;
-        virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
+        __virtio_queue_reset(vdev, i);
     }
 }
 
Signature · 1 line
2.32.0
author
#36 lore raw

PATCH v2 12/24 vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()

In reply to Kangjie Xu

Introduce vhost_net_virtqueue_restart(), which can restart the virtqueue when the vhost net started running before. If it fails to restart the virtqueue, the device will be stopped.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

---
 hw/net/vhost_net.c      | 48 +++++++++++++++++++++++++++++++++++++++++
 include/net/vhost_net.h |  2 ++
 2 files changed, 50 insertions(+)
hw/net/vhost_net.c +48 -0
index aa60dd901c..2ab67e875e 100644
@@ -535,3 +535,51 @@ void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
 
     vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
 }
+
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                                int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { };
+    int idx, r;
+
+    if (!net->dev.started) {
+        return 0;
+    }
+
+    assert(vhost_ops);
+
+    idx =  vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    r = vhost_dev_virtqueue_restart(&net->dev, vdev, idx);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        file.fd = net->backend;
+        r = vhost_net_set_backend(&net->dev, &file);
+        if (r < 0) {
+            r = -errno;
+            goto err_start;
+        }
+    }
+
+    return 0;
+
+err_start:
+    error_report("Error when restarting the queue.");
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.fd = -1;
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }
+
+    vhost_dev_stop(&net->dev, vdev);
+
+    return r;
+}
include/net/vhost_net.h +2 -0
index 9b3aaf3814..e11a297380 100644
@@ -50,4 +50,6 @@ int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
 
 void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
                               int vq_index);
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                                int vq_index);
 #endif
Signature · 1 line
2.32.0
author
#24 lore raw

PATCH v2 9/24 vhost: introduce vhost_dev_virtqueue_stop()

In reply to Kangjie Xu

Introduce vhost_dev_virtqueue_stop(), which can ummap the vrings and the desc of it.

---
 hw/virtio/vhost.c         | 9 +++++++++
 include/hw/virtio/vhost.h | 3 +++
 2 files changed, 12 insertions(+)
hw/virtio/vhost.c +9 -0
index e467dfc7bc..1bca9ff48d 100644
@@ -1904,3 +1904,12 @@ int vhost_net_set_backend(struct vhost_dev *hdev,
 
     return -ENOSYS;
 }
+
+void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
+                              int idx)
+{
+    vhost_virtqueue_unmap(hdev,
+                          vdev,
+                          hdev->vqs + idx,
+                          idx);
+}
include/hw/virtio/vhost.h +3 -0
index a346f23d13..574888440c 100644
@@ -288,4 +288,7 @@ int vhost_dev_set_inflight(struct vhost_dev *dev,
                            struct vhost_inflight *inflight);
 int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
                            struct vhost_inflight *inflight);
+
+void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
+                              int idx);
 #endif
Signature · 1 line
2.32.0
author
#28 lore raw

PATCH v2 10/24 vhost: introduce vhost_dev_virtqueue_restart()

In reply to Kangjie Xu

Introduce vhost_dev_virtqueue_restart(), which can restart the virtqueue when the vhost has already started running.

---
 hw/virtio/vhost.c         | 13 +++++++++++++
 include/hw/virtio/vhost.h |  2 ++
 2 files changed, 15 insertions(+)
hw/virtio/vhost.c +13 -0
index 1bca9ff48d..fc3f550c76 100644
@@ -1913,3 +1913,16 @@ void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                           hdev->vqs + idx,
                           idx);
 }
+
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                int idx)
+{
+    const VhostOps *vhost_ops = hdev->vhost_ops;
+
+    assert(vhost_ops);
+
+    return vhost_virtqueue_start(hdev,
+                                 vdev,
+                                 hdev->vqs + idx,
+                                 hdev->vq_index + idx);
+}
include/hw/virtio/vhost.h +2 -0
index 574888440c..b3394b6348 100644
@@ -291,4 +291,6 @@ int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
 
 void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                               int idx);
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                int idx);
 #endif
Signature · 1 line
2.32.0
author
#31 lore raw

PATCH v2 11/24 vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()

In reply to Kangjie Xu

Introduce vhost_virtqueue_stop(), which can reset the virtqueue in the device. Then it will unmap vrings and the desc of the virtqueue.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

---
 hw/net/vhost_net.c      | 21 +++++++++++++++++++++
 include/net/vhost_net.h |  2 ++
 2 files changed, 23 insertions(+)
hw/net/vhost_net.c +21 -0
index ccac5b7a64..aa60dd901c 100644
@@ -514,3 +514,24 @@ int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu)
 
     return vhost_ops->vhost_net_set_mtu(&net->dev, mtu);
 }
+
+void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { .fd = -1 };
+    int idx;
+
+    assert(vhost_ops);
+
+    idx = vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }
+
+    vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
+}
include/net/vhost_net.h +2 -0
index 387e913e4e..9b3aaf3814 100644
@@ -48,4 +48,6 @@ uint64_t vhost_net_get_acked_features(VHostNetState *net);
 
 int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
 
+void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index);
 #endif
Signature · 1 line
2.32.0
author
#54 lore raw

PATCH v2 17/24 vhost: vhost-user: update vhost_dev_virtqueue_restart()

In reply to Kangjie Xu

Update vhost_dev_virtqueue_restart() for vhost-user scenario.

---
 hw/virtio/vhost.c | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)
hw/virtio/vhost.c +22 -4
index a0d6824353..bd90cfe13a 100644
@@ -1937,11 +1937,29 @@ int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
                                 int idx)
 {
     const VhostOps *vhost_ops = hdev->vhost_ops;
+    int r;
 
     assert(vhost_ops);
 
-    return vhost_virtqueue_start(hdev,
-                                 vdev,
-                                 hdev->vqs + idx,
-                                 hdev->vq_index + idx);
+    r = vhost_virtqueue_start(hdev,
+                              vdev,
+                              hdev->vqs + idx,
+                              hdev->vq_index + idx);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    if (vhost_ops->vhost_set_single_vring_enable) {
+        r = vhost_ops->vhost_set_single_vring_enable(hdev,
+                                                     hdev->vq_index + idx,
+                                                     1);
+        if (r < 0) {
+            goto err_start;
+        }
+    }
+
+    return 0;
+
+err_start:
+    return r;
 }
Signature · 1 line
2.32.0
author
#46 lore raw

PATCH v2 15/24 vhost-user: add op to enable or disable a single vring

In reply to Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface.

---
 hw/virtio/vhost-user.c            | 26 +++++++++++++++++++-------
 include/hw/virtio/vhost-backend.h |  3 +++
 2 files changed, 22 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +19 -7
index 56033f7a92..8307976cda 100644
@@ -1199,6 +1199,22 @@ static int vhost_user_set_vring_base(struct vhost_dev *dev,
     return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
 }
 
+static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable)
+{
+    if (index < dev->vq_index || index >= dev->vq_index + dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    return vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, &state);
+}
+
 static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
 {
     int i;
@@ -1208,13 +1224,8 @@ static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
     }
 
     for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, &state);
+        int ret = vhost_user_set_single_vring_enable(dev, dev->vq_index + i,
+                                                     enable);
         if (ret < 0) {
             /*
              * Restoring the previous state is likely infeasible, as well as
@@ -2668,6 +2679,7 @@ const VhostOps user_ops = {
         .vhost_reset_vring = vhost_user_reset_vring,
         .vhost_reset_device = vhost_user_reset_device,
         .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable = vhost_user_set_single_vring_enable,
         .vhost_set_vring_enable = vhost_user_set_vring_enable,
         .vhost_requires_shm_log = vhost_user_requires_shm_log,
         .vhost_migration_done = vhost_user_migration_done,
include/hw/virtio/vhost-backend.h +3 -0
index f23bf71a8d..38f6b752ff 100644
@@ -83,6 +83,8 @@ typedef int (*vhost_reset_vring_op)(struct vhost_dev *dev,
                                     struct vhost_vring_state *ring);
 typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
 typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable);
 typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                          int enable);
 typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);
@@ -158,6 +160,7 @@ typedef struct VhostOps {
     vhost_reset_device_op vhost_reset_device;
     vhost_reset_vring_op vhost_reset_vring;
     vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
     vhost_set_vring_enable_op vhost_set_vring_enable;
     vhost_requires_shm_log_op vhost_requires_shm_log;
     vhost_migration_done_op vhost_migration_done;
Signature · 1 line
2.32.0
author
#43 lore raw

PATCH v2 14/24 vhost-user: introduce vhost_reset_vring() interface

In reply to Kangjie Xu

Introduce the interface vhost_reset_vring(). The interface is a wrapper to send a VHOST_USER_RESET_VRING message to the back-end. It will reset an individual vring in the back-end. Meanwhile, it will wait for a reply to ensure the reset has been completed.

---
 hw/virtio/vhost-user.c            | 41 +++++++++++++++++++++++++++++++
 include/hw/virtio/vhost-backend.h |  3 +++
 2 files changed, 44 insertions(+)
hw/virtio/vhost-user.c +41 -0
index 75b8df21a4..56033f7a92 100644
@@ -126,6 +126,7 @@ typedef enum VhostUserRequest {
     VHOST_USER_GET_MAX_MEM_SLOTS = 36,
     VHOST_USER_ADD_MEM_REG = 37,
     VHOST_USER_REM_MEM_REG = 38,
+    VHOST_USER_RESET_VRING = 41,
     VHOST_USER_MAX
 } VhostUserRequest;
 
@@ -1498,6 +1499,45 @@ static int vhost_user_get_max_memslots(struct vhost_dev *dev,
     return 0;
 }
 
+static int vhost_user_reset_vring(struct vhost_dev *dev,
+                                  struct vhost_vring_state *ring)
+{
+    int ret;
+    VhostUserMsg msg = {
+        .hdr.request = VHOST_USER_RESET_VRING,
+        .hdr.flags = VHOST_USER_VERSION,
+        .payload.state = *ring,
+        .hdr.size = sizeof(msg.payload.state),
+    };
+
+    if (!virtio_has_feature(dev->acked_features, VIRTIO_F_RING_RESET)) {
+        return -ENOTSUP;
+    }
+
+    ret = vhost_user_write(dev, &msg, NULL, 0);
+    if (ret < 0) {
+        return ret;
+    }
+
+    ret = vhost_user_read(dev, &msg);
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (msg.hdr.request != VHOST_USER_RESET_VRING) {
+        error_report("Received unexpected msg type. Expected %d received %d",
+                     VHOST_USER_RESET_VRING, msg.hdr.request);
+        return -EPROTO;
+    }
+
+    if (msg.hdr.size != sizeof(msg.payload.state)) {
+        error_report("Received bad msg size.");
+        return -EPROTO;
+    }
+
+    return 0;
+}
+
 static int vhost_user_reset_device(struct vhost_dev *dev)
 {
     VhostUserMsg msg = {
@@ -2625,6 +2665,7 @@ const VhostOps user_ops = {
         .vhost_set_features = vhost_user_set_features,
         .vhost_get_features = vhost_user_get_features,
         .vhost_set_owner = vhost_user_set_owner,
+        .vhost_reset_vring = vhost_user_reset_vring,
         .vhost_reset_device = vhost_user_reset_device,
         .vhost_get_vq_index = vhost_user_get_vq_index,
         .vhost_set_vring_enable = vhost_user_set_vring_enable,
include/hw/virtio/vhost-backend.h +3 -0
index eab46d7f0b..f23bf71a8d 100644
@@ -79,6 +79,8 @@ typedef int (*vhost_get_features_op)(struct vhost_dev *dev,
                                      uint64_t *features);
 typedef int (*vhost_set_backend_cap_op)(struct vhost_dev *dev);
 typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
+typedef int (*vhost_reset_vring_op)(struct vhost_dev *dev,
+                                    struct vhost_vring_state *ring);
 typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
 typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
 typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
@@ -154,6 +156,7 @@ typedef struct VhostOps {
     vhost_set_backend_cap_op vhost_set_backend_cap;
     vhost_set_owner_op vhost_set_owner;
     vhost_reset_device_op vhost_reset_device;
+    vhost_reset_vring_op vhost_reset_vring;
     vhost_get_vq_index_op vhost_get_vq_index;
     vhost_set_vring_enable_op vhost_set_vring_enable;
     vhost_requires_shm_log_op vhost_requires_shm_log;
Signature · 1 line
2.32.0
author
#66 lore raw

PATCH v2 22/24 virtio-net: support queue_enable

In reply to Kangjie Xu

virtio-net: support queue_enable for vhost

Support queue_enable in vhost scenario. It can be called when a vq reset operation is performed and the vq will be restared.

It should be noted that we can restart the vq when the vhost has already started. When launching a new vhost-user device, the vhost is not started and all vqs are not initalized until VIRTIO_PCI_COMMON_STATUS is written. Thus, we should use vhost_started to differentiate the two cases: vq reset and device start.

Currently it only supports vhost-user and vhost-kernel.

---
 hw/net/virtio-net.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
hw/net/virtio-net.c +22 -0
index e2989487ec..ba216f7b84 100644
@@ -558,6 +558,27 @@ static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
     flush_or_purge_queued_packets(nc);
 }
 
+static void virtio_net_queue_enable(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+    int r;
+
+    if (!nc->peer || !vdev->vhost_started) {
+        return;
+    }
+
+    if (get_vhost_net(nc->peer) &&
+        ((nc->peer->info->type == NET_CLIENT_DRIVER_TAP) ||
+         (nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER))) {
+        r = vhost_net_virtqueue_restart(vdev, nc, queue_index);
+        if (r < 0) {
+            error_report("unable to restart vhost net virtqueue: %d, "
+                            "when resetting the queue", queue_index);
+        }
+    }
+}
+
 static void virtio_net_reset(VirtIODevice *vdev)
 {
     VirtIONet *n = VIRTIO_NET(vdev);
@@ -3803,6 +3824,7 @@ static void virtio_net_class_init(ObjectClass *klass, void *data)
     vdc->bad_features = virtio_net_bad_features;
     vdc->reset = virtio_net_reset;
     vdc->queue_reset = virtio_net_queue_reset;
+    vdc->queue_enable = virtio_net_queue_enable;
     vdc->set_status = virtio_net_set_status;
     vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
     vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
Signature · 1 line
2.32.0
author
#51 lore raw

PATCH v2 16/24 vhost: vhost-user: update vhost_dev_virtqueue_stop()

In reply to Kangjie Xu

Update vhost_dev_virtqueue_stop() for vhost-user scenario.

---
 hw/virtio/vhost.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
hw/virtio/vhost.c +19 -0
index fc3f550c76..a0d6824353 100644
@@ -1908,10 +1908,29 @@ int vhost_net_set_backend(struct vhost_dev *hdev,
 void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                               int idx)
 {
+    const VhostOps *vhost_ops = hdev->vhost_ops;
+    struct vhost_vring_state state;
+    int r;
+
+    assert(vhost_ops);
+
+    if (vhost_ops->vhost_reset_vring) {
+        state.index = hdev->vq_index + idx;
+        r = vhost_ops->vhost_reset_vring(hdev, &state);
+        if (r < 0) {
+            goto err_queue_reset;
+        }
+    }
+
     vhost_virtqueue_unmap(hdev,
                           vdev,
                           hdev->vqs + idx,
                           idx);
+
+    return;
+
+err_queue_reset:
+    error_report("Error when stopping the qeuue.");
 }
 
 int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
Signature · 1 line
2.32.0
author
#65 lore raw

PATCH v2 21/24 virtio-net: support queue reset

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

virtio-net, vhost-kernel, vhost-user implement queue reset. Queued packets in the corresponding queue pair are flushed or purged.

---
 hw/net/virtio-net.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
hw/net/virtio-net.c +19 -0
index 27b59c0ad6..e2989487ec 100644
@@ -540,6 +540,24 @@ static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc)
     return info;
 }
 
+static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+
+    if (!nc->peer) {
+        return;
+    }
+
+    if (get_vhost_net(nc->peer) &&
+        ((nc->peer->info->type == NET_CLIENT_DRIVER_TAP) ||
+         (nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER))) {
+        vhost_net_virtqueue_stop(vdev, nc, queue_index);
+    }
+
+    flush_or_purge_queued_packets(nc);
+}
+
 static void virtio_net_reset(VirtIODevice *vdev)
 {
     VirtIONet *n = VIRTIO_NET(vdev);
@@ -3784,6 +3802,7 @@ static void virtio_net_class_init(ObjectClass *klass, void *data)
     vdc->set_features = virtio_net_set_features;
     vdc->bad_features = virtio_net_bad_features;
     vdc->reset = virtio_net_reset;
+    vdc->queue_reset = virtio_net_queue_reset;
     vdc->set_status = virtio_net_set_status;
     vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
     vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
Signature · 1 line
2.32.0
author
#57 lore raw

PATCH v2 18/24 vhost-net: vhost-user: update vhost_net_virtqueue_stop()

In reply to Kangjie Xu

Update vhost_net_virtqueue_stop() for vhost-user scenario.

---
 hw/net/vhost_net.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/net/vhost_net.c +4 -0
index 2ab67e875e..c0d408f3b4 100644
@@ -533,6 +533,10 @@ void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
         assert(r >= 0);
     }
 
+    if (net->nc->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        idx = idx - net->dev.vq_index;
+    }
+
     vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
 }
 
Signature · 1 line
2.32.0
author
#68 lore raw

PATCH v2 24/24 vhost: vhost-user: enable vq reset feature

In reply to Kangjie Xu

Add virtqueue reset feature for vhost-user.

---
 hw/net/vhost_net.c | 1 +
 1 file changed, 1 insertion(+)
hw/net/vhost_net.c +1 -0
index 9ea160e4e9..5bc89f2842 100644
@@ -74,6 +74,7 @@ static const int user_feature_bits[] = {
     VIRTIO_NET_F_MTU,
     VIRTIO_F_IOMMU_PLATFORM,
     VIRTIO_F_RING_PACKED,
+    VIRTIO_F_RING_RESET,
     VIRTIO_NET_F_RSS,
     VIRTIO_NET_F_HASH_REPORT,
 
Signature · 1 line
2.32.0
author
#64 lore raw

PATCH v2 20/24 virtio-net: introduce flush_or_purge_queued_packets()

In reply to Kangjie Xu

Introduce the fucntion flush_or_purge_queued_packets(), it will be used in device reset and virtqueue reset. Therefore, we extract the common logic as a new function.

---
 hw/net/virtio-net.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
hw/net/virtio-net.c +11 -6
index dd0d056fde..27b59c0ad6 100644
@@ -118,6 +118,16 @@ static int vq2q(int queue_index)
     return queue_index / 2;
 }
 
+static void flush_or_purge_queued_packets(NetClientState *nc)
+{
+    if (!nc->peer) {
+        return;
+    }
+
+    qemu_flush_or_purge_queued_packets(nc->peer, true);
+    assert(!virtio_net_get_subqueue(nc)->async_tx.elem);
+}
+
 /* TODO
  * - we could suppress RX interrupt if we were so inclined.
  */
@@ -560,12 +570,7 @@ static void virtio_net_reset(VirtIODevice *vdev)
 
     /* Flush any async TX */
     for (i = 0;  i < n->max_queue_pairs; i++) {
-        NetClientState *nc = qemu_get_subqueue(n->nic, i);
-
-        if (nc->peer) {
-            qemu_flush_or_purge_queued_packets(nc->peer, true);
-            assert(!virtio_net_get_subqueue(nc)->async_tx.elem);
-        }
+        flush_or_purge_queued_packets(qemu_get_subqueue(n->nic, i));
     }
 }
 
Signature · 1 line
2.32.0
author
#62 lore raw

PATCH v2 19/24 vhost-net: vhost-user: update vhost_net_virtqueue_restart()

In reply to Kangjie Xu

Update vhost_net_virtqueue_restart() for vhost-user scenario.

---
 hw/net/vhost_net.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/net/vhost_net.c +4 -0
index c0d408f3b4..778081e54a 100644
@@ -556,6 +556,10 @@ int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
 
     idx =  vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
 
+    if (net->nc->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        idx = idx - net->dev.vq_index;
+    }
+
     r = vhost_dev_virtqueue_restart(&net->dev, vdev, idx);
     if (r < 0) {
         goto err_start;
Signature · 1 line
2.32.0
#69 lore raw
In reply to Kangjie Xu
Quoted 73 lines · Kangjie Xu

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

    https://github.com/oasis-tcs/virtio-spec/issues/124
    https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of several parts:
1. Patches 1-7 are the basic interfaces for vq reset in virtio and virtio-pci.
2. Patches 8-12 support vq stop and vq restart for vhost-kernel.
3. Patches 13-19 support vq stop and vq restart for vhost-user.
4. Patches 20-22 support vq reset and re-enable for virtio-net.
5. Patches 23-24 enable the vq reset feature for vhost-kernel and vhost-user.

The process of virtqueue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue can be optionally restarted(re-enabled).

Since this patch set involves multiple modules and seems a bit messy, we briefly describe the calling process for different modes below.
virtio-net:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> set enabled, reset status of vq.

vhost-kernel:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_net_set_backend() [vhost]
                -> vhost_dev_virtqueue_stop()
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                -> vhost_net_set_backend()
    -> set enabled, reset status of vq.

vhost-user:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_dev_virtqueue_stop() [vhost]
                    -> vhost_user_reset_vring() [vhost-user]
                        -> send VHOST_USER_RESET_VRING to the device
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                    -> vhost_user_set_single_vring_enable [vhost-user]
                        -> send VHOST_USER_SET_VRING_ENABLE to the device
    -> set enabled, reset status of vq.

Test environment:
Host: 5.19.0-rc3 (With vq reset support)
Qemu: QEMU emulator version 7.0.50
Guest: 5.19.0-rc3 (With vq reset support)
DPDK: 22.07-rc1 (With vq reset support)
Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

Quoted 5 lines

The drvier can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

legacy mode testing?

Quoted 58 lines

Guest Kernel Patch:

    https://lore.kernel.org/bpf/[email protected]/

DPDK Patch:

    https://github.com/middaywords/dpdk/compare/72206323a5dd3182b13f61b25a64abdddfee595c...eabadfac7953da66bc10ffb8284b490d09bb7ec7

Host Kernel Patch:

    https://github.com/middaywords/linux/commit/19a91e0d7167b2031e46078c6215c213b89cb2c3

Looking forward to your review and comments. Thanks.

Kangjie Xu (19):
virtio: introduce virtio_queue_enable()
virtio: core: vq reset feature negotation support
virtio-pci: support queue enable
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce vhost_dev_virtqueue_stop()
vhost: introduce vhost_dev_virtqueue_restart()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()
docs: vhost-user: add VHOST_USER_RESET_VRING message
vhost-user: introduce vhost_reset_vring() interface
vhost-user: add op to enable or disable a single vring
vhost: vhost-user: update vhost_dev_virtqueue_stop()
vhost: vhost-user: update vhost_dev_virtqueue_restart()
vhost-net: vhost-user: update vhost_net_virtqueue_stop()
vhost-net: vhost-user: update vhost_net_virtqueue_restart()
virtio-net: introduce flush_or_purge_queued_packets()
virtio-net: support queue_enable
vhost: vhost-kernel: enable vq reset feature
vhost: vhost-user: enable vq reset feature

Xuan Zhuo (5):
virtio: sync relevant definitions with linux
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

 docs/interop/vhost-user.rst                   | 10 +++
 hw/core/machine.c                             |  1 +
 hw/net/vhost_net.c                            | 79 +++++++++++++++++++
 hw/net/virtio-net.c                           | 58 ++++++++++++--
 hw/virtio/vhost-user.c                        | 67 ++++++++++++++--
 hw/virtio/vhost.c                             | 79 +++++++++++++++++--
 hw/virtio/virtio-pci.c                        | 20 +++++
 hw/virtio/virtio.c                            | 62 +++++++++++----
 include/hw/virtio/vhost-backend.h             |  6 ++
 include/hw/virtio/vhost.h                     |  5 ++
 include/hw/virtio/virtio-pci.h                |  1 +
 include/hw/virtio/virtio.h                    |  8 +-
 include/net/vhost_net.h                       |  4 +
 .../standard-headers/linux/virtio_config.h    |  5 ++
 include/standard-headers/linux/virtio_pci.h   |  2 +
 15 files changed, 371 insertions(+), 36 deletions(-)
Signature · 1 line
2.32.0
#71 lore raw
In reply to Xuan Zhuo
Quoted 81 lines · Xuan Zhuo
Quoted 80 lines · Michael S. Tsirkin
Quoted 78 lines · Kangjie Xu

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

    https://github.com/oasis-tcs/virtio-spec/issues/124
    https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of several parts:
1. Patches 1-7 are the basic interfaces for vq reset in virtio and virtio-pci.
2. Patches 8-12 support vq stop and vq restart for vhost-kernel.
3. Patches 13-19 support vq stop and vq restart for vhost-user.
4. Patches 20-22 support vq reset and re-enable for virtio-net.
5. Patches 23-24 enable the vq reset feature for vhost-kernel and vhost-user.

The process of virtqueue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue can be optionally restarted(re-enabled).

Since this patch set involves multiple modules and seems a bit messy, we briefly describe the calling process for different modes below.
virtio-net:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> set enabled, reset status of vq.

vhost-kernel:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_net_set_backend() [vhost]
                -> vhost_dev_virtqueue_stop()
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                -> vhost_net_set_backend()
    -> set enabled, reset status of vq.

vhost-user:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_dev_virtqueue_stop() [vhost]
                    -> vhost_user_reset_vring() [vhost-user]
                        -> send VHOST_USER_RESET_VRING to the device
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                    -> vhost_user_set_single_vring_enable [vhost-user]
                        -> send VHOST_USER_SET_VRING_ENABLE to the device
    -> set enabled, reset status of vq.

Test environment:
Host: 5.19.0-rc3 (With vq reset support)
Qemu: QEMU emulator version 7.0.50
Guest: 5.19.0-rc3 (With vq reset support)
DPDK: 22.07-rc1 (With vq reset support)
Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

The drvier can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

Quoted 6 lines
Quoted 1 line

legacy mode testing?

legacy does not support vq reset.

Thanks.

yes but did it break with all these code changes?

Quoted 60 lines
Quoted 60 lines
Quoted 59 lines

Guest Kernel Patch:

    https://lore.kernel.org/bpf/[email protected]/

DPDK Patch:

    https://github.com/middaywords/dpdk/compare/72206323a5dd3182b13f61b25a64abdddfee595c...eabadfac7953da66bc10ffb8284b490d09bb7ec7

Host Kernel Patch:

    https://github.com/middaywords/linux/commit/19a91e0d7167b2031e46078c6215c213b89cb2c3

Looking forward to your review and comments. Thanks.

Kangjie Xu (19):
virtio: introduce virtio_queue_enable()
virtio: core: vq reset feature negotation support
virtio-pci: support queue enable
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce vhost_dev_virtqueue_stop()
vhost: introduce vhost_dev_virtqueue_restart()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()
docs: vhost-user: add VHOST_USER_RESET_VRING message
vhost-user: introduce vhost_reset_vring() interface
vhost-user: add op to enable or disable a single vring
vhost: vhost-user: update vhost_dev_virtqueue_stop()
vhost: vhost-user: update vhost_dev_virtqueue_restart()
vhost-net: vhost-user: update vhost_net_virtqueue_stop()
vhost-net: vhost-user: update vhost_net_virtqueue_restart()
virtio-net: introduce flush_or_purge_queued_packets()
virtio-net: support queue_enable
vhost: vhost-kernel: enable vq reset feature
vhost: vhost-user: enable vq reset feature

Xuan Zhuo (5):
virtio: sync relevant definitions with linux
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

 docs/interop/vhost-user.rst                   | 10 +++
 hw/core/machine.c                             |  1 +
 hw/net/vhost_net.c                            | 79 +++++++++++++++++++
 hw/net/virtio-net.c                           | 58 ++++++++++++--
 hw/virtio/vhost-user.c                        | 67 ++++++++++++++--
 hw/virtio/vhost.c                             | 79 +++++++++++++++++--
 hw/virtio/virtio-pci.c                        | 20 +++++
 hw/virtio/virtio.c                            | 62 +++++++++++----
 include/hw/virtio/vhost-backend.h             |  6 ++
 include/hw/virtio/vhost.h                     |  5 ++
 include/hw/virtio/virtio-pci.h                |  1 +
 include/hw/virtio/virtio.h                    |  8 +-
 include/net/vhost_net.h                       |  4 +
 .../standard-headers/linux/virtio_config.h    |  5 ++
 include/standard-headers/linux/virtio_pci.h   |  2 +
 15 files changed, 371 insertions(+), 36 deletions(-)

--
2.32.0

#70 lore raw
In reply to Michael S. Tsirkin
Quoted 74 lines · Michael S. Tsirkin
Quoted 73 lines · Kangjie Xu

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

    https://github.com/oasis-tcs/virtio-spec/issues/124
    https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of several parts:
1. Patches 1-7 are the basic interfaces for vq reset in virtio and virtio-pci.
2. Patches 8-12 support vq stop and vq restart for vhost-kernel.
3. Patches 13-19 support vq stop and vq restart for vhost-user.
4. Patches 20-22 support vq reset and re-enable for virtio-net.
5. Patches 23-24 enable the vq reset feature for vhost-kernel and vhost-user.

The process of virtqueue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue can be optionally restarted(re-enabled).

Since this patch set involves multiple modules and seems a bit messy, we briefly describe the calling process for different modes below.
virtio-net:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> set enabled, reset status of vq.

vhost-kernel:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_net_set_backend() [vhost]
                -> vhost_dev_virtqueue_stop()
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                -> vhost_net_set_backend()
    -> set enabled, reset status of vq.

vhost-user:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_dev_virtqueue_stop() [vhost]
                    -> vhost_user_reset_vring() [vhost-user]
                        -> send VHOST_USER_RESET_VRING to the device
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                    -> vhost_user_set_single_vring_enable [vhost-user]
                        -> send VHOST_USER_SET_VRING_ENABLE to the device
    -> set enabled, reset status of vq.

Test environment:

    Host: 5.19.0-rc3 (With vq reset support)
    Qemu: QEMU emulator version 7.0.50
    Guest: 5.19.0-rc3 (With vq reset support)
    DPDK: 22.07-rc1 (With vq reset support)
    Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;
Quoted 7 lines
Quoted 5 lines

The drvier can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

legacy mode testing?

legacy does not support vq reset.

Thanks.

Quoted 60 lines
Quoted 59 lines

Guest Kernel Patch:

    https://lore.kernel.org/bpf/[email protected]/

DPDK Patch:

    https://github.com/middaywords/dpdk/compare/72206323a5dd3182b13f61b25a64abdddfee595c...eabadfac7953da66bc10ffb8284b490d09bb7ec7

Host Kernel Patch:

    https://github.com/middaywords/linux/commit/19a91e0d7167b2031e46078c6215c213b89cb2c3

Looking forward to your review and comments. Thanks.

Kangjie Xu (19):
virtio: introduce virtio_queue_enable()
virtio: core: vq reset feature negotation support
virtio-pci: support queue enable
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce vhost_dev_virtqueue_stop()
vhost: introduce vhost_dev_virtqueue_restart()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()
docs: vhost-user: add VHOST_USER_RESET_VRING message
vhost-user: introduce vhost_reset_vring() interface
vhost-user: add op to enable or disable a single vring
vhost: vhost-user: update vhost_dev_virtqueue_stop()
vhost: vhost-user: update vhost_dev_virtqueue_restart()
vhost-net: vhost-user: update vhost_net_virtqueue_stop()
vhost-net: vhost-user: update vhost_net_virtqueue_restart()
virtio-net: introduce flush_or_purge_queued_packets()
virtio-net: support queue_enable
vhost: vhost-kernel: enable vq reset feature
vhost: vhost-user: enable vq reset feature

Xuan Zhuo (5):
virtio: sync relevant definitions with linux
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

 docs/interop/vhost-user.rst                   | 10 +++
 hw/core/machine.c                             |  1 +
 hw/net/vhost_net.c                            | 79 +++++++++++++++++++
 hw/net/virtio-net.c                           | 58 ++++++++++++--
 hw/virtio/vhost-user.c                        | 67 ++++++++++++++--
 hw/virtio/vhost.c                             | 79 +++++++++++++++++--
 hw/virtio/virtio-pci.c                        | 20 +++++
 hw/virtio/virtio.c                            | 62 +++++++++++----
 include/hw/virtio/vhost-backend.h             |  6 ++
 include/hw/virtio/vhost.h                     |  5 ++
 include/hw/virtio/virtio-pci.h                |  1 +
 include/hw/virtio/virtio.h                    |  8 +-
 include/net/vhost_net.h                       |  4 +
 .../standard-headers/linux/virtio_config.h    |  5 ++
 include/standard-headers/linux/virtio_pci.h   |  2 +
 15 files changed, 371 insertions(+), 36 deletions(-)

--
2.32.0

#72 lore raw
In reply to Michael S. Tsirkin
Quoted 82 lines · Michael S. Tsirkin
Quoted 81 lines · Xuan Zhuo
Quoted 80 lines · Michael S. Tsirkin
Quoted 78 lines · Kangjie Xu

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

    https://github.com/oasis-tcs/virtio-spec/issues/124
    https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of several parts:
1. Patches 1-7 are the basic interfaces for vq reset in virtio and virtio-pci.
2. Patches 8-12 support vq stop and vq restart for vhost-kernel.
3. Patches 13-19 support vq stop and vq restart for vhost-user.
4. Patches 20-22 support vq reset and re-enable for virtio-net.
5. Patches 23-24 enable the vq reset feature for vhost-kernel and vhost-user.

The process of virtqueue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue can be optionally restarted(re-enabled).

Since this patch set involves multiple modules and seems a bit messy, we briefly describe the calling process for different modes below.
virtio-net:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> set enabled, reset status of vq.

vhost-kernel:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_net_set_backend() [vhost]
                -> vhost_dev_virtqueue_stop()
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                -> vhost_net_set_backend()
    -> set enabled, reset status of vq.

vhost-user:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

    -> virtio_queue_reset() [virtio]
        -> virtio_net_queue_reset() [virtio-net]
            -> vhost_net_virtqueue_stop() [vhost-net]
                -> vhost_dev_virtqueue_stop() [vhost]
                    -> vhost_user_reset_vring() [vhost-user]
                        -> send VHOST_USER_RESET_VRING to the device
                    -> vhost_virtqueue_unmap()
        -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
    -> virtio_queue_enable() [virtio]
        -> virtio_net_queue_enable() [virtio-net]
            -> vhost_net_virtqueue_restart() [vhost-net]
                -> vhost_dev_virtqueue_restart() [vhost]
                    -> vhost_virtqueue_start()
                    -> vhost_user_set_single_vring_enable [vhost-user]
                        -> send VHOST_USER_SET_VRING_ENABLE to the device
    -> set enabled, reset status of vq.

Test environment:
Host: 5.19.0-rc3 (With vq reset support)
Qemu: QEMU emulator version 7.0.50
Guest: 5.19.0-rc3 (With vq reset support)
DPDK: 22.07-rc1 (With vq reset support)
Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

The drvier can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

legacy mode testing?

legacy does not support vq reset.

Thanks.

yes but did it break with all these code changes?

OK, I see, we'll test this.

Thanks.

Quoted 62 lines
Quoted 61 lines
Quoted 61 lines
Quoted 60 lines

Guest Kernel Patch:

    https://lore.kernel.org/bpf/[email protected]/

DPDK Patch:

    https://github.com/middaywords/dpdk/compare/72206323a5dd3182b13f61b25a64abdddfee595c...eabadfac7953da66bc10ffb8284b490d09bb7ec7

Host Kernel Patch:

    https://github.com/middaywords/linux/commit/19a91e0d7167b2031e46078c6215c213b89cb2c3

Looking forward to your review and comments. Thanks.

Kangjie Xu (19):
virtio: introduce virtio_queue_enable()
virtio: core: vq reset feature negotation support
virtio-pci: support queue enable
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce vhost_dev_virtqueue_stop()
vhost: introduce vhost_dev_virtqueue_restart()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()
docs: vhost-user: add VHOST_USER_RESET_VRING message
vhost-user: introduce vhost_reset_vring() interface
vhost-user: add op to enable or disable a single vring
vhost: vhost-user: update vhost_dev_virtqueue_stop()
vhost: vhost-user: update vhost_dev_virtqueue_restart()
vhost-net: vhost-user: update vhost_net_virtqueue_stop()
vhost-net: vhost-user: update vhost_net_virtqueue_restart()
virtio-net: introduce flush_or_purge_queued_packets()
virtio-net: support queue_enable
vhost: vhost-kernel: enable vq reset feature
vhost: vhost-user: enable vq reset feature

Xuan Zhuo (5):
virtio: sync relevant definitions with linux
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

 docs/interop/vhost-user.rst                   | 10 +++
 hw/core/machine.c                             |  1 +
 hw/net/vhost_net.c                            | 79 +++++++++++++++++++
 hw/net/virtio-net.c                           | 58 ++++++++++++--
 hw/virtio/vhost-user.c                        | 67 ++++++++++++++--
 hw/virtio/vhost.c                             | 79 +++++++++++++++++--
 hw/virtio/virtio-pci.c                        | 20 +++++
 hw/virtio/virtio.c                            | 62 +++++++++++----
 include/hw/virtio/vhost-backend.h             |  6 ++
 include/hw/virtio/vhost.h                     |  5 ++
 include/hw/virtio/virtio-pci.h                |  1 +
 include/hw/virtio/virtio.h                    |  8 +-
 include/net/vhost_net.h                       |  4 +
 .../standard-headers/linux/virtio_config.h    |  5 ++
 include/standard-headers/linux/virtio_pci.h   |  2 +
 15 files changed, 371 insertions(+), 36 deletions(-)

--
2.32.0

author
#73 lore raw
In reply to Michael S. Tsirkin
Quoted 79 lines · Michael S. Tsirkin
Quoted 78 lines · Xuan Zhuo
Quoted 77 lines · Michael S. Tsirkin
Quoted 76 lines · Kangjie Xu

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

     https://github.com/oasis-tcs/virtio-spec/issues/124
     https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of several parts:
1. Patches 1-7 are the basic interfaces for vq reset in virtio and virtio-pci.
2. Patches 8-12 support vq stop and vq restart for vhost-kernel.
3. Patches 13-19 support vq stop and vq restart for vhost-user.
4. Patches 20-22 support vq reset and re-enable for virtio-net.
5. Patches 23-24 enable the vq reset feature for vhost-kernel and vhost-user.

The process of virtqueue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue can be optionally restarted(re-enabled).

Since this patch set involves multiple modules and seems a bit messy, we briefly describe the calling process for different modes below.
virtio-net:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

     -> virtio_queue_reset() [virtio]
         -> virtio_net_queue_reset() [virtio-net]
         -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
     -> set enabled, reset status of vq.

vhost-kernel:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

     -> virtio_queue_reset() [virtio]
         -> virtio_net_queue_reset() [virtio-net]
             -> vhost_net_virtqueue_stop() [vhost-net]
                 -> vhost_net_set_backend() [vhost]
                 -> vhost_dev_virtqueue_stop()
                     -> vhost_virtqueue_unmap()
         -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
     -> virtio_queue_enable() [virtio]
         -> virtio_net_queue_enable() [virtio-net]
             -> vhost_net_virtqueue_restart() [vhost-net]
                 -> vhost_dev_virtqueue_restart() [vhost]
                     -> vhost_virtqueue_start()
                 -> vhost_net_set_backend()
     -> set enabled, reset status of vq.

vhost-user:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

     -> virtio_queue_reset() [virtio]
         -> virtio_net_queue_reset() [virtio-net]
             -> vhost_net_virtqueue_stop() [vhost-net]
                 -> vhost_dev_virtqueue_stop() [vhost]
                     -> vhost_user_reset_vring() [vhost-user]
                         -> send VHOST_USER_RESET_VRING to the device
                     -> vhost_virtqueue_unmap()
         -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
     -> virtio_queue_enable() [virtio]
         -> virtio_net_queue_enable() [virtio-net]
             -> vhost_net_virtqueue_restart() [vhost-net]
                 -> vhost_dev_virtqueue_restart() [vhost]
                     -> vhost_virtqueue_start()
                     -> vhost_user_set_single_vring_enable [vhost-user]
                         -> send VHOST_USER_SET_VRING_ENABLE to the device
     -> set enabled, reset status of vq.

Test environment:

     Host: 5.19.0-rc3 (With vq reset support)
     Qemu: QEMU emulator version 7.0.50
     Guest: 5.19.0-rc3 (With vq reset support)
     DPDK: 22.07-rc1 (With vq reset support)
     Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

     The drvier can resize the virtio queue, then virtio queue reset function should
     be triggered.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

legacy mode testing?

legacy does not support vq reset.

Thanks.

yes but did it break with all these code changes?

For pci legacy mode, we tested resizing the vq using 'ethtool -G', it reported an error code -ENOENT because the transport did not support the vq reset feature.

We also tested the connectivity and performance in pci legacy mode using iperf 3.9. We placed a client in the host and a server in the guest, or swapped the them.

In all the cases, the connectivity did not break down. The performance was about the same with all our code changes.

Thanks.

Quoted 58 lines
Quoted 58 lines
Quoted 58 lines
Quoted 58 lines

Guest Kernel Patch:

     https://lore.kernel.org/bpf/[email protected]/

DPDK Patch:

     https://github.com/middaywords/dpdk/compare/72206323a5dd3182b13f61b25a64abdddfee595c...eabadfac7953da66bc10ffb8284b490d09bb7ec7

Host Kernel Patch:

     https://github.com/middaywords/linux/commit/19a91e0d7167b2031e46078c6215c213b89cb2c3

Looking forward to your review and comments. Thanks.

Kangjie Xu (19):
virtio: introduce virtio_queue_enable()
virtio: core: vq reset feature negotation support
virtio-pci: support queue enable
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce vhost_dev_virtqueue_stop()
vhost: introduce vhost_dev_virtqueue_restart()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()
docs: vhost-user: add VHOST_USER_RESET_VRING message
vhost-user: introduce vhost_reset_vring() interface
vhost-user: add op to enable or disable a single vring
vhost: vhost-user: update vhost_dev_virtqueue_stop()
vhost: vhost-user: update vhost_dev_virtqueue_restart()
vhost-net: vhost-user: update vhost_net_virtqueue_stop()
vhost-net: vhost-user: update vhost_net_virtqueue_restart()
virtio-net: introduce flush_or_purge_queued_packets()
virtio-net: support queue_enable
vhost: vhost-kernel: enable vq reset feature
vhost: vhost-user: enable vq reset feature

Xuan Zhuo (5):
virtio: sync relevant definitions with linux
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

  docs/interop/vhost-user.rst                   | 10 +++
  hw/core/machine.c                             |  1 +
  hw/net/vhost_net.c                            | 79 +++++++++++++++++++
  hw/net/virtio-net.c                           | 58 ++++++++++++--
  hw/virtio/vhost-user.c                        | 67 ++++++++++++++--
  hw/virtio/vhost.c                             | 79 +++++++++++++++++--
  hw/virtio/virtio-pci.c                        | 20 +++++
  hw/virtio/virtio.c                            | 62 +++++++++++----
  include/hw/virtio/vhost-backend.h             |  6 ++
  include/hw/virtio/vhost.h                     |  5 ++
  include/hw/virtio/virtio-pci.h                |  1 +
  include/hw/virtio/virtio.h                    |  8 +-
  include/net/vhost_net.h                       |  4 +
  .../standard-headers/linux/virtio_config.h    |  5 ++
  include/standard-headers/linux/virtio_pci.h   |  2 +
  15 files changed, 371 insertions(+), 36 deletions(-)

--
2.32.0

author
#74 lore raw
In reply to Kangjie Xu

Forgot to append changelog for v2 in the cover letter, so I add it in this email.

changelog:

1. Add support for vhost-net scenario

2. Add a new vhost-user message VHOST_USER_RESET_VRING

3. Add migration compatibility for virtqueue reset

Looking forward to your review and comments to this patch set, thanks. :)

Best regards,

Kangjie

Quoted 135 lines · Kangjie Xu

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

     https://github.com/oasis-tcs/virtio-spec/issues/124
     https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of several parts:
1. Patches 1-7 are the basic interfaces for vq reset in virtio and virtio-pci.
2. Patches 8-12 support vq stop and vq restart for vhost-kernel.
3. Patches 13-19 support vq stop and vq restart for vhost-user.
4. Patches 20-22 support vq reset and re-enable for virtio-net.
5. Patches 23-24 enable the vq reset feature for vhost-kernel and vhost-user.

The process of virtqueue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue can be optionally restarted(re-enabled).

Since this patch set involves multiple modules and seems a bit messy, we briefly describe the calling process for different modes below.
virtio-net:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

     -> virtio_queue_reset() [virtio]
         -> virtio_net_queue_reset() [virtio-net]
         -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
     -> set enabled, reset status of vq.

vhost-kernel:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

     -> virtio_queue_reset() [virtio]
         -> virtio_net_queue_reset() [virtio-net]
             -> vhost_net_virtqueue_stop() [vhost-net]
                 -> vhost_net_set_backend() [vhost]
                 -> vhost_dev_virtqueue_stop()
                     -> vhost_virtqueue_unmap()
         -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
     -> virtio_queue_enable() [virtio]
         -> virtio_net_queue_enable() [virtio-net]
             -> vhost_net_virtqueue_restart() [vhost-net]
                 -> vhost_dev_virtqueue_restart() [vhost]
                     -> vhost_virtqueue_start()
                 -> vhost_net_set_backend()
     -> set enabled, reset status of vq.

vhost-user:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

     -> virtio_queue_reset() [virtio]
         -> virtio_net_queue_reset() [virtio-net]
             -> vhost_net_virtqueue_stop() [vhost-net]
                 -> vhost_dev_virtqueue_stop() [vhost]
                     -> vhost_user_reset_vring() [vhost-user]
                         -> send VHOST_USER_RESET_VRING to the device
                     -> vhost_virtqueue_unmap()
         -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
     -> virtio_queue_enable() [virtio]
         -> virtio_net_queue_enable() [virtio-net]
             -> vhost_net_virtqueue_restart() [vhost-net]
                 -> vhost_dev_virtqueue_restart() [vhost]
                     -> vhost_virtqueue_start()
                     -> vhost_user_set_single_vring_enable [vhost-user]
                         -> send VHOST_USER_SET_VRING_ENABLE to the device
     -> set enabled, reset status of vq.

Test environment:
Host: 5.19.0-rc3 (With vq reset support)
Qemu: QEMU emulator version 7.0.50
Guest: 5.19.0-rc3 (With vq reset support)
DPDK: 22.07-rc1 (With vq reset support)
Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

The drvier can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

Guest Kernel Patch:

     https://lore.kernel.org/bpf/[email protected]/

DPDK Patch:

     https://github.com/middaywords/dpdk/compare/72206323a5dd3182b13f61b25a64abdddfee595c...eabadfac7953da66bc10ffb8284b490d09bb7ec7

Host Kernel Patch:

     https://github.com/middaywords/linux/commit/19a91e0d7167b2031e46078c6215c213b89cb2c3

Looking forward to your review and comments. Thanks.

Kangjie Xu (19):
virtio: introduce virtio_queue_enable()
virtio: core: vq reset feature negotation support
virtio-pci: support queue enable
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce vhost_dev_virtqueue_stop()
vhost: introduce vhost_dev_virtqueue_restart()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()
docs: vhost-user: add VHOST_USER_RESET_VRING message
vhost-user: introduce vhost_reset_vring() interface
vhost-user: add op to enable or disable a single vring
vhost: vhost-user: update vhost_dev_virtqueue_stop()
vhost: vhost-user: update vhost_dev_virtqueue_restart()
vhost-net: vhost-user: update vhost_net_virtqueue_stop()
vhost-net: vhost-user: update vhost_net_virtqueue_restart()
virtio-net: introduce flush_or_purge_queued_packets()
virtio-net: support queue_enable
vhost: vhost-kernel: enable vq reset feature
vhost: vhost-user: enable vq reset feature

Xuan Zhuo (5):
virtio: sync relevant definitions with linux
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

  docs/interop/vhost-user.rst                   | 10 +++
  hw/core/machine.c                             |  1 +
  hw/net/vhost_net.c                            | 79 +++++++++++++++++++
  hw/net/virtio-net.c                           | 58 ++++++++++++--
  hw/virtio/vhost-user.c                        | 67 ++++++++++++++--
  hw/virtio/vhost.c                             | 79 +++++++++++++++++--
  hw/virtio/virtio-pci.c                        | 20 +++++
  hw/virtio/virtio.c                            | 62 +++++++++++----
  include/hw/virtio/vhost-backend.h             |  6 ++
  include/hw/virtio/vhost.h                     |  5 ++
  include/hw/virtio/virtio-pci.h                |  1 +
  include/hw/virtio/virtio.h                    |  8 +-
  include/net/vhost_net.h                       |  4 +
  .../standard-headers/linux/virtio_config.h    |  5 ++
  include/standard-headers/linux/virtio_pci.h   |  2 +
  15 files changed, 371 insertions(+), 36 deletions(-)
Acked-by on v2 2/24
#3 lore raw

Re: [PATCH v2 02/24] virtio: introduce __virtio_queue_reset()

In reply to Kangjie Xu
Quoted 6 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

Separate the logic of vq reset. This logic will be called directly later.

Quoted 59 lines
---
  hw/virtio/virtio.c | 37 +++++++++++++++++++++----------------
  1 file changed, 21 insertions(+), 16 deletions(-)
hw/virtio/virtio.c +21 -16
index 5d607aeaa0..67d54832a9 100644
@@ -2019,6 +2019,26 @@ static enum virtio_device_endian virtio_current_cpu_endian(void)
      }
  }
 
+static void __virtio_queue_reset(VirtIODevice *vdev, uint32_t i)
+{
+    vdev->vq[i].vring.desc = 0;
+    vdev->vq[i].vring.avail = 0;
+    vdev->vq[i].vring.used = 0;
+    vdev->vq[i].last_avail_idx = 0;
+    vdev->vq[i].shadow_avail_idx = 0;
+    vdev->vq[i].used_idx = 0;
+    vdev->vq[i].last_avail_wrap_counter = true;
+    vdev->vq[i].shadow_avail_wrap_counter = true;
+    vdev->vq[i].used_wrap_counter = true;
+    virtio_queue_set_vector(vdev, i, VIRTIO_NO_VECTOR);
+    vdev->vq[i].signalled_used = 0;
+    vdev->vq[i].signalled_used_valid = false;
+    vdev->vq[i].notification = true;
+    vdev->vq[i].vring.num = vdev->vq[i].vring.num_default;
+    vdev->vq[i].inuse = 0;
+    virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
+}
+
  void virtio_reset(void *opaque)
  {
      VirtIODevice *vdev = opaque;
@@ -2050,22 +2070,7 @@ void virtio_reset(void *opaque)
      virtio_notify_vector(vdev, vdev->config_vector);
 
      for(i = 0; i < VIRTIO_QUEUE_MAX; i++) {
-        vdev->vq[i].vring.desc = 0;
-        vdev->vq[i].vring.avail = 0;
-        vdev->vq[i].vring.used = 0;
-        vdev->vq[i].last_avail_idx = 0;
-        vdev->vq[i].shadow_avail_idx = 0;
-        vdev->vq[i].used_idx = 0;
-        vdev->vq[i].last_avail_wrap_counter = true;
-        vdev->vq[i].shadow_avail_wrap_counter = true;
-        vdev->vq[i].used_wrap_counter = true;
-        virtio_queue_set_vector(vdev, i, VIRTIO_NO_VECTOR);
-        vdev->vq[i].signalled_used = 0;
-        vdev->vq[i].signalled_used_valid = false;
-        vdev->vq[i].notification = true;
-        vdev->vq[i].vring.num = vdev->vq[i].vring.num_default;
-        vdev->vq[i].inuse = 0;
-        virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
+        __virtio_queue_reset(vdev, i);
      }
  }
 
Acked-by on v2 3/24
#5 lore raw

Re: [PATCH v2 03/24] virtio: introduce virtio_queue_reset()

In reply to Kangjie Xu
Quoted 9 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

Introduce a new interface function virtio_queue_reset() to implement reset for vq.

Add a new callback to VirtioDeviceClass for queue reset operation for each child device.

Quoted 47 lines
---
  hw/virtio/virtio.c         | 11 +++++++++++
  include/hw/virtio/virtio.h |  2 ++
  2 files changed, 13 insertions(+)
hw/virtio/virtio.c +11 -0
index 67d54832a9..0e9d41366f 100644
@@ -2039,6 +2039,17 @@ static void __virtio_queue_reset(VirtIODevice *vdev, uint32_t i)
      virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
  }
 
+void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+
+    if (k->queue_reset) {
+        k->queue_reset(vdev, queue_index);
+    }
+
+    __virtio_queue_reset(vdev, queue_index);
+}
+
  void virtio_reset(void *opaque)
  {
      VirtIODevice *vdev = opaque;
include/hw/virtio/virtio.h +2 -0
index db1c0ddf6b..879394299b 100644
@@ -130,6 +130,7 @@ struct VirtioDeviceClass {
      void (*set_config)(VirtIODevice *vdev, const uint8_t *config);
      void (*reset)(VirtIODevice *vdev);
      void (*set_status)(VirtIODevice *vdev, uint8_t val);
+    void (*queue_reset)(VirtIODevice *vdev, uint32_t queue_index);
      /* For transitional devices, this is a bitmap of features
       * that are only exposed on the legacy interface but not
       * the modern one.
@@ -268,6 +269,7 @@ int virtio_queue_set_host_notifier_mr(VirtIODevice *vdev, int n,
                                        MemoryRegion *mr, bool assign);
  int virtio_set_status(VirtIODevice *vdev, uint8_t val);
  void virtio_reset(void *opaque);
+void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
  void virtio_update_irq(VirtIODevice *vdev);
  int virtio_set_features(VirtIODevice *vdev, uint64_t val);
 
Acked-by on v2 4/24
#7 lore raw

Re: [PATCH v2 04/24] virtio: introduce virtio_queue_enable()

In reply to Kangjie Xu
Quoted 9 lines · Kangjie Xu

Introduce the interface queue_enable() in VirtioDeviceClass and the fucntion virtio_queue_enable() in virtio, it can be called when VIRTIO_PCI_COMMON_Q_ENABLE is written and related virtqueue can be started. It only supports the devices of virtio 1 or later. The not-supported devices can only start the virtqueue when DRIVER_OK.

---

Quoted 49 lines
  hw/virtio/virtio.c         | 14 ++++++++++++++
  include/hw/virtio/virtio.h |  2 ++
  2 files changed, 16 insertions(+)
hw/virtio/virtio.c +14 -0
index 0e9d41366f..141f18c633 100644
@@ -2050,6 +2050,20 @@ void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
      __virtio_queue_reset(vdev, queue_index);
  }
 
+void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+
+    if (!virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1)) {
+        error_report("queue_enable is only suppported in devices of virtio "
+                     "1.0 or later.");
+    }
+
+    if (k->queue_enable) {
+        k->queue_enable(vdev, queue_index);
+    }
+}
+
  void virtio_reset(void *opaque)
  {
      VirtIODevice *vdev = opaque;
include/hw/virtio/virtio.h +2 -0
index 879394299b..085997d8f3 100644
@@ -131,6 +131,7 @@ struct VirtioDeviceClass {
      void (*reset)(VirtIODevice *vdev);
      void (*set_status)(VirtIODevice *vdev, uint8_t val);
      void (*queue_reset)(VirtIODevice *vdev, uint32_t queue_index);
+    void (*queue_enable)(VirtIODevice *vdev, uint32_t queue_index);
      /* For transitional devices, this is a bitmap of features
       * that are only exposed on the legacy interface but not
       * the modern one.
@@ -270,6 +271,7 @@ int virtio_queue_set_host_notifier_mr(VirtIODevice *vdev, int n,
  int virtio_set_status(VirtIODevice *vdev, uint8_t val);
  void virtio_reset(void *opaque);
  void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
+void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index);
  void virtio_update_irq(VirtIODevice *vdev);
  int virtio_set_features(VirtIODevice *vdev, uint64_t val);
 
#9 lore raw

Re: [PATCH v2 05/24] virtio: core: vq reset feature negotation support

In reply to Kangjie Xu
Quoted 16 lines · Kangjie Xu

A a new command line parameter "queue_reset" is added.

Meanwhile, the vq reset feature is disabled for pre-7.1 machines.

---
  hw/core/machine.c          | 1 +
  include/hw/virtio/virtio.h | 4 +++-
  2 files changed, 4 insertions(+), 1 deletion(-)
hw/core/machine.c +0 -0
index a673302cce..8b22b4647f 100644

hw/core/machine.c line 46

Quoted 4 lines
hw/core/machine.c +1 -0
@@ -43,6 +43,7 @@
  GlobalProperty hw_compat_7_0[] = {
      { "arm-gicv3-common", "force-8-bit-prio", "on" },
      { "nvme-ns", "eui64-default", "on"},
+    { "virtio-device", "queue_reset", "false" },

7.1 is about to release so we need to do it for pre-7.2.

Thanks

Quoted 18 lines
hw/core/machine.c +0 -0
  };
  const size_t hw_compat_7_0_len = G_N_ELEMENTS(hw_compat_7_0);
 
include/hw/virtio/virtio.h +3 -1
index 085997d8f3..ed3ecbef80 100644
@@ -295,7 +295,9 @@ typedef struct VirtIORNGConf VirtIORNGConf;
      DEFINE_PROP_BIT64("iommu_platform", _state, _field, \
                        VIRTIO_F_IOMMU_PLATFORM, false), \
      DEFINE_PROP_BIT64("packed", _state, _field, \
-                      VIRTIO_F_RING_PACKED, false)
+                      VIRTIO_F_RING_PACKED, false), \
+    DEFINE_PROP_BIT64("queue_reset", _state, _field, \
+                      VIRTIO_F_RING_RESET, true)
 
  hwaddr virtio_queue_get_desc_addr(VirtIODevice *vdev, int n);
  bool virtio_queue_enabled_legacy(VirtIODevice *vdev, int n);
author
#10 lore raw

Re: [PATCH v2 05/24] virtio: core: vq reset feature negotation support

Quoted 26 lines · Jason Wang
Quoted 16 lines · Kangjie Xu

A a new command line parameter "queue_reset" is added.

Meanwhile, the vq reset feature is disabled for pre-7.1 machines.

---
  hw/core/machine.c          | 1 +
  include/hw/virtio/virtio.h | 4 +++-

  2 files changed, 4 insertions(+), 1 deletion(-)
hw/core/machine.c +0 -0
index a673302cce..8b22b4647f 100644

hw/core/machine.c line 46

Quoted 4 lines
hw/core/machine.c +1 -0
@@ -43,6 +43,7 @@
   GlobalProperty hw_compat_7_0[] = {
       { "arm-gicv3-common", "force-8-bit-prio", "on" },
       { "nvme-ns", "eui64-default", "on"},
+    { "virtio-device", "queue_reset", "false" },

7.1 is about to release so we need to do it for pre-7.2.

Quoted 2 lines

Thanks

The current version is 7.0.92, should I wait until 7.1 to be released and submit my patch after that? or just add hw_compat_7_1[] and related support.

Thanks

Quoted 17 lines
Quoted 17 lines
hw/core/machine.c +0 -0
   };
   const size_t hw_compat_7_0_len = G_N_ELEMENTS(hw_compat_7_0);

  diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h index 085997d8f3..ed3ecbef80 100644

include/hw/virtio/virtio.h +3 -1
@@ -295,7 +295,9 @@ typedef struct VirtIORNGConf VirtIORNGConf;
       DEFINE_PROP_BIT64("iommu_platform", _state, _field, \
                         VIRTIO_F_IOMMU_PLATFORM, false), \
       DEFINE_PROP_BIT64("packed", _state, _field, \
-                      VIRTIO_F_RING_PACKED, false)
+                      VIRTIO_F_RING_PACKED, false), \
+    DEFINE_PROP_BIT64("queue_reset", _state, _field, \
+                      VIRTIO_F_RING_RESET, true)
     hwaddr virtio_queue_get_desc_addr(VirtIODevice *vdev, int n);
   bool virtio_queue_enabled_legacy(VirtIODevice *vdev, int n);
#12 lore raw

Re: [PATCH v2 06/24] virtio-pci: support queue reset

In reply to Kangjie Xu
Quoted 42 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

PCI devices support vq reset.

Based on this function, the driver can adjust the size of the ring, and quickly recycle the buffer in the ring.

The migration of the virtio devices will not happen during a reset operation. This is becuase the global iothread lock is held. Migration thread also needs the lock. As a result, we do not need to migrate the reset state of VirtIOPCIQueue.

---
  hw/virtio/virtio-pci.c         | 19 +++++++++++++++++++
  include/hw/virtio/virtio-pci.h |  1 +
  2 files changed, 20 insertions(+)
hw/virtio/virtio-pci.c +4 -0
index 45327f0b31..ec8e92052f 100644
@@ -1246,6 +1246,9 @@ static uint64_t virtio_pci_common_read(void *opaque, hwaddr addr,
      case VIRTIO_PCI_COMMON_Q_USEDHI:
          val = proxy->vqs[vdev->queue_sel].used[1];
          break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        val = proxy->vqs[vdev->queue_sel].reset;
+        break;
      default:
          val = 0;
      }
@@ -1333,6 +1336,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
                         ((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 |
                         proxy->vqs[vdev->queue_sel].used[0]);
              proxy->vqs[vdev->queue_sel].enabled = 1;
+            proxy->vqs[vdev->queue_sel].reset = 0;
          } else {
              virtio_error(vdev, "wrong value for queue_enable %"PRIx64, val);
          }

hw/virtio/virtio-pci.c static void virtio_pci_common_write(void *opaque, hwaddr addr, · line 1367

Quoted 9 lines
hw/virtio/virtio-pci.c +6 -0
@@ -1355,6 +1359,20 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
      case VIRTIO_PCI_COMMON_Q_USEDHI:
          proxy->vqs[vdev->queue_sel].used[1] = val;
          break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        if (val == 1) {
+            /*
+             * With the global iothread lock taken, the migration will not
+             * happen until the virtqueue reset is done.
+             */

This comment applies to all other common cfg operation as well, So it looks not necessary?

Quoted 24 lines
hw/virtio/virtio-pci.c +9 -0
+            proxy->vqs[vdev->queue_sel].reset = 1;
+
+            virtio_queue_reset(vdev, vdev->queue_sel);
+
+            proxy->vqs[vdev->queue_sel].reset = 0;
+            proxy->vqs[vdev->queue_sel].enabled = 0;
+        }
+        break;
      default:
          break;
      }
@@ -1950,6 +1968,7 @@ static void virtio_pci_reset(DeviceState *qdev)
 
      for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
          proxy->vqs[i].enabled = 0;
+        proxy->vqs[i].reset = 0;
          proxy->vqs[i].num = 0;
          proxy->vqs[i].desc[0] = proxy->vqs[i].desc[1] = 0;
          proxy->vqs[i].avail[0] = proxy->vqs[i].avail[1] = 0;
include/hw/virtio/virtio-pci.h +0 -0
index 2446dcd9ae..e9290e2b94 100644

include/hw/virtio/virtio-pci.h typedef struct VirtIOPCIRegion { · line 120

Quoted 4 lines
include/hw/virtio/virtio-pci.h +1 -0
@@ -117,6 +117,7 @@ typedef struct VirtIOPCIRegion {
  typedef struct VirtIOPCIQueue {
    uint16_t num;
    bool enabled;
+  bool reset;

Do we need to migrate this?

Thanks

Quoted 3 lines
include/hw/virtio/virtio-pci.h +0 -0
    uint32_t desc[2];
    uint32_t avail[2];
    uint32_t used[2];
#16 lore raw

Re: [PATCH v2 07/24] virtio-pci: support queue enable

In reply to Kangjie Xu
Quoted 1 line · Kangjie Xu

PCI devices support vq enable.

Nit: it might be "support device specific vq enable"

Quoted 15 lines

Based on this function, the driver can re-enable the virtqueue after the virtqueue is reset.

---
  hw/virtio/virtio-pci.c | 1 +
  1 file changed, 1 insertion(+)
hw/virtio/virtio-pci.c +0 -0
index ec8e92052f..3d560e45ad 100644

hw/virtio/virtio-pci.c static void virtio_pci_common_write(void *opaque, hwaddr addr, · line 1340

Quoted 6 lines
hw/virtio/virtio-pci.c +1 -0
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
                         proxy->vqs[vdev->queue_sel].avail[0],
                         ((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 |
                         proxy->vqs[vdev->queue_sel].used[0]);
+            virtio_queue_enable(vdev, vdev->queue_sel);
              proxy->vqs[vdev->queue_sel].enabled = 1;
              proxy->vqs[vdev->queue_sel].reset = 0;

Any reason we do it before the assignment of 1? It probably means the device specific method can't depend on virtio_queue_enabled()?

Thanks

Quoted 1 line
hw/virtio/virtio-pci.c +0 -0
          } else {
Acked-by on v2 8/24
#23 lore raw

Re: [PATCH v2 08/24] vhost: extract the logic of unmapping the vrings and desc

In reply to Kangjie Xu
Quoted 7 lines · Kangjie Xu

Introduce vhost_virtqueue_unmap() to ummap the vrings and desc of a virtqueue.

The function will be used later.

Quoted 42 lines
---
  hw/virtio/vhost.c | 20 ++++++++++++++------
  1 file changed, 14 insertions(+), 6 deletions(-)
hw/virtio/vhost.c +14 -6
index 0827d631c0..e467dfc7bc 100644
@@ -1197,6 +1197,19 @@ fail_alloc_desc:
      return r;
  }
 
+static void vhost_virtqueue_unmap(struct vhost_dev *dev,
+                                  struct VirtIODevice *vdev,
+                                  struct vhost_virtqueue *vq,
+                                  unsigned idx)
+{
+    vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
+                       1, virtio_queue_get_used_size(vdev, idx));
+    vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
+                       0, virtio_queue_get_avail_size(vdev, idx));
+    vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
+                       0, virtio_queue_get_desc_size(vdev, idx));
+}
+
  static void vhost_virtqueue_stop(struct vhost_dev *dev,
                                      struct VirtIODevice *vdev,
                                      struct vhost_virtqueue *vq,
@@ -1235,12 +1248,7 @@ static void vhost_virtqueue_stop(struct vhost_dev *dev,
                                                  vhost_vq_index);
      }
 
-    vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
-                       1, virtio_queue_get_used_size(vdev, idx));
-    vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
-                       0, virtio_queue_get_avail_size(vdev, idx));
-    vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
-                       0, virtio_queue_get_desc_size(vdev, idx));
+    vhost_virtqueue_unmap(dev, vdev, vq, idx);
  }
 
  static void vhost_eventfd_add(MemoryListener *listener,
author
#13 lore raw

Re: [PATCH v2 06/24] virtio-pci: support queue reset

In reply to Jason Wang
Quoted 52 lines · Jason Wang
Quoted 50 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

PCI devices support vq reset.

Based on this function, the driver can adjust the size of the ring, and quickly recycle the buffer in the ring.

The migration of the virtio devices will not happen during a reset operation. This is becuase the global iothread lock is held. Migration thread also needs the lock. As a result, we do not need to migrate the reset state of VirtIOPCIQueue.

---
  hw/virtio/virtio-pci.c         | 19 +++++++++++++++++++ include/hw/virtio/virtio-pci.h |  1 +

  2 files changed, 20 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index 45327f0b31..ec8e92052f 100644
@@ -1246,6 +1246,9 @@ static uint64_t virtio_pci_common_read(void 
 *opaque, hwaddr addr,

      case VIRTIO_PCI_COMMON_Q_USEDHI:
          val = proxy->vqs[vdev->queue_sel].used[1]; break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        val = proxy->vqs[vdev->queue_sel].reset; +        break;
      default:
          val = 0;
      }

hw/virtio/virtio-pci.c +0 -0
@@ -1333,6 +1336,7 @@ static void virtio_pci_common_write(void 
 *opaque, hwaddr addr,

((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 | proxy->vqs[vdev->queue_sel].used[0]); proxy->vqs[vdev->queue_sel].enabled = 1;
+            proxy->vqs[vdev->queue_sel].reset = 0;
          } else {
              virtio_error(vdev, "wrong value for queue_enable %"PRIx64, val); }

hw/virtio/virtio-pci.c +0 -0
@@ -1355,6 +1359,20 @@ static void virtio_pci_common_write(void 
 *opaque, hwaddr addr,

      case VIRTIO_PCI_COMMON_Q_USEDHI:
          proxy->vqs[vdev->queue_sel].used[1] = val; break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        if (val == 1) {

Quoted 9 lines
Quoted 5 lines

+            /*
+             * With the global iothread lock taken, the migration will not
+             * happen until the virtqueue reset is done. +             */

This comment applies to all other common cfg operation as well, So it looks not necessary?

Get it.

Quoted 32 lines
Quoted 24 lines

+ proxy->vqs[vdev->queue_sel].reset = 1;

hw/virtio/virtio-pci.c +7 -0
+
+            virtio_queue_reset(vdev, vdev->queue_sel);
+
+            proxy->vqs[vdev->queue_sel].reset = 0;
+            proxy->vqs[vdev->queue_sel].enabled = 0;
+        }
+        break;

      default:
          break;
      }

hw/virtio/virtio-pci.c +1 -0
@@ -1950,6 +1968,7 @@ static void virtio_pci_reset(DeviceState *qdev)
         for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
           proxy->vqs[i].enabled = 0;
+        proxy->vqs[i].reset = 0;
           proxy->vqs[i].num = 0;
           proxy->vqs[i].desc[0] = proxy->vqs[i].desc[1] = 0;
           proxy->vqs[i].avail[0] = proxy->vqs[i].avail[1] = 0;
include/hw/virtio/virtio-pci.h +0 -0
index 2446dcd9ae..e9290e2b94 100644

include/hw/virtio/virtio-pci.h typedef struct VirtIOPCIRegion { · line 120

Quoted 4 lines
include/hw/virtio/virtio-pci.h +1 -0
@@ -117,6 +117,7 @@ typedef struct VirtIOPCIRegion {
   typedef struct VirtIOPCIQueue {
     uint16_t num;
     bool enabled;
+  bool reset;

Do we need to migrate this?

Quoted 2 lines

Thanks

I think we do not need to migrate this because we hold the global iothread lock when virtqueue reset is triggered. The migration of these device states also needs this lock.

On the other hand, the 'reset' state of virtqueue is same(is 0) before and after the process of resetting a virtqueue.

Thus, the migration will not happen when we are resetting a virtqueue and we do not to migrate it.

Thanks

Quoted 4 lines
Quoted 4 lines
include/hw/virtio/virtio-pci.h +0 -0
     uint32_t desc[2];
     uint32_t avail[2];
     uint32_t used[2];
#25 lore raw

Re: [PATCH v2 09/24] vhost: introduce vhost_dev_virtqueue_stop()

In reply to Kangjie Xu
Quoted 16 lines · Kangjie Xu

Introduce vhost_dev_virtqueue_stop(), which can ummap the vrings and the desc of it.

---
  hw/virtio/vhost.c         | 9 +++++++++
  include/hw/virtio/vhost.h | 3 +++
  2 files changed, 12 insertions(+)
hw/virtio/vhost.c +0 -0
index e467dfc7bc..1bca9ff48d 100644
@@ -1904,3 +1904,12 @@ int vhost_net_set_backend(struct vhost_dev *hdev,
 

hw/virtio/vhost.c int vhost_net_set_backend(struct vhost_dev *hdev, · line 1914

Quoted 10 lines
hw/virtio/vhost.c +8 -0
      return -ENOSYS;
  }
+
+void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
+                              int idx)
+{
+    vhost_virtqueue_unmap(hdev,
+                          vdev,
+                          hdev->vqs + idx,
+                          idx);

So I think the unmap is not sufficient, we need backend specific support. E.g for vhost kernel, need a SET_BACKEND here?

Thanks

Quoted 13 lines
hw/virtio/vhost.c +1 -0
+}
include/hw/virtio/vhost.h +3 -0
index a346f23d13..574888440c 100644
@@ -288,4 +288,7 @@ int vhost_dev_set_inflight(struct vhost_dev *dev,
                             struct vhost_inflight *inflight);
  int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
                             struct vhost_inflight *inflight);
+
+void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
+                              int idx);
  #endif
author
#26 lore raw

Re: [PATCH v2 09/24] vhost: introduce vhost_dev_virtqueue_stop()

Quoted 32 lines · Jason Wang
Quoted 16 lines · Kangjie Xu

Introduce vhost_dev_virtqueue_stop(), which can ummap the vrings and the desc of it.

---
  hw/virtio/vhost.c         | 9 +++++++++
  include/hw/virtio/vhost.h | 3 +++

  2 files changed, 12 insertions(+)
hw/virtio/vhost.c +0 -0
index e467dfc7bc..1bca9ff48d 100644
@@ -1904,3 +1904,12 @@ int vhost_net_set_backend(struct vhost_dev *hdev,
         return -ENOSYS;

hw/virtio/vhost.c int vhost_net_set_backend(struct vhost_dev *hdev, · line 1914

Quoted 10 lines
hw/virtio/vhost.c +8 -0
   }
+
+void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice 
 *vdev,
+                              int idx)
+{
+    vhost_virtqueue_unmap(hdev,
+                          vdev,
+                          hdev->vqs + idx,
+                          idx);

So I think the unmap is not sufficient, we need backend specific support. E.g for vhost kernel, need a SET_BACKEND here?

Quoted 2 lines

Thanks

But SET_BACKEND of vhost-net needs a parameter fd in vhost_vring_file: that is net->backend of VHostNetState.

If we add the fd parameter or struct vhost_vring_file to vhost_dev_virtqueue_stop/restart, it exposes some implementation details in the parameter list.

And that seems not good? So I put SET_BACKEND in the vhost-net module. The workflow is similar to vhost_net_start_one().

Thanks

Quoted 15 lines
Quoted 15 lines
hw/virtio/vhost.c +1 -0
+}
include/hw/virtio/vhost.h +3 -0
index a346f23d13..574888440c 100644
@@ -288,4 +288,7 @@ int vhost_dev_set_inflight(struct vhost_dev *dev,
                              struct vhost_inflight *inflight);
   int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
                              struct vhost_inflight *inflight);
+
+void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice 
 *vdev,
+                              int idx);

  #endif

author
#17 lore raw

Re: [PATCH v2 07/24] virtio-pci: support queue enable

In reply to Jason Wang
Quoted 8 lines · Jason Wang
Quoted 1 line · Kangjie Xu

PCI devices support vq enable.

Nit: it might be "support device specific vq enable"

Get it.

Quoted 26 lines
Quoted 17 lines

Based on this function, the driver can re-enable the virtqueue after the virtqueue is reset.

---
  hw/virtio/virtio-pci.c | 1 +

  1 file changed, 1 insertion(+)
hw/virtio/virtio-pci.c +0 -0
index ec8e92052f..3d560e45ad 100644
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void 
 *opaque, hwaddr addr,
 proxy->vqs[vdev->queue_sel].avail[0],
Quoted 5 lines

((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 | proxy->vqs[vdev->queue_sel].used[0]);
+            virtio_queue_enable(vdev, vdev->queue_sel); proxy->vqs[vdev->queue_sel].enabled = 1; proxy->vqs[vdev->queue_sel].reset = 0;

Any reason we do it before the assignment of 1? It probably means the device specific method can't depend on virtio_queue_enabled()?

Quoted 2 lines

Thanks

Sorry, I don't get why device specific method can't depend on virtio_queue_enabled().

Before virtio_queue_enable() is done, virtqueue should always be not ready and disabled.

Otherwise, If we put it after the assignment of enabled to 1, the virtqueue may be accessed illegally and may cause panic, because the virtqueue is still being intialized and being configured.

Thanks

Quoted 2 lines
Quoted 2 lines

          } else {

#29 lore raw

Re: [PATCH v2 10/24] vhost: introduce vhost_dev_virtqueue_restart()

In reply to Kangjie Xu
Quoted 15 lines · Kangjie Xu

Introduce vhost_dev_virtqueue_restart(), which can restart the virtqueue when the vhost has already started running.

---
  hw/virtio/vhost.c         | 13 +++++++++++++
  include/hw/virtio/vhost.h |  2 ++
  2 files changed, 15 insertions(+)
hw/virtio/vhost.c +0 -0
index 1bca9ff48d..fc3f550c76 100644

hw/virtio/vhost.c void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev, · line 1922

Quoted 10 lines
hw/virtio/vhost.c +7 -0
@@ -1913,3 +1913,16 @@ void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                            hdev->vqs + idx,
                            idx);
  }
+
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                int idx)
+{
+    const VhostOps *vhost_ops = hdev->vhost_ops;
+
+    assert(vhost_ops);

So we had the comment like:

    /* should only be called after backend is connected */

in vhost_virtqueue_mask().

If this assert has the same reason, let's add a comment here.

hw/virtio/vhost.c void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev, · line 1927

Quoted 5 lines
hw/virtio/vhost.c +5 -0
+
+    return vhost_virtqueue_start(hdev,
+                                 vdev,
+                                 hdev->vqs + idx,
+                                 hdev->vq_index + idx);

So it just a wrapper of vhost_virtqueue_start(), any value to have a re-start wrapper?

Thanks

Quoted 12 lines
hw/virtio/vhost.c +1 -0
+}
include/hw/virtio/vhost.h +2 -0
index 574888440c..b3394b6348 100644
@@ -291,4 +291,6 @@ int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
 
  void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                                int idx);
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                int idx);
  #endif
#32 lore raw

Re: [PATCH v2 11/24] vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()

In reply to Kangjie Xu
Quoted 30 lines · Kangjie Xu

Introduce vhost_virtqueue_stop(), which can reset the virtqueue in the device. Then it will unmap vrings and the desc of the virtqueue.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

---
  hw/net/vhost_net.c      | 21 +++++++++++++++++++++
  include/net/vhost_net.h |  2 ++
  2 files changed, 23 insertions(+)
hw/net/vhost_net.c +8 -0
index ccac5b7a64..aa60dd901c 100644
@@ -514,3 +514,24 @@ int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu)
 
      return vhost_ops->vhost_net_set_mtu(&net->dev, mtu);
  }
+
+void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { .fd = -1 };
+    int idx;

hw/net/vhost_net.c int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu) · line 534

Quoted 10 lines
hw/net/vhost_net.c +10 -0
+
+    assert(vhost_ops);
+
+    idx = vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }

Let's have a vhost_ops here instead of open code it.

Thanks

Quoted 14 lines
hw/net/vhost_net.c +3 -0
+
+    vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
+}
include/net/vhost_net.h +2 -0
index 387e913e4e..9b3aaf3814 100644
@@ -48,4 +48,6 @@ uint64_t vhost_net_get_acked_features(VHostNetState *net);
 
  int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
 
+void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index);
  #endif
#27 lore raw

Re: [PATCH v2 09/24] vhost: introduce vhost_dev_virtqueue_stop()

In reply to Kangjie Xu
Quoted 35 lines · Kangjie Xu
Quoted 33 lines · Jason Wang
Quoted 20 lines · Kangjie Xu

Introduce vhost_dev_virtqueue_stop(), which can ummap the vrings and the desc of it.

---
  hw/virtio/vhost.c         | 9 +++++++++
  include/hw/virtio/vhost.h | 3 +++

  2 files changed, 12 insertions(+)
hw/virtio/vhost.c +2 -0
index e467dfc7bc..1bca9ff48d 100644
@@ -1904,3 +1904,12 @@ int vhost_net_set_backend(struct vhost_dev 
 *hdev,
         return -ENOSYS;
   }
+
+void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice 
Quoted 7 lines

*vdev,
+                              int idx)
+{
+    vhost_virtqueue_unmap(hdev,
+                          vdev,
+                          hdev->vqs + idx,
+                          idx);

So I think the unmap is not sufficient, we need backend specific support. E.g for vhost kernel, need a SET_BACKEND here?

Quoted 7 lines
Quoted 2 lines

Thanks

But SET_BACKEND of vhost-net needs a parameter fd in vhost_vring_file: that is net->backend of VHostNetState.

If we add the fd parameter or struct vhost_vring_file to vhost_dev_virtqueue_stop/restart, it exposes some implementation details in the parameter list.

And that seems not good? So I put SET_BACKEND in the vhost-net module. The workflow is similar to vhost_net_start_one().

That looks fine.

Thanks

Quoted 19 lines

Thanks

Quoted 15 lines
Quoted 15 lines
hw/virtio/vhost.c +1 -0
+}
include/hw/virtio/vhost.h +3 -0
index a346f23d13..574888440c 100644
@@ -288,4 +288,7 @@ int vhost_dev_set_inflight(struct vhost_dev *dev,
                              struct vhost_inflight *inflight);
   int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
                              struct vhost_inflight *inflight);
+
+void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice 
 *vdev,
+                              int idx);

  #endif

#37 lore raw

Re: [PATCH v2 12/24] vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()

In reply to Kangjie Xu
Quoted 9 lines · Kangjie Xu

Introduce vhost_net_virtqueue_restart(), which can restart the virtqueue when the vhost net started running before. If it fails to restart the virtqueue, the device will be stopped.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

I would explain why current vhost_net_start_one()/vhost_net_stop_one() can't work. Is it because it works at queue pair level? If yes can we restructure the code and try to reuse ?

Thanks

Quoted 72 lines
---
  hw/net/vhost_net.c      | 48 +++++++++++++++++++++++++++++++++++++++++
  include/net/vhost_net.h |  2 ++
  2 files changed, 50 insertions(+)
hw/net/vhost_net.c +48 -0
index aa60dd901c..2ab67e875e 100644
@@ -535,3 +535,51 @@ void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
 
      vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
  }
+
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                                int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { };
+    int idx, r;
+
+    if (!net->dev.started) {
+        return 0;
+    }
+
+    assert(vhost_ops);
+
+    idx =  vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    r = vhost_dev_virtqueue_restart(&net->dev, vdev, idx);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        file.fd = net->backend;
+        r = vhost_net_set_backend(&net->dev, &file);
+        if (r < 0) {
+            r = -errno;
+            goto err_start;
+        }
+    }
+
+    return 0;
+
+err_start:
+    error_report("Error when restarting the queue.");
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.fd = -1;
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }
+
+    vhost_dev_stop(&net->dev, vdev);
+
+    return r;
+}
include/net/vhost_net.h +2 -0
index 9b3aaf3814..e11a297380 100644
@@ -50,4 +50,6 @@ int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
 
  void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
                                int vq_index);
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                                int vq_index);
  #endif
author
#30 lore raw

Re: [PATCH v2 10/24] vhost: introduce vhost_dev_virtqueue_restart()

In reply to Jason Wang
Quoted 33 lines · Jason Wang
Quoted 23 lines · Kangjie Xu

Introduce vhost_dev_virtqueue_restart(), which can restart the virtqueue when the vhost has already started running.

---
  hw/virtio/vhost.c         | 13 +++++++++++++
  include/hw/virtio/vhost.h |  2 ++

  2 files changed, 15 insertions(+)
hw/virtio/vhost.c +0 -0
index 1bca9ff48d..fc3f550c76 100644
@@ -1913,3 +1913,16 @@ void vhost_dev_virtqueue_stop(struct vhost_dev 
 *hdev, VirtIODevice *vdev,
                             hdev->vqs + idx,
                             idx);

  }

hw/virtio/vhost.c +2 -0
+
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice 

*vdev,
+                                int idx)

hw/virtio/vhost.c void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev, · line 1922

Quoted 4 lines
hw/virtio/vhost.c +4 -0
+{
+    const VhostOps *vhost_ops = hdev->vhost_ops;
+
+    assert(vhost_ops);

So we had the comment like:

Quoted 7 lines

    /* should only be called after backend is connected */

in vhost_virtqueue_mask().

If this assert has the same reason, let's add a comment here.

Get it.

Quoted 11 lines
Quoted 5 lines

+
+    return vhost_virtqueue_start(hdev,
+                                 vdev,
+                                 hdev->vqs + idx, +                                 hdev->vq_index + idx);

So it just a wrapper of vhost_virtqueue_start(), any value to have a re-start wrapper?

Thanks

Because in subsequent patches, to support vhost-user, we will add vhost_ops->vhost_set_single_vring_enable() in this function.

Thanks.

Quoted 14 lines
Quoted 14 lines
hw/virtio/vhost.c +1 -0
+}
include/hw/virtio/vhost.h +2 -0
index 574888440c..b3394b6348 100644
@@ -291,4 +291,6 @@ int vhost_dev_get_inflight(struct vhost_dev *dev, 
 uint16_t queue_size,
     void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                                 int idx);
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice 
 *vdev,
+                                int idx);

  #endif

Acked-by on v2 13/24
#42 lore raw

Re: [PATCH v2 13/24] docs: vhost-user: add VHOST_USER_RESET_VRING message

In reply to Kangjie Xu
Quoted 8 lines · Kangjie Xu

To support the reset operation for an individual virtqueue, we introduce a new message VHOST_USER_RESET_VRING. This message is submitted by the front-end to reset an individual virtqueue to initial states in the back-end. The reply is needed to ensure that the reset operation is complete.

Quoted 25 lines
---
  docs/interop/vhost-user.rst | 10 ++++++++++
  1 file changed, 10 insertions(+)
docs/interop/vhost-user.rst +10 -0
index 3f18ab424e..ce7991b9d3 100644
@@ -1422,6 +1422,16 @@ Front-end message types
    query the back-end for its device status as defined in the Virtio
    specification.
 
+``VHOST_USER_RESET_VRING``
+  :id: 41
+  :equivalent ioctl: N/A
+  :request payload: vring state description
+  :reply payload: ``u64``
+
+  When the feature ``VIRTIO_F_RING_RESET`` feature has been successfully
+  negotiated, this message is submitted by the front-end to reset an
+  individual virtqueue to initial states in the back-end. It will ask
+  for a reply to ensure the virtqueue is successfully reset in the back-end.
 
  Back-end message types
  ----------------------
author
#33 lore raw

Re: [PATCH v2 11/24] vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()

In reply to Jason Wang
Quoted 46 lines · Jason Wang
Quoted 30 lines · Kangjie Xu

Introduce vhost_virtqueue_stop(), which can reset the virtqueue in the device. Then it will unmap vrings and the desc of the virtqueue.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

---
  hw/net/vhost_net.c      | 21 +++++++++++++++++++++ include/net/vhost_net.h |  2 ++

  2 files changed, 23 insertions(+)
hw/net/vhost_net.c +8 -0
index ccac5b7a64..aa60dd901c 100644
@@ -514,3 +514,24 @@ int vhost_net_set_mtu(struct vhost_net *net, 
 uint16_t mtu)
         return vhost_ops->vhost_net_set_mtu(&net->dev, mtu);
   }
+
+void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { .fd = -1 };
+    int idx;

hw/net/vhost_net.c int vhost_net_set_mtu(struct vhost_net *net, · line 534

Quoted 10 lines
hw/net/vhost_net.c +10 -0
+
+    assert(vhost_ops);
+
+    idx = vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }

Let's have a vhost_ops here instead of open code it.

Quoted 2 lines

Thanks

I share your view.

Thanks

Quoted 14 lines
Quoted 14 lines
hw/net/vhost_net.c +3 -0
+
+    vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
+}
include/net/vhost_net.h +1 -0
index 387e913e4e..9b3aaf3814 100644
@@ -48,4 +48,6 @@ uint64_t vhost_net_get_acked_features(VHostNetState 
 *net);
     int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
   +void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index);
   #endif
#44 lore raw

Re: [PATCH v2 14/24] vhost-user: introduce vhost_reset_vring() interface

In reply to Kangjie Xu
Quoted 25 lines · Kangjie Xu

Introduce the interface vhost_reset_vring(). The interface is a wrapper to send a VHOST_USER_RESET_VRING message to the back-end. It will reset an individual vring in the back-end. Meanwhile, it will wait for a reply to ensure the reset has been completed.

---
  hw/virtio/vhost-user.c            | 41 +++++++++++++++++++++++++++++++
  include/hw/virtio/vhost-backend.h |  3 +++
  2 files changed, 44 insertions(+)
hw/virtio/vhost-user.c +1 -0
index 75b8df21a4..56033f7a92 100644
@@ -126,6 +126,7 @@ typedef enum VhostUserRequest {
      VHOST_USER_GET_MAX_MEM_SLOTS = 36,
      VHOST_USER_ADD_MEM_REG = 37,
      VHOST_USER_REM_MEM_REG = 38,
+    VHOST_USER_RESET_VRING = 41,
      VHOST_USER_MAX
  } VhostUserRequest;
 

hw/virtio/vhost-user.c static int vhost_user_get_max_memslots(struct vhost_dev *dev, · line 1508

Quoted 10 lines
hw/virtio/vhost-user.c +7 -0
@@ -1498,6 +1499,45 @@ static int vhost_user_get_max_memslots(struct vhost_dev *dev,
      return 0;
  }
 
+static int vhost_user_reset_vring(struct vhost_dev *dev,
+                                  struct vhost_vring_state *ring)
+{
+    int ret;
+    VhostUserMsg msg = {
+        .hdr.request = VHOST_USER_RESET_VRING,
+        .hdr.flags = VHOST_USER_VERSION,

Do we need VHOST_USER_NEED_REPLY_MASK here?

Other looks good.

Thanks

Quoted 64 lines
hw/virtio/vhost-user.c +33 -0
+        .payload.state = *ring,
+        .hdr.size = sizeof(msg.payload.state),
+    };
+
+    if (!virtio_has_feature(dev->acked_features, VIRTIO_F_RING_RESET)) {
+        return -ENOTSUP;
+    }
+
+    ret = vhost_user_write(dev, &msg, NULL, 0);
+    if (ret < 0) {
+        return ret;
+    }
+
+    ret = vhost_user_read(dev, &msg);
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (msg.hdr.request != VHOST_USER_RESET_VRING) {
+        error_report("Received unexpected msg type. Expected %d received %d",
+                     VHOST_USER_RESET_VRING, msg.hdr.request);
+        return -EPROTO;
+    }
+
+    if (msg.hdr.size != sizeof(msg.payload.state)) {
+        error_report("Received bad msg size.");
+        return -EPROTO;
+    }
+
+    return 0;
+}
+
  static int vhost_user_reset_device(struct vhost_dev *dev)
  {
      VhostUserMsg msg = {
@@ -2625,6 +2665,7 @@ const VhostOps user_ops = {
          .vhost_set_features = vhost_user_set_features,
          .vhost_get_features = vhost_user_get_features,
          .vhost_set_owner = vhost_user_set_owner,
+        .vhost_reset_vring = vhost_user_reset_vring,
          .vhost_reset_device = vhost_user_reset_device,
          .vhost_get_vq_index = vhost_user_get_vq_index,
          .vhost_set_vring_enable = vhost_user_set_vring_enable,
include/hw/virtio/vhost-backend.h +3 -0
index eab46d7f0b..f23bf71a8d 100644
@@ -79,6 +79,8 @@ typedef int (*vhost_get_features_op)(struct vhost_dev *dev,
                                       uint64_t *features);
  typedef int (*vhost_set_backend_cap_op)(struct vhost_dev *dev);
  typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
+typedef int (*vhost_reset_vring_op)(struct vhost_dev *dev,
+                                    struct vhost_vring_state *ring);
  typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
  typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
  typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
@@ -154,6 +156,7 @@ typedef struct VhostOps {
      vhost_set_backend_cap_op vhost_set_backend_cap;
      vhost_set_owner_op vhost_set_owner;
      vhost_reset_device_op vhost_reset_device;
+    vhost_reset_vring_op vhost_reset_vring;
      vhost_get_vq_index_op vhost_get_vq_index;
      vhost_set_vring_enable_op vhost_set_vring_enable;
      vhost_requires_shm_log_op vhost_requires_shm_log;
author
#38 lore raw

Re: [PATCH v2 12/24] vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()

In reply to Jason Wang
Quoted 9 lines · Jason Wang
Quoted 7 lines · Kangjie Xu

Introduce vhost_net_virtqueue_restart(), which can restart the virtqueue when the vhost net started running before. If it fails to restart the virtqueue, the device will be stopped.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

Quoted 8 lines
Quoted 2 lines

I would explain why current vhost_net_start_one()/vhost_net_stop_one() can't work. Is it because it works at queue pair level? If yes can we restructure the code and try to reuse ?

Thanks

Because vhost_net_start_one()/vhost_net_stop_one() works at device level.

The queue pair level start/stop are vhost_virtqueue_start() and vhost_virtqueue_stop().

What we can reuse is the vhost_virtqueue_start(). vhost_virtqueue_stop() cannot be reused because it will destroy device.

I think we do not need to restructure because we've already had an abstraction vhost_virtqueue_start().

Thanks.

Quoted 73 lines
Quoted 73 lines

---
  hw/net/vhost_net.c      | 48 +++++++++++++++++++++++++++++++++++++++++ include/net/vhost_net.h |  2 ++

  2 files changed, 50 insertions(+)
hw/net/vhost_net.c +48 -0
index aa60dd901c..2ab67e875e 100644
@@ -535,3 +535,51 @@ void vhost_net_virtqueue_stop(VirtIODevice 
 *vdev, NetClientState *nc,
         vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
   }
+
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                                int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { };
+    int idx, r;
+
+    if (!net->dev.started) {
+        return 0;
+    }
+
+    assert(vhost_ops);
+
+    idx =  vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    r = vhost_dev_virtqueue_restart(&net->dev, vdev, idx);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        file.fd = net->backend;
+        r = vhost_net_set_backend(&net->dev, &file);
+        if (r < 0) {
+            r = -errno;
+            goto err_start;
+        }
+    }
+
+    return 0;
+
+err_start:
+    error_report("Error when restarting the queue.");
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.fd = -1;
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }
+
+    vhost_dev_stop(&net->dev, vdev);
+
+    return r;
+}
include/net/vhost_net.h +2 -0
index 9b3aaf3814..e11a297380 100644
@@ -50,4 +50,6 @@ int vhost_net_set_mtu(struct vhost_net *net, 
 uint16_t mtu);
     void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
                                 int vq_index);
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                                int vq_index);
   #endif
#47 lore raw

Re: [PATCH v2 15/24] vhost-user: add op to enable or disable a single vring

In reply to Kangjie Xu
Quoted 41 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface.

---
  hw/virtio/vhost-user.c            | 26 +++++++++++++++++++-------
  include/hw/virtio/vhost-backend.h |  3 +++
  2 files changed, 22 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +16 -0
index 56033f7a92..8307976cda 100644
@@ -1199,6 +1199,22 @@ static int vhost_user_set_vring_base(struct vhost_dev *dev,
      return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
  }
 
+static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable)
+{
+    if (index < dev->vq_index || index >= dev->vq_index + dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    return vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, &state);
+}
+
  static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
  {
      int i;

hw/virtio/vhost-user.c static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable) · line 1217

Quoted 10 lines
hw/virtio/vhost-user.c +0 -7
@@ -1208,13 +1224,8 @@ static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
      }
 
      for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, &state);

Then I'd squash this into previous patch or re-roder to let this patch (vhost_user_set_single_vring_enable()) to be first.

Thanks

Quoted 34 lines
hw/virtio/vhost-user.c +3 -0
+        int ret = vhost_user_set_single_vring_enable(dev, dev->vq_index + i,
+                                                     enable);
          if (ret < 0) {
              /*
               * Restoring the previous state is likely infeasible, as well as
@@ -2668,6 +2679,7 @@ const VhostOps user_ops = {
          .vhost_reset_vring = vhost_user_reset_vring,
          .vhost_reset_device = vhost_user_reset_device,
          .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable = vhost_user_set_single_vring_enable,
          .vhost_set_vring_enable = vhost_user_set_vring_enable,
          .vhost_requires_shm_log = vhost_user_requires_shm_log,
          .vhost_migration_done = vhost_user_migration_done,
include/hw/virtio/vhost-backend.h +3 -0
index f23bf71a8d..38f6b752ff 100644
@@ -83,6 +83,8 @@ typedef int (*vhost_reset_vring_op)(struct vhost_dev *dev,
                                      struct vhost_vring_state *ring);
  typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
  typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable);
  typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                           int enable);
  typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);
@@ -158,6 +160,7 @@ typedef struct VhostOps {
      vhost_reset_device_op vhost_reset_device;
      vhost_reset_vring_op vhost_reset_vring;
      vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
      vhost_set_vring_enable_op vhost_set_vring_enable;
      vhost_requires_shm_log_op vhost_requires_shm_log;
      vhost_migration_done_op vhost_migration_done;
author
#45 lore raw

Re: [PATCH v2 14/24] vhost-user: introduce vhost_reset_vring() interface

In reply to Jason Wang
Quoted 39 lines · Jason Wang
Quoted 25 lines · Kangjie Xu

Introduce the interface vhost_reset_vring(). The interface is a wrapper to send a VHOST_USER_RESET_VRING message to the back-end. It will reset an individual vring in the back-end. Meanwhile, it will wait for a reply to ensure the reset has been completed.

---
  hw/virtio/vhost-user.c            | 41 +++++++++++++++++++++++++++++++ include/hw/virtio/vhost-backend.h |  3 +++

  2 files changed, 44 insertions(+)
hw/virtio/vhost-user.c +1 -0
index 75b8df21a4..56033f7a92 100644
@@ -126,6 +126,7 @@ typedef enum VhostUserRequest {
       VHOST_USER_GET_MAX_MEM_SLOTS = 36,
       VHOST_USER_ADD_MEM_REG = 37,
       VHOST_USER_REM_MEM_REG = 38,
+    VHOST_USER_RESET_VRING = 41,

      VHOST_USER_MAX
  } VhostUserRequest;
  @@ -1498,6 +1499,45 @@ static int vhost_user_get_max_memslots(struct vhost_dev *dev, return 0;

Quoted 8 lines

  }
  +static int vhost_user_reset_vring(struct vhost_dev *dev, +                                  struct vhost_vring_state *ring) +{
+    int ret;
+    VhostUserMsg msg = {
+        .hdr.request = VHOST_USER_RESET_VRING,
+        .hdr.flags = VHOST_USER_VERSION,

Do we need VHOST_USER_NEED_REPLY_MASK here?

Quoted 4 lines

Other looks good.

Thanks

No, it works the same as vhost_user_get_features() or vhost_user_get_vring_base().

Thanks

Quoted 68 lines
Quoted 68 lines

+        .payload.state = *ring,
+        .hdr.size = sizeof(msg.payload.state),
+    };
+
+    if (!virtio_has_feature(dev->acked_features,
VIRTIO_F_RING_RESET)) {
+        return -ENOTSUP;
+    }

hw/virtio/vhost-user.c +13 -0
+
+    ret = vhost_user_write(dev, &msg, NULL, 0);
+    if (ret < 0) {
+        return ret;
+    }
+
+    ret = vhost_user_read(dev, &msg);
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (msg.hdr.request != VHOST_USER_RESET_VRING) {
+        error_report("Received unexpected msg type. Expected %d 

received %d",
+                     VHOST_USER_RESET_VRING, msg.hdr.request); +        return -EPROTO;
+    }

hw/virtio/vhost-user.c +9 -0
+
+    if (msg.hdr.size != sizeof(msg.payload.state)) {
+        error_report("Received bad msg size.");
+        return -EPROTO;
+    }
+
+    return 0;
+}
+

  static int vhost_user_reset_device(struct vhost_dev *dev) {
      VhostUserMsg msg = {

hw/virtio/vhost-user.c +1 -0
@@ -2625,6 +2665,7 @@ const VhostOps user_ops = {
           .vhost_set_features = vhost_user_set_features,
           .vhost_get_features = vhost_user_get_features,
           .vhost_set_owner = vhost_user_set_owner,
+        .vhost_reset_vring = vhost_user_reset_vring,
           .vhost_reset_device = vhost_user_reset_device,
           .vhost_get_vq_index = vhost_user_get_vq_index,
           .vhost_set_vring_enable = vhost_user_set_vring_enable,
include/hw/virtio/vhost-backend.h +2 -0
index eab46d7f0b..f23bf71a8d 100644
@@ -79,6 +79,8 @@ typedef int (*vhost_get_features_op)(struct 
 vhost_dev *dev,
                                        uint64_t *features);
   typedef int (*vhost_set_backend_cap_op)(struct vhost_dev *dev);
   typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
+typedef int (*vhost_reset_vring_op)(struct vhost_dev *dev,
+                                    struct vhost_vring_state *ring);
   typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
   typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);

  typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,

include/hw/virtio/vhost-backend.h +1 -0
@@ -154,6 +156,7 @@ typedef struct VhostOps {
       vhost_set_backend_cap_op vhost_set_backend_cap;
       vhost_set_owner_op vhost_set_owner;
       vhost_reset_device_op vhost_reset_device;
+    vhost_reset_vring_op vhost_reset_vring;
       vhost_get_vq_index_op vhost_get_vq_index;
       vhost_set_vring_enable_op vhost_set_vring_enable;
       vhost_requires_shm_log_op vhost_requires_shm_log;
author
#48 lore raw

Re: [PATCH v2 15/24] vhost-user: add op to enable or disable a single vring

In reply to Jason Wang
Quoted 58 lines · Jason Wang
Quoted 42 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface.

---
  hw/virtio/vhost-user.c            | 26 +++++++++++++++++++------- include/hw/virtio/vhost-backend.h |  3 +++

  2 files changed, 22 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +15 -0
index 56033f7a92..8307976cda 100644
@@ -1199,6 +1199,22 @@ static int vhost_user_set_vring_base(struct 
 vhost_dev *dev,
       return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
   }
   +static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable)
+{
+    if (index < dev->vq_index || index >= dev->vq_index + dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    return vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, &state);
+}
+
   static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
   {

      int i;

hw/virtio/vhost-user.c +0 -0
@@ -1208,13 +1224,8 @@ static int vhost_user_set_vring_enable(struct 
 vhost_dev *dev, int enable)

hw/virtio/vhost-user.c static int vhost_user_set_vring_enable(struct · line 1227

Quoted 10 lines
hw/virtio/vhost-user.c +0 -7
       }
         for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, 
 &state);

Then I'd squash this into previous patch or re-roder to let this patch (vhost_user_set_single_vring_enable()) to be first.

Quoted 3 lines

Thanks

Sorry, I don't get why we should re-order them, since these two patches are independent.

Thanks

Quoted 37 lines
Quoted 37 lines
hw/virtio/vhost-user.c +2 -0
+        int ret = vhost_user_set_single_vring_enable(dev, 
 dev->vq_index + i,
+                                                     enable);
           if (ret < 0) {

              /*
               * Restoring the previous state is likely infeasible, as well as

hw/virtio/vhost-user.c +1 -0
@@ -2668,6 +2679,7 @@ const VhostOps user_ops = {
           .vhost_reset_vring = vhost_user_reset_vring,
           .vhost_reset_device = vhost_user_reset_device,
           .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable = 
 vhost_user_set_single_vring_enable,
           .vhost_set_vring_enable = vhost_user_set_vring_enable,
           .vhost_requires_shm_log = vhost_user_requires_shm_log,

          .vhost_migration_done = vhost_user_migration_done,

include/hw/virtio/vhost-backend.h +2 -0
index f23bf71a8d..38f6b752ff 100644
@@ -83,6 +83,8 @@ typedef int (*vhost_reset_vring_op)(struct 
 vhost_dev *dev,
                                       struct vhost_vring_state *ring);
   typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
   typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable);
   typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                            int enable);

  typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);

include/hw/virtio/vhost-backend.h +1 -0
@@ -158,6 +160,7 @@ typedef struct VhostOps {
       vhost_reset_device_op vhost_reset_device;
       vhost_reset_vring_op vhost_reset_vring;
       vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
       vhost_set_vring_enable_op vhost_set_vring_enable;
       vhost_requires_shm_log_op vhost_requires_shm_log;
       vhost_migration_done_op vhost_migration_done;
author
#34 lore raw

Re: [PATCH v2 11/24] vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()

In reply to Jason Wang
Quoted 46 lines · Jason Wang
Quoted 30 lines · Kangjie Xu

Introduce vhost_virtqueue_stop(), which can reset the virtqueue in the device. Then it will unmap vrings and the desc of the virtqueue.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

---
  hw/net/vhost_net.c      | 21 +++++++++++++++++++++ include/net/vhost_net.h |  2 ++

  2 files changed, 23 insertions(+)
hw/net/vhost_net.c +8 -0
index ccac5b7a64..aa60dd901c 100644
@@ -514,3 +514,24 @@ int vhost_net_set_mtu(struct vhost_net *net, 
 uint16_t mtu)
         return vhost_ops->vhost_net_set_mtu(&net->dev, mtu);
   }
+
+void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { .fd = -1 };
+    int idx;

hw/net/vhost_net.c int vhost_net_set_mtu(struct vhost_net *net, · line 534

Quoted 10 lines
hw/net/vhost_net.c +10 -0
+
+    assert(vhost_ops);
+
+    idx = vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }

Let's have a vhost_ops here instead of open code it.

Quoted 2 lines

Thanks

I double-checked it, vhost_net_set_backend is already a wrapper of vhost_ops->vhost_net_set_backend().

It seems that, to further simplify it, we can only add idx and fd to the parameter list of vhost_net_set_backend().

Thanks

Quoted 14 lines
Quoted 14 lines
hw/net/vhost_net.c +3 -0
+
+    vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
+}
include/net/vhost_net.h +1 -0
index 387e913e4e..9b3aaf3814 100644
@@ -48,4 +48,6 @@ uint64_t vhost_net_get_acked_features(VHostNetState 
 *net);
     int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
   +void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index);
   #endif
#52 lore raw

Re: [PATCH v2 16/24] vhost: vhost-user: update vhost_dev_virtqueue_stop()

In reply to Kangjie Xu
Quoted 20 lines · Kangjie Xu

Update vhost_dev_virtqueue_stop() for vhost-user scenario.

---
  hw/virtio/vhost.c | 19 +++++++++++++++++++
  1 file changed, 19 insertions(+)
hw/virtio/vhost.c +4 -0
index fc3f550c76..a0d6824353 100644
@@ -1908,10 +1908,29 @@ int vhost_net_set_backend(struct vhost_dev *hdev,
  void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                                int idx)
  {
+    const VhostOps *vhost_ops = hdev->vhost_ops;
+    struct vhost_vring_state state;
+    int r;
+

hw/virtio/vhost.c int vhost_net_set_backend(struct vhost_dev *hdev, · line 1924

Quoted 10 lines
hw/virtio/vhost.c +10 -0
+    assert(vhost_ops);
+
+    if (vhost_ops->vhost_reset_vring) {
+        state.index = hdev->vq_index + idx;
+        r = vhost_ops->vhost_reset_vring(hdev, &state);
+        if (r < 0) {
+            goto err_queue_reset;
+        }
+    }
+

So this worries me:

1) having two similar functions

vhost_virtqueue_stop()

and

vhost_dev_virtqueue_stop()

It can easily confuse the people who want stop the device.

I think we need rename vhost_dev_virtqueue_stop() to vhost_virqtueue_reset() since it has different semantic compared to stop:

1) stop means the virtqueue state is reserved, e.g the index could be synced via get_vring_base()
2) reset means the virqtueue state is lost

Thanks

Quoted 12 lines
hw/virtio/vhost.c +5 -0
      vhost_virtqueue_unmap(hdev,
                            vdev,
                            hdev->vqs + idx,
                            idx);
+
+    return;
+
+err_queue_reset:
+    error_report("Error when stopping the qeuue.");
  }
 
  int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
#55 lore raw

Re: [PATCH v2 17/24] vhost: vhost-user: update vhost_dev_virtqueue_restart()

In reply to Kangjie Xu
Quoted 26 lines · Kangjie Xu

Update vhost_dev_virtqueue_restart() for vhost-user scenario.

---
  hw/virtio/vhost.c | 26 ++++++++++++++++++++++----
  1 file changed, 22 insertions(+), 4 deletions(-)
hw/virtio/vhost.c +3 -4
index a0d6824353..bd90cfe13a 100644
@@ -1937,11 +1937,29 @@ int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
                                  int idx)
  {
      const VhostOps *vhost_ops = hdev->vhost_ops;
+    int r;
 
      assert(vhost_ops);
 
-    return vhost_virtqueue_start(hdev,
-                                 vdev,
-                                 hdev->vqs + idx,
-                                 hdev->vq_index + idx);
+    r = vhost_virtqueue_start(hdev,
+                              vdev,

hw/virtio/vhost.c int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev, · line 1955

Quoted 10 lines
hw/virtio/vhost.c +10 -0
+                              hdev->vqs + idx,
+                              hdev->vq_index + idx);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    if (vhost_ops->vhost_set_single_vring_enable) {
+        r = vhost_ops->vhost_set_single_vring_enable(hdev,
+                                                     hdev->vq_index + idx,
+                                                     1);

Rethink about the name, I think the current names is kind of confusing:

        .vhost_set_single_vring_enable = vhost_user_set_single_vring_enable, .vhost_set_vring_enable = vhost_user_set_vring_enable,

I wonder if it's better:

1) rename vhost_set_vring_enable to vhost_set_dev_enable()

then we can

2) use vhost_set_vring_enable() for per vq enabling?

Thanks

Quoted 10 lines
hw/virtio/vhost.c +9 -0
+        if (r < 0) {
+            goto err_start;
+        }
+    }
+
+    return 0;
+
+err_start:
+    return r;
  }
#58 lore raw

Re: [PATCH v2 18/24] vhost-net: vhost-user: update vhost_net_virtqueue_stop()

In reply to Kangjie Xu
Quoted 1 line · Kangjie Xu

Update vhost_net_virtqueue_stop() for vhost-user scenario.

Let's explain why it is needed now or why it doesn't cause any issue or it's a bug fix or not.

Thanks

Quoted 22 lines
---
  hw/net/vhost_net.c | 4 ++++
  1 file changed, 4 insertions(+)
hw/net/vhost_net.c +4 -0
index 2ab67e875e..c0d408f3b4 100644
@@ -533,6 +533,10 @@ void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
          assert(r >= 0);
      }
 
+    if (net->nc->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        idx = idx - net->dev.vq_index;
+    }
+
      vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
  }
 
#63 lore raw

Re: [PATCH v2 19/24] vhost-net: vhost-user: update vhost_net_virtqueue_restart()

In reply to Kangjie Xu
Quoted 1 line · Kangjie Xu

Update vhost_net_virtqueue_restart() for vhost-user scenario.

Similar to previous patch, let's explain why it it needed.

Thanks

Quoted 22 lines
---
  hw/net/vhost_net.c | 4 ++++
  1 file changed, 4 insertions(+)
hw/net/vhost_net.c +4 -0
index c0d408f3b4..778081e54a 100644
@@ -556,6 +556,10 @@ int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
 
      idx =  vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
 
+    if (net->nc->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        idx = idx - net->dev.vq_index;
+    }
+
      r = vhost_dev_virtqueue_restart(&net->dev, vdev, idx);
      if (r < 0) {
          goto err_start;
#75 lore raw
In reply to Kangjie Xu
Quoted 82 lines · Kangjie Xu

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

     https://github.com/oasis-tcs/virtio-spec/issues/124
     https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of several parts:
1. Patches 1-7 are the basic interfaces for vq reset in virtio and virtio-pci.
2. Patches 8-12 support vq stop and vq restart for vhost-kernel.
3. Patches 13-19 support vq stop and vq restart for vhost-user.
4. Patches 20-22 support vq reset and re-enable for virtio-net.
5. Patches 23-24 enable the vq reset feature for vhost-kernel and vhost-user.

The process of virtqueue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue can be optionally restarted(re-enabled).

Since this patch set involves multiple modules and seems a bit messy, we briefly describe the calling process for different modes below.
virtio-net:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

     -> virtio_queue_reset() [virtio]
         -> virtio_net_queue_reset() [virtio-net]
         -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
     -> set enabled, reset status of vq.

vhost-kernel:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

     -> virtio_queue_reset() [virtio]
         -> virtio_net_queue_reset() [virtio-net]
             -> vhost_net_virtqueue_stop() [vhost-net]
                 -> vhost_net_set_backend() [vhost]
                 -> vhost_dev_virtqueue_stop()
                     -> vhost_virtqueue_unmap()
         -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
     -> virtio_queue_enable() [virtio]
         -> virtio_net_queue_enable() [virtio-net]
             -> vhost_net_virtqueue_restart() [vhost-net]
                 -> vhost_dev_virtqueue_restart() [vhost]
                     -> vhost_virtqueue_start()
                 -> vhost_net_set_backend()
     -> set enabled, reset status of vq.

vhost-user:
1. VIRTIO_PCI_COMMON_Q_RESET is written [virtio-pci]

     -> virtio_queue_reset() [virtio]
         -> virtio_net_queue_reset() [virtio-net]
             -> vhost_net_virtqueue_stop() [vhost-net]
                 -> vhost_dev_virtqueue_stop() [vhost]
                     -> vhost_user_reset_vring() [vhost-user]
                         -> send VHOST_USER_RESET_VRING to the device
                     -> vhost_virtqueue_unmap()
         -> __virtio_queue_reset()
2. VIRTIO_PCI_COMMON_Q_ENABLE is written [virtio-pci]
     -> virtio_queue_enable() [virtio]
         -> virtio_net_queue_enable() [virtio-net]
             -> vhost_net_virtqueue_restart() [vhost-net]
                 -> vhost_dev_virtqueue_restart() [vhost]
                     -> vhost_virtqueue_start()
                     -> vhost_user_set_single_vring_enable [vhost-user]
                         -> send VHOST_USER_SET_VRING_ENABLE to the device
     -> set enabled, reset status of vq.

Test environment:
Host: 5.19.0-rc3 (With vq reset support)
Qemu: QEMU emulator version 7.0.50
Guest: 5.19.0-rc3 (With vq reset support)
DPDK: 22.07-rc1 (With vq reset support)
Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

The drvier can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

Guest Kernel Patch:

     https://lore.kernel.org/bpf/[email protected]/
Quoted 7 lines

DPDK Patch:

     https://github.com/middaywords/dpdk/compare/72206323a5dd3182b13f61b25a64abdddfee595c...eabadfac7953da66bc10ffb8284b490d09bb7ec7

Host Kernel Patch:

     https://github.com/middaywords/linux/commit/19a91e0d7167b2031e46078c6215c213b89cb2c3

Looking forward to your review and comments. Thanks.

I suggest to split vhost-user part into another series since:

1) vhost-net kernel is ready for this future

2) vhost-user probably requires some extensions of the current protocol

Then it can speed up the merging and reduce the rebasing of a huge series.

Thanks

Quoted 46 lines

Kangjie Xu (19):
virtio: introduce virtio_queue_enable()
virtio: core: vq reset feature negotation support
virtio-pci: support queue enable
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce vhost_dev_virtqueue_stop()
vhost: introduce vhost_dev_virtqueue_restart()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()
vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()
docs: vhost-user: add VHOST_USER_RESET_VRING message
vhost-user: introduce vhost_reset_vring() interface
vhost-user: add op to enable or disable a single vring
vhost: vhost-user: update vhost_dev_virtqueue_stop()
vhost: vhost-user: update vhost_dev_virtqueue_restart()
vhost-net: vhost-user: update vhost_net_virtqueue_stop()
vhost-net: vhost-user: update vhost_net_virtqueue_restart()
virtio-net: introduce flush_or_purge_queued_packets()
virtio-net: support queue_enable
vhost: vhost-kernel: enable vq reset feature
vhost: vhost-user: enable vq reset feature

Xuan Zhuo (5):
virtio: sync relevant definitions with linux
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

  docs/interop/vhost-user.rst                   | 10 +++
  hw/core/machine.c                             |  1 +
  hw/net/vhost_net.c                            | 79 +++++++++++++++++++
  hw/net/virtio-net.c                           | 58 ++++++++++++--
  hw/virtio/vhost-user.c                        | 67 ++++++++++++++--
  hw/virtio/vhost.c                             | 79 +++++++++++++++++--
  hw/virtio/virtio-pci.c                        | 20 +++++
  hw/virtio/virtio.c                            | 62 +++++++++++----
  include/hw/virtio/vhost-backend.h             |  6 ++
  include/hw/virtio/vhost.h                     |  5 ++
  include/hw/virtio/virtio-pci.h                |  1 +
  include/hw/virtio/virtio.h                    |  8 +-
  include/net/vhost_net.h                       |  4 +
  .../standard-headers/linux/virtio_config.h    |  5 ++
  include/standard-headers/linux/virtio_pci.h   |  2 +
  15 files changed, 371 insertions(+), 36 deletions(-)
author
#59 lore raw

Re: [PATCH v2 18/24] vhost-net: vhost-user: update vhost_net_virtqueue_stop()

In reply to Jason Wang
Quoted 9 lines · Jason Wang
Quoted 1 line · Kangjie Xu

Update vhost_net_virtqueue_stop() for vhost-user scenario.

Let's explain why it is needed now or why it doesn't cause any issue or it's a bug fix or not.

Thanks

This patch is to suppport vq reset for vhost-user.

We need this simply because the behavior of vhost_ops->get_vq_index() is different in vhost-user and vhost-kernel.

vhost_user_get_vq_index(dev, idx) simply returns "idx".

vhost_kernel_get_vq_index(dev, idx) returns "idx - dev->vq_index".

Thanks

Quoted 22 lines
Quoted 22 lines

---
  hw/net/vhost_net.c | 4 ++++

  1 file changed, 4 insertions(+)
hw/net/vhost_net.c +3 -0
index 2ab67e875e..c0d408f3b4 100644
@@ -533,6 +533,10 @@ void vhost_net_virtqueue_stop(VirtIODevice 
 *vdev, NetClientState *nc,
           assert(r >= 0);
       }
   +    if (net->nc->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        idx = idx - net->dev.vq_index;
+    }
+
       vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
   }
author
#53 lore raw

Re: [PATCH v2 16/24] vhost: vhost-user: update vhost_dev_virtqueue_stop()

In reply to Jason Wang
Quoted 45 lines · Jason Wang
Quoted 21 lines · Kangjie Xu

Update vhost_dev_virtqueue_stop() for vhost-user scenario.

---
  hw/virtio/vhost.c | 19 +++++++++++++++++++

  1 file changed, 19 insertions(+)
hw/virtio/vhost.c +4 -0
index fc3f550c76..a0d6824353 100644
@@ -1908,10 +1908,29 @@ int vhost_net_set_backend(struct vhost_dev 
 *hdev,
   void vhost_dev_virtqueue_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                                 int idx)
   {
+    const VhostOps *vhost_ops = hdev->vhost_ops;
+    struct vhost_vring_state state;
+    int r;
+

hw/virtio/vhost.c int vhost_net_set_backend(struct vhost_dev · line 1925

Quoted 10 lines
hw/virtio/vhost.c +10 -0
+    assert(vhost_ops);
+
+    if (vhost_ops->vhost_reset_vring) {
+        state.index = hdev->vq_index + idx;
+        r = vhost_ops->vhost_reset_vring(hdev, &state);
+        if (r < 0) {
+            goto err_queue_reset;
+        }
+    }
+

So this worries me:

1) having two similar functions

vhost_virtqueue_stop()

and

vhost_dev_virtqueue_stop()

Quoted 10 lines

It can easily confuse the people who want stop the device.

I think we need rename vhost_dev_virtqueue_stop() to vhost_virqtueue_reset() since it has different semantic compared to stop:

1) stop means the virtqueue state is reserved, e.g the index could be synced via get_vring_base()
2) reset means the virqtueue state is lost

Thanks

Totally agree, will fix.

Thanks

Quoted 11 lines
Quoted 11 lines
hw/virtio/vhost.c +5 -0
       vhost_virtqueue_unmap(hdev,
                             vdev,
                             hdev->vqs + idx,
                             idx);
+
+    return;
+
+err_queue_reset:
+    error_report("Error when stopping the qeuue.");
   }
     int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
author
#56 lore raw

Re: [PATCH v2 17/24] vhost: vhost-user: update vhost_dev_virtqueue_restart()

In reply to Jason Wang
Quoted 46 lines · Jason Wang
Quoted 25 lines · Kangjie Xu

Update vhost_dev_virtqueue_restart() for vhost-user scenario.

---
  hw/virtio/vhost.c | 26 ++++++++++++++++++++++----

  1 file changed, 22 insertions(+), 4 deletions(-)
hw/virtio/vhost.c +3 -3
index a0d6824353..bd90cfe13a 100644
@@ -1937,11 +1937,29 @@ int vhost_dev_virtqueue_restart(struct 
 vhost_dev *hdev, VirtIODevice *vdev,
                                   int idx)
   {
       const VhostOps *vhost_ops = hdev->vhost_ops;
+    int r;
         assert(vhost_ops);
   -    return vhost_virtqueue_start(hdev,
-                                 vdev,
-                                 hdev->vqs + idx,
-                                 hdev->vq_index + idx);
+    r = vhost_virtqueue_start(hdev,
+                              vdev,

hw/virtio/vhost.c int vhost_dev_virtqueue_restart(struct · line 1955

Quoted 10 lines
hw/virtio/vhost.c +10 -0
+                              hdev->vqs + idx,
+                              hdev->vq_index + idx);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    if (vhost_ops->vhost_set_single_vring_enable) {
+        r = vhost_ops->vhost_set_single_vring_enable(hdev,
+ hdev->vq_index + idx,
+                                                     1);

Rethink about the name, I think the current names is kind of confusing:

        .vhost_set_single_vring_enable = vhost_user_set_single_vring_enable, .vhost_set_vring_enable = vhost_user_set_vring_enable,

I wonder if it's better:

Quoted 9 lines

1) rename vhost_set_vring_enable to vhost_set_dev_enable()

then we can

2) use vhost_set_vring_enable() for per vq enabling?

Thanks

I agree. It looks better, I will refactor this part.

Thanks.

Quoted 10 lines
Quoted 10 lines
hw/virtio/vhost.c +9 -0
+        if (r < 0) {
+            goto err_start;
+        }
+    }
+
+    return 0;
+
+err_start:
+    return r;
   }
#35 lore raw

Re: [PATCH v2 11/24] vhost-net: vhost-kernel: introduce vhost_net_virtqueue_stop()

In reply to Kangjie Xu
Quoted 48 lines · Kangjie Xu
Quoted 46 lines · Jason Wang
Quoted 30 lines · Kangjie Xu

Introduce vhost_virtqueue_stop(), which can reset the virtqueue in the device. Then it will unmap vrings and the desc of the virtqueue.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

---
  hw/net/vhost_net.c      | 21 +++++++++++++++++++++ include/net/vhost_net.h |  2 ++

  2 files changed, 23 insertions(+)
hw/net/vhost_net.c +8 -0
index ccac5b7a64..aa60dd901c 100644
@@ -514,3 +514,24 @@ int vhost_net_set_mtu(struct vhost_net *net, 
 uint16_t mtu)
         return vhost_ops->vhost_net_set_mtu(&net->dev, mtu);
   }
+
+void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { .fd = -1 };
+    int idx;

hw/net/vhost_net.c int vhost_net_set_mtu(struct vhost_net *net, · line 534

Quoted 10 lines
hw/net/vhost_net.c +10 -0
+
+    assert(vhost_ops);
+
+    idx = vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }

Let's have a vhost_ops here instead of open code it.

Quoted 5 lines
Quoted 2 lines

Thanks

I double-checked it, vhost_net_set_backend is already a wrapper of vhost_ops->vhost_net_set_backend().

It seems that, to further simplify it, we can only add idx and fd to the parameter list of vhost_net_set_backend().

Ok, so we can leave it as is. (Probably need a vhost_net_ops in the future).

Thanks

Quoted 18 lines

Thanks

Quoted 14 lines
Quoted 14 lines
hw/net/vhost_net.c +3 -0
+
+    vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
+}
include/net/vhost_net.h +1 -0
index 387e913e4e..9b3aaf3814 100644
@@ -48,4 +48,6 @@ uint64_t 
 vhost_net_get_acked_features(VHostNetState *net);
     int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
   +void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                              int vq_index);
   #endif
#18 lore raw

Re: [PATCH v2 07/24] virtio-pci: support queue enable

In reply to Kangjie Xu
Quoted 37 lines · Kangjie Xu
Quoted 8 lines · Jason Wang
Quoted 1 line · Kangjie Xu

PCI devices support vq enable.

Nit: it might be "support device specific vq enable"

Get it.

Quoted 26 lines
Quoted 17 lines

Based on this function, the driver can re-enable the virtqueue after the virtqueue is reset.

---
  hw/virtio/virtio-pci.c | 1 +

  1 file changed, 1 insertion(+)
hw/virtio/virtio-pci.c +0 -0
index ec8e92052f..3d560e45ad 100644
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void 
 *opaque, hwaddr addr,
 proxy->vqs[vdev->queue_sel].avail[0],
Quoted 5 lines

((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 | proxy->vqs[vdev->queue_sel].used[0]);
+            virtio_queue_enable(vdev, vdev->queue_sel); proxy->vqs[vdev->queue_sel].enabled = 1; proxy->vqs[vdev->queue_sel].reset = 0;

Any reason we do it before the assignment of 1? It probably means the device specific method can't depend on virtio_queue_enabled()?

Quoted 3 lines
Quoted 2 lines

Thanks

Sorry, I don't get why device specific method can't depend on virtio_queue_enabled().

I meant if the device specific method call virtio_queue_enabled() it will return false in this case, is this intended?

Quoted 4 lines

Before virtio_queue_enable() is done, virtqueue should always be not ready and disabled.

Otherwise, If we put it after the assignment of enabled to 1, the virtqueue may be accessed illegally and may cause panic, because the virtqueue is still being intialized and being configured.

How? Shouldn't we make transport ready before making device virtqueue(device) ready?

Thanks

Quoted 6 lines

Thanks

Quoted 2 lines
Quoted 2 lines

          } else {

#14 lore raw

Re: [PATCH v2 06/24] virtio-pci: support queue reset

In reply to Kangjie Xu
Quoted 97 lines · Kangjie Xu
Quoted 52 lines · Jason Wang
Quoted 50 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

PCI devices support vq reset.

Based on this function, the driver can adjust the size of the ring, and quickly recycle the buffer in the ring.

The migration of the virtio devices will not happen during a reset operation. This is becuase the global iothread lock is held. Migration thread also needs the lock. As a result, we do not need to migrate the reset state of VirtIOPCIQueue.

---
  hw/virtio/virtio-pci.c         | 19 +++++++++++++++++++ include/hw/virtio/virtio-pci.h |  1 +

  2 files changed, 20 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index 45327f0b31..ec8e92052f 100644
@@ -1246,6 +1246,9 @@ static uint64_t virtio_pci_common_read(void 
 *opaque, hwaddr addr,

      case VIRTIO_PCI_COMMON_Q_USEDHI:
          val = proxy->vqs[vdev->queue_sel].used[1]; break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        val = proxy->vqs[vdev->queue_sel].reset; +        break;
      default:
          val = 0;
      }

hw/virtio/virtio-pci.c +0 -0
@@ -1333,6 +1336,7 @@ static void virtio_pci_common_write(void 
 *opaque, hwaddr addr,

((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 | proxy->vqs[vdev->queue_sel].used[0]); proxy->vqs[vdev->queue_sel].enabled = 1;
+            proxy->vqs[vdev->queue_sel].reset = 0;
          } else {
              virtio_error(vdev, "wrong value for queue_enable %"PRIx64, val); }

hw/virtio/virtio-pci.c +0 -0
@@ -1355,6 +1359,20 @@ static void virtio_pci_common_write(void 
 *opaque, hwaddr addr,

      case VIRTIO_PCI_COMMON_Q_USEDHI:
          proxy->vqs[vdev->queue_sel].used[1] = val; break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        if (val == 1) {

Quoted 9 lines
Quoted 5 lines

+            /*
+             * With the global iothread lock taken, the migration will not
+             * happen until the virtqueue reset is done. +             */

This comment applies to all other common cfg operation as well, So it looks not necessary?

Get it.

Quoted 32 lines
Quoted 24 lines

+ proxy->vqs[vdev->queue_sel].reset = 1;

hw/virtio/virtio-pci.c +7 -0
+
+            virtio_queue_reset(vdev, vdev->queue_sel);
+
+            proxy->vqs[vdev->queue_sel].reset = 0;
+            proxy->vqs[vdev->queue_sel].enabled = 0;
+        }
+        break;

      default:
          break;
      }

hw/virtio/virtio-pci.c +1 -0
@@ -1950,6 +1968,7 @@ static void virtio_pci_reset(DeviceState *qdev)
         for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
           proxy->vqs[i].enabled = 0;
+        proxy->vqs[i].reset = 0;
           proxy->vqs[i].num = 0;
           proxy->vqs[i].desc[0] = proxy->vqs[i].desc[1] = 0;
           proxy->vqs[i].avail[0] = proxy->vqs[i].avail[1] = 0;
include/hw/virtio/virtio-pci.h +0 -0
index 2446dcd9ae..e9290e2b94 100644

include/hw/virtio/virtio-pci.h typedef struct VirtIOPCIRegion { · line 120

Quoted 4 lines
include/hw/virtio/virtio-pci.h +1 -0
@@ -117,6 +117,7 @@ typedef struct VirtIOPCIRegion {
   typedef struct VirtIOPCIQueue {
     uint16_t num;
     bool enabled;
+  bool reset;

Do we need to migrate this?

Quoted 7 lines
Quoted 2 lines

Thanks

I think we do not need to migrate this because we hold the global iothread lock when virtqueue reset is triggered. The migration of these device states also needs this lock.

On the other hand, the 'reset' state of virtqueue is same(is 0) before and after the process of resetting a virtqueue.

Thus, the migration will not happen when we are resetting a virtqueue and we do not to migrate it.

Ok, let's add a comment above reset to explain this.

Thanks

Quoted 8 lines

Thanks

Quoted 4 lines
Quoted 4 lines
include/hw/virtio/virtio-pci.h +0 -0
     uint32_t desc[2];
     uint32_t avail[2];
     uint32_t used[2];
#60 lore raw

Re: [PATCH v2 18/24] vhost-net: vhost-user: update vhost_net_virtqueue_stop()

In reply to Kangjie Xu
Quoted 13 lines · Kangjie Xu
Quoted 9 lines · Jason Wang
Quoted 1 line · Kangjie Xu

Update vhost_net_virtqueue_stop() for vhost-user scenario.

Let's explain why it is needed now or why it doesn't cause any issue or it's a bug fix or not.

Thanks

This patch is to suppport vq reset for vhost-user.

Quoted 7 lines

We need this simply because the behavior of vhost_ops->get_vq_index() is different in vhost-user and vhost-kernel.

vhost_user_get_vq_index(dev, idx) simply returns "idx".

vhost_kernel_get_vq_index(dev, idx) returns "idx - dev->vq_index".

Thanks

Let's add them in the change-log in the next version.

But the question still, is this a bug fix (requires a Fixes tag)? If not why do we need this now?

Thanks

Quoted 24 lines
Quoted 23 lines
Quoted 23 lines

---
  hw/net/vhost_net.c | 4 ++++

  1 file changed, 4 insertions(+)
hw/net/vhost_net.c +3 -0
index 2ab67e875e..c0d408f3b4 100644
@@ -533,6 +533,10 @@ void vhost_net_virtqueue_stop(VirtIODevice 
 *vdev, NetClientState *nc,
           assert(r >= 0);
       }
   +    if (net->nc->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        idx = idx - net->dev.vq_index;
+    }
+
       vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
   }
#39 lore raw

Re: [PATCH v2 12/24] vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()

In reply to Kangjie Xu
Quoted 17 lines · Kangjie Xu
Quoted 15 lines · Jason Wang
Quoted 9 lines · Kangjie Xu

Introduce vhost_net_virtqueue_restart(), which can restart the virtqueue when the vhost net started running before. If it fails to restart the virtqueue, the device will be stopped.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

I would explain why current vhost_net_start_one()/vhost_net_stop_one() can't work. Is it because it works at queue pair level? If yes can we restructure the code and try to reuse ?

Quoted 7 lines
Quoted 2 lines

Thanks

Because vhost_net_start_one()/vhost_net_stop_one() works at device level.

The queue pair level start/stop are vhost_virtqueue_start() and vhost_virtqueue_stop().

What we can reuse is the vhost_virtqueue_start(). vhost_virtqueue_stop() cannot be reused because it will destroy device.

Let's add this in the changelog or a comment in the code.

Thanks

Quoted 82 lines

I think we do not need to restructure because we've already had an abstraction vhost_virtqueue_start().

Thanks.

Quoted 76 lines
Quoted 76 lines

---
  hw/net/vhost_net.c      | 48
+++++++++++++++++++++++++++++++++++++++++
  include/net/vhost_net.h |  2 ++

  2 files changed, 50 insertions(+)
hw/net/vhost_net.c +2 -0
index aa60dd901c..2ab67e875e 100644
@@ -535,3 +535,51 @@ void vhost_net_virtqueue_stop(VirtIODevice 
 *vdev, NetClientState *nc,
         vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
   }
+
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState 

*nc,
+                                int vq_index)

hw/net/vhost_net.c +45 -0
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { };
+    int idx, r;
+
+    if (!net->dev.started) {
+        return 0;
+    }
+
+    assert(vhost_ops);
+
+    idx =  vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    r = vhost_dev_virtqueue_restart(&net->dev, vdev, idx);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        file.fd = net->backend;
+        r = vhost_net_set_backend(&net->dev, &file);
+        if (r < 0) {
+            r = -errno;
+            goto err_start;
+        }
+    }
+
+    return 0;
+
+err_start:
+    error_report("Error when restarting the queue.");
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.fd = -1;
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }
+
+    vhost_dev_stop(&net->dev, vdev);
+
+    return r;
+}
include/net/vhost_net.h +2 -0
index 9b3aaf3814..e11a297380 100644
@@ -50,4 +50,6 @@ int vhost_net_set_mtu(struct vhost_net *net, 
 uint16_t mtu);
     void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
                                 int vq_index);
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState 
 *nc,
+                                int vq_index);

  #endif

author
#40 lore raw

Re: [PATCH v2 12/24] vhost-net: vhost-kernel: introduce vhost_net_virtqueue_restart()

Quoted 23 lines · Jason Wang
Quoted 21 lines · Kangjie Xu
Quoted 9 lines · Jason Wang
Quoted 7 lines · Kangjie Xu

Introduce vhost_net_virtqueue_restart(), which can restart the virtqueue when the vhost net started running before. If it fails to restart the virtqueue, the device will be stopped.

This patch only considers the case for vhost-kernel, when NetClientDriver is NET_CLIENT_DRIVER_TAP.

Quoted 8 lines
Quoted 2 lines

I would explain why current vhost_net_start_one()/vhost_net_stop_one() can't work. Is it because it works at queue pair level? If yes can we restructure the code and try to reuse ?

Thanks

Because vhost_net_start_one()/vhost_net_stop_one() works at device level.

Quoted 10 lines
Quoted 3 lines

The queue pair level start/stop are vhost_virtqueue_start() and vhost_virtqueue_stop().

What we can reuse is the vhost_virtqueue_start(). vhost_virtqueue_stop() cannot be reused because it will destroy device.

Let's add this in the changelog or a comment in the code.

Thanks

Will fix.

Thanks.

Quoted 82 lines
Quoted 82 lines

I think we do not need to restructure because we've already had an abstraction vhost_virtqueue_start().

Thanks.

Quoted 76 lines
Quoted 76 lines

---
  hw/net/vhost_net.c      | 48
+++++++++++++++++++++++++++++++++++++++++
  include/net/vhost_net.h |  2 ++

  2 files changed, 50 insertions(+)
hw/net/vhost_net.c +2 -0
index aa60dd901c..2ab67e875e 100644
@@ -535,3 +535,51 @@ void vhost_net_virtqueue_stop(VirtIODevice 
 *vdev, NetClientState *nc,
         vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
   }
+
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState 

*nc,
+                                int vq_index)

hw/net/vhost_net.c +45 -0
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    struct vhost_vring_file file = { };
+    int idx, r;
+
+    if (!net->dev.started) {
+        return 0;
+    }
+
+    assert(vhost_ops);
+
+    idx = vhost_ops->vhost_get_vq_index(&net->dev, vq_index);
+
+    r = vhost_dev_virtqueue_restart(&net->dev, vdev, idx);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.index = idx;
+        file.fd = net->backend;
+        r = vhost_net_set_backend(&net->dev, &file);
+        if (r < 0) {
+            r = -errno;
+            goto err_start;
+        }
+    }
+
+    return 0;
+
+err_start:
+    error_report("Error when restarting the queue.");
+
+    if (net->nc->info->type == NET_CLIENT_DRIVER_TAP) {
+        file.fd = -1;
+        file.index = idx;
+        int r = vhost_net_set_backend(&net->dev, &file);
+        assert(r >= 0);
+    }
+
+    vhost_dev_stop(&net->dev, vdev);
+
+    return r;
+}
include/net/vhost_net.h +2 -0
index 9b3aaf3814..e11a297380 100644
@@ -50,4 +50,6 @@ int vhost_net_set_mtu(struct vhost_net *net, 
 uint16_t mtu);
     void vhost_net_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
                                 int vq_index);
+int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState 
 *nc,
+                                int vq_index);

  #endif

#49 lore raw

Re: [PATCH v2 15/24] vhost-user: add op to enable or disable a single vring

In reply to Kangjie Xu
Quoted 60 lines · Kangjie Xu
Quoted 58 lines · Jason Wang
Quoted 42 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface.

---
  hw/virtio/vhost-user.c            | 26 +++++++++++++++++++------- include/hw/virtio/vhost-backend.h |  3 +++

  2 files changed, 22 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +15 -0
index 56033f7a92..8307976cda 100644
@@ -1199,6 +1199,22 @@ static int vhost_user_set_vring_base(struct 
 vhost_dev *dev,
       return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
   }
   +static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable)
+{
+    if (index < dev->vq_index || index >= dev->vq_index + dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    return vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, &state);
+}
+
   static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
   {

      int i;

hw/virtio/vhost-user.c +0 -0
@@ -1208,13 +1224,8 @@ static int vhost_user_set_vring_enable(struct 
 vhost_dev *dev, int enable)

hw/virtio/vhost-user.c static int vhost_user_set_vring_enable(struct · line 1227

Quoted 10 lines
hw/virtio/vhost-user.c +0 -7
       }
         for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, 
 &state);

Then I'd squash this into previous patch or re-roder to let this patch (vhost_user_set_single_vring_enable()) to be first.

Quoted 4 lines
Quoted 3 lines

Thanks

Sorry, I don't get why we should re-order them, since these two patches are independent.

I meant it's not good to introduce some codes in patch 14 but delete them in patch 15 (the above part for example).

Thanks

Quoted 42 lines

Thanks

Quoted 38 lines
Quoted 38 lines
hw/virtio/vhost-user.c +2 -0
+        int ret = vhost_user_set_single_vring_enable(dev, 
 dev->vq_index + i,
+                                                     enable);
           if (ret < 0) {

              /*
               * Restoring the previous state is likely infeasible, as well as

hw/virtio/vhost-user.c +1 -0
@@ -2668,6 +2679,7 @@ const VhostOps user_ops = {
           .vhost_reset_vring = vhost_user_reset_vring,
           .vhost_reset_device = vhost_user_reset_device,
           .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable = 
 vhost_user_set_single_vring_enable,
           .vhost_set_vring_enable = vhost_user_set_vring_enable,
           .vhost_requires_shm_log = vhost_user_requires_shm_log,

          .vhost_migration_done = vhost_user_migration_done,

include/hw/virtio/vhost-backend.h +2 -0
index f23bf71a8d..38f6b752ff 100644
@@ -83,6 +83,8 @@ typedef int (*vhost_reset_vring_op)(struct 
 vhost_dev *dev,
                                       struct vhost_vring_state *ring);
   typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
   typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int 
 enable);
   typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,

                                           int enable); typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);

include/hw/virtio/vhost-backend.h +1 -0
@@ -158,6 +160,7 @@ typedef struct VhostOps {
       vhost_reset_device_op vhost_reset_device;
       vhost_reset_vring_op vhost_reset_vring;
       vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
       vhost_set_vring_enable_op vhost_set_vring_enable;
       vhost_requires_shm_log_op vhost_requires_shm_log;
       vhost_migration_done_op vhost_migration_done;
author
#61 lore raw

Re: [PATCH v2 18/24] vhost-net: vhost-user: update vhost_net_virtqueue_stop()

In reply to Jason Wang
Quoted 17 lines · Jason Wang
Quoted 15 lines · Kangjie Xu
Quoted 9 lines · Jason Wang
Quoted 1 line · Kangjie Xu

Update vhost_net_virtqueue_stop() for vhost-user scenario.

Let's explain why it is needed now or why it doesn't cause any issue or it's a bug fix or not.

Thanks

This patch is to suppport vq reset for vhost-user.

We need this simply because the behavior of vhost_ops->get_vq_index() is different in vhost-user and vhost-kernel.

Quoted 8 lines
Quoted 5 lines

vhost_user_get_vq_index(dev, idx) simply returns "idx".

vhost_kernel_get_vq_index(dev, idx) returns "idx - dev->vq_index".

Thanks

Let's add them in the change-log in the next version.

Sorry, i don't get what to be changed here, could you explain it?

Quoted 5 lines

But the question still, is this a bug fix (requires a Fixes tag)? If not why do we need this now?

Thanks

Actually, it is not a bugfix, it is simply intended to support vhost-user.

Because vhost_ops->get_vq_index returns different values for vhost-kernel and vhost-user.

To align vhost-kernel and vhost-user and reuse the following code,

    vhost_dev_virtqueue_stop(&net->dev, vdev, idx);

we process the 'idx' here for vhost-user specifically.

Thanks.

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 24 lines

---
  hw/net/vhost_net.c | 4 ++++

  1 file changed, 4 insertions(+)
hw/net/vhost_net.c +3 -0
index 2ab67e875e..c0d408f3b4 100644
@@ -533,6 +533,10 @@ void vhost_net_virtqueue_stop(VirtIODevice 
 *vdev, NetClientState *nc,
           assert(r >= 0);
       }
   +    if (net->nc->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        idx = idx - net->dev.vq_index;
+    }
+
       vhost_dev_virtqueue_stop(&net->dev, vdev, idx);
   }
author
#50 lore raw

Re: [PATCH v2 15/24] vhost-user: add op to enable or disable a single vring

In reply to Jason Wang
Quoted 66 lines · Jason Wang
Quoted 64 lines · Kangjie Xu
Quoted 62 lines · Jason Wang
Quoted 43 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface.

---
  hw/virtio/vhost-user.c            | 26 +++++++++++++++++++------- include/hw/virtio/vhost-backend.h |  3 +++

  2 files changed, 22 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +15 -0
index 56033f7a92..8307976cda 100644
@@ -1199,6 +1199,22 @@ static int vhost_user_set_vring_base(struct 
 vhost_dev *dev,
       return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
   }
   +static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable)
+{
+    if (index < dev->vq_index || index >= dev->vq_index + 
 dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    return vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, &state);
+}
+
   static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)

  {
      int i;

hw/virtio/vhost-user.c +0 -0
@@ -1208,13 +1224,8 @@ static int 
 vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)

hw/virtio/vhost-user.c static int · line 1227

Quoted 10 lines
hw/virtio/vhost-user.c +0 -7
       }
         for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, 
 &state);

Then I'd squash this into previous patch or re-roder to let this patch (vhost_user_set_single_vring_enable()) to be first.

Thanks

Quoted 6 lines
Quoted 1 line

Sorry, I don't get why we should re-order them, since these two patches are independent.

I meant it's not good to introduce some codes in patch 14 but delete them in patch 15 (the above part for example).

Thanks

I get your point, but in fact, it seems that the deleded codes here in patch 15 do not appear in patch 14....

Patch 14 is about vhost_user_reset_vring(), patch 15 is about vhost_user_set_vring_enable().

Thanks.

Quoted 43 lines
Quoted 43 lines

Thanks

Quoted 39 lines
Quoted 39 lines
hw/virtio/vhost-user.c +2 -0
+        int ret = vhost_user_set_single_vring_enable(dev, 
 dev->vq_index + i,
+ enable);
           if (ret < 0) {

              /*
               * Restoring the previous state is likely infeasible, as well as

hw/virtio/vhost-user.c +1 -0
@@ -2668,6 +2679,7 @@ const VhostOps user_ops = {
           .vhost_reset_vring = vhost_user_reset_vring,
           .vhost_reset_device = vhost_user_reset_device,
           .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable = 
 vhost_user_set_single_vring_enable,
           .vhost_set_vring_enable = vhost_user_set_vring_enable,
           .vhost_requires_shm_log = vhost_user_requires_shm_log,

          .vhost_migration_done = vhost_user_migration_done,

include/hw/virtio/vhost-backend.h +2 -0
index f23bf71a8d..38f6b752ff 100644
@@ -83,6 +83,8 @@ typedef int (*vhost_reset_vring_op)(struct 
 vhost_dev *dev,
                                       struct vhost_vring_state *ring);
   typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
   typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev 
 *dev,
+                                                int index, int 
 enable);

  typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev, int enable); typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);

include/hw/virtio/vhost-backend.h +1 -0
@@ -158,6 +160,7 @@ typedef struct VhostOps {
       vhost_reset_device_op vhost_reset_device;
       vhost_reset_vring_op vhost_reset_vring;
       vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
       vhost_set_vring_enable_op vhost_set_vring_enable;
       vhost_requires_shm_log_op vhost_requires_shm_log;
       vhost_migration_done_op vhost_migration_done;
author
#19 lore raw

Re: [PATCH v2 07/24] virtio-pci: support queue enable

In reply to Jason Wang
Quoted 39 lines · Jason Wang
Quoted 37 lines · Kangjie Xu
Quoted 8 lines · Jason Wang
Quoted 1 line · Kangjie Xu

PCI devices support vq enable.

Nit: it might be "support device specific vq enable"

Get it.

Quoted 26 lines
Quoted 17 lines

Based on this function, the driver can re-enable the virtqueue after the virtqueue is reset.

---
  hw/virtio/virtio-pci.c | 1 +

  1 file changed, 1 insertion(+)
hw/virtio/virtio-pci.c +0 -0
index ec8e92052f..3d560e45ad 100644
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void 
 *opaque, hwaddr addr,
 proxy->vqs[vdev->queue_sel].avail[0],
Quoted 5 lines

((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 | proxy->vqs[vdev->queue_sel].used[0]);
+            virtio_queue_enable(vdev, vdev->queue_sel); proxy->vqs[vdev->queue_sel].enabled = 1; proxy->vqs[vdev->queue_sel].reset = 0;

Any reason we do it before the assignment of 1? It probably means the device specific method can't depend on virtio_queue_enabled()?

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Thanks

Sorry, I don't get why device specific method can't depend on virtio_queue_enabled().

I meant if the device specific method call virtio_queue_enabled() it will return false in this case, is this intended?

Yes, I intend it to behave in this way.

Quoted 11 lines
Quoted 5 lines

Before virtio_queue_enable() is done, virtqueue should always be not ready and disabled.

Otherwise, If we put it after the assignment of enabled to 1, the virtqueue may be accessed illegally and may cause panic, because the virtqueue is still being intialized and being configured.

How? Shouldn't we make transport ready before making device virtqueue(device) ready?

Thanks

I am not experienced in this field, could you tell me why we should make the transport ready first?

I make the transport ready later than making device ready for two aspects:

1. In QEMU, the virtio_queue_enabled() is used only when we start the device/queue pair (vhost_dev_start_one), or reading VIRTIO_PCI_COMMON_Q_ENABLE. These two operations and resetting the queue will *be synchronized* using iothread lock, so we do not need to worry about the case currently.

2. Suppose we use virtio_queue_enabled() or access the enabled status asynchronously, and we make the transport ready first.

After enabled set to true, and before virtio_queue_enable() is finished, somewhere  that call virtio_queue_enabled()  or access the enabled status of VirtIOPCIQueue. Then the caller will consider the virtqueue as enabled(enabled = true in VirtIOPCIQueue). The caller might access the virtqueue(access avail ring / desc table). But I think *the access here is illegal* because the virtqueue might still be *unintialized* status.

Thus, from my perspective, to prevent illegal access, we need to make transport ready after virtio_queue_enable().

Thanks

Quoted 7 lines
Quoted 7 lines

Thanks

Quoted 2 lines
Quoted 2 lines

          } else {

#20 lore raw

Re: [PATCH v2 07/24] virtio-pci: support queue enable

Quoted 66 lines · Kangjie Xu

在 2022/8/24 16:59, Jason Wang 写道:

在 2022/8/23 16:20, Kangjie Xu 写道:

在 2022/8/23 15:44, Jason Wang 写道:

在 2022/8/16 09:06, Kangjie Xu 写道:

PCI devices support vq enable.

Nit: it might be "support device specific vq enable"

Get it.

Based on this function, the driver can re-enable the virtqueue after the virtqueue is reset.

---
  hw/virtio/virtio-pci.c | 1 +
  1 file changed, 1 insertion(+)
hw/virtio/virtio-pci.c +0 -0
index ec8e92052f..3d560e45ad 100644
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
 proxy->vqs[vdev->queue_sel].avail[0],

((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 | proxy->vqs[vdev->queue_sel].used[0]);

+            virtio_queue_enable(vdev, vdev->queue_sel);
              proxy->vqs[vdev->queue_sel].enabled = 1;
              proxy->vqs[vdev->queue_sel].reset = 0;

Any reason we do it before the assignment of 1? It probably means the device specific method can't depend on virtio_queue_enabled()?

Thanks

Sorry, I don't get why device specific method can't depend on virtio_queue_enabled().

I meant if the device specific method call virtio_queue_enabled() it will return false in this case, is this intended?

Yes, I intend it to behave in this way.

Before virtio_queue_enable() is done, virtqueue should always be not ready and disabled.

Otherwise, If we put it after the assignment of enabled to 1, the virtqueue may be accessed illegally and may cause panic, because the virtqueue is still being intialized and being configured.

Quoted 5 lines

How? Shouldn't we make transport ready before making device virtqueue(device) ready?

Thanks

I am not experienced in this field, could you tell me why we should make the transport ready first?

That's a must for the device to work.

E.g for PCI device, I can't image the device is ready to work before PCI is ready.

Quoted 4 lines

I make the transport ready later than making device ready for two aspects:

1. In QEMU, the virtio_queue_enabled() is used only when we start the device/queue pair (vhost_dev_start_one), or reading VIRTIO_PCI_COMMON_Q_ENABLE. These two operations and resetting the queue will be synchronized using iothread lock, so we do not need to worry about the case currently.

Note that virtio_queue_enabled() is an exported helper, you can't assume how it will be used in the future.

Quoted 5 lines

2. Suppose we use virtio_queue_enabled() or access the enabled status asynchronously, and we make the transport ready first.

After enabled set to true, and before virtio_queue_enable() is finished, somewhere that call virtio_queue_enabled() or access the enabled status of VirtIOPCIQueue. Then the caller will consider the virtqueue as enabled(enabled = true in VirtIOPCIQueue). The caller might access the virtqueue(access avail ring / desc table). But I think the access here is illegal because the virtqueue might still be unintialized status.

How can this happen, won't we call device specific method after we set queue_enabled as true? It's the charge of the device specific method to synchronize the necessary external I/O in this case.

Thanks

Quoted 12 lines

Thus, from my perspective, to prevent illegal access, we need to make transport ready after virtio_queue_enable().

Thanks

Thanks

          } else {
author
#21 lore raw

Re: [PATCH v2 07/24] virtio-pci: support queue enable

Quoted 84 lines · Jason Wang
Quoted 65 lines · Kangjie Xu

在 2022/8/24 16:59, Jason Wang 写道:

在 2022/8/23 16:20, Kangjie Xu 写道:

在 2022/8/23 15:44, Jason Wang 写道:

在 2022/8/16 09:06, Kangjie Xu 写道:

PCI devices support vq enable.

Nit: it might be "support device specific vq enable"

Get it.

Based on this function, the driver can re-enable the virtqueue after the virtqueue is reset.

---
   hw/virtio/virtio-pci.c | 1 +
   1 file changed, 1 insertion(+)
hw/virtio/virtio-pci.c +0 -0
index ec8e92052f..3d560e45ad 100644
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
 proxy->vqs[vdev->queue_sel].avail[0],

((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 | proxy->vqs[vdev->queue_sel].used[0]);

+            virtio_queue_enable(vdev, vdev->queue_sel);
               proxy->vqs[vdev->queue_sel].enabled = 1;
               proxy->vqs[vdev->queue_sel].reset = 0;

Any reason we do it before the assignment of 1? It probably means the device specific method can't depend on virtio_queue_enabled()?

Thanks

Sorry, I don't get why device specific method can't depend on virtio_queue_enabled().

I meant if the device specific method call virtio_queue_enabled() it will return false in this case, is this intended?

Yes, I intend it to behave in this way.

Before virtio_queue_enable() is done, virtqueue should always be not ready and disabled.

Otherwise, If we put it after the assignment of enabled to 1, the virtqueue may be accessed illegally and may cause panic, because the virtqueue is still being intialized and being configured.

Quoted 5 lines

How? Shouldn't we make transport ready before making device virtqueue(device) ready?

Thanks

I am not experienced in this field, could you tell me why we should make the transport ready first?

That's a must for the device to work.

E.g for PCI device, I can't image the device is ready to work before PCI is ready.

Quoted 3 lines

I make the transport ready later than making device ready for two aspects:

1. In QEMU, the virtio_queue_enabled() is used only when we start the device/queue pair (vhost_dev_start_one), or reading VIRTIO_PCI_COMMON_Q_ENABLE. These two operations and resetting the queue will be synchronized using iothread lock, so we do not need to worry about the case currently.

Note that virtio_queue_enabled() is an exported helper, you can't assume how it will be used in the future.

Quoted 2 lines

2. Suppose we use virtio_queue_enabled() or access the enabled status asynchronously, and we make the transport ready first.

Quoted 7 lines
Quoted 2 lines

After enabled set to true, and before virtio_queue_enable() is finished, somewhere that call virtio_queue_enabled() or access the enabled status of VirtIOPCIQueue. Then the caller will consider the virtqueue as enabled(enabled = true in VirtIOPCIQueue). The caller might access the virtqueue(access avail ring / desc table). But I think the access here is illegal because the virtqueue might still be unintialized status.

How can this happen, won't we call device specific method after we set queue_enabled as true? It's the charge of the device specific method to synchronize the necessary external I/O in this case.

Thanks

I get your point.

You are right. I realized that I ignored the external I/O.

Will fix it.

Thanks

Quoted 13 lines
Quoted 13 lines

Thus, from my perspective, to prevent illegal access, we need to make transport ready after virtio_queue_enable().

Thanks

Thanks

           } else {