Skip to contentSkip to conversation

Support VIRTIO_F_RING_RESET for virtio-net and vhost-user in virtio pci

PATCH In review Kangjie Xu 2 participants 55 messages Jul 18, 2022 – Jul 29, 2022 2 messages not in the archive Networking

lore
Share & tools
lore

Kangjie Xu

author
#0 lore raw

cover Support VIRTIO_F_RING_RESET for virtio-net and vhost-user in virtio pci

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

    https://github.com/oasis-tcs/virtio-spec/issues/124
    https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of two parts: virtio-net and vhost-user. The patches 1-7 are the implementation for virtio-net and the patches 8-16 are for vhost-user.

The process of virt queue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue is restarted after the information of vrings is passed to QEMU and VIRTIO_PCI_COMMON_Q_ENABLE is written.

Test environment:
Host: 5.4.189
Qemu: QEMU emulator version 7.0.50
Guest: 5.19.0-rc3 (With vq reset support)
DPDK: 22.07-rc1 (With vq reset support)
Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

The frontend can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

Guest Kernel Patch:

    https://lore.kernel.org/bpf/[email protected]/

DPDK Code:

    https://github.com/middaywords/dpdk/commit/098c8e1dfae10b747da8dd8950a82890efca7bee

Please review. Thanks.

Kangjie Xu (9):
vhost: add op to enable or disable a single vring
vhost-user: enable/disable a single vring
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce restart and release for vhost_dev's vqs
vhost-net: introduce restart and stop for vhost_net's vqs
virtio: introduce queue_enable in virtio
virtio-net: support queue_enable for vhost-user
virtio-net: support queue_reset for vhost-user
vhost-net: vq reset feature bit support

Xuan Zhuo (7):
virtio-pci: virtio_pci_common_cfg add queue_notify_data
virtio: add VIRTIO_F_RING_RESET
virtio: pci: virtio_pci_common_cfg add queue_reset
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

 hw/net/vhost_net.c                            | 56 ++++++++++++++++++
 hw/net/virtio-net.c                           | 40 +++++++++++++
 hw/virtio/vhost-user.c                        | 55 +++++++++++++++---
 hw/virtio/vhost.c                             | 49 ++++++++++++++--
 hw/virtio/virtio-pci.c                        | 17 ++++++
 hw/virtio/virtio.c                            | 57 +++++++++++++------
 include/hw/virtio/vhost-backend.h             |  4 ++
 include/hw/virtio/vhost.h                     |  6 ++
 include/hw/virtio/virtio-pci.h                |  1 +
 include/hw/virtio/virtio.h                    |  4 ++
 include/net/vhost_net.h                       |  5 ++
 .../standard-headers/linux/virtio_config.h    |  5 ++
 include/standard-headers/linux/virtio_pci.h   |  4 ++
 13 files changed, 274 insertions(+), 29 deletions(-)
Signature · 1 line
2.32.0
author
#1 lore raw

PATCH 1/16 virtio-pci: virtio_pci_common_cfg add queue_notify_data

From: Xuan Zhuo <[email protected]>

Add queue_notify_data in struct virtio_pci_common_cfg, which comes from here https://github.com/oasis-tcs/virtio-spec/issues/89

Since I want to add queue_reset after queue_notify_data, I submitted this patch first.

---
 include/standard-headers/linux/virtio_pci.h | 2 ++
 1 file changed, 2 insertions(+)
include/standard-headers/linux/virtio_pci.h +2 -0
index db7a8e2fcb..598ebe9825 100644
@@ -164,6 +164,7 @@ struct virtio_pci_common_cfg {
 	uint32_t queue_avail_hi;		/* read-write */
 	uint32_t queue_used_lo;		/* read-write */
 	uint32_t queue_used_hi;		/* read-write */
+	uint16_t queue_notify_data;	/* read-write */
 };
 
 /* Fields in VIRTIO_PCI_CAP_PCI_CFG: */
@@ -202,6 +203,7 @@ struct virtio_pci_cfg_cap {
 #define VIRTIO_PCI_COMMON_Q_AVAILHI	44
 #define VIRTIO_PCI_COMMON_Q_USEDLO	48
 #define VIRTIO_PCI_COMMON_Q_USEDHI	52
+#define VIRTIO_PCI_COMMON_Q_NOTIFY_DATA	56
 
 #endif /* VIRTIO_PCI_NO_MODERN */
 
Signature · 1 line
2.32.0
#2 lore raw

Re: [PATCH 01/16] virtio-pci: virtio_pci_common_cfg add queue_notify_data

Quoted 9 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

Add queue_notify_data in struct virtio_pci_common_cfg, which comes from here https://github.com/oasis-tcs/virtio-spec/issues/89

Since I want to add queue_reset after queue_notify_data, I submitted this patch first.

This should be done by script/update-linux-headers.sh.

Thanks

Quoted 24 lines
---
  include/standard-headers/linux/virtio_pci.h | 2 ++
  1 file changed, 2 insertions(+)
include/standard-headers/linux/virtio_pci.h +2 -0
index db7a8e2fcb..598ebe9825 100644
@@ -164,6 +164,7 @@ struct virtio_pci_common_cfg {
  	uint32_t queue_avail_hi;		/* read-write */
  	uint32_t queue_used_lo;		/* read-write */
  	uint32_t queue_used_hi;		/* read-write */
+	uint16_t queue_notify_data;	/* read-write */
  };
 
  /* Fields in VIRTIO_PCI_CAP_PCI_CFG: */
@@ -202,6 +203,7 @@ struct virtio_pci_cfg_cap {
  #define VIRTIO_PCI_COMMON_Q_AVAILHI	44
  #define VIRTIO_PCI_COMMON_Q_USEDLO	48
  #define VIRTIO_PCI_COMMON_Q_USEDHI	52
+#define VIRTIO_PCI_COMMON_Q_NOTIFY_DATA	56
 
  #endif /* VIRTIO_PCI_NO_MODERN */
 
author
#3 lore raw

Re: [PATCH 01/16] virtio-pci: virtio_pci_common_cfg add queue_notify_data

Quoted 10 lines · Jason Wang
Quoted 8 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

Add queue_notify_data in struct virtio_pci_common_cfg, which comes from here https://github.com/oasis-tcs/virtio-spec/issues/89

Since I want to add queue_reset after queue_notify_data, I submitted this patch first.

Quoted 7 lines
Quoted 1 line

This should be done by script/update-linux-headers.sh.

Thanks

Will fix.

Thanks

Quoted 22 lines
Quoted 22 lines

---
  include/standard-headers/linux/virtio_pci.h | 2 ++

  1 file changed, 2 insertions(+)
include/standard-headers/linux/virtio_pci.h +0 -0
index db7a8e2fcb..598ebe9825 100644
@@ -164,6 +164,7 @@ struct virtio_pci_common_cfg {

      uint32_t queue_avail_hi;        /* read-write */ uint32_t queue_used_lo;        /* read-write */ uint32_t queue_used_hi;        /* read-write */ +    uint16_t queue_notify_data;    /* read-write */ };
    /* Fields in VIRTIO_PCI_CAP_PCI_CFG: */

include/standard-headers/linux/virtio_pci.h +1 -0
@@ -202,6 +203,7 @@ struct virtio_pci_cfg_cap {
   #define VIRTIO_PCI_COMMON_Q_AVAILHI    44
   #define VIRTIO_PCI_COMMON_Q_USEDLO    48
   #define VIRTIO_PCI_COMMON_Q_USEDHI    52
+#define VIRTIO_PCI_COMMON_Q_NOTIFY_DATA    56
     #endif /* VIRTIO_PCI_NO_MODERN */
author
#4 lore raw

PATCH 2/16 virtio: add VIRTIO_F_RING_RESET

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

Added VIRTIO_F_RING_RESET, it came from here
https://github.com/oasis-tcs/virtio-spec/issues/124

This feature indicates that the driver can reset a queue individually.

---
 include/standard-headers/linux/virtio_config.h | 5 +++++
 1 file changed, 5 insertions(+)
include/standard-headers/linux/virtio_config.h +5 -0
index 7acd8d4abc..47a7eef5e4 100644
@@ -96,4 +96,9 @@
  * Does the device support Single Root I/O Virtualization?
  */
 #define VIRTIO_F_SR_IOV			37
+
+/*
+ * This feature indicates that the driver can reset a queue individually.
+ */
+#define VIRTIO_F_RING_RESET		40
 #endif /* _LINUX_VIRTIO_CONFIG_H */
Signature · 1 line
2.32.0
author
#5 lore raw

PATCH 3/16 virtio: pci: virtio_pci_common_cfg add queue_reset

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

Add queue_reset in virtio_pci_common_cfg.

---
 include/standard-headers/linux/virtio_pci.h | 2 ++
 1 file changed, 2 insertions(+)
include/standard-headers/linux/virtio_pci.h +2 -0
index 598ebe9825..9feb865636 100644
@@ -165,6 +165,7 @@ struct virtio_pci_common_cfg {
 	uint32_t queue_used_lo;		/* read-write */
 	uint32_t queue_used_hi;		/* read-write */
 	uint16_t queue_notify_data;	/* read-write */
+	uint16_t queue_reset;		/* read-write */
 };
 
 /* Fields in VIRTIO_PCI_CAP_PCI_CFG: */
@@ -204,6 +205,7 @@ struct virtio_pci_cfg_cap {
 #define VIRTIO_PCI_COMMON_Q_USEDLO	48
 #define VIRTIO_PCI_COMMON_Q_USEDHI	52
 #define VIRTIO_PCI_COMMON_Q_NOTIFY_DATA	56
+#define VIRTIO_PCI_COMMON_Q_RESET	58
 
 #endif /* VIRTIO_PCI_NO_MODERN */
 
Signature · 1 line
2.32.0
author
#6 lore raw

PATCH 4/16 virtio: introduce __virtio_queue_reset()

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

Separate the logic of vq reset. This logic will be called directly later.

---
 hw/virtio/virtio.c | 37 +++++++++++++++++++++----------------
 1 file changed, 21 insertions(+), 16 deletions(-)
hw/virtio/virtio.c +21 -16
index 5d607aeaa0..67d54832a9 100644
@@ -2019,6 +2019,26 @@ static enum virtio_device_endian virtio_current_cpu_endian(void)
     }
 }
 
+static void __virtio_queue_reset(VirtIODevice *vdev, uint32_t i)
+{
+    vdev->vq[i].vring.desc = 0;
+    vdev->vq[i].vring.avail = 0;
+    vdev->vq[i].vring.used = 0;
+    vdev->vq[i].last_avail_idx = 0;
+    vdev->vq[i].shadow_avail_idx = 0;
+    vdev->vq[i].used_idx = 0;
+    vdev->vq[i].last_avail_wrap_counter = true;
+    vdev->vq[i].shadow_avail_wrap_counter = true;
+    vdev->vq[i].used_wrap_counter = true;
+    virtio_queue_set_vector(vdev, i, VIRTIO_NO_VECTOR);
+    vdev->vq[i].signalled_used = 0;
+    vdev->vq[i].signalled_used_valid = false;
+    vdev->vq[i].notification = true;
+    vdev->vq[i].vring.num = vdev->vq[i].vring.num_default;
+    vdev->vq[i].inuse = 0;
+    virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
+}
+
 void virtio_reset(void *opaque)
 {
     VirtIODevice *vdev = opaque;
@@ -2050,22 +2070,7 @@ void virtio_reset(void *opaque)
     virtio_notify_vector(vdev, vdev->config_vector);
 
     for(i = 0; i < VIRTIO_QUEUE_MAX; i++) {
-        vdev->vq[i].vring.desc = 0;
-        vdev->vq[i].vring.avail = 0;
-        vdev->vq[i].vring.used = 0;
-        vdev->vq[i].last_avail_idx = 0;
-        vdev->vq[i].shadow_avail_idx = 0;
-        vdev->vq[i].used_idx = 0;
-        vdev->vq[i].last_avail_wrap_counter = true;
-        vdev->vq[i].shadow_avail_wrap_counter = true;
-        vdev->vq[i].used_wrap_counter = true;
-        virtio_queue_set_vector(vdev, i, VIRTIO_NO_VECTOR);
-        vdev->vq[i].signalled_used = 0;
-        vdev->vq[i].signalled_used_valid = false;
-        vdev->vq[i].notification = true;
-        vdev->vq[i].vring.num = vdev->vq[i].vring.num_default;
-        vdev->vq[i].inuse = 0;
-        virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
+        __virtio_queue_reset(vdev, i);
     }
 }
 
Signature · 1 line
2.32.0
Acked-by on v1 4/16
#7 lore raw

Re: [PATCH 04/16] virtio: introduce __virtio_queue_reset()

Quoted 6 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

Separate the logic of vq reset. This logic will be called directly later.

Quoted 59 lines
---
  hw/virtio/virtio.c | 37 +++++++++++++++++++++----------------
  1 file changed, 21 insertions(+), 16 deletions(-)
hw/virtio/virtio.c +21 -16
index 5d607aeaa0..67d54832a9 100644
@@ -2019,6 +2019,26 @@ static enum virtio_device_endian virtio_current_cpu_endian(void)
      }
  }
 
+static void __virtio_queue_reset(VirtIODevice *vdev, uint32_t i)
+{
+    vdev->vq[i].vring.desc = 0;
+    vdev->vq[i].vring.avail = 0;
+    vdev->vq[i].vring.used = 0;
+    vdev->vq[i].last_avail_idx = 0;
+    vdev->vq[i].shadow_avail_idx = 0;
+    vdev->vq[i].used_idx = 0;
+    vdev->vq[i].last_avail_wrap_counter = true;
+    vdev->vq[i].shadow_avail_wrap_counter = true;
+    vdev->vq[i].used_wrap_counter = true;
+    virtio_queue_set_vector(vdev, i, VIRTIO_NO_VECTOR);
+    vdev->vq[i].signalled_used = 0;
+    vdev->vq[i].signalled_used_valid = false;
+    vdev->vq[i].notification = true;
+    vdev->vq[i].vring.num = vdev->vq[i].vring.num_default;
+    vdev->vq[i].inuse = 0;
+    virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
+}
+
  void virtio_reset(void *opaque)
  {
      VirtIODevice *vdev = opaque;
@@ -2050,22 +2070,7 @@ void virtio_reset(void *opaque)
      virtio_notify_vector(vdev, vdev->config_vector);
 
      for(i = 0; i < VIRTIO_QUEUE_MAX; i++) {
-        vdev->vq[i].vring.desc = 0;
-        vdev->vq[i].vring.avail = 0;
-        vdev->vq[i].vring.used = 0;
-        vdev->vq[i].last_avail_idx = 0;
-        vdev->vq[i].shadow_avail_idx = 0;
-        vdev->vq[i].used_idx = 0;
-        vdev->vq[i].last_avail_wrap_counter = true;
-        vdev->vq[i].shadow_avail_wrap_counter = true;
-        vdev->vq[i].used_wrap_counter = true;
-        virtio_queue_set_vector(vdev, i, VIRTIO_NO_VECTOR);
-        vdev->vq[i].signalled_used = 0;
-        vdev->vq[i].signalled_used_valid = false;
-        vdev->vq[i].notification = true;
-        vdev->vq[i].vring.num = vdev->vq[i].vring.num_default;
-        vdev->vq[i].inuse = 0;
-        virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
+        __virtio_queue_reset(vdev, i);
      }
  }
 
author
#8 lore raw

PATCH 5/16 virtio: introduce virtio_queue_reset()

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

Introduce a new interface function virtio_queue_reset() to implement reset for vq.

Add a new callback to VirtioDeviceClass for queue reset operation for each child device.

---
 hw/virtio/virtio.c         | 11 +++++++++++
 include/hw/virtio/virtio.h |  2 ++
 2 files changed, 13 insertions(+)
hw/virtio/virtio.c +11 -0
index 67d54832a9..0e9d41366f 100644
@@ -2039,6 +2039,17 @@ static void __virtio_queue_reset(VirtIODevice *vdev, uint32_t i)
     virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
 }
 
+void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+
+    if (k->queue_reset) {
+        k->queue_reset(vdev, queue_index);
+    }
+
+    __virtio_queue_reset(vdev, queue_index);
+}
+
 void virtio_reset(void *opaque)
 {
     VirtIODevice *vdev = opaque;
include/hw/virtio/virtio.h +2 -0
index db1c0ddf6b..879394299b 100644
@@ -130,6 +130,7 @@ struct VirtioDeviceClass {
     void (*set_config)(VirtIODevice *vdev, const uint8_t *config);
     void (*reset)(VirtIODevice *vdev);
     void (*set_status)(VirtIODevice *vdev, uint8_t val);
+    void (*queue_reset)(VirtIODevice *vdev, uint32_t queue_index);
     /* For transitional devices, this is a bitmap of features
      * that are only exposed on the legacy interface but not
      * the modern one.
@@ -268,6 +269,7 @@ int virtio_queue_set_host_notifier_mr(VirtIODevice *vdev, int n,
                                       MemoryRegion *mr, bool assign);
 int virtio_set_status(VirtIODevice *vdev, uint8_t val);
 void virtio_reset(void *opaque);
+void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
 void virtio_update_irq(VirtIODevice *vdev);
 int virtio_set_features(VirtIODevice *vdev, uint64_t val);
 
Signature · 1 line
2.32.0
Acked-by on v1 5/16
#9 lore raw

Re: [PATCH 05/16] virtio: introduce virtio_queue_reset()

Quoted 9 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

Introduce a new interface function virtio_queue_reset() to implement reset for vq.

Add a new callback to VirtioDeviceClass for queue reset operation for each child device.

Quoted 47 lines
---
  hw/virtio/virtio.c         | 11 +++++++++++
  include/hw/virtio/virtio.h |  2 ++
  2 files changed, 13 insertions(+)
hw/virtio/virtio.c +11 -0
index 67d54832a9..0e9d41366f 100644
@@ -2039,6 +2039,17 @@ static void __virtio_queue_reset(VirtIODevice *vdev, uint32_t i)
      virtio_virtqueue_reset_region_cache(&vdev->vq[i]);
  }
 
+void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+
+    if (k->queue_reset) {
+        k->queue_reset(vdev, queue_index);
+    }
+
+    __virtio_queue_reset(vdev, queue_index);
+}
+
  void virtio_reset(void *opaque)
  {
      VirtIODevice *vdev = opaque;
include/hw/virtio/virtio.h +2 -0
index db1c0ddf6b..879394299b 100644
@@ -130,6 +130,7 @@ struct VirtioDeviceClass {
      void (*set_config)(VirtIODevice *vdev, const uint8_t *config);
      void (*reset)(VirtIODevice *vdev);
      void (*set_status)(VirtIODevice *vdev, uint8_t val);
+    void (*queue_reset)(VirtIODevice *vdev, uint32_t queue_index);
      /* For transitional devices, this is a bitmap of features
       * that are only exposed on the legacy interface but not
       * the modern one.
@@ -268,6 +269,7 @@ int virtio_queue_set_host_notifier_mr(VirtIODevice *vdev, int n,
                                        MemoryRegion *mr, bool assign);
  int virtio_set_status(VirtIODevice *vdev, uint8_t val);
  void virtio_reset(void *opaque);
+void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
  void virtio_update_irq(VirtIODevice *vdev);
  int virtio_set_features(VirtIODevice *vdev, uint64_t val);
 
author
#10 lore raw

PATCH 6/16 virtio-pci: support queue reset

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

PCI devices support vq reset.

Based on this function, the driver can adjust the size of the ring, and quickly recycle the buffer in the ring.

---
 hw/virtio/virtio-pci.c         | 16 ++++++++++++++++
 include/hw/virtio/virtio-pci.h |  1 +
 2 files changed, 17 insertions(+)
hw/virtio/virtio-pci.c +16 -0
index 45327f0b31..35e8a5101a 100644
@@ -1246,6 +1246,9 @@ static uint64_t virtio_pci_common_read(void *opaque, hwaddr addr,
     case VIRTIO_PCI_COMMON_Q_USEDHI:
         val = proxy->vqs[vdev->queue_sel].used[1];
         break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        val = proxy->vqs[vdev->queue_sel].reset;
+        break;
     default:
         val = 0;
     }
@@ -1333,6 +1336,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
                        ((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 |
                        proxy->vqs[vdev->queue_sel].used[0]);
             proxy->vqs[vdev->queue_sel].enabled = 1;
+            proxy->vqs[vdev->queue_sel].reset = 0;
         } else {
             virtio_error(vdev, "wrong value for queue_enable %"PRIx64, val);
         }
@@ -1355,6 +1359,17 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
     case VIRTIO_PCI_COMMON_Q_USEDHI:
         proxy->vqs[vdev->queue_sel].used[1] = val;
         break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        if (val == 1) {
+            proxy->vqs[vdev->queue_sel].reset = 1;
+
+            virtio_queue_reset(vdev, vdev->queue_sel);
+
+            /* mark reset complete */
+            proxy->vqs[vdev->queue_sel].reset = 0;
+            proxy->vqs[vdev->queue_sel].enabled = 0;
+        }
+        break;
     default:
         break;
     }
@@ -1950,6 +1965,7 @@ static void virtio_pci_reset(DeviceState *qdev)
 
     for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
         proxy->vqs[i].enabled = 0;
+        proxy->vqs[i].reset = 0;
         proxy->vqs[i].num = 0;
         proxy->vqs[i].desc[0] = proxy->vqs[i].desc[1] = 0;
         proxy->vqs[i].avail[0] = proxy->vqs[i].avail[1] = 0;
include/hw/virtio/virtio-pci.h +1 -0
index 2446dcd9ae..e9290e2b94 100644
@@ -117,6 +117,7 @@ typedef struct VirtIOPCIRegion {
 typedef struct VirtIOPCIQueue {
   uint16_t num;
   bool enabled;
+  bool reset;
   uint32_t desc[2];
   uint32_t avail[2];
   uint32_t used[2];
Signature · 1 line
2.32.0
#11 lore raw

Re: [PATCH 06/16] virtio-pci: support queue reset

Quoted 36 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

PCI devices support vq reset.

Based on this function, the driver can adjust the size of the ring, and quickly recycle the buffer in the ring.

---
  hw/virtio/virtio-pci.c         | 16 ++++++++++++++++
  include/hw/virtio/virtio-pci.h |  1 +
  2 files changed, 17 insertions(+)
hw/virtio/virtio-pci.c +4 -0
index 45327f0b31..35e8a5101a 100644
@@ -1246,6 +1246,9 @@ static uint64_t virtio_pci_common_read(void *opaque, hwaddr addr,
      case VIRTIO_PCI_COMMON_Q_USEDHI:
          val = proxy->vqs[vdev->queue_sel].used[1];
          break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        val = proxy->vqs[vdev->queue_sel].reset;
+        break;
      default:
          val = 0;
      }
@@ -1333,6 +1336,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
                         ((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 |
                         proxy->vqs[vdev->queue_sel].used[0]);
              proxy->vqs[vdev->queue_sel].enabled = 1;
+            proxy->vqs[vdev->queue_sel].reset = 0;
          } else {
              virtio_error(vdev, "wrong value for queue_enable %"PRIx64, val);
          }

hw/virtio/virtio-pci.c static void virtio_pci_common_write(void *opaque, hwaddr addr, · line 1368

Quoted 10 lines
hw/virtio/virtio-pci.c +7 -0
@@ -1355,6 +1359,17 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
      case VIRTIO_PCI_COMMON_Q_USEDHI:
          proxy->vqs[vdev->queue_sel].used[1] = val;
          break;
+    case VIRTIO_PCI_COMMON_Q_RESET:
+        if (val == 1) {
+            proxy->vqs[vdev->queue_sel].reset = 1;
+
+            virtio_queue_reset(vdev, vdev->queue_sel);
+
+            /* mark reset complete */

The code explain itself, so the comment could be removed.

Quoted 20 lines
hw/virtio/virtio-pci.c +5 -0
+            proxy->vqs[vdev->queue_sel].reset = 0;
+            proxy->vqs[vdev->queue_sel].enabled = 0;
+        }
+        break;
      default:
          break;
      }
@@ -1950,6 +1965,7 @@ static void virtio_pci_reset(DeviceState *qdev)
 
      for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
          proxy->vqs[i].enabled = 0;
+        proxy->vqs[i].reset = 0;
          proxy->vqs[i].num = 0;
          proxy->vqs[i].desc[0] = proxy->vqs[i].desc[1] = 0;
          proxy->vqs[i].avail[0] = proxy->vqs[i].avail[1] = 0;
include/hw/virtio/virtio-pci.h +0 -0
index 2446dcd9ae..e9290e2b94 100644

include/hw/virtio/virtio-pci.h typedef struct VirtIOPCIRegion { · line 120

Quoted 4 lines
include/hw/virtio/virtio-pci.h +1 -0
@@ -117,6 +117,7 @@ typedef struct VirtIOPCIRegion {
  typedef struct VirtIOPCIQueue {
    uint16_t num;
    bool enabled;
+  bool reset;

How is this migrated?

Thanks

Quoted 3 lines
include/hw/virtio/virtio-pci.h +0 -0
    uint32_t desc[2];
    uint32_t avail[2];
    uint32_t used[2];
author
#12 lore raw

PATCH 7/16 virtio-net: support queue reset

In reply to Kangjie Xu

From: Xuan Zhuo <[email protected]>

virtio-net implements queue reset. Queued packets in the corresponding queue pair are flushed or purged.

Queue reset is currently only implemented for non-vhosts.

---
 hw/net/virtio-net.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
hw/net/virtio-net.c +15 -0
index 7ad948ee7c..8396e21a67 100644
@@ -531,6 +531,19 @@ static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc)
     return info;
 }
 
+static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+
+    if (!nc->peer) {
+        return;
+    }
+
+    qemu_flush_or_purge_queued_packets(nc->peer, true);
+    assert(!virtio_net_get_subqueue(nc)->async_tx.elem);
+}
+
 static void virtio_net_reset(VirtIODevice *vdev)
 {
     VirtIONet *n = VIRTIO_NET(vdev);
@@ -741,6 +754,7 @@ static uint64_t virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
     }
 
     if (!get_vhost_net(nc->peer)) {
+        virtio_add_feature(&features, VIRTIO_F_RING_RESET);
         return features;
     }
 
@@ -3766,6 +3780,7 @@ static void virtio_net_class_init(ObjectClass *klass, void *data)
     vdc->set_features = virtio_net_set_features;
     vdc->bad_features = virtio_net_bad_features;
     vdc->reset = virtio_net_reset;
+    vdc->queue_reset = virtio_net_queue_reset;
     vdc->set_status = virtio_net_set_status;
     vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
     vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
Signature · 1 line
2.32.0
#13 lore raw

Re: [PATCH 07/16] virtio-net: support queue reset

Quoted 21 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

virtio-net implements queue reset. Queued packets in the corresponding queue pair are flushed or purged.

Queue reset is currently only implemented for non-vhosts.

---
  hw/net/virtio-net.c | 15 +++++++++++++++
  1 file changed, 15 insertions(+)
hw/net/virtio-net.c +1 -0
index 7ad948ee7c..8396e21a67 100644
@@ -531,6 +531,19 @@ static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc)
      return info;
  }
 
+static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t queue_index)

hw/net/virtio-net.c static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc) · line 544

Quoted 10 lines
hw/net/virtio-net.c +10 -0
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+
+    if (!nc->peer) {
+        return;
+    }
+
+    qemu_flush_or_purge_queued_packets(nc->peer, true);
+    assert(!virtio_net_get_subqueue(nc)->async_tx.elem);

Let's try to reuse this function in virtio_net_reset().

hw/net/virtio-net.c static uint64_t virtio_net_get_features(VirtIODevice *vdev, uint64_t features, · line 757

Quoted 10 lines
hw/net/virtio-net.c +3 -0
+}
+
  static void virtio_net_reset(VirtIODevice *vdev)
  {
      VirtIONet *n = VIRTIO_NET(vdev);
@@ -741,6 +754,7 @@ static uint64_t virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
      }
 
      if (!get_vhost_net(nc->peer)) {
+        virtio_add_feature(&features, VIRTIO_F_RING_RESET);

This breaks migration compatibility.

We probably need:

1) a new command line parameter
2) make it disabled for pre-7.2 machine

Thanks

Quoted 11 lines
hw/net/virtio-net.c +1 -0
          return features;
      }
 
@@ -3766,6 +3780,7 @@ static void virtio_net_class_init(ObjectClass *klass, void *data)
      vdc->set_features = virtio_net_set_features;
      vdc->bad_features = virtio_net_bad_features;
      vdc->reset = virtio_net_reset;
+    vdc->queue_reset = virtio_net_queue_reset;
      vdc->set_status = virtio_net_set_status;
      vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
      vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
author
#14 lore raw

Re: [PATCH 07/16] virtio-net: support queue reset

Quoted 28 lines · Jason Wang
Quoted 26 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

virtio-net implements queue reset. Queued packets in the corresponding queue pair are flushed or purged.

Queue reset is currently only implemented for non-vhosts.

---
  hw/net/virtio-net.c | 15 +++++++++++++++

  1 file changed, 15 insertions(+)
hw/net/virtio-net.c +5 -0
index 7ad948ee7c..8396e21a67 100644
@@ -531,6 +531,19 @@ static RxFilterInfo 
 *virtio_net_query_rxfilter(NetClientState *nc)
       return info;
   }
   +static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+
+    if (!nc->peer) {
Quoted 9 lines

hw/net/virtio-net.c static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc) · line 544

Quoted 5 lines
hw/net/virtio-net.c +5 -0
+        return;
+    }
+
+    qemu_flush_or_purge_queued_packets(nc->peer, true);
+    assert(!virtio_net_get_subqueue(nc)->async_tx.elem);

Let's try to reuse this function in virtio_net_reset().

Yeah, I'll fix it.

Thanks.

Quoted 15 lines

hw/net/virtio-net.c static uint64_t · line 757

Quoted 11 lines
hw/net/virtio-net.c +2 -0
+}
+
   static void virtio_net_reset(VirtIODevice *vdev)
   {

      VirtIONet *n = VIRTIO_NET(vdev);

hw/net/virtio-net.c +1 -0
@@ -741,6 +754,7 @@ static uint64_t 
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
       }
         if (!get_vhost_net(nc->peer)) {
+        virtio_add_feature(&features, VIRTIO_F_RING_RESET);

This breaks migration compatibility.

Quoted 8 lines

We probably need:

1) a new command line parameter
2) make it disabled for pre-7.2 machine

Thanks

Sorry, I don't get what is the meaning of "pre-7.2 machine". Could you please explain it?

Thanks

Quoted 10 lines
Quoted 10 lines
hw/net/virtio-net.c +0 -0
           return features;
       }
   @@ -3766,6 +3780,7 @@ static void virtio_net_class_init(ObjectClass *klass, void *data)

      vdc->set_features = virtio_net_set_features; vdc->bad_features = virtio_net_bad_features; vdc->reset = virtio_net_reset;
+    vdc->queue_reset = virtio_net_queue_reset; vdc->set_status = virtio_net_set_status; vdc->guest_notifier_mask = virtio_net_guest_notifier_mask; vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;

#15 lore raw

Re: [PATCH 07/16] virtio-net: support queue reset

Quoted 62 lines · Kangjie Xu
Quoted 29 lines · Jason Wang
Quoted 27 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

virtio-net implements queue reset. Queued packets in the corresponding queue pair are flushed or purged.

Queue reset is currently only implemented for non-vhosts.

---
  hw/net/virtio-net.c | 15 +++++++++++++++
  1 file changed, 15 insertions(+)
hw/net/virtio-net.c +5 -0
index 7ad948ee7c..8396e21a67 100644
@@ -531,6 +531,19 @@ static RxFilterInfo
 *virtio_net_query_rxfilter(NetClientState *nc)
      return info;
  }
  +static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t
 queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+
+    if (!nc->peer) {
Quoted 9 lines

hw/net/virtio-net.c static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc) · line 544

Quoted 5 lines
hw/net/virtio-net.c +5 -0
+        return;
+    }
+
+    qemu_flush_or_purge_queued_packets(nc->peer, true);
+    assert(!virtio_net_get_subqueue(nc)->async_tx.elem);

Let's try to reuse this function in virtio_net_reset().

Yeah, I'll fix it.

Thanks.

Quoted 17 lines

hw/net/virtio-net.c static uint64_t · line 757

Quoted 11 lines
hw/net/virtio-net.c +3 -0
+}
+
  static void virtio_net_reset(VirtIODevice *vdev)
  {
      VirtIONet *n = VIRTIO_NET(vdev);
@@ -741,6 +754,7 @@ static uint64_t
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
      }
        if (!get_vhost_net(nc->peer)) {
+        virtio_add_feature(&features, VIRTIO_F_RING_RESET);

This breaks migration compatibility.

We probably need:

Quoted 8 lines
Quoted 6 lines

1) a new command line parameter
2) make it disabled for pre-7.2 machine

Thanks

Sorry, I don't get what is the meaning of "pre-7.2 machine". Could you please explain it?

I meant for pre 7.2 machine type, we should make reset fault off by default.

Otherwise we break migration compatibility.

Thanks

Quoted 15 lines

Thanks

Quoted 11 lines
Quoted 11 lines
hw/net/virtio-net.c +0 -0
          return features;
      }
  @@ -3766,6 +3780,7 @@ static void virtio_net_class_init(ObjectClass

*klass, void *data)

      vdc->set_features = virtio_net_set_features;
      vdc->bad_features = virtio_net_bad_features;
      vdc->reset = virtio_net_reset;
+    vdc->queue_reset = virtio_net_queue_reset;
      vdc->set_status = virtio_net_set_status;
      vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
      vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
author
#16 lore raw

Re: [PATCH 07/16] virtio-net: support queue reset

Quoted 64 lines · Jason Wang
Quoted 63 lines · Kangjie Xu
Quoted 27 lines · Jason Wang
Quoted 26 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

virtio-net implements queue reset. Queued packets in the corresponding queue pair are flushed or purged.

Queue reset is currently only implemented for non-vhosts.

---
   hw/net/virtio-net.c | 15 +++++++++++++++
   1 file changed, 15 insertions(+)
hw/net/virtio-net.c +4 -0
index 7ad948ee7c..8396e21a67 100644
@@ -531,6 +531,19 @@ static RxFilterInfo
 *virtio_net_query_rxfilter(NetClientState *nc)
       return info;
   }
   +static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t
 queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+
Quoted 9 lines

hw/net/virtio-net.c static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc) · line 544

Quoted 6 lines
hw/net/virtio-net.c +6 -0
+    if (!nc->peer) {
+        return;
+    }
+
+    qemu_flush_or_purge_queued_packets(nc->peer, true);
+    assert(!virtio_net_get_subqueue(nc)->async_tx.elem);

Let's try to reuse this function in virtio_net_reset().

Yeah, I'll fix it.

Thanks.

Quoted 21 lines

hw/net/virtio-net.c static uint64_t · line 757

Quoted 10 lines
hw/net/virtio-net.c +3 -0
+}
+
   static void virtio_net_reset(VirtIODevice *vdev)
   {
       VirtIONet *n = VIRTIO_NET(vdev);
@@ -741,6 +754,7 @@ static uint64_t
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
       }
         if (!get_vhost_net(nc->peer)) {
+        virtio_add_feature(&features, VIRTIO_F_RING_RESET);

This breaks migration compatibility.

We probably need:

1) a new command line parameter
2) make it disabled for pre-7.2 machine

Thanks

Quoted 7 lines
Quoted 2 lines

Sorry, I don't get what is the meaning of "pre-7.2 machine". Could you please explain it?

I meant for pre 7.2 machine type, we should make reset fault off by default.

Otherwise we break migration compatibility.

Thanks

Sorry, I did not express myself clearly. Is "7.2" the version of a system or a module? If it is a system version, what is the system?

I did not have backgrond knowledge related to this part and will investigate migration issues afterwards.

Thanks.

Quoted 13 lines
Quoted 13 lines

Thanks

Quoted 11 lines
Quoted 11 lines
hw/net/virtio-net.c +0 -0
           return features;
       }
   @@ -3766,6 +3780,7 @@ static void virtio_net_class_init(ObjectClass

*klass, void *data)

       vdc->set_features = virtio_net_set_features;
       vdc->bad_features = virtio_net_bad_features;
       vdc->reset = virtio_net_reset;
+    vdc->queue_reset = virtio_net_queue_reset;
       vdc->set_status = virtio_net_set_status;
       vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
       vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
#17 lore raw

Re: [PATCH 07/16] virtio-net: support queue reset

Quoted 69 lines · Kangjie Xu
Quoted 66 lines · Jason Wang
Quoted 65 lines · Kangjie Xu
Quoted 27 lines · Jason Wang
Quoted 26 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

virtio-net implements queue reset. Queued packets in the corresponding queue pair are flushed or purged.

Queue reset is currently only implemented for non-vhosts.

---
   hw/net/virtio-net.c | 15 +++++++++++++++
   1 file changed, 15 insertions(+)
hw/net/virtio-net.c +4 -0
index 7ad948ee7c..8396e21a67 100644
@@ -531,6 +531,19 @@ static RxFilterInfo
 *virtio_net_query_rxfilter(NetClientState *nc)
       return info;
   }
   +static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t
 queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+
Quoted 9 lines

hw/net/virtio-net.c static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc) · line 544

Quoted 6 lines
hw/net/virtio-net.c +6 -0
+    if (!nc->peer) {
+        return;
+    }
+
+    qemu_flush_or_purge_queued_packets(nc->peer, true);
+    assert(!virtio_net_get_subqueue(nc)->async_tx.elem);

Let's try to reuse this function in virtio_net_reset().

Yeah, I'll fix it.

Thanks.

Quoted 13 lines

hw/net/virtio-net.c static uint64_t · line 757

Quoted 10 lines
hw/net/virtio-net.c +3 -0
+}
+
   static void virtio_net_reset(VirtIODevice *vdev)
   {
       VirtIONet *n = VIRTIO_NET(vdev);
@@ -741,6 +754,7 @@ static uint64_t
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
       }
         if (!get_vhost_net(nc->peer)) {
+        virtio_add_feature(&features, VIRTIO_F_RING_RESET);

This breaks migration compatibility.

Quoted 8 lines

We probably need:

1) a new command line parameter
2) make it disabled for pre-7.2 machine

Thanks

Sorry, I don't get what is the meaning of "pre-7.2 machine". Could you please explain it?

Quoted 8 lines
Quoted 5 lines

I meant for pre 7.2 machine type, we should make reset fault off by default.

Otherwise we break migration compatibility.

Thanks

Sorry, I did not express myself clearly. Is "7.2" the version of a system or a module? If it is a system version, what is the system?

It's the machine type to make sure the migration can work. (you can get the list of those types via qemu -machine ?)

E.g you can start a 7.0 machine on Qemu 7.1 and it guarantees to be migrated to 7.0 machine on Qemu 7.0.

Quoted 3 lines

I did not have backgrond knowledge related to this part and will investigate migration issues afterwards.

Git grep hw_compat_7_0 may give you more hints.

Thanks

Quoted 17 lines

Thanks.

Quoted 13 lines
Quoted 13 lines

Thanks

Quoted 11 lines
Quoted 11 lines
hw/net/virtio-net.c +0 -0
           return features;
       }
   @@ -3766,6 +3780,7 @@ static void virtio_net_class_init(ObjectClass

*klass, void *data)

       vdc->set_features = virtio_net_set_features;
       vdc->bad_features = virtio_net_bad_features;
       vdc->reset = virtio_net_reset;
+    vdc->queue_reset = virtio_net_queue_reset;
       vdc->set_status = virtio_net_set_status;
       vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
       vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
author
#18 lore raw

Re: [PATCH 07/16] virtio-net: support queue reset

Quoted 76 lines · Jason Wang
Quoted 65 lines · Kangjie Xu
Quoted 63 lines · Jason Wang
Quoted 62 lines · Kangjie Xu
Quoted 26 lines · Jason Wang
Quoted 25 lines · Kangjie Xu

From: Xuan Zhuo <[email protected]>

virtio-net implements queue reset. Queued packets in the corresponding queue pair are flushed or purged.

Queue reset is currently only implemented for non-vhosts.

---
    hw/net/virtio-net.c | 15 +++++++++++++++
    1 file changed, 15 insertions(+)
hw/net/virtio-net.c +3 -0
index 7ad948ee7c..8396e21a67 100644
@@ -531,6 +531,19 @@ static RxFilterInfo
 *virtio_net_query_rxfilter(NetClientState *nc)
        return info;
    }
    +static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t
 queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
Quoted 9 lines

hw/net/virtio-net.c static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc) · line 544

Quoted 7 lines
hw/net/virtio-net.c +7 -0
+
+    if (!nc->peer) {
+        return;
+    }
+
+    qemu_flush_or_purge_queued_packets(nc->peer, true);
+    assert(!virtio_net_get_subqueue(nc)->async_tx.elem);

Let's try to reuse this function in virtio_net_reset().

Yeah, I'll fix it.

Thanks.

Quoted 12 lines

hw/net/virtio-net.c static uint64_t · line 757

Quoted 10 lines
hw/net/virtio-net.c +3 -0
+}
+
    static void virtio_net_reset(VirtIODevice *vdev)
    {
        VirtIONet *n = VIRTIO_NET(vdev);
@@ -741,6 +754,7 @@ static uint64_t
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
        }
          if (!get_vhost_net(nc->peer)) {
+        virtio_add_feature(&features, VIRTIO_F_RING_RESET);

This breaks migration compatibility.

Quoted 8 lines

We probably need:

1) a new command line parameter
2) make it disabled for pre-7.2 machine

Thanks

Sorry, I don't get what is the meaning of "pre-7.2 machine". Could you please explain it?

Quoted 7 lines
Quoted 5 lines

I meant for pre 7.2 machine type, we should make reset fault off by default.

Otherwise we break migration compatibility.

Thanks

Sorry, I did not express myself clearly. Is "7.2" the version of a system or a module? If it is a system version, what is the system?

It's the machine type to make sure the migration can work. (you can get the list of those types via qemu -machine ?)

Quoted 8 lines

E.g you can start a 7.0 machine on Qemu 7.1 and it guarantees to be migrated to 7.0 machine on Qemu 7.0.

Quoted 2 lines

I did not have backgrond knowledge related to this part and will investigate migration issues afterwards.

Git grep hw_compat_7_0 may give you more hints.

Thanks

Okay, I get it.

Thanks for your guidelines.

Quoted 16 lines
Quoted 16 lines

Thanks.

Quoted 13 lines
Quoted 13 lines

Thanks

Quoted 11 lines
Quoted 11 lines
hw/net/virtio-net.c +0 -0
            return features;
        }
    @@ -3766,6 +3780,7 @@ static void virtio_net_class_init(ObjectClass

*klass, void *data)

        vdc->set_features = virtio_net_set_features;
        vdc->bad_features = virtio_net_bad_features;
        vdc->reset = virtio_net_reset;
+    vdc->queue_reset = virtio_net_queue_reset;
        vdc->set_status = virtio_net_set_status;
        vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
        vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
author
#19 lore raw

PATCH 8/16 vhost: add op to enable or disable a single vring

In reply to Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface. It requires a reply to indicate that the reset operation is finished, so the parameter, wait_for_reply, is added.

---
 include/hw/virtio/vhost-backend.h | 4 ++++
 1 file changed, 4 insertions(+)
include/hw/virtio/vhost-backend.h +4 -0
index eab46d7f0b..7bddd1e9a0 100644
@@ -81,6 +81,9 @@ typedef int (*vhost_set_backend_cap_op)(struct vhost_dev *dev);
 typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
 typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
 typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable,
+                                                bool wait_for_reply);
 typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                          int enable);
 typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);
@@ -155,6 +158,7 @@ typedef struct VhostOps {
     vhost_set_owner_op vhost_set_owner;
     vhost_reset_device_op vhost_reset_device;
     vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
     vhost_set_vring_enable_op vhost_set_vring_enable;
     vhost_requires_shm_log_op vhost_requires_shm_log;
     vhost_migration_done_op vhost_migration_done;
Signature · 1 line
2.32.0
#20 lore raw

Re: [PATCH 08/16] vhost: add op to enable or disable a single vring

Quoted 7 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface. It requires a reply to indicate that the reset operation is finished, so the parameter, wait_for_reply, is added.

The wait reply seems to be a implementation specific thing. Can we hide it?

Thanks

Quoted 29 lines
---
  include/hw/virtio/vhost-backend.h | 4 ++++
  1 file changed, 4 insertions(+)
include/hw/virtio/vhost-backend.h +4 -0
index eab46d7f0b..7bddd1e9a0 100644
@@ -81,6 +81,9 @@ typedef int (*vhost_set_backend_cap_op)(struct vhost_dev *dev);
  typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
  typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
  typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable,
+                                                bool wait_for_reply);
  typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                           int enable);
  typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);
@@ -155,6 +158,7 @@ typedef struct VhostOps {
      vhost_set_owner_op vhost_set_owner;
      vhost_reset_device_op vhost_reset_device;
      vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
      vhost_set_vring_enable_op vhost_set_vring_enable;
      vhost_requires_shm_log_op vhost_requires_shm_log;
      vhost_migration_done_op vhost_migration_done;
author
#21 lore raw

Re: [PATCH 08/16] vhost: add op to enable or disable a single vring

Quoted 6 lines · Jason Wang
Quoted 4 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Quoted 9 lines
Quoted 3 lines

Resetting a single vq will rely on this interface. It requires a reply to indicate that the reset operation is finished, so the parameter, wait_for_reply, is added.

The wait reply seems to be a implementation specific thing. Can we hide it?

Thanks

I do not hide wait_for_reply here because when stopping the queue, a reply is needed to ensure that the message has been processed and queue has been disabled.

When restarting the queue, we do not need a reply, which is the same as what qemu does in vhost_dev_start().

So I add this parameter to distinguish the two cases.

I think one alternative implementation is to add a interface in VhostOps: queue_reset(). In this way details can be hidden. What do you think of this solution? Does it look better?

Thanks

Quoted 31 lines
Quoted 31 lines

---
  include/hw/virtio/vhost-backend.h | 4 ++++

  1 file changed, 4 insertions(+)
include/hw/virtio/vhost-backend.h +3 -0
index eab46d7f0b..7bddd1e9a0 100644
@@ -81,6 +81,9 @@ typedef int (*vhost_set_backend_cap_op)(struct 
 vhost_dev *dev);
   typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
   typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
   typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable,
+                                                bool wait_for_reply);
   typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                            int enable);

  typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);

include/hw/virtio/vhost-backend.h +1 -0
@@ -155,6 +158,7 @@ typedef struct VhostOps {
       vhost_set_owner_op vhost_set_owner;
       vhost_reset_device_op vhost_reset_device;
       vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
       vhost_set_vring_enable_op vhost_set_vring_enable;
       vhost_requires_shm_log_op vhost_requires_shm_log;
       vhost_migration_done_op vhost_migration_done;
#22 lore raw

Re: [PATCH 08/16] vhost: add op to enable or disable a single vring

Quoted 14 lines · Kangjie Xu
Quoted 11 lines · Jason Wang
Quoted 7 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface. It requires a reply to indicate that the reset operation is finished, so the parameter, wait_for_reply, is added.

Quoted 8 lines
Quoted 5 lines

The wait reply seems to be a implementation specific thing. Can we hide it?

Thanks

I do not hide wait_for_reply here because when stopping the queue, a reply is needed to ensure that the message has been processed and queue has been disabled.

This needs to be done at vhost-backend level instead of the general vhost code.

E.g vhost-kernel or vDPA is using ioctl() which is synchronous.

Quoted 9 lines

When restarting the queue, we do not need a reply, which is the same as what qemu does in vhost_dev_start().

So I add this parameter to distinguish the two cases.

I think one alternative implementation is to add a interface in
VhostOps: queue_reset(). In this way details can be hidden. What do you think of this solution? Does it look better?

Let me ask it differently, under which case can we call this function with wait_for_reply = false?

Thanks

Quoted 36 lines

Thanks

Quoted 32 lines
Quoted 32 lines
---
  include/hw/virtio/vhost-backend.h | 4 ++++
  1 file changed, 4 insertions(+)
include/hw/virtio/vhost-backend.h +3 -0
index eab46d7f0b..7bddd1e9a0 100644
@@ -81,6 +81,9 @@ typedef int (*vhost_set_backend_cap_op)(struct
 vhost_dev *dev);
  typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
  typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
  typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable,
+                                                bool wait_for_reply);
  typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                           int enable);
  typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);
include/hw/virtio/vhost-backend.h +1 -0
@@ -155,6 +158,7 @@ typedef struct VhostOps {
      vhost_set_owner_op vhost_set_owner;
      vhost_reset_device_op vhost_reset_device;
      vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
      vhost_set_vring_enable_op vhost_set_vring_enable;
      vhost_requires_shm_log_op vhost_requires_shm_log;
      vhost_migration_done_op vhost_migration_done;
author
#23 lore raw

Re: [PATCH 08/16] vhost: add op to enable or disable a single vring

Quoted 15 lines · Jason Wang
Quoted 14 lines · Kangjie Xu
Quoted 12 lines · Jason Wang
Quoted 7 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface. It requires a reply to indicate that the reset operation is finished, so the parameter, wait_for_reply, is added.

The wait reply seems to be a implementation specific thing. Can we hide it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Thanks

I do not hide wait_for_reply here because when stopping the queue, a reply is needed to ensure that the message has been processed and queue has been disabled.

This needs to be done at vhost-backend level instead of the general vhost code.

E.g vhost-kernel or vDPA is using ioctl() which is synchronous.

Yeah, I understand your concerns, will fix it in the next version.

Quoted 5 lines
Quoted 5 lines

When restarting the queue, we do not need a reply, which is the same as what qemu does in vhost_dev_start().

So I add this parameter to distinguish the two cases.

Quoted 7 lines
Quoted 3 lines

I think one alternative implementation is to add a interface in
VhostOps: queue_reset(). In this way details can be hidden. What do you think of this solution? Does it look better?

Let me ask it differently, under which case can we call this function with wait_for_reply = false?

Thanks

Cases when we do not need to wait until that the reply has been processed. Actually in dev_start(), or dev_stop(), they do not wait for replies when enabling/disabling vqs.

Specifically, vhost_set_vring_enable() will call it with wait_for_reply = false.

In our vq reset scenario, it should be synchronous because the driver need to modify queues after the device stop using the vq in the backend. Undefined errors will occur If the device is still using the queue when the driver is making modifications to it,

Back to our implementation, we will not expose this parameter in the next version.

Thanks.

Quoted 32 lines
Quoted 32 lines

Thanks

Quoted 30 lines
Quoted 30 lines
---
   include/hw/virtio/vhost-backend.h | 4 ++++
   1 file changed, 4 insertions(+)
include/hw/virtio/vhost-backend.h +3 -0
index eab46d7f0b..7bddd1e9a0 100644
@@ -81,6 +81,9 @@ typedef int (*vhost_set_backend_cap_op)(struct
 vhost_dev *dev);
   typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
   typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
   typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable,
+                                                bool wait_for_reply);
   typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                            int enable);
   typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);
include/hw/virtio/vhost-backend.h +1 -0
@@ -155,6 +158,7 @@ typedef struct VhostOps {
       vhost_set_owner_op vhost_set_owner;
       vhost_reset_device_op vhost_reset_device;
       vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
       vhost_set_vring_enable_op vhost_set_vring_enable;
       vhost_requires_shm_log_op vhost_requires_shm_log;
       vhost_migration_done_op vhost_migration_done;
#24 lore raw

Re: [PATCH 08/16] vhost: add op to enable or disable a single vring

Quoted 35 lines · Kangjie Xu
Quoted 15 lines · Jason Wang
Quoted 14 lines · Kangjie Xu
Quoted 12 lines · Jason Wang
Quoted 7 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface. It requires a reply to indicate that the reset operation is finished, so the parameter, wait_for_reply, is added.

The wait reply seems to be a implementation specific thing. Can we hide it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Thanks

I do not hide wait_for_reply here because when stopping the queue, a reply is needed to ensure that the message has been processed and queue has been disabled.

This needs to be done at vhost-backend level instead of the general vhost code.

E.g vhost-kernel or vDPA is using ioctl() which is synchronous.

Yeah, I understand your concerns, will fix it in the next version.

Quoted 8 lines
Quoted 8 lines

When restarting the queue, we do not need a reply, which is the same as what qemu does in vhost_dev_start().

So I add this parameter to distinguish the two cases.

I think one alternative implementation is to add a interface in
VhostOps: queue_reset(). In this way details can be hidden. What do you think of this solution? Does it look better?

Quoted 7 lines
Quoted 4 lines

Let me ask it differently, under which case can we call this function with wait_for_reply = false?

Thanks

Cases when we do not need to wait until that the reply has been processed. Actually in dev_start(), or dev_stop(),

But we don't need to send virtqueue reset requests in those cases?

Quoted 6 lines

they do not wait for
replies when enabling/disabling vqs.

Specifically, vhost_set_vring_enable() will call it with wait_for_reply = false.

Quoted 7 lines

In our vq reset scenario, it should be synchronous because the driver need to modify queues after the device stop using the vq in the backend. Undefined errors will occur If the device is still using the queue when the driver is making modifications to it,

Back to our implementation, we will not expose this parameter in the next version.

Ok.

Thanks

Quoted 36 lines

Thanks.

Quoted 32 lines
Quoted 32 lines

Thanks

Quoted 30 lines
Quoted 30 lines
---
   include/hw/virtio/vhost-backend.h | 4 ++++
   1 file changed, 4 insertions(+)
include/hw/virtio/vhost-backend.h +3 -0
index eab46d7f0b..7bddd1e9a0 100644
@@ -81,6 +81,9 @@ typedef int (*vhost_set_backend_cap_op)(struct
 vhost_dev *dev);
   typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
   typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
   typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable,
+                                                bool wait_for_reply);
   typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                            int enable);
   typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);
include/hw/virtio/vhost-backend.h +1 -0
@@ -155,6 +158,7 @@ typedef struct VhostOps {
       vhost_set_owner_op vhost_set_owner;
       vhost_reset_device_op vhost_reset_device;
       vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
       vhost_set_vring_enable_op vhost_set_vring_enable;
       vhost_requires_shm_log_op vhost_requires_shm_log;
       vhost_migration_done_op vhost_migration_done;
author
#25 lore raw

Re: [PATCH 08/16] vhost: add op to enable or disable a single vring

Quoted 33 lines · Jason Wang
Quoted 32 lines · Kangjie Xu
Quoted 13 lines · Jason Wang
Quoted 12 lines · Kangjie Xu
Quoted 11 lines · Jason Wang
Quoted 7 lines · Kangjie Xu

The interface to set enable status for a single vring is lacked in VhostOps, since the vhost_set_vring_enable_op will manipulate all virtqueues in a device.

Resetting a single vq will rely on this interface. It requires a reply to indicate that the reset operation is finished, so the parameter, wait_for_reply, is added.

The wait reply seems to be a implementation specific thing. Can we hide it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Thanks

I do not hide wait_for_reply here because when stopping the queue, a reply is needed to ensure that the message has been processed and queue has been disabled.

This needs to be done at vhost-backend level instead of the general vhost code.

E.g vhost-kernel or vDPA is using ioctl() which is synchronous.

Yeah, I understand your concerns, will fix it in the next version.

Quoted 8 lines
Quoted 8 lines

When restarting the queue, we do not need a reply, which is the same as what qemu does in vhost_dev_start().

So I add this parameter to distinguish the two cases.

I think one alternative implementation is to add a interface in
VhostOps: queue_reset(). In this way details can be hidden. What do you think of this solution? Does it look better?

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

Let me ask it differently, under which case can we call this function with wait_for_reply = false?

Thanks

Cases when we do not need to wait until that the reply has been processed. Actually in dev_start(), or dev_stop(),

But we don't need to send virtqueue reset requests in those cases?

You're right, we don't need to do this in those cases.

Thanks

Quoted 51 lines
Quoted 6 lines

they do not wait for
replies when enabling/disabling vqs.

Specifically, vhost_set_vring_enable() will call it with wait_for_reply = false.

Quoted 7 lines

In our vq reset scenario, it should be synchronous because the driver need to modify queues after the device stop using the vq in the backend. Undefined errors will occur If the device is still using the queue when the driver is making modifications to it,

Back to our implementation, we will not expose this parameter in the next version.

Ok.

Thanks

Quoted 34 lines

Thanks.

Quoted 32 lines
Quoted 32 lines

Thanks

Quoted 30 lines
Quoted 30 lines
---
    include/hw/virtio/vhost-backend.h | 4 ++++
    1 file changed, 4 insertions(+)
include/hw/virtio/vhost-backend.h +3 -0
index eab46d7f0b..7bddd1e9a0 100644
@@ -81,6 +81,9 @@ typedef int (*vhost_set_backend_cap_op)(struct
 vhost_dev *dev);
    typedef int (*vhost_set_owner_op)(struct vhost_dev *dev);
    typedef int (*vhost_reset_device_op)(struct vhost_dev *dev);
    typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx);
+typedef int (*vhost_set_single_vring_enable_op)(struct vhost_dev *dev,
+                                                int index, int enable,
+                                                bool wait_for_reply);
    typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev,
                                             int enable);
    typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev);
include/hw/virtio/vhost-backend.h +1 -0
@@ -155,6 +158,7 @@ typedef struct VhostOps {
        vhost_set_owner_op vhost_set_owner;
        vhost_reset_device_op vhost_reset_device;
        vhost_get_vq_index_op vhost_get_vq_index;
+    vhost_set_single_vring_enable_op vhost_set_single_vring_enable;
        vhost_set_vring_enable_op vhost_set_vring_enable;
        vhost_requires_shm_log_op vhost_requires_shm_log;
        vhost_migration_done_op vhost_migration_done;
author
#26 lore raw

PATCH 9/16 vhost-user: enable/disable a single vring

In reply to Kangjie Xu

Implement the vhost_set_single_vring_enable, which is to enable or disable a single vring.

The parameter wait_for_reply is added to help for some cases such as vq reset.

Meanwhile, vhost_user_set_vring_enable() is refactored.

---
 hw/virtio/vhost-user.c | 55 ++++++++++++++++++++++++++++++++++++------
 1 file changed, 48 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +48 -7
index 75b8df21a4..5a80a415f0 100644
@@ -267,6 +267,8 @@ struct scrub_regions {
     int fd_idx;
 };
 
+static int enforce_reply(struct vhost_dev *dev, const VhostUserMsg *msg);
+
 static bool ioeventfd_enabled(void)
 {
     return !kvm_enabled() || kvm_eventfds_enabled();
@@ -1198,6 +1200,49 @@ static int vhost_user_set_vring_base(struct vhost_dev *dev,
     return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
 }
 
+
+static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable,
+                                              bool wait_for_reply)
+{
+    int ret;
+
+    if (index < dev->vq_index || index >= dev->vq_index + dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    VhostUserMsg msg = {
+        .hdr.request = VHOST_USER_SET_VRING_ENABLE,
+        .hdr.flags = VHOST_USER_VERSION,
+        .payload.state = state,
+        .hdr.size = sizeof(msg.payload.state),
+    };
+
+    bool reply_supported = virtio_has_feature(dev->protocol_features,
+                                              VHOST_USER_PROTOCOL_F_REPLY_ACK);
+
+    if (reply_supported && wait_for_reply) {
+        msg.hdr.flags |= VHOST_USER_NEED_REPLY_MASK;
+    }
+
+    ret = vhost_user_write(dev, &msg, NULL, 0);
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (wait_for_reply) {
+        return enforce_reply(dev, &msg);
+    }
+
+    return ret;
+}
+
 static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
 {
     int i;
@@ -1207,13 +1252,8 @@ static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
     }
 
     for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, &state);
+        int ret = vhost_user_set_single_vring_enable(dev, dev->vq_index + i,
+                                                     enable, false);
         if (ret < 0) {
             /*
              * Restoring the previous state is likely infeasible, as well as
@@ -2627,6 +2667,7 @@ const VhostOps user_ops = {
         .vhost_set_owner = vhost_user_set_owner,
         .vhost_reset_device = vhost_user_reset_device,
         .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable = vhost_user_set_single_vring_enable,
         .vhost_set_vring_enable = vhost_user_set_vring_enable,
         .vhost_requires_shm_log = vhost_user_requires_shm_log,
         .vhost_migration_done = vhost_user_migration_done,
Signature · 1 line
2.32.0
#27 lore raw

Re: [PATCH 09/16] vhost-user: enable/disable a single vring

Quoted 51 lines · Kangjie Xu

Implement the vhost_set_single_vring_enable, which is to enable or disable a single vring.

The parameter wait_for_reply is added to help for some cases such as vq reset.

Meanwhile, vhost_user_set_vring_enable() is refactored.

---
  hw/virtio/vhost-user.c | 55 ++++++++++++++++++++++++++++++++++++------
  1 file changed, 48 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +22 -0
index 75b8df21a4..5a80a415f0 100644
@@ -267,6 +267,8 @@ struct scrub_regions {
      int fd_idx;
  };
 
+static int enforce_reply(struct vhost_dev *dev, const VhostUserMsg *msg);
+
  static bool ioeventfd_enabled(void)
  {
      return !kvm_enabled() || kvm_eventfds_enabled();
@@ -1198,6 +1200,49 @@ static int vhost_user_set_vring_base(struct vhost_dev *dev,
      return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
  }
 
+
+static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable,
+                                              bool wait_for_reply)
+{
+    int ret;
+
+    if (index < dev->vq_index || index >= dev->vq_index + dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    VhostUserMsg msg = {
+        .hdr.request = VHOST_USER_SET_VRING_ENABLE,
+        .hdr.flags = VHOST_USER_VERSION,

hw/virtio/vhost-user.c static int vhost_user_set_vring_base(struct vhost_dev *dev, · line 1232

Quoted 10 lines
hw/virtio/vhost-user.c +10 -0
+        .payload.state = state,
+        .hdr.size = sizeof(msg.payload.state),
+    };
+
+    bool reply_supported = virtio_has_feature(dev->protocol_features,
+                                              VHOST_USER_PROTOCOL_F_REPLY_ACK);
+
+    if (reply_supported && wait_for_reply) {
+        msg.hdr.flags |= VHOST_USER_NEED_REPLY_MASK;
+    }

Do we need to fail if !realy_supported && wait_for_reply?

Thanks

Quoted 40 lines
hw/virtio/vhost-user.c +16 -7
+
+    ret = vhost_user_write(dev, &msg, NULL, 0);
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (wait_for_reply) {
+        return enforce_reply(dev, &msg);
+    }
+
+    return ret;
+}
+
  static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
  {
      int i;
@@ -1207,13 +1252,8 @@ static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable)
      }
 
      for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, &state);
+        int ret = vhost_user_set_single_vring_enable(dev, dev->vq_index + i,
+                                                     enable, false);
          if (ret < 0) {
              /*
               * Restoring the previous state is likely infeasible, as well as
@@ -2627,6 +2667,7 @@ const VhostOps user_ops = {
          .vhost_set_owner = vhost_user_set_owner,
          .vhost_reset_device = vhost_user_reset_device,
          .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable = vhost_user_set_single_vring_enable,
          .vhost_set_vring_enable = vhost_user_set_vring_enable,
          .vhost_requires_shm_log = vhost_user_requires_shm_log,
          .vhost_migration_done = vhost_user_migration_done,
author
#28 lore raw

Re: [PATCH 09/16] vhost-user: enable/disable a single vring

Quoted 66 lines · Jason Wang
Quoted 50 lines · Kangjie Xu

Implement the vhost_set_single_vring_enable, which is to enable or disable a single vring.

The parameter wait_for_reply is added to help for some cases such as vq reset.

Meanwhile, vhost_user_set_vring_enable() is refactored.

---
  hw/virtio/vhost-user.c | 55 ++++++++++++++++++++++++++++++++++++------

  1 file changed, 48 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +1 -0
index 75b8df21a4..5a80a415f0 100644
@@ -267,6 +267,8 @@ struct scrub_regions {
       int fd_idx;
   };
   +static int enforce_reply(struct vhost_dev *dev, const VhostUserMsg *msg);
+
   static bool ioeventfd_enabled(void)
   {
       return !kvm_enabled() || kvm_eventfds_enabled();
@@ -1198,6 +1200,49 @@ static int vhost_user_set_vring_base(struct 
 vhost_dev *dev,
       return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
   }

  +

hw/virtio/vhost-user.c +19 -0
+static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable,
+                                              bool wait_for_reply)
+{
+    int ret;
+
+    if (index < dev->vq_index || index >= dev->vq_index + dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    VhostUserMsg msg = {
+        .hdr.request = VHOST_USER_SET_VRING_ENABLE,
+        .hdr.flags = VHOST_USER_VERSION,

hw/virtio/vhost-user.c static int vhost_user_set_vring_base(struct vhost_dev *dev, · line 1232

Quoted 10 lines
hw/virtio/vhost-user.c +10 -0
+        .payload.state = state,
+        .hdr.size = sizeof(msg.payload.state),
+    };
+
+    bool reply_supported = virtio_has_feature(dev->protocol_features,
+ VHOST_USER_PROTOCOL_F_REPLY_ACK);
+
+    if (reply_supported && wait_for_reply) {
+        msg.hdr.flags |= VHOST_USER_NEED_REPLY_MASK;
+    }

Do we need to fail if !realy_supported && wait_for_reply?

Quoted 3 lines

Thanks

I guess you mean "should we fail if VHOST_USER_PROTOCOL_F_REPLY_ACK feature is not supported?".

The implementation here is similar to that in vhost_user_set_vring_addr().

If this feature is not supported, it will call enforce_reply(), then call vhost_user_get_features() to get a reply.

Since the messages will be processed sequentailly in DPDK, success of enforce_reply() means the previous message VHOST_USER_SET_VRING_ENABLE has been processed.

Thanks

Quoted 44 lines
Quoted 44 lines
hw/virtio/vhost-user.c +13 -0
+
+    ret = vhost_user_write(dev, &msg, NULL, 0);
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (wait_for_reply) {
+        return enforce_reply(dev, &msg);
+    }
+
+    return ret;
+}
+

  static int vhost_user_set_vring_enable(struct vhost_dev *dev, int enable) {
      int i;

hw/virtio/vhost-user.c +2 -7
@@ -1207,13 +1252,8 @@ static int vhost_user_set_vring_enable(struct 
 vhost_dev *dev, int enable)
       }
         for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE, 
 &state);
+        int ret = vhost_user_set_single_vring_enable(dev, 
 dev->vq_index + i,
+                                                     enable, false);
           if (ret < 0) {

              /*
               * Restoring the previous state is likely infeasible, as well as

hw/virtio/vhost-user.c +1 -0
@@ -2627,6 +2667,7 @@ const VhostOps user_ops = {
           .vhost_set_owner = vhost_user_set_owner,
           .vhost_reset_device = vhost_user_reset_device,
           .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable = 
 vhost_user_set_single_vring_enable,
           .vhost_set_vring_enable = vhost_user_set_vring_enable,
           .vhost_requires_shm_log = vhost_user_requires_shm_log,

          .vhost_migration_done = vhost_user_migration_done,

#29 lore raw

Re: [PATCH 09/16] vhost-user: enable/disable a single vring

Quoted 73 lines · Kangjie Xu
Quoted 67 lines · Jason Wang
Quoted 51 lines · Kangjie Xu

Implement the vhost_set_single_vring_enable, which is to enable or disable a single vring.

The parameter wait_for_reply is added to help for some cases such as vq reset.

Meanwhile, vhost_user_set_vring_enable() is refactored.

---
  hw/virtio/vhost-user.c | 55 ++++++++++++++++++++++++++++++++++++------
  1 file changed, 48 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +20 -0
index 75b8df21a4..5a80a415f0 100644
@@ -267,6 +267,8 @@ struct scrub_regions {
      int fd_idx;
  };
  +static int enforce_reply(struct vhost_dev *dev, const VhostUserMsg
 *msg);
+
  static bool ioeventfd_enabled(void)
  {
      return !kvm_enabled() || kvm_eventfds_enabled();
@@ -1198,6 +1200,49 @@ static int vhost_user_set_vring_base(struct
 vhost_dev *dev,
      return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
  }
  +
+static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable,
+                                              bool wait_for_reply)
+{
+    int ret;
+
+    if (index < dev->vq_index || index >= dev->vq_index + dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    VhostUserMsg msg = {
+        .hdr.request = VHOST_USER_SET_VRING_ENABLE,
+        .hdr.flags = VHOST_USER_VERSION,

hw/virtio/vhost-user.c static int vhost_user_set_vring_base(struct · line 1232

Quoted 10 lines
hw/virtio/vhost-user.c +10 -0
+        .payload.state = state,
+        .hdr.size = sizeof(msg.payload.state),
+    };
+
+    bool reply_supported = virtio_has_feature(dev->protocol_features,
+ VHOST_USER_PROTOCOL_F_REPLY_ACK);
+
+    if (reply_supported && wait_for_reply) {
+        msg.hdr.flags |= VHOST_USER_NEED_REPLY_MASK;
+    }

Do we need to fail if !realy_supported && wait_for_reply?

Quoted 3 lines

Thanks

Quoted 7 lines

I guess you mean "should we fail if VHOST_USER_PROTOCOL_F_REPLY_ACK feature is not supported?".

The implementation here is similar to that in vhost_user_set_vring_addr().

If this feature is not supported, it will call enforce_reply(), then call vhost_user_get_features() to get a reply.

Ok, so you meant we can then fallback to VHOST_USER_GET_FEATURES? I wonder how robust is this, e.g is the bakcend required to process vhost-user request in order?

Thanks

Quoted 54 lines

Since the messages will be processed sequentailly in DPDK, success of enforce_reply() means the previous message VHOST_USER_SET_VRING_ENABLE has been processed.

Thanks

Quoted 46 lines
Quoted 46 lines
hw/virtio/vhost-user.c +15 -7
+
+    ret = vhost_user_write(dev, &msg, NULL, 0);
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (wait_for_reply) {
+        return enforce_reply(dev, &msg);
+    }
+
+    return ret;
+}
+
  static int vhost_user_set_vring_enable(struct vhost_dev *dev, int
 enable)
  {
      int i;
@@ -1207,13 +1252,8 @@ static int vhost_user_set_vring_enable(struct
 vhost_dev *dev, int enable)
      }
        for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE,
 &state);
+        int ret = vhost_user_set_single_vring_enable(dev,
 dev->vq_index + i,
+                                                     enable, false);
          if (ret < 0) {

/*
* Restoring the previous state is likely infeasible,
as well as

hw/virtio/vhost-user.c +1 -0
@@ -2627,6 +2667,7 @@ const VhostOps user_ops = {
          .vhost_set_owner = vhost_user_set_owner,
          .vhost_reset_device = vhost_user_reset_device,
          .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable =
 vhost_user_set_single_vring_enable,
          .vhost_set_vring_enable = vhost_user_set_vring_enable,
          .vhost_requires_shm_log = vhost_user_requires_shm_log,

.vhost_migration_done = vhost_user_migration_done,

author
#30 lore raw

Re: [PATCH 09/16] vhost-user: enable/disable a single vring

Quoted 76 lines · Jason Wang
Quoted 75 lines · Kangjie Xu
Quoted 65 lines · Jason Wang
Quoted 51 lines · Kangjie Xu

Implement the vhost_set_single_vring_enable, which is to enable or disable a single vring.

The parameter wait_for_reply is added to help for some cases such as vq reset.

Meanwhile, vhost_user_set_vring_enable() is refactored.

---
   hw/virtio/vhost-user.c | 55 ++++++++++++++++++++++++++++++++++++------
   1 file changed, 48 insertions(+), 7 deletions(-)
hw/virtio/vhost-user.c +20 -0
index 75b8df21a4..5a80a415f0 100644
@@ -267,6 +267,8 @@ struct scrub_regions {
       int fd_idx;
   };
   +static int enforce_reply(struct vhost_dev *dev, const VhostUserMsg
 *msg);
+
   static bool ioeventfd_enabled(void)
   {
       return !kvm_enabled() || kvm_eventfds_enabled();
@@ -1198,6 +1200,49 @@ static int vhost_user_set_vring_base(struct
 vhost_dev *dev,
       return vhost_set_vring(dev, VHOST_USER_SET_VRING_BASE, ring);
   }
   +
+static int vhost_user_set_single_vring_enable(struct vhost_dev *dev,
+                                              int index,
+                                              int enable,
+                                              bool wait_for_reply)
+{
+    int ret;
+
+    if (index < dev->vq_index || index >= dev->vq_index + dev->nvqs) {
+        return -EINVAL;
+    }
+
+    struct vhost_vring_state state = {
+        .index = index,
+        .num   = enable,
+    };
+
+    VhostUserMsg msg = {
+        .hdr.request = VHOST_USER_SET_VRING_ENABLE,
+        .hdr.flags = VHOST_USER_VERSION,

hw/virtio/vhost-user.c static int vhost_user_set_vring_base(struct · line 1232

Quoted 10 lines
hw/virtio/vhost-user.c +10 -0
+        .payload.state = state,
+        .hdr.size = sizeof(msg.payload.state),
+    };
+
+    bool reply_supported = virtio_has_feature(dev->protocol_features,
+ VHOST_USER_PROTOCOL_F_REPLY_ACK);
+
+    if (reply_supported && wait_for_reply) {
+        msg.hdr.flags |= VHOST_USER_NEED_REPLY_MASK;
+    }

Do we need to fail if !realy_supported && wait_for_reply?

Quoted 3 lines

Thanks

I guess you mean "should we fail if VHOST_USER_PROTOCOL_F_REPLY_ACK feature is not supported?".

The implementation here is similar to that in vhost_user_set_vring_addr().

Quoted 7 lines
Quoted 2 lines

If this feature is not supported, it will call enforce_reply(), then call vhost_user_get_features() to get a reply.

Ok, so you meant we can then fallback to VHOST_USER_GET_FEATURES? I wonder how robust is this, e.g is the bakcend required to process vhost-user request in order?

Thanks

Yes, we rely on VHOST_USER_GET_FEATURES message to ensure that VHOST_USER_SET_VRING_ENABLE has been processed.

It's not robust. I reviewed the vhost-user protocol in qemu doc, actually it does not specify that the backend should process them in order.

I think adding a new vhost protocol message can fix this issue. The new invented message should reset the queue, and have a blocked read to ensure the message has been processed.

Thanks

Quoted 51 lines
Quoted 51 lines

Since the messages will be processed sequentailly in DPDK, success of enforce_reply() means the previous message VHOST_USER_SET_VRING_ENABLE has been processed.

Thanks

Quoted 45 lines
Quoted 45 lines
hw/virtio/vhost-user.c +15 -7
+
+    ret = vhost_user_write(dev, &msg, NULL, 0);
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (wait_for_reply) {
+        return enforce_reply(dev, &msg);
+    }
+
+    return ret;
+}
+
   static int vhost_user_set_vring_enable(struct vhost_dev *dev, int
 enable)
   {
       int i;
@@ -1207,13 +1252,8 @@ static int vhost_user_set_vring_enable(struct
 vhost_dev *dev, int enable)
       }
         for (i = 0; i < dev->nvqs; ++i) {
-        int ret;
-        struct vhost_vring_state state = {
-            .index = dev->vq_index + i,
-            .num   = enable,
-        };
-
-        ret = vhost_set_vring(dev, VHOST_USER_SET_VRING_ENABLE,
 &state);
+        int ret = vhost_user_set_single_vring_enable(dev,
 dev->vq_index + i,
+                                                     enable, false);
           if (ret < 0) {

/*
* Restoring the previous state is likely infeasible,
as well as

hw/virtio/vhost-user.c +1 -0
@@ -2627,6 +2667,7 @@ const VhostOps user_ops = {
           .vhost_set_owner = vhost_user_set_owner,
           .vhost_reset_device = vhost_user_reset_device,
           .vhost_get_vq_index = vhost_user_get_vq_index,
+        .vhost_set_single_vring_enable =
 vhost_user_set_single_vring_enable,
           .vhost_set_vring_enable = vhost_user_set_vring_enable,
           .vhost_requires_shm_log = vhost_user_requires_shm_log,

.vhost_migration_done = vhost_user_migration_done,

author
#31 lore raw

PATCH 10/16 vhost: extract the logic of unmapping the vrings and desc

In reply to Kangjie Xu

Introduce vhost_virtqueue_unmap() to ummap the vrings and desc of a virtqueue.

The function will be used later.

---
 hw/virtio/vhost.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
hw/virtio/vhost.c +14 -6
index 0827d631c0..e467dfc7bc 100644
@@ -1197,6 +1197,19 @@ fail_alloc_desc:
     return r;
 }
 
+static void vhost_virtqueue_unmap(struct vhost_dev *dev,
+                                  struct VirtIODevice *vdev,
+                                  struct vhost_virtqueue *vq,
+                                  unsigned idx)
+{
+    vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
+                       1, virtio_queue_get_used_size(vdev, idx));
+    vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
+                       0, virtio_queue_get_avail_size(vdev, idx));
+    vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
+                       0, virtio_queue_get_desc_size(vdev, idx));
+}
+
 static void vhost_virtqueue_stop(struct vhost_dev *dev,
                                     struct VirtIODevice *vdev,
                                     struct vhost_virtqueue *vq,
@@ -1235,12 +1248,7 @@ static void vhost_virtqueue_stop(struct vhost_dev *dev,
                                                 vhost_vq_index);
     }
 
-    vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
-                       1, virtio_queue_get_used_size(vdev, idx));
-    vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
-                       0, virtio_queue_get_avail_size(vdev, idx));
-    vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
-                       0, virtio_queue_get_desc_size(vdev, idx));
+    vhost_virtqueue_unmap(dev, vdev, vq, idx);
 }
 
 static void vhost_eventfd_add(MemoryListener *listener,
Signature · 1 line
2.32.0
Acked-by on v1 10/16
#32 lore raw

Re: [PATCH 10/16] vhost: extract the logic of unmapping the vrings and desc

Quoted 7 lines · Kangjie Xu

Introduce vhost_virtqueue_unmap() to ummap the vrings and desc of a virtqueue.

The function will be used later.

Quoted 42 lines
---
  hw/virtio/vhost.c | 20 ++++++++++++++------
  1 file changed, 14 insertions(+), 6 deletions(-)
hw/virtio/vhost.c +14 -6
index 0827d631c0..e467dfc7bc 100644
@@ -1197,6 +1197,19 @@ fail_alloc_desc:
      return r;
  }
 
+static void vhost_virtqueue_unmap(struct vhost_dev *dev,
+                                  struct VirtIODevice *vdev,
+                                  struct vhost_virtqueue *vq,
+                                  unsigned idx)
+{
+    vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
+                       1, virtio_queue_get_used_size(vdev, idx));
+    vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
+                       0, virtio_queue_get_avail_size(vdev, idx));
+    vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
+                       0, virtio_queue_get_desc_size(vdev, idx));
+}
+
  static void vhost_virtqueue_stop(struct vhost_dev *dev,
                                      struct VirtIODevice *vdev,
                                      struct vhost_virtqueue *vq,
@@ -1235,12 +1248,7 @@ static void vhost_virtqueue_stop(struct vhost_dev *dev,
                                                  vhost_vq_index);
      }
 
-    vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx),
-                       1, virtio_queue_get_used_size(vdev, idx));
-    vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx),
-                       0, virtio_queue_get_avail_size(vdev, idx));
-    vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx),
-                       0, virtio_queue_get_desc_size(vdev, idx));
+    vhost_virtqueue_unmap(dev, vdev, vq, idx);
  }
 
  static void vhost_eventfd_add(MemoryListener *listener,
author
#33 lore raw

PATCH 11/16 vhost: introduce restart and release for vhost_dev's vqs

In reply to Kangjie Xu

Introduce vhost_dev_virtqueue_restart(), which can restart the virtqueue when the vhost has already started running.

Meanwhile, vhost_dev_virtqueue_release(), which can ummap the vrings and the desc of a specific vq of a device.

Combining the two functions, we can reset a virtqueue with a started vhost.

---
 hw/virtio/vhost.c         | 29 +++++++++++++++++++++++++++++
 include/hw/virtio/vhost.h |  6 ++++++
 2 files changed, 35 insertions(+)
hw/virtio/vhost.c +29 -0
index e467dfc7bc..d158d71866 100644
@@ -1904,3 +1904,32 @@ int vhost_net_set_backend(struct vhost_dev *hdev,
 
     return -ENOSYS;
 }
+
+void vhost_dev_virtqueue_release(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                 int vq_index)
+{
+    int idx = vq_index - hdev->vq_index;
+
+    idx = hdev->vhost_ops->vhost_get_vq_index(hdev, idx);
+
+    vhost_virtqueue_unmap(hdev,
+                          vdev,
+                          hdev->vqs + idx,
+                          hdev->vq_index + idx);
+}
+
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                int vq_index)
+{
+    int idx = vq_index - hdev->vq_index;
+    int r = 0;
+
+    idx = hdev->vhost_ops->vhost_get_vq_index(hdev, idx);
+
+    r = vhost_virtqueue_start(hdev,
+                              vdev,
+                              hdev->vqs + idx,
+                              hdev->vq_index + idx);
+
+    return r;
+}
include/hw/virtio/vhost.h +6 -0
index a346f23d13..7df7dbe24d 100644
@@ -277,6 +277,12 @@ bool vhost_has_free_slot(void);
 int vhost_net_set_backend(struct vhost_dev *hdev,
                           struct vhost_vring_file *file);
 
+
+void vhost_dev_virtqueue_release(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                 int vq_index);
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                int vq_index);
+
 int vhost_device_iotlb_miss(struct vhost_dev *dev, uint64_t iova, int write);
 
 void vhost_dev_reset_inflight(struct vhost_inflight *inflight);
Signature · 1 line
2.32.0
#34 lore raw

Re: [PATCH 11/16] vhost: introduce restart and release for vhost_dev's vqs

Quoted 27 lines · Kangjie Xu

Introduce vhost_dev_virtqueue_restart(), which can restart the virtqueue when the vhost has already started running.

Meanwhile, vhost_dev_virtqueue_release(), which can ummap the vrings and the desc of a specific vq of a device.

Combining the two functions, we can reset a virtqueue with a started vhost.

---
  hw/virtio/vhost.c         | 29 +++++++++++++++++++++++++++++
  include/hw/virtio/vhost.h |  6 ++++++
  2 files changed, 35 insertions(+)
hw/virtio/vhost.c +3 -0
index e467dfc7bc..d158d71866 100644
@@ -1904,3 +1904,32 @@ int vhost_net_set_backend(struct vhost_dev *hdev,
 
      return -ENOSYS;
  }
+
+void vhost_dev_virtqueue_release(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                 int vq_index)

hw/virtio/vhost.c int vhost_net_set_backend(struct vhost_dev *hdev, · line 1919

Quoted 10 lines
hw/virtio/vhost.c +10 -0
+{
+    int idx = vq_index - hdev->vq_index;
+
+    idx = hdev->vhost_ops->vhost_get_vq_index(hdev, idx);
+
+    vhost_virtqueue_unmap(hdev,
+                          vdev,
+                          hdev->vqs + idx,
+                          hdev->vq_index + idx);
+}

Anything wrong that makes you can't use vhost_virtqueue_stop() here?

Thanks

Quoted 33 lines
hw/virtio/vhost.c +16 -0
+
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                int vq_index)
+{
+    int idx = vq_index - hdev->vq_index;
+    int r = 0;
+
+    idx = hdev->vhost_ops->vhost_get_vq_index(hdev, idx);
+
+    r = vhost_virtqueue_start(hdev,
+                              vdev,
+                              hdev->vqs + idx,
+                              hdev->vq_index + idx);
+
+    return r;
+}
include/hw/virtio/vhost.h +6 -0
index a346f23d13..7df7dbe24d 100644
@@ -277,6 +277,12 @@ bool vhost_has_free_slot(void);
  int vhost_net_set_backend(struct vhost_dev *hdev,
                            struct vhost_vring_file *file);
 
+
+void vhost_dev_virtqueue_release(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                 int vq_index);
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                int vq_index);
+
  int vhost_device_iotlb_miss(struct vhost_dev *dev, uint64_t iova, int write);
 
  void vhost_dev_reset_inflight(struct vhost_inflight *inflight);
author
#35 lore raw

Re: [PATCH 11/16] vhost: introduce restart and release for vhost_dev's vqs

Quoted 43 lines · Jason Wang
Quoted 27 lines · Kangjie Xu

Introduce vhost_dev_virtqueue_restart(), which can restart the virtqueue when the vhost has already started running.

Meanwhile, vhost_dev_virtqueue_release(), which can ummap the vrings and the desc of a specific vq of a device.

Combining the two functions, we can reset a virtqueue with a started vhost.

---
  hw/virtio/vhost.c         | 29 +++++++++++++++++++++++++++++ include/hw/virtio/vhost.h |  6 ++++++

  2 files changed, 35 insertions(+)
hw/virtio/vhost.c +3 -0
index e467dfc7bc..d158d71866 100644
@@ -1904,3 +1904,32 @@ int vhost_net_set_backend(struct vhost_dev *hdev,
         return -ENOSYS;
   }
+
+void vhost_dev_virtqueue_release(struct vhost_dev *hdev, 
 VirtIODevice *vdev,
+                                 int vq_index)

hw/virtio/vhost.c int vhost_net_set_backend(struct vhost_dev *hdev, · line 1919

Quoted 10 lines
hw/virtio/vhost.c +10 -0
+{
+    int idx = vq_index - hdev->vq_index;
+
+    idx = hdev->vhost_ops->vhost_get_vq_index(hdev, idx);
+
+    vhost_virtqueue_unmap(hdev,
+                          vdev,
+                          hdev->vqs + idx,
+                          hdev->vq_index + idx);
+}

Anything wrong that makes you can't use vhost_virtqueue_stop() here?

Quoted 3 lines

Thanks

Yeah, vhost_virtqueue_stop() will result in all the queues in the device being stopped.

In the scenario of DPDK-OVS, vhost_virtqueue_stop will send VHOST_USER_GET_VRING_BASE to DPDK. This message is meant to destroy the queue.

However, In DPDK, VHOST_USER_GET_VRING_BASE message will lead to vhost_destroy_device_notify(). This function will disable all the queues in the device.

Thus, when restarting the queue, other queues except the restarted one are still disabled.

On the other hand, I think it may be not necessary to destroy the queue. We can simply disable it. When restarting the queue, it will be intialized again.

I'm sorry I forgot to cc in the last email, so I resend this email.

Thanks

Quoted 34 lines
Quoted 34 lines
hw/virtio/vhost.c +2 -0
+
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice 

*vdev,
+                                int vq_index)

hw/virtio/vhost.c +13 -0
+{
+    int idx = vq_index - hdev->vq_index;
+    int r = 0;
+
+    idx = hdev->vhost_ops->vhost_get_vq_index(hdev, idx);
+
+    r = vhost_virtqueue_start(hdev,
+                              vdev,
+                              hdev->vqs + idx,
+                              hdev->vq_index + idx);
+
+    return r;
+}
include/hw/virtio/vhost.h +0 -0
index a346f23d13..7df7dbe24d 100644
@@ -277,6 +277,12 @@ bool vhost_has_free_slot(void);
   int vhost_net_set_backend(struct vhost_dev *hdev,
                             struct vhost_vring_file *file);

  +
+void vhost_dev_virtqueue_release(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                 int vq_index);
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice *vdev,
+                                int vq_index);
+
  int vhost_device_iotlb_miss(struct vhost_dev *dev, uint64_t iova, int write); void vhost_dev_reset_inflight(struct vhost_inflight *inflight);

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): CACGkMEtxXSm8Qc1LpKJJYm9cQ-F+eU5Lqecr62maRPxq1tM5rg@mail.gmail.com

author
#38 lore raw

Re: [PATCH 11/16] vhost: introduce restart and release for vhost_dev's vqs

Quoted 44 lines · Jason Wang
Quoted 43 lines · Kangjie Xu
Quoted 41 lines · Jason Wang
Quoted 27 lines · Kangjie Xu

Introduce vhost_dev_virtqueue_restart(), which can restart the virtqueue when the vhost has already started running.

Meanwhile, vhost_dev_virtqueue_release(), which can ummap the vrings and the desc of a specific vq of a device.

Combining the two functions, we can reset a virtqueue with a started vhost.

---
   hw/virtio/vhost.c         | 29 +++++++++++++++++++++++++++++
   include/hw/virtio/vhost.h |  6 ++++++
   2 files changed, 35 insertions(+)
hw/virtio/vhost.c +3 -0
index e467dfc7bc..d158d71866 100644
@@ -1904,3 +1904,32 @@ int vhost_net_set_backend(struct vhost_dev *hdev,
         return -ENOSYS;
   }
+
+void vhost_dev_virtqueue_release(struct vhost_dev *hdev,
 VirtIODevice *vdev,
+                                 int vq_index)

hw/virtio/vhost.c int vhost_net_set_backend(struct vhost_dev *hdev, · line 1919

Quoted 10 lines
hw/virtio/vhost.c +10 -0
+{
+    int idx = vq_index - hdev->vq_index;
+
+    idx = hdev->vhost_ops->vhost_get_vq_index(hdev, idx);
+
+    vhost_virtqueue_unmap(hdev,
+                          vdev,
+                          hdev->vqs + idx,
+                          hdev->vq_index + idx);
+}

Anything wrong that makes you can't use vhost_virtqueue_stop() here?

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Thanks

Yeah, vhost_virtqueue_stop() will result in all the queues in the device being stopped.

This sounds a defect of the backend. I think we'd better not workaround it in the qemu.

Yeah, I agree.

Quoted 11 lines
Quoted 3 lines

In the scenario of DPDK-OVS, vhost_virtqueue_stop will send VHOST_USER_GET_VRING_BASE to DPDK. This message is meant to destroy the queue.

That's tricky.

Quoted 3 lines

However, In DPDK, VHOST_USER_GET_VRING_BASE message will lead to vhost_destroy_device_notify(). This function will disable all the queues in the device.

Ok. I wonder how the reset is implemented in DPDK? It looks to me a new type of request is required.

Currently, DPDK does not support reset of queue level, but it supports device-level reset. VHOST_USER_RESET_DEVICE request exists in vhost protocol but it is not used in vhost-net module in QEMU.

It seems like there is no way for the driver to directly reset the device in vhost user net (If we use ethtool -r <eth_name>, we will get operation not supported.)

Actually we can plug the driver in and out to perform a device reset operation, but it will be applied to all the devices. I take it for example here.

If you plug out the driver(modprobe -r virtio-net), vhost_dev_stop() will be called in qemu. DPDK will receive VHOST_USER_GET_VRING_BASE messages for each virtqueue.

Then you can plug in the driver(modprobe virtio-net), vhost_dev_start() will be called in qemu. The process is same as when we boot the machine, DPDK will receive:

    VHOST_USER_SET_VRING_NUM, VHOST_USER_SET_VRING_BASE, VHOST_USER_SET_VRING_ADDR, VHOST_USER_SET_VRING_KICK, VHOST_USER_SET_VRING_CALL

in order to start a virtqueue.

Our vq reset implemenation does not need any change to the DPDK except adding the feature bit, since we reuse all the request types in vhost-user protocol.

I agree that we should add a new request in vhost-user protocol to support vq reset, we'll implement it in the next version.

Quoted 11 lines
Quoted 6 lines

Thus, when restarting the queue, other queues except the restarted one are still disabled.

On the other hand, I think it may be not necessary to destroy the queue. We can simply disable it. When restarting the queue, it will be intialized again.

So for vhost-net kernel, we can simply set NULL as the backend for a specific virtqueue. We just need to refactor the stop codes (which did it at vhost device level instead of queue level currently).

Thanks

Yes, I think so.

Thanks

Quoted 37 lines
Quoted 37 lines

Thanks

Quoted 35 lines
Quoted 35 lines
hw/virtio/vhost.c +2 -0
+
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice

*vdev,

hw/virtio/vhost.c +14 -0
+                                int vq_index)
+{
+    int idx = vq_index - hdev->vq_index;
+    int r = 0;
+
+    idx = hdev->vhost_ops->vhost_get_vq_index(hdev, idx);
+
+    r = vhost_virtqueue_start(hdev,
+                              vdev,
+                              hdev->vqs + idx,
+                              hdev->vq_index + idx);
+
+    return r;
+}
include/hw/virtio/vhost.h +5 -0
index a346f23d13..7df7dbe24d 100644
@@ -277,6 +277,12 @@ bool vhost_has_free_slot(void);
   int vhost_net_set_backend(struct vhost_dev *hdev,
                             struct vhost_vring_file *file);
   +
+void vhost_dev_virtqueue_release(struct vhost_dev *hdev,
 VirtIODevice *vdev,
+                                 int vq_index);
+int vhost_dev_virtqueue_restart(struct vhost_dev *hdev, VirtIODevice
 *vdev,
+                                int vq_index);
+
   int vhost_device_iotlb_miss(struct vhost_dev *dev, uint64_t iova,
int write);
     void vhost_dev_reset_inflight(struct vhost_inflight *inflight);
author
#39 lore raw

PATCH 12/16 vhost-net: introduce restart and stop for vhost_net's vqs

In reply to Kangjie Xu

Introduce vhost_virtqueue_restart(), which can restart the virtqueue when the vhost net started running before.

Introduce vhost_virtqueue_stop(), which can disable the vq and unmap vrings and the desc of the vq. When disabling the vq, the function is blocked and waits for a reply.

Combining the two functions, we can reset a virtqueue with a started vhost net.

---
 hw/net/vhost_net.c      | 55 +++++++++++++++++++++++++++++++++++++++++
 include/net/vhost_net.h |  5 ++++
 2 files changed, 60 insertions(+)
hw/net/vhost_net.c +55 -0
index ccac5b7a64..4f5f034c11 100644
@@ -514,3 +514,58 @@ int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu)
 
     return vhost_ops->vhost_net_set_mtu(&net->dev, mtu);
 }
+
+void vhost_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                          int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    int r;
+
+    assert(vhost_ops);
+
+    r = vhost_ops->vhost_set_single_vring_enable(&net->dev, vq_index, 0, true);
+    if (r < 0) {
+        goto err_queue_disable;
+    }
+
+    vhost_dev_virtqueue_release(&net->dev, vdev, vq_index);
+
+    return;
+
+err_queue_disable:
+    error_report("Error when releasing the qeuue.");
+}
+
+int vhost_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                            int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    int r;
+
+    if (!net->dev.started) {
+        return 0;
+    }
+
+    assert(vhost_ops);
+
+    r = vhost_dev_virtqueue_restart(&net->dev, vdev, vq_index);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    r = vhost_ops->vhost_set_single_vring_enable(&net->dev, vq_index, 1,
+                                                 false);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    return 0;
+
+err_start:
+    error_report("Error when restarting the queue.");
+    vhost_dev_stop(&net->dev, vdev);
+
+    return r;
+}
include/net/vhost_net.h +5 -0
index 387e913e4e..fcb09e36ef 100644
@@ -48,4 +48,9 @@ uint64_t vhost_net_get_acked_features(VHostNetState *net);
 
 int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
 
+void vhost_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                          int vq_index);
+int vhost_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                            int vq_index);
+
 #endif
Signature · 1 line
2.32.0
#40 lore raw

Re: [PATCH 12/16] vhost-net: introduce restart and stop for vhost_net's vqs

Quoted 22 lines · Kangjie Xu

Introduce vhost_virtqueue_restart(), which can restart the virtqueue when the vhost net started running before.

Introduce vhost_virtqueue_stop(), which can disable the vq and unmap vrings and the desc of the vq. When disabling the vq, the function is blocked and waits for a reply.

Combining the two functions, we can reset a virtqueue with a started vhost net.

---
  hw/net/vhost_net.c      | 55 +++++++++++++++++++++++++++++++++++++++++
  include/net/vhost_net.h |  5 ++++
  2 files changed, 60 insertions(+)
hw/net/vhost_net.c +0 -0
index ccac5b7a64..4f5f034c11 100644

hw/net/vhost_net.c int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu) · line 519

Quoted 6 lines
hw/net/vhost_net.c +3 -0
@@ -514,3 +514,58 @@ int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu)
 
      return vhost_ops->vhost_net_set_mtu(&net->dev, mtu);
  }
+
+void vhost_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                          int vq_index)

Let's rename this as vhost_net_virtqueue_stop()

Quoted 30 lines
hw/net/vhost_net.c +30 -0
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    int r;
+
+    assert(vhost_ops);
+
+    r = vhost_ops->vhost_set_single_vring_enable(&net->dev, vq_index, 0, true);
+    if (r < 0) {
+        goto err_queue_disable;
+    }
+
+    vhost_dev_virtqueue_release(&net->dev, vdev, vq_index);
+
+    return;
+
+err_queue_disable:
+    error_report("Error when releasing the qeuue.");
+}
+
+int vhost_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                            int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    int r;
+
+    if (!net->dev.started) {
+        return 0;
+    }

hw/net/vhost_net.c int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu) · line 559

Quoted 10 lines
hw/net/vhost_net.c +10 -0
+
+    assert(vhost_ops);
+
+    r = vhost_dev_virtqueue_restart(&net->dev, vdev, vq_index);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    r = vhost_ops->vhost_set_single_vring_enable(&net->dev, vq_index, 1,
+                                                 false);

So it looks nothing vhost_net specific but vhost. And why not do set_single_vring_enable in vhost_dev_virtqueue_restart() (btw, the name of this function looks confusing).

Thanks

Quoted 26 lines
hw/net/vhost_net.c +12 -0
+    if (r < 0) {
+        goto err_start;
+    }
+
+    return 0;
+
+err_start:
+    error_report("Error when restarting the queue.");
+    vhost_dev_stop(&net->dev, vdev);
+
+    return r;
+}
include/net/vhost_net.h +5 -0
index 387e913e4e..fcb09e36ef 100644
@@ -48,4 +48,9 @@ uint64_t vhost_net_get_acked_features(VHostNetState *net);
 
  int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
 
+void vhost_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                          int vq_index);
+int vhost_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                            int vq_index);
+
  #endif
author
#41 lore raw

Re: [PATCH 12/16] vhost-net: introduce restart and stop for vhost_net's vqs

Quoted 19 lines · Jason Wang
Quoted 17 lines · Kangjie Xu

Introduce vhost_virtqueue_restart(), which can restart the virtqueue when the vhost net started running before.

Introduce vhost_virtqueue_stop(), which can disable the vq and unmap vrings and the desc of the vq. When disabling the vq, the function is blocked and waits for a reply.

Combining the two functions, we can reset a virtqueue with a started vhost net.

---
  hw/net/vhost_net.c      | 55 +++++++++++++++++++++++++++++++++++++++++ include/net/vhost_net.h |  5 ++++

  2 files changed, 60 insertions(+)
Quoted 15 lines

hw/net/vhost_net.c int vhost_net_set_mtu(struct vhost_net *net, · line 519

Quoted 11 lines
hw/net/vhost_net.c +3 -0
index ccac5b7a64..4f5f034c11 100644
@@ -514,3 +514,58 @@ int vhost_net_set_mtu(struct vhost_net *net, 
 uint16_t mtu)
         return vhost_ops->vhost_net_set_mtu(&net->dev, mtu);
   }
+
+void vhost_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                          int vq_index)

Let's rename this as vhost_net_virtqueue_stop()

Yeah, I agree.

Thanks

Quoted 47 lines
Quoted 41 lines
hw/net/vhost_net.c +8 -0
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    int r;
+
+    assert(vhost_ops);
+
+    r = vhost_ops->vhost_set_single_vring_enable(&net->dev, 

vq_index, 0, true);
+    if (r < 0) {
+        goto err_queue_disable;
+    }

hw/net/vhost_net.c +28 -0
+
+    vhost_dev_virtqueue_release(&net->dev, vdev, vq_index);
+
+    return;
+
+err_queue_disable:
+    error_report("Error when releasing the qeuue.");
+}
+
+int vhost_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                            int vq_index)
+{
+    VHostNetState *net = get_vhost_net(nc->peer);
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    int r;
+
+    if (!net->dev.started) {
+        return 0;
+    }
+
+    assert(vhost_ops);
+
+    r = vhost_dev_virtqueue_restart(&net->dev, vdev, vq_index);
+    if (r < 0) {
+        goto err_start;
+    }
+
+    r = vhost_ops->vhost_set_single_vring_enable(&net->dev, 
Quoted 2 lines

vq_index, 1,
+                                                 false);

So it looks nothing vhost_net specific but vhost. And why not do set_single_vring_enable in vhost_dev_virtqueue_restart() (btw, the name of this function looks confusing).

Quoted 3 lines

Thanks

I thought it was a convention that virtio-net will call vhost_net first, and cannot call vhost directly. So I implement it in this way.

It's no probem to move it to vhost. I'll fix it.

Thanks

Quoted 25 lines
Quoted 25 lines

+    if (r < 0) {
+        goto err_start;
+    }

hw/net/vhost_net.c +9 -0
+
+    return 0;
+
+err_start:
+    error_report("Error when restarting the queue.");
+    vhost_dev_stop(&net->dev, vdev);
+
+    return r;
+}
include/net/vhost_net.h +4 -0
index 387e913e4e..fcb09e36ef 100644
@@ -48,4 +48,9 @@ uint64_t vhost_net_get_acked_features(VHostNetState 
 *net);
     int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu);
   +void vhost_virtqueue_stop(VirtIODevice *vdev, NetClientState *nc,
+                          int vq_index);
+int vhost_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
+                            int vq_index);
+
   #endif
author
#42 lore raw

PATCH 13/16 virtio: introduce queue_enable in virtio

In reply to Kangjie Xu

Introduce the interface queue_enable() in VirtioDeviceClass and the fucntion virtio_queue_enable() in virtio, it can be called when VIRTIO_PCI_COMMON_Q_ENABLE is written.

---
 hw/virtio/virtio-pci.c     | 1 +
 hw/virtio/virtio.c         | 9 +++++++++
 include/hw/virtio/virtio.h | 2 ++
 3 files changed, 12 insertions(+)
hw/virtio/virtio-pci.c +1 -0
index 35e8a5101a..85e1840479 100644
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
                        proxy->vqs[vdev->queue_sel].avail[0],
                        ((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 |
                        proxy->vqs[vdev->queue_sel].used[0]);
+            virtio_queue_enable(vdev, vdev->queue_sel);
             proxy->vqs[vdev->queue_sel].enabled = 1;
             proxy->vqs[vdev->queue_sel].reset = 0;
         } else {
hw/virtio/virtio.c +9 -0
index 0e9d41366f..82eb9dd4f2 100644
@@ -2050,6 +2050,15 @@ void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
     __virtio_queue_reset(vdev, queue_index);
 }
 
+void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+
+    if (k->queue_enable) {
+        k->queue_enable(vdev, queue_index);
+    }
+}
+
 void virtio_reset(void *opaque)
 {
     VirtIODevice *vdev = opaque;
include/hw/virtio/virtio.h +2 -0
index 879394299b..085997d8f3 100644
@@ -131,6 +131,7 @@ struct VirtioDeviceClass {
     void (*reset)(VirtIODevice *vdev);
     void (*set_status)(VirtIODevice *vdev, uint8_t val);
     void (*queue_reset)(VirtIODevice *vdev, uint32_t queue_index);
+    void (*queue_enable)(VirtIODevice *vdev, uint32_t queue_index);
     /* For transitional devices, this is a bitmap of features
      * that are only exposed on the legacy interface but not
      * the modern one.
@@ -270,6 +271,7 @@ int virtio_queue_set_host_notifier_mr(VirtIODevice *vdev, int n,
 int virtio_set_status(VirtIODevice *vdev, uint8_t val);
 void virtio_reset(void *opaque);
 void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
+void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index);
 void virtio_update_irq(VirtIODevice *vdev);
 int virtio_set_features(VirtIODevice *vdev, uint64_t val);
 
Signature · 1 line
2.32.0
#43 lore raw

Re: [PATCH 13/16] virtio: introduce queue_enable in virtio

Quoted 3 lines · Kangjie Xu

Introduce the interface queue_enable() in VirtioDeviceClass and the fucntion virtio_queue_enable() in virtio, it can be called when VIRTIO_PCI_COMMON_Q_ENABLE is written.

I'd suggest to split this series into two.

1) queue_enable

2) queue_reset

The patch looks good to me.

Thanks

Quoted 61 lines
---
  hw/virtio/virtio-pci.c     | 1 +
  hw/virtio/virtio.c         | 9 +++++++++
  include/hw/virtio/virtio.h | 2 ++
  3 files changed, 12 insertions(+)
hw/virtio/virtio-pci.c +1 -0
index 35e8a5101a..85e1840479 100644
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void *opaque, hwaddr addr,
                         proxy->vqs[vdev->queue_sel].avail[0],
                         ((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 |
                         proxy->vqs[vdev->queue_sel].used[0]);
+            virtio_queue_enable(vdev, vdev->queue_sel);
              proxy->vqs[vdev->queue_sel].enabled = 1;
              proxy->vqs[vdev->queue_sel].reset = 0;
          } else {
hw/virtio/virtio.c +9 -0
index 0e9d41366f..82eb9dd4f2 100644
@@ -2050,6 +2050,15 @@ void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
      __virtio_queue_reset(vdev, queue_index);
  }
 
+void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+
+    if (k->queue_enable) {
+        k->queue_enable(vdev, queue_index);
+    }
+}
+
  void virtio_reset(void *opaque)
  {
      VirtIODevice *vdev = opaque;
include/hw/virtio/virtio.h +2 -0
index 879394299b..085997d8f3 100644
@@ -131,6 +131,7 @@ struct VirtioDeviceClass {
      void (*reset)(VirtIODevice *vdev);
      void (*set_status)(VirtIODevice *vdev, uint8_t val);
      void (*queue_reset)(VirtIODevice *vdev, uint32_t queue_index);
+    void (*queue_enable)(VirtIODevice *vdev, uint32_t queue_index);
      /* For transitional devices, this is a bitmap of features
       * that are only exposed on the legacy interface but not
       * the modern one.
@@ -270,6 +271,7 @@ int virtio_queue_set_host_notifier_mr(VirtIODevice *vdev, int n,
  int virtio_set_status(VirtIODevice *vdev, uint8_t val);
  void virtio_reset(void *opaque);
  void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
+void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index);
  void virtio_update_irq(VirtIODevice *vdev);
  int virtio_set_features(VirtIODevice *vdev, uint64_t val);
 
author
#44 lore raw

Re: [PATCH 13/16] virtio: introduce queue_enable in virtio

Quoted 9 lines · Jason Wang
Quoted 3 lines · Kangjie Xu

Introduce the interface queue_enable() in VirtioDeviceClass and the fucntion virtio_queue_enable() in virtio, it can be called when VIRTIO_PCI_COMMON_Q_ENABLE is written.

I'd suggest to split this series into two.

Quoted 8 lines

1) queue_enable

2) queue_reset

The patch looks good to me.

Thanks

Okay, will fix.

Thanks

Quoted 63 lines
Quoted 63 lines

---
  hw/virtio/virtio-pci.c     | 1 +
  hw/virtio/virtio.c         | 9 +++++++++
  include/hw/virtio/virtio.h | 2 ++

  3 files changed, 12 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index 35e8a5101a..85e1840479 100644
@@ -1335,6 +1335,7 @@ static void virtio_pci_common_write(void 
 *opaque, hwaddr addr,
 proxy->vqs[vdev->queue_sel].avail[0],

((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << 32 | proxy->vqs[vdev->queue_sel].used[0]);
+            virtio_queue_enable(vdev, vdev->queue_sel); proxy->vqs[vdev->queue_sel].enabled = 1; proxy->vqs[vdev->queue_sel].reset = 0; } else {

hw/virtio/virtio.c +8 -0
index 0e9d41366f..82eb9dd4f2 100644
@@ -2050,6 +2050,15 @@ void virtio_queue_reset(VirtIODevice *vdev, 
 uint32_t queue_index)
       __virtio_queue_reset(vdev, queue_index);
   }
   +void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+
+    if (k->queue_enable) {
+        k->queue_enable(vdev, queue_index);
+    }
+}
+
   void virtio_reset(void *opaque)
   {

      VirtIODevice *vdev = opaque;

include/hw/virtio/virtio.h +2 -0
index 879394299b..085997d8f3 100644
@@ -131,6 +131,7 @@ struct VirtioDeviceClass {
       void (*reset)(VirtIODevice *vdev);
       void (*set_status)(VirtIODevice *vdev, uint8_t val);
       void (*queue_reset)(VirtIODevice *vdev, uint32_t queue_index);
+    void (*queue_enable)(VirtIODevice *vdev, uint32_t queue_index);
       /* For transitional devices, this is a bitmap of features
        * that are only exposed on the legacy interface but not
        * the modern one.
@@ -270,6 +271,7 @@ int 
 virtio_queue_set_host_notifier_mr(VirtIODevice *vdev, int n,
   int virtio_set_status(VirtIODevice *vdev, uint8_t val);
   void virtio_reset(void *opaque);
   void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
+void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index);
   void virtio_update_irq(VirtIODevice *vdev);
   int virtio_set_features(VirtIODevice *vdev, uint64_t val);
author
#45 lore raw

PATCH 14/16 virtio-net: support queue_enable for vhost-user

In reply to Kangjie Xu

Support queue enable in vhost-user scenario. It will be called when a vq reset operation is performed and the vq will be restared.

It should be noted that we can restart the vq when the vhost has already started. When launching a new vhost-user device, the vhost is not started and all vqs are not initalized until VIRTIO_PCI_COMMON_STATUS is written. Thus, we should use vhost_started to differentiate the two cases: vq reset and device start.

---
 hw/net/virtio-net.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
hw/net/virtio-net.c +20 -0
index 8396e21a67..2c26e2ef73 100644
@@ -544,6 +544,25 @@ static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
     assert(!virtio_net_get_subqueue(nc)->async_tx.elem);
 }
 
+static void virtio_net_queue_enable(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+    int r;
+
+    if (!nc->peer || !vdev->vhost_started) {
+        return;
+    }
+
+    if (nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        r = vhost_virtqueue_restart(vdev, nc, queue_index);
+        if (r < 0) {
+            error_report("unable to restart vhost net virtqueue: %d, "
+                            "when resetting the queue", queue_index);
+        }
+    }
+}
+
 static void virtio_net_reset(VirtIODevice *vdev)
 {
     VirtIONet *n = VIRTIO_NET(vdev);
@@ -3781,6 +3800,7 @@ static void virtio_net_class_init(ObjectClass *klass, void *data)
     vdc->bad_features = virtio_net_bad_features;
     vdc->reset = virtio_net_reset;
     vdc->queue_reset = virtio_net_queue_reset;
+    vdc->queue_enable = virtio_net_queue_enable;
     vdc->set_status = virtio_net_set_status;
     vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
     vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
Signature · 1 line
2.32.0
#46 lore raw

Re: [PATCH 14/16] virtio-net: support queue_enable for vhost-user

Quoted 31 lines · Kangjie Xu

Support queue enable in vhost-user scenario. It will be called when a vq reset operation is performed and the vq will be restared.

It should be noted that we can restart the vq when the vhost has already started. When launching a new vhost-user device, the vhost is not started and all vqs are not initalized until VIRTIO_PCI_COMMON_STATUS is written. Thus, we should use vhost_started to differentiate the two cases: vq reset and device start.

---
  hw/net/virtio-net.c | 20 ++++++++++++++++++++
  1 file changed, 20 insertions(+)
hw/net/virtio-net.c +8 -0
index 8396e21a67..2c26e2ef73 100644
@@ -544,6 +544,25 @@ static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
      assert(!virtio_net_get_subqueue(nc)->async_tx.elem);
  }
 
+static void virtio_net_queue_enable(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+    int r;
+
+    if (!nc->peer || !vdev->vhost_started) {
+        return;

hw/net/virtio-net.c static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t queue_index) · line 564

Quoted 10 lines
hw/net/virtio-net.c +10 -0
+    }
+
+    if (nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        r = vhost_virtqueue_restart(vdev, nc, queue_index);
+        if (r < 0) {
+            error_report("unable to restart vhost net virtqueue: %d, "
+                            "when resetting the queue", queue_index);
+        }
+    }
+}

So we don't check queue_enable in vhost_dev_start(), does this mean the queue_enable is actually meaningless (since the virtqueue is already started there)?

And another issue is

peet_attach/peer_detach() "abuse" vhost_set_vring_enable(). This probably means we need to invent new type of request instead of re-using VHOST_USER_SET_VRING_ENABLE.

Thanks

Quoted 12 lines
hw/net/virtio-net.c +2 -0
+
  static void virtio_net_reset(VirtIODevice *vdev)
  {
      VirtIONet *n = VIRTIO_NET(vdev);
@@ -3781,6 +3800,7 @@ static void virtio_net_class_init(ObjectClass *klass, void *data)
      vdc->bad_features = virtio_net_bad_features;
      vdc->reset = virtio_net_reset;
      vdc->queue_reset = virtio_net_queue_reset;
+    vdc->queue_enable = virtio_net_queue_enable;
      vdc->set_status = virtio_net_set_status;
      vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
      vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
author
#47 lore raw

Re: [PATCH 14/16] virtio-net: support queue_enable for vhost-user

Quoted 47 lines · Jason Wang
Quoted 31 lines · Kangjie Xu

Support queue enable in vhost-user scenario. It will be called when a vq reset operation is performed and the vq will be restared.

It should be noted that we can restart the vq when the vhost has already started. When launching a new vhost-user device, the vhost is not started and all vqs are not initalized until VIRTIO_PCI_COMMON_STATUS is written. Thus, we should use vhost_started to differentiate the two cases: vq reset and device start.

---
  hw/net/virtio-net.c | 20 ++++++++++++++++++++

  1 file changed, 20 insertions(+)
hw/net/virtio-net.c +7 -0
index 8396e21a67..2c26e2ef73 100644
@@ -544,6 +544,25 @@ static void virtio_net_queue_reset(VirtIODevice 
 *vdev, uint32_t queue_index)
       assert(!virtio_net_get_subqueue(nc)->async_tx.elem);
   }
   +static void virtio_net_queue_enable(VirtIODevice *vdev, uint32_t queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+    int r;
+
+    if (!nc->peer || !vdev->vhost_started) {
+        return;

hw/net/virtio-net.c static void virtio_net_queue_reset(VirtIODevice · line 564

Quoted 10 lines
hw/net/virtio-net.c +10 -0
+    }
+
+    if (nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        r = vhost_virtqueue_restart(vdev, nc, queue_index);
+        if (r < 0) {
+            error_report("unable to restart vhost net virtqueue: %d, "
+                            "when resetting the queue", queue_index);
+        }
+    }
+}

So we don't check queue_enable in vhost_dev_start(), does this mean the queue_enable is actually meaningless (since the virtqueue is already started there)?

Quoted 7 lines

And another issue is

peet_attach/peer_detach() "abuse" vhost_set_vring_enable(). This probably means we need to invent new type of request instead of re-using VHOST_USER_SET_VRING_ENABLE.

Thanks

1. Yes, we don't need queue_enable in vhost_dev_start(), queue_enable is only useful when restarting the queue. I name it as queue_enable() simply because it is called when VIRTIO_PCI_COMMON_Q_ENABLE is written. Would it look better if we rename it as "queue_reenable"?

2. I think inventing a new type of vhost-protocol message can be a good choice. However, I don't know much about the vhost protocol. If we want to add a new message in vhost protocol, except the documentation and the code in qemu, Do we need to submit patches to other projects, e.g. some projects like virtio-spec?

Thanks

Quoted 13 lines
Quoted 13 lines
hw/net/virtio-net.c +1 -0
+
   static void virtio_net_reset(VirtIODevice *vdev)
   {

      VirtIONet *n = VIRTIO_NET(vdev);

hw/net/virtio-net.c +1 -0
@@ -3781,6 +3800,7 @@ static void virtio_net_class_init(ObjectClass 
 *klass, void *data)
       vdc->bad_features = virtio_net_bad_features;
       vdc->reset = virtio_net_reset;
       vdc->queue_reset = virtio_net_queue_reset;
+    vdc->queue_enable = virtio_net_queue_enable;
       vdc->set_status = virtio_net_set_status;
       vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;

      vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;

#48 lore raw

Re: [PATCH 14/16] virtio-net: support queue_enable for vhost-user

Quoted 59 lines · Kangjie Xu
Quoted 56 lines · Jason Wang
Quoted 32 lines · Kangjie Xu

Support queue enable in vhost-user scenario. It will be called when a vq reset operation is performed and the vq will be restared.

It should be noted that we can restart the vq when the vhost has already started. When launching a new vhost-user device, the vhost is not started and all vqs are not initalized until VIRTIO_PCI_COMMON_STATUS is written. Thus, we should use vhost_started to differentiate the two cases: vq reset and device start.

---
  hw/net/virtio-net.c | 20 ++++++++++++++++++++
  1 file changed, 20 insertions(+)
hw/net/virtio-net.c +7 -0
index 8396e21a67..2c26e2ef73 100644
@@ -544,6 +544,25 @@ static void virtio_net_queue_reset(VirtIODevice
 *vdev, uint32_t queue_index)
      assert(!virtio_net_get_subqueue(nc)->async_tx.elem);
  }
  +static void virtio_net_queue_enable(VirtIODevice *vdev, uint32_t
 queue_index)
+{
+    VirtIONet *n = VIRTIO_NET(vdev);
+    NetClientState *nc = qemu_get_subqueue(n->nic, vq2q(queue_index));
+    int r;
+
+    if (!nc->peer || !vdev->vhost_started) {
+        return;

hw/net/virtio-net.c static void virtio_net_queue_reset(VirtIODevice · line 565

Quoted 10 lines
hw/net/virtio-net.c +10 -0
+    }
+
+    if (nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        r = vhost_virtqueue_restart(vdev, nc, queue_index);
+        if (r < 0) {
+            error_report("unable to restart vhost net virtqueue: %d, "
+                            "when resetting the queue", queue_index);
+        }
+    }
+}

So we don't check queue_enable in vhost_dev_start(), does this mean the queue_enable is actually meaningless (since the virtqueue is already started there)?

And another issue is

peet_attach/peer_detach() "abuse" vhost_set_vring_enable(). This probably means we need to invent new type of request instead of re-using VHOST_USER_SET_VRING_ENABLE.

Quoted 7 lines
Quoted 3 lines

Thanks

1. Yes, we don't need queue_enable in vhost_dev_start(), queue_enable is only useful when restarting the queue. I name it as queue_enable() simply because it is called when VIRTIO_PCI_COMMON_Q_ENABLE is written. Would it look better if we rename it as "queue_reenable"?

I think the right approach is probably:

1) when VERSION_1 is negotiated, only start the virtqueue when queue_enable is 1
2) when VERSION_1 is not negotiated, start virtqueue when DRIVER_OK (vhost_dev_start())

?

Quoted 6 lines

2. I think inventing a new type of vhost-protocol message can be a good choice. However, I don't know much about the vhost protocol. If we want to add a new message in vhost protocol, except the documentation and the code in qemu, Do we need to submit patches to other projects, e.g. some projects like virtio-spec?

Probably not since vhost-user doesn't belong to the spec currently. The doc in qemu should be sufficient.

Thanks

Quoted 17 lines

Thanks

Quoted 13 lines
Quoted 13 lines
hw/net/virtio-net.c +2 -0
+
  static void virtio_net_reset(VirtIODevice *vdev)
  {
      VirtIONet *n = VIRTIO_NET(vdev);
@@ -3781,6 +3800,7 @@ static void virtio_net_class_init(ObjectClass
 *klass, void *data)
      vdc->bad_features = virtio_net_bad_features;
      vdc->reset = virtio_net_reset;
      vdc->queue_reset = virtio_net_queue_reset;
+    vdc->queue_enable = virtio_net_queue_enable;
      vdc->set_status = virtio_net_set_status;
      vdc->guest_notifier_mask = virtio_net_guest_notifier_mask;
      vdc->guest_notifier_pending = virtio_net_guest_notifier_pending;
author
#49 lore raw

PATCH 15/16 virtio-net: support queue_reset for vhost-user

In reply to Kangjie Xu

Support queue reset in vhost-user scenario.

---
 hw/net/virtio-net.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/net/virtio-net.c +4 -0
index 2c26e2ef73..0747ffe71c 100644
@@ -540,6 +540,10 @@ static void virtio_net_queue_reset(VirtIODevice *vdev, uint32_t queue_index)
         return;
     }
 
+    if (nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+        vhost_virtqueue_stop(vdev, nc, queue_index);
+    }
+
     qemu_flush_or_purge_queued_packets(nc->peer, true);
     assert(!virtio_net_get_subqueue(nc)->async_tx.elem);
 }
Signature · 1 line
2.32.0
author
#50 lore raw

PATCH 16/16 vhost-net: vq reset feature bit support

In reply to Kangjie Xu

Add support for negotation of vq reset feature bit.

---
 hw/net/vhost_net.c  | 1 +
 hw/net/virtio-net.c | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)
hw/net/vhost_net.c +1 -0
index 4f5f034c11..de910f6466 100644
@@ -73,6 +73,7 @@ static const int user_feature_bits[] = {
     VIRTIO_NET_F_MTU,
     VIRTIO_F_IOMMU_PLATFORM,
     VIRTIO_F_RING_PACKED,
+    VIRTIO_F_RING_RESET,
     VIRTIO_NET_F_RSS,
     VIRTIO_NET_F_HASH_REPORT,
 
hw/net/virtio-net.c +2 -1
index 0747ffe71c..a8b299067a 100644
@@ -757,6 +757,8 @@ static uint64_t virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
 
     virtio_add_feature(&features, VIRTIO_NET_F_MAC);
 
+    virtio_add_feature(&features, VIRTIO_F_RING_RESET);
+
     if (!peer_has_vnet_hdr(n)) {
         virtio_clear_feature(&features, VIRTIO_NET_F_CSUM);
         virtio_clear_feature(&features, VIRTIO_NET_F_HOST_TSO4);
@@ -777,7 +779,6 @@ static uint64_t virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
     }
 
     if (!get_vhost_net(nc->peer)) {
-        virtio_add_feature(&features, VIRTIO_F_RING_RESET);
         return features;
     }
 
Signature · 1 line
2.32.0
#51 lore raw

Re: [PATCH 16/16] vhost-net: vq reset feature bit support

Quoted 4 lines · Kangjie Xu

Add support for negotation of vq reset feature bit.

I'd suggest to add support for vhost-net kernel as well. It looks much more easier than vhost-user (I guess a stop/start would do the trick).

Thanks

Quoted 38 lines
---
  hw/net/vhost_net.c  | 1 +
  hw/net/virtio-net.c | 3 ++-
  2 files changed, 3 insertions(+), 1 deletion(-)
hw/net/vhost_net.c +1 -0
index 4f5f034c11..de910f6466 100644
@@ -73,6 +73,7 @@ static const int user_feature_bits[] = {
      VIRTIO_NET_F_MTU,
      VIRTIO_F_IOMMU_PLATFORM,
      VIRTIO_F_RING_PACKED,
+    VIRTIO_F_RING_RESET,
      VIRTIO_NET_F_RSS,
      VIRTIO_NET_F_HASH_REPORT,
 
hw/net/virtio-net.c +2 -1
index 0747ffe71c..a8b299067a 100644
@@ -757,6 +757,8 @@ static uint64_t virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
 
      virtio_add_feature(&features, VIRTIO_NET_F_MAC);
 
+    virtio_add_feature(&features, VIRTIO_F_RING_RESET);
+
      if (!peer_has_vnet_hdr(n)) {
          virtio_clear_feature(&features, VIRTIO_NET_F_CSUM);
          virtio_clear_feature(&features, VIRTIO_NET_F_HOST_TSO4);
@@ -777,7 +779,6 @@ static uint64_t virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
      }
 
      if (!get_vhost_net(nc->peer)) {
-        virtio_add_feature(&features, VIRTIO_F_RING_RESET);
          return features;
      }
 
author
#52 lore raw

Re: [PATCH 16/16] vhost-net: vq reset feature bit support

Quoted 4 lines · Jason Wang
Quoted 2 lines · Kangjie Xu

Add support for negotation of vq reset feature bit.

Quoted 9 lines
Quoted 2 lines

I'd suggest to add support for vhost-net kernel as well. It looks much more easier than vhost-user (I guess a stop/start would do the trick).

Thanks

Yeah, we've planned to support it in the future.

Thanks

Quoted 35 lines
Quoted 35 lines

---
  hw/net/vhost_net.c  | 1 +
  hw/net/virtio-net.c | 3 ++-

  2 files changed, 3 insertions(+), 1 deletion(-)
hw/net/vhost_net.c +1 -0
index 4f5f034c11..de910f6466 100644
@@ -73,6 +73,7 @@ static const int user_feature_bits[] = {
       VIRTIO_NET_F_MTU,
       VIRTIO_F_IOMMU_PLATFORM,
       VIRTIO_F_RING_PACKED,
+    VIRTIO_F_RING_RESET,
       VIRTIO_NET_F_RSS,
       VIRTIO_NET_F_HASH_REPORT,

  diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index 0747ffe71c..a8b299067a 100644

hw/net/virtio-net.c +1 -1
@@ -757,6 +757,8 @@ static uint64_t 
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
         virtio_add_feature(&features, VIRTIO_NET_F_MAC);
   +    virtio_add_feature(&features, VIRTIO_F_RING_RESET);
+
       if (!peer_has_vnet_hdr(n)) {
           virtio_clear_feature(&features, VIRTIO_NET_F_CSUM);
           virtio_clear_feature(&features, VIRTIO_NET_F_HOST_TSO4);
@@ -777,7 +779,6 @@ static uint64_t 
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
       }
         if (!get_vhost_net(nc->peer)) {
-        virtio_add_feature(&features, VIRTIO_F_RING_RESET);
           return features;
       }
#53 lore raw

Re: [PATCH 16/16] vhost-net: vq reset feature bit support

Quoted 11 lines · Kangjie Xu
Quoted 8 lines · Jason Wang
Quoted 4 lines · Kangjie Xu

Add support for negotation of vq reset feature bit.

Quoted 7 lines
Quoted 6 lines

I'd suggest to add support for vhost-net kernel as well. It looks much more easier than vhost-user (I guess a stop/start would do the trick).

Thanks

Yeah, we've planned to support it in the future.

If it's possible, I suggest to implement in this series. It would be easier since it current kernel support it already (via SET_BACKEND).

Thanks

Quoted 39 lines

Thanks

Quoted 35 lines
Quoted 35 lines
---
  hw/net/vhost_net.c  | 1 +
  hw/net/virtio-net.c | 3 ++-
  2 files changed, 3 insertions(+), 1 deletion(-)
hw/net/vhost_net.c +1 -0
index 4f5f034c11..de910f6466 100644
@@ -73,6 +73,7 @@ static const int user_feature_bits[] = {
      VIRTIO_NET_F_MTU,
      VIRTIO_F_IOMMU_PLATFORM,
      VIRTIO_F_RING_PACKED,
+    VIRTIO_F_RING_RESET,
      VIRTIO_NET_F_RSS,
      VIRTIO_NET_F_HASH_REPORT,
  diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c

index 0747ffe71c..a8b299067a 100644

hw/net/virtio-net.c +1 -1
@@ -757,6 +757,8 @@ static uint64_t
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
        virtio_add_feature(&features, VIRTIO_NET_F_MAC);
  +    virtio_add_feature(&features, VIRTIO_F_RING_RESET);
+
      if (!peer_has_vnet_hdr(n)) {
          virtio_clear_feature(&features, VIRTIO_NET_F_CSUM);
          virtio_clear_feature(&features, VIRTIO_NET_F_HOST_TSO4);
@@ -777,7 +779,6 @@ static uint64_t
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
      }
        if (!get_vhost_net(nc->peer)) {
-        virtio_add_feature(&features, VIRTIO_F_RING_RESET);
          return features;
      }
author
#54 lore raw

Re: [PATCH 16/16] vhost-net: vq reset feature bit support

Quoted 12 lines · Jason Wang
Quoted 11 lines · Kangjie Xu
Quoted 9 lines · Jason Wang
Quoted 4 lines · Kangjie Xu

Add support for negotation of vq reset feature bit.

I'd suggest to add support for vhost-net kernel as well. It looks much more easier than vhost-user (I guess a stop/start would do the trick).

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Thanks

Yeah, we've planned to support it in the future.

If it's possible, I suggest to implement in this series. It would be easier since it current kernel support it already (via SET_BACKEND).

Thanks

Okay, in the next version, we will add the implementation for vhost-kernel vq reset.

Thanks

Quoted 37 lines
Quoted 37 lines

Thanks

Quoted 35 lines
Quoted 35 lines
---
   hw/net/vhost_net.c  | 1 +
   hw/net/virtio-net.c | 3 ++-
   2 files changed, 3 insertions(+), 1 deletion(-)
hw/net/vhost_net.c +1 -0
index 4f5f034c11..de910f6466 100644
@@ -73,6 +73,7 @@ static const int user_feature_bits[] = {
       VIRTIO_NET_F_MTU,
       VIRTIO_F_IOMMU_PLATFORM,
       VIRTIO_F_RING_PACKED,
+    VIRTIO_F_RING_RESET,
       VIRTIO_NET_F_RSS,
       VIRTIO_NET_F_HASH_REPORT,
   diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c

index 0747ffe71c..a8b299067a 100644

hw/net/virtio-net.c +1 -1
@@ -757,6 +757,8 @@ static uint64_t
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
         virtio_add_feature(&features, VIRTIO_NET_F_MAC);
   +    virtio_add_feature(&features, VIRTIO_F_RING_RESET);
+
       if (!peer_has_vnet_hdr(n)) {
           virtio_clear_feature(&features, VIRTIO_NET_F_CSUM);
           virtio_clear_feature(&features, VIRTIO_NET_F_HOST_TSO4);
@@ -777,7 +779,6 @@ static uint64_t
 virtio_net_get_features(VirtIODevice *vdev, uint64_t features,
       }
         if (!get_vhost_net(nc->peer)) {
-        virtio_add_feature(&features, VIRTIO_F_RING_RESET);
           return features;
       }
author
#55 lore raw
In reply to Kangjie Xu

Do you have any comments or suggestions about improvements to this patch set?

Regards,

Kangjie

Quoted 71 lines · Kangjie Xu

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

     https://github.com/oasis-tcs/virtio-spec/issues/124
     https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of two parts: virtio-net and vhost-user. The patches 1-7 are the implementation for virtio-net and the patches 8-16 are for vhost-user.

The process of virt queue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue is restarted after the information of vrings is passed to QEMU and VIRTIO_PCI_COMMON_Q_ENABLE is written.

Test environment:
Host: 5.4.189
Qemu: QEMU emulator version 7.0.50
Guest: 5.19.0-rc3 (With vq reset support)
DPDK: 22.07-rc1 (With vq reset support)
Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

The frontend can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

Guest Kernel Patch:

     https://lore.kernel.org/bpf/[email protected]/

DPDK Code:

     https://github.com/middaywords/dpdk/commit/098c8e1dfae10b747da8dd8950a82890efca7bee

Please review. Thanks.

Kangjie Xu (9):
vhost: add op to enable or disable a single vring
vhost-user: enable/disable a single vring
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce restart and release for vhost_dev's vqs
vhost-net: introduce restart and stop for vhost_net's vqs
virtio: introduce queue_enable in virtio
virtio-net: support queue_enable for vhost-user
virtio-net: support queue_reset for vhost-user
vhost-net: vq reset feature bit support

Xuan Zhuo (7):
virtio-pci: virtio_pci_common_cfg add queue_notify_data
virtio: add VIRTIO_F_RING_RESET
virtio: pci: virtio_pci_common_cfg add queue_reset
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

  hw/net/vhost_net.c                            | 56 ++++++++++++++++++
  hw/net/virtio-net.c                           | 40 +++++++++++++
  hw/virtio/vhost-user.c                        | 55 +++++++++++++++---
  hw/virtio/vhost.c                             | 49 ++++++++++++++--
  hw/virtio/virtio-pci.c                        | 17 ++++++
  hw/virtio/virtio.c                            | 57 +++++++++++++------
  include/hw/virtio/vhost-backend.h             |  4 ++
  include/hw/virtio/vhost.h                     |  6 ++
  include/hw/virtio/virtio-pci.h                |  1 +
  include/hw/virtio/virtio.h                    |  4 ++
  include/net/vhost_net.h                       |  5 ++
  .../standard-headers/linux/virtio_config.h    |  5 ++
  include/standard-headers/linux/virtio_pci.h   |  4 ++
  13 files changed, 274 insertions(+), 29 deletions(-)
#56 lore raw
Quoted 3 lines · Kangjie Xu

Do you have any comments or suggestions about improvements to this patch set?

Will have a look and give feedback no later than the end of this week.

Thanks

Quoted 79 lines

Regards,

Kangjie

Quoted 71 lines · Kangjie Xu

The virtio queue reset function has already been defined in the virtio spec 1.2. The relevant virtio spec information is here:

     https://github.com/oasis-tcs/virtio-spec/issues/124
     https://github.com/oasis-tcs/virtio-spec/issues/139

This patch set is to support this function in QEMU. It consists of two parts: virtio-net and vhost-user. The patches 1-7 are the implementation for virtio-net and the patches 8-16 are for vhost-user.

The process of virt queue reset can be concluded as:
1. The virtqueue is disabled when VIRTIO_PCI_COMMON_Q_RESET is written.
2. Then the virtqueue is restarted after the information of vrings is passed to QEMU and VIRTIO_PCI_COMMON_Q_ENABLE is written.

Test environment:
Host: 5.4.189
Qemu: QEMU emulator version 7.0.50
Guest: 5.19.0-rc3 (With vq reset support)
DPDK: 22.07-rc1 (With vq reset support)
Test Cmd: ethtool -g eth1; ethtool -G eth1 rx $1 tx $2; ethtool -g eth1;

The frontend can resize the virtio queue, then virtio queue reset function should be triggered.

The default is split mode, modify Qemu virtio-net to add PACKED feature to test packed mode.

Guest Kernel Patch:

     https://lore.kernel.org/bpf/[email protected]/

DPDK Code:

     https://github.com/middaywords/dpdk/commit/098c8e1dfae10b747da8dd8950a82890efca7bee

Please review. Thanks.

Kangjie Xu (9):
vhost: add op to enable or disable a single vring
vhost-user: enable/disable a single vring
vhost: extract the logic of unmapping the vrings and desc
vhost: introduce restart and release for vhost_dev's vqs
vhost-net: introduce restart and stop for vhost_net's vqs
virtio: introduce queue_enable in virtio
virtio-net: support queue_enable for vhost-user
virtio-net: support queue_reset for vhost-user
vhost-net: vq reset feature bit support

Xuan Zhuo (7):
virtio-pci: virtio_pci_common_cfg add queue_notify_data
virtio: add VIRTIO_F_RING_RESET
virtio: pci: virtio_pci_common_cfg add queue_reset
virtio: introduce __virtio_queue_reset()
virtio: introduce virtio_queue_reset()
virtio-pci: support queue reset
virtio-net: support queue reset

  hw/net/vhost_net.c                            | 56 ++++++++++++++++++
  hw/net/virtio-net.c                           | 40 +++++++++++++
  hw/virtio/vhost-user.c                        | 55 +++++++++++++++---
  hw/virtio/vhost.c                             | 49 ++++++++++++++--
  hw/virtio/virtio-pci.c                        | 17 ++++++
  hw/virtio/virtio.c                            | 57 +++++++++++++------
  include/hw/virtio/vhost-backend.h             |  4 ++
  include/hw/virtio/vhost.h                     |  6 ++
  include/hw/virtio/virtio-pci.h                |  1 +
  include/hw/virtio/virtio.h                    |  4 ++
  include/net/vhost_net.h                       |  5 ++
  .../standard-headers/linux/virtio_config.h    |  5 ++
  include/standard-headers/linux/virtio_pci.h   |  4 ++
  13 files changed, 274 insertions(+), 29 deletions(-)