Skip to contentSkip to conversation

Guest announce feature emulation using Shadow VirtQueue

PATCH In review Eugenio Pérez 2 participants 9 messages Oct 20, 2022 – Oct 26, 2022 Networking

lore
Share & tools
lore
#0 lore raw

cover Guest announce feature emulation using Shadow VirtQueue

A gratuitous ARP is recommended after a live migration to reduce the amount of time needed by the network links to be aware of the new location. A hypervisor may not have the knowledge of the guest network configuration, and this is especially true on passthrough devices, so its simpler to ask the guest to do it.

However, the device control part of this feature can be totally emulated by qemu and shadow virtqueue, not needing any special feature from the actual vdpa device.

VIRTIO_NET_F_STATUS is also needed for the guest to access the status of virtio net config where announcement status bit is set. Emulating it as always active in case backend does not support it.

This series need two fixes applied:
* https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03242.html
* https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03241.html

v1:
* Move code from vhost_net_get_config to virtio_net_get_config.

RFC v2:
* Add VIRTIO_NET_F_STATUS emulation.

Eugenio Pérez (7):
vdpa: Save emulated features list in vhost_vdpa
virtio_net: Modify virtio_net_get_config to early return
virtio_net: Handle _F_STATUS emulation in virtio_net_get_config
vdpa: Expose VIRTIO_NET_F_STATUS unconditionally
vdpa: handle VIRTIO_NET_CTRL_ANNOUNCE in
vhost_vdpa_net_handle_ctrl_avail
virtio_net: copy VIRTIO_NET_S_ANNOUNCE if device model has it
vdpa: Offer VIRTIO_NET_F_GUEST_ANNOUNCE feature if SVQ is enabled

 include/hw/virtio/vhost-vdpa.h |  2 ++
 hw/net/virtio-net.c            | 40 +++++++++++++++++++++-------------
 hw/virtio/vhost-vdpa.c         |  8 +++----
 net/vhost-vdpa.c               | 25 ++++++++++++++++++---
 4 files changed, 53 insertions(+), 22 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH 1/7 vdpa: Save emulated features list in vhost_vdpa

At this moment only _F_LOG is added there.

However future patches add features that depend on the kind of device. In particular, only net devices can add VIRTIO_F_GUEST_ANNOUNCE. So let's allow vhost_vdpa creator to set custom emulated device features.

---
 include/hw/virtio/vhost-vdpa.h | 2 ++
 hw/virtio/vhost-vdpa.c         | 8 ++++----
 net/vhost-vdpa.c               | 4 ++++
 3 files changed, 10 insertions(+), 4 deletions(-)
include/hw/virtio/vhost-vdpa.h +2 -0
index 1111d85643..50083e1e3b 100644
@@ -31,6 +31,8 @@ typedef struct vhost_vdpa {
     bool iotlb_batch_begin_sent;
     MemoryListener listener;
     struct vhost_vdpa_iova_range iova_range;
+    /* VirtIO device features that can be emulated by qemu */
+    uint64_t added_features;
     uint64_t acked_features;
     bool shadow_vqs_enabled;
     /* IOVA mapping used by the Shadow Virtqueue */
hw/virtio/vhost-vdpa.c +4 -4
index 7468e44b87..ddb5e29288 100644
@@ -660,8 +660,8 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,
 
         v->acked_features = features;
 
-        /* We must not ack _F_LOG if SVQ is enabled */
-        features &= ~BIT_ULL(VHOST_F_LOG_ALL);
+        /* Do not ack features emulated by qemu */
+        features &= ~v->added_features;
     }
 
     trace_vhost_vdpa_set_features(dev, features);
@@ -1244,8 +1244,8 @@ static int vhost_vdpa_get_features(struct vhost_dev *dev,
     int ret = vhost_vdpa_get_dev_features(dev, features);
 
     if (ret == 0 && v->shadow_vqs_enabled) {
-        /* Add SVQ logging capabilities */
-        *features |= BIT_ULL(VHOST_F_LOG_ALL);
+        /* Add emulated capabilities */
+        *features |= v->added_features;
     }
 
     return ret;
net/vhost-vdpa.c +4 -0
index 6d64000202..e7db40b7cd 100644
@@ -551,6 +551,10 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
     s->vhost_vdpa.index = queue_pair_index;
     s->vhost_vdpa.shadow_vqs_enabled = svq;
     s->vhost_vdpa.iova_tree = iova_tree;
+    if (svq) {
+        /* Add SVQ logging capabilities */
+        s->vhost_vdpa.added_features |= BIT_ULL(VHOST_F_LOG_ALL);
+    }
     if (!is_datapath) {
         s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
                                             vhost_vdpa_net_cvq_cmd_page_len());
Signature · 1 line
2.31.1
#2 lore raw

PATCH 2/7 virtio_net: Modify virtio_net_get_config to early return

In reply to Eugenio Pérez

Next patches introduce more code on vhost-vdpa branch, with already have too much indentation.

---
 hw/net/virtio-net.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)
hw/net/virtio-net.c +15 -13
index e9f696b4cf..56ff219196 100644
@@ -158,20 +158,22 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
     if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
         ret = vhost_net_get_config(get_vhost_net(nc->peer), (uint8_t *)&netcfg,
                                    n->config_size);
-        if (ret != -1) {
-            /*
-             * Some NIC/kernel combinations present 0 as the mac address.  As
-             * that is not a legal address, try to proceed with the
-             * address from the QEMU command line in the hope that the
-             * address has been configured correctly elsewhere - just not
-             * reported by the device.
-             */
-            if (memcmp(&netcfg.mac, &zero, sizeof(zero)) == 0) {
-                info_report("Zero hardware mac address detected. Ignoring.");
-                memcpy(netcfg.mac, n->mac, ETH_ALEN);
-            }
-            memcpy(config, &netcfg, n->config_size);
+        if (ret == -1) {
+            return;
         }
+
+        /*
+         * Some NIC/kernel combinations present 0 as the mac address.  As that
+         * is not a legal address, try to proceed with the address from the
+         * QEMU command line in the hope that the address has been configured
+         * correctly elsewhere - just not reported by the device.
+         */
+        if (memcmp(&netcfg.mac, &zero, sizeof(zero)) == 0) {
+            info_report("Zero hardware mac address detected. Ignoring.");
+            memcpy(netcfg.mac, n->mac, ETH_ALEN);
+        }
+
+        memcpy(config, &netcfg, n->config_size);
     }
 }
 
Signature · 1 line
2.31.1
#3 lore raw

PATCH 3/7 virtio_net: Handle _F_STATUS emulation in virtio_net_get_config

In reply to Eugenio Pérez

At this moment this code path is not reached, but vdpa devices can offer VIRTIO_NET_F_STATUS unconditionally. While the guest must assume that link is always up by the standard, qemu will set the status bit to 1 always in this case.

This makes little use by itself, but VIRTIO_NET_F_STATUS is needed for the guest to read status bit VIRTIO_NET_F_GUEST_ANNOUNCE, used by feature VIRTIO_NET_F_GUEST_ANNOUNCE. So qemu must emulate status feature in case it needs to emulate the guest announce feature.

---
 hw/net/virtio-net.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
hw/net/virtio-net.c +9 -2
index 56ff219196..51fe6d5c1a 100644
@@ -156,8 +156,9 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
      * disconnect/reconnect a VDPA peer.
      */
     if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
-        ret = vhost_net_get_config(get_vhost_net(nc->peer), (uint8_t *)&netcfg,
-                                   n->config_size);
+        struct vhost_net *net = get_vhost_net(nc->peer);
+
+        ret = vhost_net_get_config(net, (uint8_t *)&netcfg, n->config_size);
         if (ret == -1) {
             return;
         }
@@ -173,6 +174,12 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
             memcpy(netcfg.mac, n->mac, ETH_ALEN);
         }
 
+        if (vdev->guest_features & BIT_ULL(VIRTIO_NET_F_STATUS) &&
+            !(net->dev.features & BIT_ULL(VIRTIO_NET_F_STATUS))) {
+            /* Emulating link up in qemu */
+            netcfg.status |= VIRTIO_NET_S_LINK_UP;
+        }
+
         memcpy(config, &netcfg, n->config_size);
     }
 }
Signature · 1 line
2.31.1
#4 lore raw

PATCH 4/7 vdpa: Expose VIRTIO_NET_F_STATUS unconditionally

In reply to Eugenio Pérez

Now that qemu can handle and emulate it if the vdpa backend does not support it we can offer it always.

---
 net/vhost-vdpa.c | 3 +++
 1 file changed, 3 insertions(+)
net/vhost-vdpa.c +3 -0
index e7db40b7cd..3374c21b4d 100644
@@ -554,6 +554,9 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
     if (svq) {
         /* Add SVQ logging capabilities */
         s->vhost_vdpa.added_features |= BIT_ULL(VHOST_F_LOG_ALL);
+
+        /* VIRTIO_NET_F_STATUS is mandatory for _F_GUEST_ANNOUNCE. */
+        s->vhost_vdpa.added_features |= BIT_ULL(VIRTIO_NET_F_STATUS);
     }
     if (!is_datapath) {
         s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
Signature · 1 line
2.31.1
#5 lore raw

PATCH 5/7 vdpa: handle VIRTIO_NET_CTRL_ANNOUNCE in vhost_vdpa_net_handle_ctrl_avail

In reply to Eugenio Pérez

Since this capability is emulated by qemu shadowed CVQ cannot forward it to the device. Process all that command within qemu.

---
 net/vhost-vdpa.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
net/vhost-vdpa.c +12 -3
index 3374c21b4d..5fda405a66 100644
@@ -488,9 +488,18 @@ static int vhost_vdpa_net_handle_ctrl_avail(VhostShadowVirtqueue *svq,
     out.iov_len = iov_to_buf(elem->out_sg, elem->out_num, 0,
                              s->cvq_cmd_out_buffer,
                              vhost_vdpa_net_cvq_cmd_len());
-    dev_written = vhost_vdpa_net_cvq_add(s, out.iov_len, sizeof(status));
-    if (unlikely(dev_written < 0)) {
-        goto out;
+    if (*(uint8_t *)s->cvq_cmd_out_buffer == VIRTIO_NET_CTRL_ANNOUNCE) {
+        /*
+         * Guest announce capability is emulated by qemu, so dont forward to
+         * the device.
+         */
+        dev_written = sizeof(status);
+        *s->status = VIRTIO_NET_OK;
+    } else {
+        dev_written = vhost_vdpa_net_cvq_add(s, out.iov_len, sizeof(status));
+        if (unlikely(dev_written < 0)) {
+            goto out;
+        }
     }
 
     if (unlikely(dev_written < sizeof(status))) {
Signature · 1 line
2.31.1
#6 lore raw

PATCH 6/7 virtio_net: copy VIRTIO_NET_S_ANNOUNCE if device model has it

In reply to Eugenio Pérez

Status part of the emulated feature. It will follow device model, so we must copy it as long as NIC device model has it set.

---
 hw/net/virtio-net.c | 1 +
 1 file changed, 1 insertion(+)
hw/net/virtio-net.c +1 -0
index 51fe6d5c1a..49674dcf32 100644
@@ -180,6 +180,7 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
             netcfg.status |= VIRTIO_NET_S_LINK_UP;
         }
 
+        netcfg.status |= (n->status & VIRTIO_NET_S_ANNOUNCE);
         memcpy(config, &netcfg, n->config_size);
     }
 }
Signature · 1 line
2.31.1
#7 lore raw

PATCH 7/7 vdpa: Offer VIRTIO_NET_F_GUEST_ANNOUNCE feature if SVQ is enabled

In reply to Eugenio Pérez

So qemu emulates it in case the device does not support it.

---
 net/vhost-vdpa.c | 3 +++
 1 file changed, 3 insertions(+)
net/vhost-vdpa.c +3 -0
index 5fda405a66..64442e8455 100644
@@ -566,6 +566,9 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
 
         /* VIRTIO_NET_F_STATUS is mandatory for _F_GUEST_ANNOUNCE. */
         s->vhost_vdpa.added_features |= BIT_ULL(VIRTIO_NET_F_STATUS);
+
+        /* We can emulate guest announce shadowing CVQ */
+        s->vhost_vdpa.added_features |= BIT_ULL(VIRTIO_NET_F_GUEST_ANNOUNCE);
     }
     if (!is_datapath) {
         s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
Signature · 1 line
2.31.1
Reviewed-by on the v1 series
#8 lore raw
In reply to Eugenio Pérez
Quoted 10 lines · Eugenio Pérez

A gratuitous ARP is recommended after a live migration to reduce the amount of time needed by the network links to be aware of the new location. A hypervisor may not have the knowledge of the guest network configuration, and this is especially true on passthrough devices, so its simpler to ask the guest to do it.

However, the device control part of this feature can be totally emulated by qemu and shadow virtqueue, not needing any special feature from the actual vdpa device.

Quoted 7 lines

VIRTIO_NET_F_STATUS is also needed for the guest to access the status of virtio net config where announcement status bit is set. Emulating it as always active in case backend does not support it.

This series need two fixes applied:
* https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03242.html
* https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03241.html

Jason you seem to have queued these so you will have to take this patchset too.

Quoted 25 lines

v1:
* Move code from vhost_net_get_config to virtio_net_get_config.

RFC v2:
* Add VIRTIO_NET_F_STATUS emulation.

Eugenio Pérez (7):
vdpa: Save emulated features list in vhost_vdpa
virtio_net: Modify virtio_net_get_config to early return
virtio_net: Handle _F_STATUS emulation in virtio_net_get_config
vdpa: Expose VIRTIO_NET_F_STATUS unconditionally
vdpa: handle VIRTIO_NET_CTRL_ANNOUNCE in
vhost_vdpa_net_handle_ctrl_avail
virtio_net: copy VIRTIO_NET_S_ANNOUNCE if device model has it
vdpa: Offer VIRTIO_NET_F_GUEST_ANNOUNCE feature if SVQ is enabled

 include/hw/virtio/vhost-vdpa.h |  2 ++
 hw/net/virtio-net.c            | 40 +++++++++++++++++++++-------------
 hw/virtio/vhost-vdpa.c         |  8 +++----
 net/vhost-vdpa.c               | 25 ++++++++++++++++++---
 4 files changed, 53 insertions(+), 22 deletions(-)
Signature · 1 line
2.31.1