Skip to contentSkip to conversation

Vhost-vdpa Shadow Virtqueue Offloads support

PATCH Superseded Hawkins Jiawei 3 participants 13 messages Jun 1, 2023 Networking

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover Vhost-vdpa Shadow Virtqueue Offloads support

This series enables shadowed CVQ to intercept Offloads commands through shadowed CVQ, update the virtio NIC device model so qemu send it in a migration, and the restore of that Offloads state in the destination.

Changelog
=========
v2:
- make some function arguments const
- reuse virtio_vdev_has_feature() suggested by Eugenio and Jason
- avoid sending CVQ command in default state suggested by Eugenio

v1: https://lore.kernel.org/all/[email protected]/

Hawkins Jiawei (6):
include/hw/virtio: make some VirtIODevice const
vdpa: reuse virtio_vdev_has_feature() hw/net/virtio-net: make some VirtIONet const
virtio-net: expose virtio_net_supported_guest_offloads()
vdpa: Add vhost_vdpa_net_load_offloads()
vdpa: Allow VIRTIO_NET_F_CTRL_GUEST_OFFLOADS in SVQ

 hw/net/virtio-net.c            |  2 +-
 include/hw/virtio/virtio-net.h |  1 +
 include/hw/virtio/virtio.h     |  2 +-
 net/vhost-vdpa.c               | 45 +++++++++++++++++++++++++++++++---
 4 files changed, 44 insertions(+), 6 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH v2 1/6 include/hw/virtio: make some VirtIODevice const

The VirtIODevice structure is not modified in virtio_vdev_has_feature(). Therefore, make it const to allow this function to accept const variables.

---
 include/hw/virtio/virtio.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/hw/virtio/virtio.h +1 -1
index af86ed7249..0492d26900 100644
@@ -402,7 +402,7 @@ static inline bool virtio_has_feature(uint64_t features, unsigned int fbit)
     return !!(features & (1ULL << fbit));
 }
 
-static inline bool virtio_vdev_has_feature(VirtIODevice *vdev,
+static inline bool virtio_vdev_has_feature(const VirtIODevice *vdev,
                                            unsigned int fbit)
 {
     return virtio_has_feature(vdev->guest_features, fbit);
Signature · 1 line
2.25.1
#2 lore raw

PATCH v2 2/6 vdpa: reuse virtio_vdev_has_feature()

In reply to Hawkins Jiawei

We can use virtio_vdev_has_feature() instead of manually accessing the features.

---
 net/vhost-vdpa.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
net/vhost-vdpa.c +2 -4
index 37cdc84562..e907a3c792 100644
@@ -643,8 +643,7 @@ static ssize_t vhost_vdpa_net_load_cmd(VhostVDPAState *s, uint8_t class,
 
 static int vhost_vdpa_net_load_mac(VhostVDPAState *s, const VirtIONet *n)
 {
-    uint64_t features = n->parent_obj.guest_features;
-    if (features & BIT_ULL(VIRTIO_NET_F_CTRL_MAC_ADDR)) {
+    if (virtio_vdev_has_feature(&n->parent_obj, VIRTIO_NET_F_CTRL_MAC_ADDR)) {
         ssize_t dev_written = vhost_vdpa_net_load_cmd(s, VIRTIO_NET_CTRL_MAC,
                                                   VIRTIO_NET_CTRL_MAC_ADDR_SET,
                                                   n->mac, sizeof(n->mac));
@@ -662,10 +661,9 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
                                   const VirtIONet *n)
 {
     struct virtio_net_ctrl_mq mq;
-    uint64_t features = n->parent_obj.guest_features;
     ssize_t dev_written;
 
-    if (!(features & BIT_ULL(VIRTIO_NET_F_MQ))) {
+    if (!virtio_vdev_has_feature(&n->parent_obj, VIRTIO_NET_F_MQ)) {
         return 0;
     }
 
Signature · 1 line
2.25.1
#3 lore raw

PATCH v2 3/6 hw/net/virtio-net: make some VirtIONet const

In reply to Hawkins Jiawei

The VirtIONet structure is not modified in virtio_net_supported_guest_offloads(). Therefore, make it const to allow this function to accept const variables.

---
 hw/net/virtio-net.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/net/virtio-net.c +1 -1
index 6df6b7329d..7b27dad6c4 100644
@@ -874,7 +874,7 @@ static uint64_t virtio_net_guest_offloads_by_features(uint32_t features)
     return guest_offloads_mask & features;
 }
 
-static inline uint64_t virtio_net_supported_guest_offloads(VirtIONet *n)
+static inline uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
     return virtio_net_guest_offloads_by_features(vdev->guest_features);
Signature · 1 line
2.25.1
#4 lore raw

PATCH v2 4/6 virtio-net: expose virtio_net_supported_guest_offloads()

In reply to Hawkins Jiawei

To support restoring offloads state in vdpa, need to expose the function virtio_net_supported_guest_offloads(), then vdpa uses this function to get the guest supported offloads.

---
 hw/net/virtio-net.c            | 2 +-
 include/hw/virtio/virtio-net.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +1 -1
index 7b27dad6c4..7e8897a8bc 100644
@@ -874,7 +874,7 @@ static uint64_t virtio_net_guest_offloads_by_features(uint32_t features)
     return guest_offloads_mask & features;
 }
 
-static inline uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
     return virtio_net_guest_offloads_by_features(vdev->guest_features);
include/hw/virtio/virtio-net.h +1 -0
index ef234ffe7e..5f5dcb4572 100644
@@ -227,5 +227,6 @@ size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
                                   unsigned out_num);
 void virtio_net_set_netclient_name(VirtIONet *n, const char *name,
                                    const char *type);
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n);
 
 #endif
Signature · 1 line
2.25.1
#5 lore raw

Re: [PATCH v2 4/6] virtio-net: expose virtio_net_supported_guest_offloads()

Quoted 4 lines · Hawkins Jiawei

To support restoring offloads state in vdpa, need to expose the function virtio_net_supported_guest_offloads(), then vdpa uses this function to get the guest supported offloads.

Here it should be changed to "then QEMU uses this function to get the guest supported offloads.". I will correct the commit message in the v3 patch.

Thanks!

Quoted 34 lines
---
 hw/net/virtio-net.c            | 2 +-
 include/hw/virtio/virtio-net.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +1 -1
index 7b27dad6c4..7e8897a8bc 100644
@@ -874,7 +874,7 @@ static uint64_t virtio_net_guest_offloads_by_features(uint32_t features)
     return guest_offloads_mask & features;
 }
 
-static inline uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
     return virtio_net_guest_offloads_by_features(vdev->guest_features);
include/hw/virtio/virtio-net.h +1 -1
index ef234ffe7e..5f5dcb4572 100644
@@ -227,5 +227,6 @@ size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
                                   unsigned out_num);
 void virtio_net_set_netclient_name(VirtIONet *n, const char *name,
                                    const char *type);
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n);
 
 #endif
--

2.25.1

#6 lore raw

Re: [PATCH v2 4/6] virtio-net: expose virtio_net_supported_guest_offloads()

Quoted 11 lines · Hawkins Jiawei
Quoted 4 lines · Hawkins Jiawei

To support restoring offloads state in vdpa, need to expose the function virtio_net_supported_guest_offloads(), then vdpa uses this function to get the guest supported offloads.

Here it should be changed to "then QEMU uses this function to get the guest supported offloads.". I will correct the commit message in the v3 patch.

Maybe "to get the device supported offloads allow qemu to know the defaults, so it can skip the control message sending if they match with the driver's configuration"?

We can also add "This will be the default at guest's startup, these values can mismatch only at live migration".

What do you think?

Thanks!

Quoted 37 lines

Thanks!

Quoted 34 lines
---
 hw/net/virtio-net.c            | 2 +-
 include/hw/virtio/virtio-net.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +1 -1
index 7b27dad6c4..7e8897a8bc 100644
@@ -874,7 +874,7 @@ static uint64_t virtio_net_guest_offloads_by_features(uint32_t features)
     return guest_offloads_mask & features;
 }
 
-static inline uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
     return virtio_net_guest_offloads_by_features(vdev->guest_features);
include/hw/virtio/virtio-net.h +1 -1
index ef234ffe7e..5f5dcb4572 100644
@@ -227,5 +227,6 @@ size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
                                   unsigned out_num);
 void virtio_net_set_netclient_name(VirtIONet *n, const char *name,
                                    const char *type);
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n);
 
 #endif
--

2.25.1

#7 lore raw

Re: [PATCH v2 4/6] virtio-net: expose virtio_net_supported_guest_offloads()

Quoted 13 lines · Eugenio Perez Martin
Quoted 11 lines · Hawkins Jiawei
Quoted 4 lines · Hawkins Jiawei

To support restoring offloads state in vdpa, need to expose the function virtio_net_supported_guest_offloads(), then vdpa uses this function to get the guest supported offloads.

Here it should be changed to "then QEMU uses this function to get the guest supported offloads.". I will correct the commit message in the v3 patch.

Quoted 6 lines

Maybe "to get the device supported offloads allow qemu to know the defaults, so it can skip the control message sending if they match with the driver's configuration"?

We can also add "This will be the default at guest's startup, these values can mismatch only at live migration".

Hi Eugenio,

I sent the v3 patch before seeing this email:(.

I refactor the commit message to
"To support restoring offloads state in vdpa, need to expose the function virtio_net_supported_guest_offloads(). QEMU uses this function to get the guest supported offloads and no needs to send the corresponding CVQ command if guest enables all supported features." in v3 patch.

Do you think there is still some refactoring needed here?

Thanks!

Quoted 43 lines

What do you think?

Thanks!

Quoted 37 lines

Thanks!

Quoted 34 lines
---
  hw/net/virtio-net.c            | 2 +-
  include/hw/virtio/virtio-net.h | 1 +
  2 files changed, 2 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +1 -1
index 7b27dad6c4..7e8897a8bc 100644
@@ -874,7 +874,7 @@ static uint64_t virtio_net_guest_offloads_by_features(uint32_t features)
      return guest_offloads_mask & features;
  }
 
-static inline uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
  {
      VirtIODevice *vdev = VIRTIO_DEVICE(n);
      return virtio_net_guest_offloads_by_features(vdev->guest_features);
include/hw/virtio/virtio-net.h +1 -1
index ef234ffe7e..5f5dcb4572 100644
@@ -227,5 +227,6 @@ size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
                                    unsigned out_num);
  void virtio_net_set_netclient_name(VirtIONet *n, const char *name,
                                     const char *type);
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n);
 
  #endif
--

2.25.1

#8 lore raw

Re: [PATCH v2 4/6] virtio-net: expose virtio_net_supported_guest_offloads()

Quoted 19 lines · Hawkins Jiawei
Quoted 17 lines · Eugenio Perez Martin
Quoted 11 lines · Hawkins Jiawei
Quoted 4 lines · Hawkins Jiawei

To support restoring offloads state in vdpa, need to expose the function virtio_net_supported_guest_offloads(), then vdpa uses this function to get the guest supported offloads.

Here it should be changed to "then QEMU uses this function to get the guest supported offloads.". I will correct the commit message in the v3 patch.

Maybe "to get the device supported offloads allow qemu to know the defaults, so it can skip the control message sending if they match with the driver's configuration"?

Quoted 7 lines
Quoted 2 lines

We can also add "This will be the default at guest's startup, these values can mismatch only at live migration".

Hi Eugenio,

I sent the v3 patch before seeing this email:(.

Moving the discussion to v3 then :).

Thanks!

Quoted 56 lines

I refactor the commit message to
"To support restoring offloads state in vdpa, need to expose the function virtio_net_supported_guest_offloads(). QEMU uses this function to get the guest supported offloads and no needs to send the corresponding CVQ command if guest enables all supported features." in v3 patch.

Do you think there is still some refactoring needed here?

Thanks!

Quoted 43 lines

What do you think?

Thanks!

Quoted 37 lines

Thanks!

Quoted 34 lines
---
  hw/net/virtio-net.c            | 2 +-
  include/hw/virtio/virtio-net.h | 1 +
  2 files changed, 2 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +1 -1
index 7b27dad6c4..7e8897a8bc 100644
@@ -874,7 +874,7 @@ static uint64_t virtio_net_guest_offloads_by_features(uint32_t features)
      return guest_offloads_mask & features;
  }
 
-static inline uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n)
  {
      VirtIODevice *vdev = VIRTIO_DEVICE(n);
      return virtio_net_guest_offloads_by_features(vdev->guest_features);
include/hw/virtio/virtio-net.h +1 -1
index ef234ffe7e..5f5dcb4572 100644
@@ -227,5 +227,6 @@ size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
                                    unsigned out_num);
  void virtio_net_set_netclient_name(VirtIONet *n, const char *name,
                                     const char *type);
+uint64_t virtio_net_supported_guest_offloads(const VirtIONet *n);
 
  #endif
--

2.25.1

#9 lore raw

PATCH v2 5/6 vdpa: Add vhost_vdpa_net_load_offloads()

In reply to Hawkins Jiawei

This patch introduces vhost_vdpa_net_load_offloads() to restore offloads state at device's startup.

---
 net/vhost-vdpa.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
net/vhost-vdpa.c +38 -0
index e907a3c792..0e647886d1 100644
@@ -678,6 +678,40 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
     return *s->status != VIRTIO_NET_OK;
 }
 
+static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,
+                                        const VirtIONet *n)
+{
+    uint64_t offloads;
+    ssize_t dev_written;
+
+    if (!virtio_vdev_has_feature(&n->parent_obj,
+                                 VIRTIO_NET_F_CTRL_GUEST_OFFLOADS)) {
+        return 0;
+    }
+
+    offloads = cpu_to_le64(n->curr_guest_offloads);
+
+    if (offloads == virtio_net_supported_guest_offloads(n)) {
+        /*
+         * According to VirtIO standard, "Upon feature negotiation
+         * corresponding offload gets enabled to preserve
+         * backward compatibility."
+         * So we do not need to send this CVQ command if the guest
+         * also enables all supported offloads.
+         */
+        return 0;
+    }
+
+    dev_written = vhost_vdpa_net_load_cmd(s, VIRTIO_NET_CTRL_GUEST_OFFLOADS,
+                                          VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET,
+                                          &offloads, sizeof(offloads));
+    if (unlikely(dev_written < 0)) {
+        return dev_written;
+    }
+
+    return *s->status != VIRTIO_NET_OK;
+}
+
 static int vhost_vdpa_net_load(NetClientState *nc)
 {
     VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -700,6 +734,10 @@ static int vhost_vdpa_net_load(NetClientState *nc)
     if (unlikely(r)) {
         return r;
     }
+    r = vhost_vdpa_net_load_offloads(s, n);
+    if (unlikely(r)) {
+        return r;
+    }
 
     return 0;
 }
Signature · 1 line
2.25.1
#10 lore raw

PATCH v2 6/6 vdpa: Allow VIRTIO_NET_F_CTRL_GUEST_OFFLOADS in SVQ

In reply to Hawkins Jiawei

Enable SVQ with VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature.

---
 net/vhost-vdpa.c | 1 +
 1 file changed, 1 insertion(+)
net/vhost-vdpa.c +1 -0
index 0e647886d1..4cc24a2997 100644
@@ -85,6 +85,7 @@ const int vdpa_feature_bits[] = {
 static const uint64_t vdpa_svq_device_features =
     BIT_ULL(VIRTIO_NET_F_CSUM) |
     BIT_ULL(VIRTIO_NET_F_GUEST_CSUM) |
+    BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS) |
     BIT_ULL(VIRTIO_NET_F_MTU) |
     BIT_ULL(VIRTIO_NET_F_MAC) |
     BIT_ULL(VIRTIO_NET_F_GUEST_TSO4) |
Signature · 1 line
2.25.1
#11 lore raw
In reply to Hawkins Jiawei

I'm a QE responsible for vhost_vdpa parts. Could you please provide me with the test steps for this series? I can test it in my environment and update the test results.

Quoted 33 lines · Hawkins Jiawei

This series enables shadowed CVQ to intercept Offloads commands through shadowed CVQ, update the virtio NIC device model so qemu send it in a migration, and the restore of that Offloads state in the destination.

Changelog
=========
v2:
- make some function arguments const
- reuse virtio_vdev_has_feature() suggested by Eugenio and Jason
- avoid sending CVQ command in default state suggested by Eugenio

v1: https://lore.kernel.org/all/[email protected]/

Hawkins Jiawei (6):
include/hw/virtio: make some VirtIODevice const
vdpa: reuse virtio_vdev_has_feature() hw/net/virtio-net: make some VirtIONet const
virtio-net: expose virtio_net_supported_guest_offloads()
vdpa: Add vhost_vdpa_net_load_offloads()
vdpa: Allow VIRTIO_NET_F_CTRL_GUEST_OFFLOADS in SVQ

 hw/net/virtio-net.c            |  2 +-
 include/hw/virtio/virtio-net.h |  1 +
 include/hw/virtio/virtio.h     |  2 +-
 net/vhost-vdpa.c               | 45 +++++++++++++++++++++++++++++++---
 4 files changed, 44 insertions(+), 6 deletions(-)

--
2.25.1

#12 lore raw
Quoted 4 lines · Lei Yang

I'm a QE responsible for vhost_vdpa parts. Could you please provide me with the test steps for this series? I can test it in my environment and update the test results.

Hi Lei,

I am not sure if these are enough, but you can test it like this:

1. you can merge these patches into the latest master branch and compile the source. It should not trigger any error or warning.

2. start QEMU with the vdpa device and enable the ctrl_guest_offloads feature on. The command line should like the following:

        -netdev type=vhost-vdpa,id=vdpa0,vhostdev=/dev/vhost-vdpa-0,x-svq=true \
        -device virtio-net-pci,netdev=vdpa0,ctrl_guest_offloads=on,...

It should also not trigger any error or warning. Without this series, QEMU should fail with "vdpa svq does not work with features 4" error.

Regarding migration, it seems that testing it is not feasible at the moment due to the migration blocker.

And you can test on my v3 patch, I will send the v3 patch to the mailing lists later.

Thank you for your efforts!

Quoted 39 lines · Hawkins Jiawei
Quoted 33 lines

This series enables shadowed CVQ to intercept Offloads commands through shadowed CVQ, update the virtio NIC device model so qemu send it in a migration, and the restore of that Offloads state in the destination.

Changelog
=========
v2:
- make some function arguments const
- reuse virtio_vdev_has_feature() suggested by Eugenio and Jason
- avoid sending CVQ command in default state suggested by Eugenio

v1: https://lore.kernel.org/all/[email protected]/

Hawkins Jiawei (6):
include/hw/virtio: make some VirtIODevice const
vdpa: reuse virtio_vdev_has_feature() hw/net/virtio-net: make some VirtIONet const
virtio-net: expose virtio_net_supported_guest_offloads()
vdpa: Add vhost_vdpa_net_load_offloads()
vdpa: Allow VIRTIO_NET_F_CTRL_GUEST_OFFLOADS in SVQ

  hw/net/virtio-net.c            |  2 +-
  include/hw/virtio/virtio-net.h |  1 +
  include/hw/virtio/virtio.h     |  2 +-
  net/vhost-vdpa.c               | 45 +++++++++++++++++++++++++++++++---
  4 files changed, 44 insertions(+), 6 deletions(-)

--
2.25.1