Skip to contentSkip to conversation

Vhost-vdpa Shadow Virtqueue Offloads support

PATCH Superseded Hawkins Jiawei 3 participants 12 messages May 29, 2023 – May 31, 2023 Networking

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover Vhost-vdpa Shadow Virtqueue Offloads support

This series enables shadowed CVQ to intercept Offloads commands through shadowed CVQ, update the virtio NIC device model so qemu send it in a migration, and the restore of that Offloads state in the destination.

Hawkins Jiawei (2):
vdpa: Add vhost_vdpa_net_load_offloads
vdpa: Allow VIRTIO_NET_F_CTRL_GUEST_OFFLOADS in SVQ

 net/vhost-vdpa.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/2 vdpa: Add vhost_vdpa_net_load_offloads

This patch introduces vhost_vdpa_net_load_offloads() to restore offloads state at device's startup.

---
 net/vhost-vdpa.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
net/vhost-vdpa.c +26 -0
index 37cdc84562..682c749b19 100644
@@ -680,6 +680,28 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
     return *s->status != VIRTIO_NET_OK;
 }
 
+static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,
+                                        const VirtIONet *n)
+{
+    uint64_t features, offloads;
+    ssize_t dev_written;
+
+    features = n->parent_obj.guest_features;
+    if (!(features & BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))) {
+        return 0;
+    }
+
+    offloads = cpu_to_le64(n->curr_guest_offloads);
+    dev_written = vhost_vdpa_net_load_cmd(s, VIRTIO_NET_CTRL_GUEST_OFFLOADS,
+                                          VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET,
+                                          &offloads, sizeof(offloads));
+    if (unlikely(dev_written < 0)) {
+        return dev_written;
+    }
+
+    return *s->status != VIRTIO_NET_OK;
+}
+
 static int vhost_vdpa_net_load(NetClientState *nc)
 {
     VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -702,6 +724,10 @@ static int vhost_vdpa_net_load(NetClientState *nc)
     if (unlikely(r)) {
         return r;
     }
+    r = vhost_vdpa_net_load_offloads(s, n);
+    if (unlikely(r)) {
+        return r;
+    }
 
     return 0;
 }
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH 1/2] vdpa: Add vhost_vdpa_net_load_offloads

Quoted 18 lines · Hawkins Jiawei

This patch introduces vhost_vdpa_net_load_offloads() to restore offloads state at device's startup.

---
 net/vhost-vdpa.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
net/vhost-vdpa.c +1 -0
index 37cdc84562..682c749b19 100644
@@ -680,6 +680,28 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
     return *s->status != VIRTIO_NET_OK;
 }
 
+static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,

net/vhost-vdpa.c static int vhost_vdpa_net_load_mq(VhostVDPAState *s, · line 693

Quoted 10 lines
net/vhost-vdpa.c +10 -0
+                                        const VirtIONet *n)
+{
+    uint64_t features, offloads;
+    ssize_t dev_written;
+
+    features = n->parent_obj.guest_features;
+    if (!(features & BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))) {
+        return 0;
+    }
+

Maybe we can avoid sending this CVQ command if the guest already uses the default values?

By default all features are enabled if I'm not wrong. I think the best way is to expose virtio_net_supported_guest_offloads or virtio_net_guest_offloads_by_features and then check if n->curr_guest_offloads is the same.

We should do the same with vhost_vdpa_net_load_mq, but that is out of the scope of this series.

Thanks!

Quoted 28 lines
net/vhost-vdpa.c +15 -1
+    offloads = cpu_to_le64(n->curr_guest_offloads);
+    dev_written = vhost_vdpa_net_load_cmd(s, VIRTIO_NET_CTRL_GUEST_OFFLOADS,
+                                          VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET,
+                                          &offloads, sizeof(offloads));
+    if (unlikely(dev_written < 0)) {
+        return dev_written;
+    }
+
+    return *s->status != VIRTIO_NET_OK;
+}
+
 static int vhost_vdpa_net_load(NetClientState *nc)
 {
     VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -702,6 +724,10 @@ static int vhost_vdpa_net_load(NetClientState *nc)
     if (unlikely(r)) {
         return r;
     }
+    r = vhost_vdpa_net_load_offloads(s, n);
+    if (unlikely(r)) {
+        return r;
+    }
 
     return 0;
 }
--

2.25.1

#3 lore raw

Re: [PATCH 1/2] vdpa: Add vhost_vdpa_net_load_offloads

Quoted 18 lines · Eugenio Perez Martin
Quoted 17 lines · Hawkins Jiawei

This patch introduces vhost_vdpa_net_load_offloads() to restore offloads state at device's startup.

---
  net/vhost-vdpa.c | 26 ++++++++++++++++++++++++++
  1 file changed, 26 insertions(+)
net/vhost-vdpa.c +0 -0
index 37cdc84562..682c749b19 100644
@@ -680,6 +680,28 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
      return *s->status != VIRTIO_NET_OK;
  }
 
Quoted 14 lines

net/vhost-vdpa.c static int vhost_vdpa_net_load_mq(VhostVDPAState *s, · line 693

Quoted 11 lines
net/vhost-vdpa.c +11 -0
+static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,
+                                        const VirtIONet *n)
+{
+    uint64_t features, offloads;
+    ssize_t dev_written;
+
+    features = n->parent_obj.guest_features;
+    if (!(features & BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))) {
+        return 0;
+    }
+

Maybe we can avoid sending this CVQ command if the guest already uses the default values?

Hi Eugenio,

Thanks for the review. However, I'm curious why we don't need to send this CVQ command if the guest is using the default values. Is it because the device automatically applies these default offloads, when the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is negotiated and QEMU doesn't send the CVQ command?

Thanks!

Quoted 40 lines

By default all features are enabled if I'm not wrong. I think the best way is to expose virtio_net_supported_guest_offloads or virtio_net_guest_offloads_by_features and then check if n->curr_guest_offloads is the same.

We should do the same with vhost_vdpa_net_load_mq, but that is out of the scope of this series.

Thanks!

Quoted 28 lines
net/vhost-vdpa.c +15 -1
+    offloads = cpu_to_le64(n->curr_guest_offloads);
+    dev_written = vhost_vdpa_net_load_cmd(s, VIRTIO_NET_CTRL_GUEST_OFFLOADS,
+                                          VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET,
+                                          &offloads, sizeof(offloads));
+    if (unlikely(dev_written < 0)) {
+        return dev_written;
+    }
+
+    return *s->status != VIRTIO_NET_OK;
+}
+
  static int vhost_vdpa_net_load(NetClientState *nc)
  {
      VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -702,6 +724,10 @@ static int vhost_vdpa_net_load(NetClientState *nc)
      if (unlikely(r)) {
          return r;
      }
+    r = vhost_vdpa_net_load_offloads(s, n);
+    if (unlikely(r)) {
+        return r;
+    }
 
      return 0;
  }
--

2.25.1

#4 lore raw

Re: [PATCH 1/2] vdpa: Add vhost_vdpa_net_load_offloads

Quoted 35 lines · Hawkins Jiawei
Quoted 18 lines · Eugenio Perez Martin
Quoted 17 lines · Hawkins Jiawei

This patch introduces vhost_vdpa_net_load_offloads() to restore offloads state at device's startup.

---
  net/vhost-vdpa.c | 26 ++++++++++++++++++++++++++
  1 file changed, 26 insertions(+)
net/vhost-vdpa.c +0 -0
index 37cdc84562..682c749b19 100644
@@ -680,6 +680,28 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
      return *s->status != VIRTIO_NET_OK;
  }
 
Quoted 14 lines

net/vhost-vdpa.c static int vhost_vdpa_net_load_mq(VhostVDPAState *s, · line 693

Quoted 11 lines
net/vhost-vdpa.c +11 -0
+static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,
+                                        const VirtIONet *n)
+{
+    uint64_t features, offloads;
+    ssize_t dev_written;
+
+    features = n->parent_obj.guest_features;
+    if (!(features & BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))) {
+        return 0;
+    }
+

Maybe we can avoid sending this CVQ command if the guest already uses the default values?

Quoted 8 lines

Hi Eugenio,

Thanks for the review. However, I'm curious why we don't need to send this CVQ command if the guest is using the default values. Is it because the device automatically applies these default offloads, when the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is negotiated and QEMU doesn't send the CVQ command?

Exactly. You can check that either by the standard text or (sometimes easier) qemu virtio or virtio-net device code.

The standard text says that:
"Upon feature negotiation corresponding offload gets enabled to preserve backward compatibility."

And you can check in the qemu code by hw/net/virtio-net:virtio_net_set_features(vdev, features), this chunk of code:
n->curr_guest_offloads = virtio_net_guest_offloads_by_features(features); virtio_net_apply_guest_offloads(n);

Thanks!

Quoted 44 lines

Thanks!

Quoted 40 lines

By default all features are enabled if I'm not wrong. I think the best way is to expose virtio_net_supported_guest_offloads or virtio_net_guest_offloads_by_features and then check if n->curr_guest_offloads is the same.

We should do the same with vhost_vdpa_net_load_mq, but that is out of the scope of this series.

Thanks!

Quoted 28 lines
net/vhost-vdpa.c +15 -1
+    offloads = cpu_to_le64(n->curr_guest_offloads);
+    dev_written = vhost_vdpa_net_load_cmd(s, VIRTIO_NET_CTRL_GUEST_OFFLOADS,
+                                          VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET,
+                                          &offloads, sizeof(offloads));
+    if (unlikely(dev_written < 0)) {
+        return dev_written;
+    }
+
+    return *s->status != VIRTIO_NET_OK;
+}
+
  static int vhost_vdpa_net_load(NetClientState *nc)
  {
      VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -702,6 +724,10 @@ static int vhost_vdpa_net_load(NetClientState *nc)
      if (unlikely(r)) {
          return r;
      }
+    r = vhost_vdpa_net_load_offloads(s, n);
+    if (unlikely(r)) {
+        return r;
+    }
 
      return 0;
  }
--

2.25.1

#5 lore raw

Re: [PATCH 1/2] vdpa: Add vhost_vdpa_net_load_offloads

In reply to Hawkins Jiawei
Quoted 14 lines · Hawkins Jiawei

This patch introduces vhost_vdpa_net_load_offloads() to restore offloads state at device's startup.

---
 net/vhost-vdpa.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
net/vhost-vdpa.c +0 -0
index 37cdc84562..682c749b19 100644

net/vhost-vdpa.c static int vhost_vdpa_net_load_mq(VhostVDPAState *s, · line 689

Quoted 10 lines
net/vhost-vdpa.c +7 -0
@@ -680,6 +680,28 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
     return *s->status != VIRTIO_NET_OK;
 }
 
+static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,
+                                        const VirtIONet *n)
+{
+    uint64_t features, offloads;
+    ssize_t dev_written;
+
+    features = n->parent_obj.guest_features;

Any reason you need to do tricks like this instead of using virtio_xxx_has_features()?

Quoted 32 lines
net/vhost-vdpa.c +19 -1
+    if (!(features & BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))) {
+        return 0;
+    }
+
+    offloads = cpu_to_le64(n->curr_guest_offloads);
+    dev_written = vhost_vdpa_net_load_cmd(s, VIRTIO_NET_CTRL_GUEST_OFFLOADS,
+                                          VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET,
+                                          &offloads, sizeof(offloads));
+    if (unlikely(dev_written < 0)) {
+        return dev_written;
+    }
+
+    return *s->status != VIRTIO_NET_OK;
+}
+
 static int vhost_vdpa_net_load(NetClientState *nc)
 {
     VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -702,6 +724,10 @@ static int vhost_vdpa_net_load(NetClientState *nc)
     if (unlikely(r)) {
         return r;
     }
+    r = vhost_vdpa_net_load_offloads(s, n);
+    if (unlikely(r)) {
+        return r;
+    }
 
     return 0;
 }
--

2.25.1

#6 lore raw

Re: [PATCH 1/2] vdpa: Add vhost_vdpa_net_load_offloads

Quoted 19 lines · Jason Wang
Quoted 17 lines · Hawkins Jiawei

This patch introduces vhost_vdpa_net_load_offloads() to restore offloads state at device's startup.

---
 net/vhost-vdpa.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
net/vhost-vdpa.c +0 -0
index 37cdc84562..682c749b19 100644
@@ -680,6 +680,28 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
     return *s->status != VIRTIO_NET_OK;
 }
 
Quoted 11 lines

net/vhost-vdpa.c static int vhost_vdpa_net_load_mq(VhostVDPAState *s, · line 689

Quoted 7 lines
net/vhost-vdpa.c +7 -0
+static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,
+                                        const VirtIONet *n)
+{
+    uint64_t features, offloads;
+    ssize_t dev_written;
+
+    features = n->parent_obj.guest_features;

Any reason you need to do tricks like this instead of using virtio_xxx_has_features()?

It can be replaced by virtio_vdev_has_feature, yes.

Current code of vhost_vdpa_net_load_mac and vhost_vdpa_net_load_mq access to guest_features directly too, so I think we should change all of them at once.

Thanks!

Quoted 33 lines
Quoted 32 lines
net/vhost-vdpa.c +19 -1
+    if (!(features & BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))) {
+        return 0;
+    }
+
+    offloads = cpu_to_le64(n->curr_guest_offloads);
+    dev_written = vhost_vdpa_net_load_cmd(s, VIRTIO_NET_CTRL_GUEST_OFFLOADS,
+                                          VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET,
+                                          &offloads, sizeof(offloads));
+    if (unlikely(dev_written < 0)) {
+        return dev_written;
+    }
+
+    return *s->status != VIRTIO_NET_OK;
+}
+
 static int vhost_vdpa_net_load(NetClientState *nc)
 {
     VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -702,6 +724,10 @@ static int vhost_vdpa_net_load(NetClientState *nc)
     if (unlikely(r)) {
         return r;
     }
+    r = vhost_vdpa_net_load_offloads(s, n);
+    if (unlikely(r)) {
+        return r;
+    }
 
     return 0;
 }
--

2.25.1

#7 lore raw

Re: [PATCH 1/2] vdpa: Add vhost_vdpa_net_load_offloads

Quoted 32 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 17 lines · Hawkins Jiawei

This patch introduces vhost_vdpa_net_load_offloads() to restore offloads state at device's startup.

---
  net/vhost-vdpa.c | 26 ++++++++++++++++++++++++++
  1 file changed, 26 insertions(+)
net/vhost-vdpa.c +0 -0
index 37cdc84562..682c749b19 100644
@@ -680,6 +680,28 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
      return *s->status != VIRTIO_NET_OK;
  }
 
Quoted 11 lines

net/vhost-vdpa.c static int vhost_vdpa_net_load_mq(VhostVDPAState *s, · line 689

Quoted 7 lines
net/vhost-vdpa.c +7 -0
+static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,
+                                        const VirtIONet *n)
+{
+    uint64_t features, offloads;
+    ssize_t dev_written;
+
+    features = n->parent_obj.guest_features;

Any reason you need to do tricks like this instead of using virtio_xxx_has_features()?

Quoted 5 lines

It can be replaced by virtio_vdev_has_feature, yes.

Current code of vhost_vdpa_net_load_mac and vhost_vdpa_net_load_mq access to guest_features directly too, so I think we should change all of them at once.

Yes, I agree with you and Jason.

I will refactor the patch as you and Jason have suggested.

Thanks!

Quoted 37 lines

Thanks!

Quoted 33 lines
Quoted 32 lines
net/vhost-vdpa.c +19 -1
+    if (!(features & BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))) {
+        return 0;
+    }
+
+    offloads = cpu_to_le64(n->curr_guest_offloads);
+    dev_written = vhost_vdpa_net_load_cmd(s, VIRTIO_NET_CTRL_GUEST_OFFLOADS,
+                                          VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET,
+                                          &offloads, sizeof(offloads));
+    if (unlikely(dev_written < 0)) {
+        return dev_written;
+    }
+
+    return *s->status != VIRTIO_NET_OK;
+}
+
  static int vhost_vdpa_net_load(NetClientState *nc)
  {
      VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -702,6 +724,10 @@ static int vhost_vdpa_net_load(NetClientState *nc)
      if (unlikely(r)) {
          return r;
      }
+    r = vhost_vdpa_net_load_offloads(s, n);
+    if (unlikely(r)) {
+        return r;
+    }
 
      return 0;
  }
--

2.25.1

#8 lore raw

PATCH 2/2 vdpa: Allow VIRTIO_NET_F_CTRL_GUEST_OFFLOADS in SVQ

In reply to Hawkins Jiawei

Enable SVQ with VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature.

---
 net/vhost-vdpa.c | 1 +
 1 file changed, 1 insertion(+)
net/vhost-vdpa.c +1 -0
index 682c749b19..cc52b7f0ad 100644
@@ -85,6 +85,7 @@ const int vdpa_feature_bits[] = {
 static const uint64_t vdpa_svq_device_features =
     BIT_ULL(VIRTIO_NET_F_CSUM) |
     BIT_ULL(VIRTIO_NET_F_GUEST_CSUM) |
+    BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS) |
     BIT_ULL(VIRTIO_NET_F_MTU) |
     BIT_ULL(VIRTIO_NET_F_MAC) |
     BIT_ULL(VIRTIO_NET_F_GUEST_TSO4) |
Signature · 1 line
2.25.1
Acked-by on v1 2/2
#9 lore raw

Re: [PATCH 2/2] vdpa: Allow VIRTIO_NET_F_CTRL_GUEST_OFFLOADS in SVQ

Quoted 4 lines · Hawkins Jiawei

Enable SVQ with VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature.

Thanks

Quoted 19 lines
---
 net/vhost-vdpa.c | 1 +
 1 file changed, 1 insertion(+)
net/vhost-vdpa.c +1 -1
index 682c749b19..cc52b7f0ad 100644
@@ -85,6 +85,7 @@ const int vdpa_feature_bits[] = {
 static const uint64_t vdpa_svq_device_features =
     BIT_ULL(VIRTIO_NET_F_CSUM) |
     BIT_ULL(VIRTIO_NET_F_GUEST_CSUM) |
+    BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS) |
     BIT_ULL(VIRTIO_NET_F_MTU) |
     BIT_ULL(VIRTIO_NET_F_MAC) |
     BIT_ULL(VIRTIO_NET_F_GUEST_TSO4) |
--

2.25.1

#10 lore raw
In reply to Hawkins Jiawei
Quoted 6 lines · Hawkins Jiawei

This series enables shadowed CVQ to intercept Offloads commands through shadowed CVQ, update the virtio NIC device model so qemu send it in a migration, and the restore of that Offloads state in the destination.

Quoted 7 lines

Hawkins Jiawei (2):
vdpa: Add vhost_vdpa_net_load_offloads
vdpa: Allow VIRTIO_NET_F_CTRL_GUEST_OFFLOADS in SVQ

 net/vhost-vdpa.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

CCing MST too.

Thanks!

#11 lore raw
Quoted 11 lines · Eugenio Perez Martin
Quoted 10 lines · Hawkins Jiawei

This series enables shadowed CVQ to intercept Offloads commands through shadowed CVQ, update the virtio NIC device model so qemu send it in a migration, and the restore of that Offloads state in the destination.

Hawkins Jiawei (2):
vdpa: Add vhost_vdpa_net_load_offloads
vdpa: Allow VIRTIO_NET_F_CTRL_GUEST_OFFLOADS in SVQ

Quoted 5 lines
Quoted 3 lines
  net/vhost-vdpa.c | 27 +++++++++++++++++++++++++++
  1 file changed, 27 insertions(+)

CCing MST too.

OK, I will cc MST in the future patches for this part.

Thanks!

Quoted 3 lines

Thanks!