Skip to contentSkip to conversation

virtio_net: support multi buffer xdp

PATCH Superseded Heng Qi 3 participants 41 messages Dec 20, 2022 – Dec 28, 2022

Subsystems (4)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Heng Qi

author
#0 lore raw

v2 cover virtio_net: support multi buffer xdp

Changes since RFC:
- Using headroom instead of vi->xdp_enabled to avoid re-reading in add_recvbuf_mergeable();
- Disable GRO_HW and keep linearization for single buffer xdp;
- Renamed to virtnet_build_xdp_buff_mrg();
- pr_debug() to netdev_dbg();
- Adjusted the order of the patch series.

Currently, virtio net only supports xdp for single-buffer packets or linearized multi-buffer packets. This patchset supports xdp for multi-buffer packets, then larger MTU can be used if xdp sets the xdp.frags. This does not affect single buffer handling.

In order to build multi-buffer xdp neatly, we integrated the code into virtnet_build_xdp_buff_mrg() for xdp. The first buffer is used for prepared xdp buff, and the rest of the buffers are added to its skb_shared_info structure. This structure can also be conveniently converted during XDP_PASS to get the corresponding skb.

Since virtio net uses comp pages, and bpf_xdp_frags_increase_tail() is based on the assumption of the page pool, (rxq->frag_size - skb_frag_size(frag) - skb_frag_off(frag)) is negative in most cases. So we didn't set xdp_rxq->frag_size in virtnet_open() to disable the tail increase.

Heng Qi (9):
virtio_net: disable the hole mechanism for xdp
virtio_net: set up xdp for multi buffer packets
virtio_net: update bytes calculation for xdp_frame
virtio_net: build xdp_buff with multi buffers
virtio_net: construct multi-buffer xdp in mergeable
virtio_net: transmit the multi-buffer xdp
virtio_net: build skb from multi-buffer xdp
virtio_net: remove xdp related info from page_to_skb()
virtio_net: support multi-buffer xdp

 drivers/net/virtio_net.c | 332 ++++++++++++++++++++++++++-------------
 1 file changed, 219 insertions(+), 113 deletions(-)
Signature · 1 line
2.19.1.6.gb485710b
author
#1 lore raw

PATCH v2 1/9 virtio_net: disable the hole mechanism for xdp

XDP core assumes that the frame_size of xdp_buff and the length of the frag are PAGE_SIZE. The hole may cause the processing of xdp to fail, so we disable the hole mechanism when xdp is set.

---
 drivers/net/virtio_net.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +4 -1
index 9cce7dec7366..443aa7b8f0ad 100644
@@ -1419,8 +1419,11 @@ static int add_recvbuf_mergeable(struct virtnet_info *vi,
 		/* To avoid internal fragmentation, if there is very likely not
 		 * enough space for another buffer, add the remaining space to
 		 * the current buffer.
+		 * XDP core assumes that frame_size of xdp_buff and the length
+		 * of the frag are PAGE_SIZE, so we disable the hole mechanism.
 		 */
-		len += hole;
+		if (!headroom)
+			len += hole;
 		alloc_frag->offset += hole;
 	}
 
Signature · 1 line
2.19.1.6.gb485710b
#2 lore raw

Re: [PATCH v2 1/9] virtio_net: disable the hole mechanism for xdp

Quoted 15 lines · Heng Qi

XDP core assumes that the frame_size of xdp_buff and the length of the frag are PAGE_SIZE. The hole may cause the processing of xdp to fail, so we disable the hole mechanism when xdp is set.

---
  drivers/net/virtio_net.c | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +0 -0
index 9cce7dec7366..443aa7b8f0ad 100644

drivers/net/virtio_net.c static int add_recvbuf_mergeable(struct virtnet_info *vi, · line 1426

Quoted 9 lines
drivers/net/virtio_net.c +4 -1
@@ -1419,8 +1419,11 @@ static int add_recvbuf_mergeable(struct virtnet_info *vi,
  		/* To avoid internal fragmentation, if there is very likely not
  		 * enough space for another buffer, add the remaining space to
  		 * the current buffer.
+		 * XDP core assumes that frame_size of xdp_buff and the length
+		 * of the frag are PAGE_SIZE, so we disable the hole mechanism.
  		 */
-		len += hole;
+		if (!headroom)
+			len += hole;

Is this only a requirement of multi-buffer XDP? If not, it need to be backported to stable.

Thanks

Quoted 3 lines
drivers/net/virtio_net.c +0 -0
  		alloc_frag->offset += hole;
  	}
 
author
#3 lore raw

Re: [PATCH v2 1/9] virtio_net: disable the hole mechanism for xdp

Quoted 22 lines · Jason Wang
Quoted 20 lines · Heng Qi

XDP core assumes that the frame_size of xdp_buff and the length of the frag are PAGE_SIZE. The hole may cause the processing of xdp to fail, so we disable the hole mechanism when xdp is set.

---
  drivers/net/virtio_net.c | 5 ++++-

  1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +1 -0
index 9cce7dec7366..443aa7b8f0ad 100644
@@ -1419,8 +1419,11 @@ static int add_recvbuf_mergeable(struct 
 virtnet_info *vi,
           /* To avoid internal fragmentation, if there is very likely not
            * enough space for another buffer, add the remaining space to
            * the current buffer.
+         * XDP core assumes that frame_size of xdp_buff and the length
Quoted 8 lines

drivers/net/virtio_net.c static int add_recvbuf_mergeable(struct virtnet_info *vi, · line 1426

Quoted 5 lines
drivers/net/virtio_net.c +3 -1
+         * of the frag are PAGE_SIZE, so we disable the hole mechanism.
            */
-        len += hole;
+        if (!headroom)
+            len += hole;

Is this only a requirement of multi-buffer XDP? If not, it need to be backported to stable.

It applies to single buffer xdp and multi-buffer xdp, but even if single buffer xdp has a hole mechanism, there will be no problem (limiting mtu and turning off GUEST GSO), so there is no need to backport it.

Thanks.

Quoted 6 lines

Thanks

Quoted 2 lines
drivers/net/virtio_net.c +0 -0
           alloc_frag->offset += hole;
       }
Acked-by
#4 lore raw

Re: [PATCH v2 1/9] virtio_net: disable the hole mechanism for xdp

Quoted 35 lines · Heng Qi
Quoted 23 lines · Jason Wang
Quoted 21 lines · Heng Qi

XDP core assumes that the frame_size of xdp_buff and the length of the frag are PAGE_SIZE. The hole may cause the processing of xdp to fail, so we disable the hole mechanism when xdp is set.

---
  drivers/net/virtio_net.c | 5 ++++-

  1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +2 -0
index 9cce7dec7366..443aa7b8f0ad 100644
@@ -1419,8 +1419,11 @@ static int add_recvbuf_mergeable(struct 
 virtnet_info *vi,
           /* To avoid internal fragmentation, if there is very likely not
            * enough space for another buffer, add the remaining space to
            * the current buffer.
+         * XDP core assumes that frame_size of xdp_buff and the length
+         * of the frag are PAGE_SIZE, so we disable the hole 
Quoted 8 lines
Quoted 5 lines

mechanism.
           */
-        len += hole;
+        if (!headroom)
+            len += hole;

Is this only a requirement of multi-buffer XDP? If not, it need to be backported to stable.

Quoted 3 lines

It applies to single buffer xdp and multi-buffer xdp, but even if single buffer xdp has a hole mechanism, there will be no problem (limiting mtu and turning off GUEST GSO), so there is no need to backport it.

Let's add this in the changelog.

With that,

Thanks

Quoted 10 lines

Thanks.

Quoted 6 lines

Thanks

Quoted 2 lines

          alloc_frag->offset += hole;
      }

author
#5 lore raw

Re: [PATCH v2 1/9] virtio_net: disable the hole mechanism for xdp

Quoted 38 lines · Jason Wang
Quoted 36 lines · Heng Qi
Quoted 32 lines · Jason Wang
Quoted 20 lines · Heng Qi

XDP core assumes that the frame_size of xdp_buff and the length of the frag are PAGE_SIZE. The hole may cause the processing of xdp to fail, so we disable the hole mechanism when xdp is set.

---
  drivers/net/virtio_net.c | 5 ++++-

  1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +1 -0
index 9cce7dec7366..443aa7b8f0ad 100644
@@ -1419,8 +1419,11 @@ static int add_recvbuf_mergeable(struct 
 virtnet_info *vi,
           /* To avoid internal fragmentation, if there is very likely not
            * enough space for another buffer, add the remaining space to
            * the current buffer.
+         * XDP core assumes that frame_size of xdp_buff and the 
Quoted 7 lines

length
+         * of the frag are PAGE_SIZE, so we disable the hole mechanism.
           */
-        len += hole;
+        if (!headroom)
+            len += hole;

Is this only a requirement of multi-buffer XDP? If not, it need to be backported to stable.

Quoted 6 lines
Quoted 3 lines

It applies to single buffer xdp and multi-buffer xdp, but even if single buffer xdp has a hole mechanism, there will be no problem (limiting mtu and turning off GUEST GSO), so there is no need to backport it.

Let's add this in the changelog.

Ok, thanks for your energy.

Quoted 18 lines

With that,

Thanks

Quoted 10 lines

Thanks.

Quoted 6 lines

Thanks

Quoted 2 lines

          alloc_frag->offset += hole;
      }

author
#6 lore raw

PATCH v2 2/9 virtio_net: set up xdp for multi buffer packets

In reply to Heng Qi

When the xdp program sets xdp.frags, which means it can process multi-buffer packets over larger MTU, so we continue to support xdp. But for single-buffer xdp, we should keep checking for MTU.

---
 drivers/net/virtio_net.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +2 -2
index 443aa7b8f0ad..c5c4e9db4ed3 100644
@@ -3095,8 +3095,8 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog,
 		return -EINVAL;
 	}
 
-	if (dev->mtu > max_sz) {
-		NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP");
+	if (prog && !prog->aux->xdp_has_frags && dev->mtu > max_sz) {
+		NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP without frags");
 		netdev_warn(dev, "XDP requires MTU less than %lu\n", max_sz);
 		return -EINVAL;
 	}
Signature · 1 line
2.19.1.6.gb485710b
#7 lore raw

Re: [PATCH v2 2/9] virtio_net: set up xdp for multi buffer packets

Quoted 15 lines · Heng Qi

When the xdp program sets xdp.frags, which means it can process multi-buffer packets over larger MTU, so we continue to support xdp. But for single-buffer xdp, we should keep checking for MTU.

---
  drivers/net/virtio_net.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +0 -0
index 443aa7b8f0ad..c5c4e9db4ed3 100644

drivers/net/virtio_net.c static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog, · line 3098

Quoted 6 lines
drivers/net/virtio_net.c +1 -2
@@ -3095,8 +3095,8 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog,
  		return -EINVAL;
  	}
 
-	if (dev->mtu > max_sz) {
-		NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP");
+	if (prog && !prog->aux->xdp_has_frags && dev->mtu > max_sz) {

Not related to this patch, but I see:

        unsigned long int max_sz = PAGE_SIZE - sizeof(struct padded_vnet_hdr);

Which is suspicious, do we need to count reserved headroom/tailroom as well?

Thanks

Quoted 4 lines
drivers/net/virtio_net.c +1 -0
+		NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP without frags");
  		netdev_warn(dev, "XDP requires MTU less than %lu\n", max_sz);
  		return -EINVAL;
  	}
author
#8 lore raw

Re: [PATCH v2 2/9] virtio_net: set up xdp for multi buffer packets

Quoted 27 lines · Jason Wang
Quoted 15 lines · Heng Qi

When the xdp program sets xdp.frags, which means it can process multi-buffer packets over larger MTU, so we continue to support xdp. But for single-buffer xdp, we should keep checking for MTU.

---
  drivers/net/virtio_net.c | 4 ++--

  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +0 -0
index 443aa7b8f0ad..c5c4e9db4ed3 100644

drivers/net/virtio_net.c static int virtnet_xdp_set(struct net_device · line 3099

Quoted 6 lines
drivers/net/virtio_net.c +1 -1
@@ -3095,8 +3095,8 @@ static int virtnet_xdp_set(struct net_device 
 *dev, struct bpf_prog *prog,
           return -EINVAL;
       }
   -    if (dev->mtu > max_sz) {
-        NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP");
+    if (prog && !prog->aux->xdp_has_frags && dev->mtu > max_sz) {

Not related to this patch, but I see:

Quoted 3 lines

        unsigned long int max_sz = PAGE_SIZE - sizeof(struct padded_vnet_hdr);

Which is suspicious, do we need to count reserved headroom/tailroom as well?

This seems to be suspicious. After loading xdp, the size of the filled avail buffer is (PAGE_SIZE - headroom - tailroom), so the size of the received used buffer, ie MTU, should also be (PAGE_SIZE - headroom - tailroom).

Thanks.

Quoted 9 lines

Thanks

Quoted 5 lines
drivers/net/virtio_net.c +1 -0
+        NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP 
 without frags");
           netdev_warn(dev, "XDP requires MTU less than %lu\n", max_sz);

          return -EINVAL;
      }

author
#9 lore raw

Re: [PATCH v2 2/9] virtio_net: set up xdp for multi buffer packets

Quoted 30 lines · Heng Qi
Quoted 27 lines · Jason Wang
Quoted 15 lines · Heng Qi

When the xdp program sets xdp.frags, which means it can process multi-buffer packets over larger MTU, so we continue to support xdp. But for single-buffer xdp, we should keep checking for MTU.

---
  drivers/net/virtio_net.c | 4 ++--

  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +0 -0
index 443aa7b8f0ad..c5c4e9db4ed3 100644

drivers/net/virtio_net.c static int virtnet_xdp_set(struct net_device · line 3099

Quoted 6 lines
drivers/net/virtio_net.c +1 -1
@@ -3095,8 +3095,8 @@ static int virtnet_xdp_set(struct net_device 
 *dev, struct bpf_prog *prog,
           return -EINVAL;
       }
   -    if (dev->mtu > max_sz) {
-        NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP");
+    if (prog && !prog->aux->xdp_has_frags && dev->mtu > max_sz) {

Not related to this patch, but I see:

Quoted 7 lines
Quoted 3 lines

        unsigned long int max_sz = PAGE_SIZE - sizeof(struct padded_vnet_hdr);

Which is suspicious, do we need to count reserved headroom/tailroom as well?

This seems to be suspicious. After loading xdp, the size of the filled avail buffer is (PAGE_SIZE - headroom - tailroom), so the size of the received used buffer, ie MTU, should also be (PAGE_SIZE - headroom - tailroom).

Hi Jason, this is indeed a problem. After verification, packet drop will indeed occur.  To avoid this, the size of MTU should be (PAGE_SIZE - headroom - tailroom - ethhdr = 4096 - 256 -320 - 14 =3506). Because when there is xdp, each filling is 3520 (PAGE_SIZE - room), if the value of (MTU + 14) is greater than 3520 (because the MTU does not contain the ethernet header), then the packet with a length greater than 3520 will come in, so num_buf will still be greater than or equal to 2, and then xdp_linearize_page() will be performed and the packet will be dropped because the total length is greater than PAGE_SIZE.

I will make a separate bugfix patch to fix this later.

Thanks.

Quoted 12 lines

Thanks.

Quoted 9 lines

Thanks

Quoted 5 lines
drivers/net/virtio_net.c +1 -0
+        NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP 
 without frags");
           netdev_warn(dev, "XDP requires MTU less than %lu\n", max_sz);

          return -EINVAL;
      }

#10 lore raw

Re: [PATCH v2 2/9] virtio_net: set up xdp for multi buffer packets

Quoted 49 lines · Heng Qi
Quoted 30 lines · Heng Qi
Quoted 27 lines · Jason Wang
Quoted 15 lines · Heng Qi

When the xdp program sets xdp.frags, which means it can process multi-buffer packets over larger MTU, so we continue to support xdp. But for single-buffer xdp, we should keep checking for MTU.

---
  drivers/net/virtio_net.c | 4 ++--

  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +0 -0
index 443aa7b8f0ad..c5c4e9db4ed3 100644

drivers/net/virtio_net.c static int virtnet_xdp_set(struct net_device · line 3099

Quoted 6 lines
drivers/net/virtio_net.c +1 -1
@@ -3095,8 +3095,8 @@ static int virtnet_xdp_set(struct net_device 
 *dev, struct bpf_prog *prog,
           return -EINVAL;
       }
   -    if (dev->mtu > max_sz) {
-        NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP");
+    if (prog && !prog->aux->xdp_has_frags && dev->mtu > max_sz) {

Not related to this patch, but I see:

Quoted 7 lines
Quoted 3 lines

        unsigned long int max_sz = PAGE_SIZE - sizeof(struct padded_vnet_hdr);

Which is suspicious, do we need to count reserved headroom/tailroom as well?

This seems to be suspicious. After loading xdp, the size of the filled avail buffer is (PAGE_SIZE - headroom - tailroom), so the size of the received used buffer, ie MTU, should also be (PAGE_SIZE - headroom - tailroom).

Hi Jason, this is indeed a problem. After verification, packet drop will indeed occur.  To avoid this, the size of MTU should be (PAGE_SIZE - headroom - tailroom - ethhdr = 4096 - 256 -320 - 14 =3506). Because when there is xdp, each filling is 3520 (PAGE_SIZE - room), if the value of (MTU + 14) is greater than 3520 (because the MTU does not contain the ethernet header), then the packet with a length greater than 3520 will come in, so num_buf will still be greater than or equal to 2, and then xdp_linearize_page() will be performed and the packet will be dropped because the total length is greater than PAGE_SIZE.

Quoted 1 line

I will make a separate bugfix patch to fix this later.

Great.

Thanks

Quoted 16 lines

Thanks.

Quoted 12 lines

Thanks.

Quoted 9 lines

Thanks

Quoted 5 lines
drivers/net/virtio_net.c +1 -0
+        NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP 
 without frags");
           netdev_warn(dev, "XDP requires MTU less than %lu\n", max_sz);

          return -EINVAL;
      }

author
#11 lore raw

PATCH v2 3/9 virtio_net: update bytes calculation for xdp_frame

In reply to Heng Qi

Update relative record value for xdp_frame as basis for multi-buffer xdp transmission.

---
 drivers/net/virtio_net.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +2 -2
index c5c4e9db4ed3..08f209d7b0bf 100644
@@ -658,7 +658,7 @@ static int virtnet_xdp_xmit(struct net_device *dev,
 		if (likely(is_xdp_frame(ptr))) {
 			struct xdp_frame *frame = ptr_to_xdp(ptr);
 
-			bytes += frame->len;
+			bytes += xdp_get_frame_len(frame);
 			xdp_return_frame(frame);
 		} else {
 			struct sk_buff *skb = ptr;
@@ -1604,7 +1604,7 @@ static void free_old_xmit_skbs(struct send_queue *sq, bool in_napi)
 		} else {
 			struct xdp_frame *frame = ptr_to_xdp(ptr);
 
-			bytes += frame->len;
+			bytes += xdp_get_frame_len(frame);
 			xdp_return_frame(frame);
 		}
 		packets++;
Signature · 1 line
2.19.1.6.gb485710b
author
#12 lore raw

PATCH v2 4/9 virtio_net: build xdp_buff with multi buffers

In reply to Heng Qi

Support xdp for multi buffer packets in mergeable mode.

Putting the first buffer as the linear part for xdp_buff, and the rest of the buffers as non-linear fragments to struct skb_shared_info in the tailroom belonging to xdp_buff.

---
 drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
drivers/net/virtio_net.c +78 -0
index 08f209d7b0bf..8fc3b1841d92 100644
@@ -931,6 +931,84 @@ static struct sk_buff *receive_big(struct net_device *dev,
 	return NULL;
 }
 
+/* TODO: build xdp in big mode */
+static int virtnet_build_xdp_buff_mrg(struct net_device *dev,
+				      struct virtnet_info *vi,
+				      struct receive_queue *rq,
+				      struct xdp_buff *xdp,
+				      void *buf,
+				      unsigned int len,
+				      unsigned int frame_sz,
+				      u16 *num_buf,
+				      unsigned int *xdp_frags_truesize,
+				      struct virtnet_rq_stats *stats)
+{
+	unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+	struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
+	unsigned int truesize, headroom;
+	struct skb_shared_info *shinfo;
+	unsigned int xdp_frags_truesz = 0;
+	unsigned int cur_frag_size;
+	struct page *page;
+	skb_frag_t *frag;
+	int offset;
+	void *ctx;
+
+	xdp_init_buff(xdp, frame_sz, &rq->xdp_rxq);
+	xdp_prepare_buff(xdp, buf - VIRTIO_XDP_HEADROOM,
+			 VIRTIO_XDP_HEADROOM + vi->hdr_len, len - vi->hdr_len, true);
+
+	if (*num_buf > 1) {
+		shinfo = xdp_get_shared_info_from_buff(xdp);
+		shinfo->nr_frags = 0;
+		shinfo->xdp_frags_size = 0;
+	}
+
+	if ((*num_buf - 1) > MAX_SKB_FRAGS)
+		return -EINVAL;
+
+	while ((--*num_buf) >= 1) {
+		buf = virtqueue_get_buf_ctx(rq->vq, &len, &ctx);
+		if (unlikely(!buf)) {
+			pr_debug("%s: rx error: %d buffers out of %d missing\n",
+				 dev->name, *num_buf,
+				 virtio16_to_cpu(vi->vdev, hdr->num_buffers));
+			dev->stats.rx_length_errors++;
+			return -EINVAL;
+		}
+
+		if (!xdp_buff_has_frags(xdp))
+			xdp_buff_set_frags_flag(xdp);
+
+		stats->bytes += len;
+		page = virt_to_head_page(buf);
+		offset = buf - page_address(page);
+		truesize = mergeable_ctx_to_truesize(ctx);
+		headroom = mergeable_ctx_to_headroom(ctx);
+
+		cur_frag_size = truesize + (headroom ? (headroom + tailroom) : 0);
+		xdp_frags_truesz += cur_frag_size;
+		if (unlikely(len > truesize || cur_frag_size > PAGE_SIZE)) {
+			pr_debug("%s: rx error: len %u exceeds truesize %lu\n",
+				 dev->name, len, (unsigned long)ctx);
+			dev->stats.rx_length_errors++;
+			return -EINVAL;
+		}
+
+		frag = &shinfo->frags[shinfo->nr_frags++];
+		__skb_frag_set_page(frag, page);
+		skb_frag_off_set(frag, offset);
+		skb_frag_size_set(frag, len);
+		if (page_is_pfmemalloc(page))
+			xdp_buff_set_frag_pfmemalloc(xdp);
+
+		shinfo->xdp_frags_size += len;
+	}
+
+	*xdp_frags_truesize = xdp_frags_truesz;
+	return 0;
+}
+
 static struct sk_buff *receive_mergeable(struct net_device *dev,
 					 struct virtnet_info *vi,
 					 struct receive_queue *rq,
Signature · 1 line
2.19.1.6.gb485710b
#13 lore raw

Re: [PATCH v2 4/9] virtio_net: build xdp_buff with multi buffers

Quoted 58 lines · Heng Qi

Support xdp for multi buffer packets in mergeable mode.

Putting the first buffer as the linear part for xdp_buff, and the rest of the buffers as non-linear fragments to struct skb_shared_info in the tailroom belonging to xdp_buff.

---
  drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++++++++++++++++++
  1 file changed, 78 insertions(+)
drivers/net/virtio_net.c +38 -0
index 08f209d7b0bf..8fc3b1841d92 100644
@@ -931,6 +931,84 @@ static struct sk_buff *receive_big(struct net_device *dev,
  	return NULL;
  }
 
+/* TODO: build xdp in big mode */
+static int virtnet_build_xdp_buff_mrg(struct net_device *dev,
+				      struct virtnet_info *vi,
+				      struct receive_queue *rq,
+				      struct xdp_buff *xdp,
+				      void *buf,
+				      unsigned int len,
+				      unsigned int frame_sz,
+				      u16 *num_buf,
+				      unsigned int *xdp_frags_truesize,
+				      struct virtnet_rq_stats *stats)
+{
+	unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+	struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
+	unsigned int truesize, headroom;
+	struct skb_shared_info *shinfo;
+	unsigned int xdp_frags_truesz = 0;
+	unsigned int cur_frag_size;
+	struct page *page;
+	skb_frag_t *frag;
+	int offset;
+	void *ctx;
+
+	xdp_init_buff(xdp, frame_sz, &rq->xdp_rxq);
+	xdp_prepare_buff(xdp, buf - VIRTIO_XDP_HEADROOM,
+			 VIRTIO_XDP_HEADROOM + vi->hdr_len, len - vi->hdr_len, true);
+
+	if (*num_buf > 1) {
+		shinfo = xdp_get_shared_info_from_buff(xdp);
+		shinfo->nr_frags = 0;
+		shinfo->xdp_frags_size = 0;
+	}
+
+	if ((*num_buf - 1) > MAX_SKB_FRAGS)
+		return -EINVAL;
+
+	while ((--*num_buf) >= 1) {
+		buf = virtqueue_get_buf_ctx(rq->vq, &len, &ctx);

drivers/net/virtio_net.c static struct sk_buff *receive_big(struct net_device *dev, · line 981

Quoted 10 lines
drivers/net/virtio_net.c +10 -0
+		if (unlikely(!buf)) {
+			pr_debug("%s: rx error: %d buffers out of %d missing\n",
+				 dev->name, *num_buf,
+				 virtio16_to_cpu(vi->vdev, hdr->num_buffers));
+			dev->stats.rx_length_errors++;
+			return -EINVAL;
+		}
+
+		if (!xdp_buff_has_frags(xdp))
+			xdp_buff_set_frags_flag(xdp);

Any reason to put this inside the loop?

drivers/net/virtio_net.c static struct sk_buff *receive_big(struct net_device *dev, · line 990

Quoted 9 lines
drivers/net/virtio_net.c +9 -0
+
+		stats->bytes += len;
+		page = virt_to_head_page(buf);
+		offset = buf - page_address(page);
+		truesize = mergeable_ctx_to_truesize(ctx);
+		headroom = mergeable_ctx_to_headroom(ctx);
+
+		cur_frag_size = truesize + (headroom ? (headroom + tailroom) : 0);
+		xdp_frags_truesz += cur_frag_size;

Not related to this patch, but it would easily confuse the future readers that the we need another math for truesize. I think at least we need some comments for this or

I guess the root cause is in get_mergeable_buf_len:

static unsigned int get_mergeable_buf_len(struct receive_queue *rq, struct ewma_pkt_len *avg_pkt_len, unsigned int room) {
        struct virtnet_info *vi = rq->vq->vdev->priv;
        const size_t hdr_len = vi->hdr_len;
        unsigned int len;

        if (room)
        return PAGE_SIZE - room;

And we do

    len = get_mergeable_buf_len(rq, &rq->mrg_avg_pkt_len, room);

    ...

    ctx = mergeable_len_to_ctx(len, headroom);

I wonder if it's better to pack the real truesize (PAGE_SIZE) here. This may ease a lot of things.

Thanks

Quoted 24 lines
drivers/net/virtio_net.c +21 -0
+		if (unlikely(len > truesize || cur_frag_size > PAGE_SIZE)) {
+			pr_debug("%s: rx error: len %u exceeds truesize %lu\n",
+				 dev->name, len, (unsigned long)ctx);
+			dev->stats.rx_length_errors++;
+			return -EINVAL;
+		}
+
+		frag = &shinfo->frags[shinfo->nr_frags++];
+		__skb_frag_set_page(frag, page);
+		skb_frag_off_set(frag, offset);
+		skb_frag_size_set(frag, len);
+		if (page_is_pfmemalloc(page))
+			xdp_buff_set_frag_pfmemalloc(xdp);
+
+		shinfo->xdp_frags_size += len;
+	}
+
+	*xdp_frags_truesize = xdp_frags_truesz;
+	return 0;
+}
+
  static struct sk_buff *receive_mergeable(struct net_device *dev,
  					 struct virtnet_info *vi,
  					 struct receive_queue *rq,
author
#14 lore raw

Re: [PATCH v2 4/9] virtio_net: build xdp_buff with multi buffers

Quoted 67 lines · Jason Wang
Quoted 65 lines · Heng Qi

Support xdp for multi buffer packets in mergeable mode.

Putting the first buffer as the linear part for xdp_buff, and the rest of the buffers as non-linear fragments to struct skb_shared_info in the tailroom belonging to xdp_buff.

---
  drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++++++++++++++++++

  1 file changed, 78 insertions(+)
drivers/net/virtio_net.c +0 -0
index 08f209d7b0bf..8fc3b1841d92 100644
@@ -931,6 +931,84 @@ static struct sk_buff *receive_big(struct 
 net_device *dev,
       return NULL;
   }

  +/* TODO: build xdp in big mode */

drivers/net/virtio_net.c +12 -0
+static int virtnet_build_xdp_buff_mrg(struct net_device *dev,
+                      struct virtnet_info *vi,
+                      struct receive_queue *rq,
+                      struct xdp_buff *xdp,
+                      void *buf,
+                      unsigned int len,
+                      unsigned int frame_sz,
+                      u16 *num_buf,
+                      unsigned int *xdp_frags_truesize,
+                      struct virtnet_rq_stats *stats)
+{
+    unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct 

skb_shared_info));
+    struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
+    unsigned int truesize, headroom;
+    struct skb_shared_info *shinfo;
+    unsigned int xdp_frags_truesz = 0;
+    unsigned int cur_frag_size;
+    struct page *page;
+    skb_frag_t *frag;
+    int offset;
+    void *ctx;
+
+    xdp_init_buff(xdp, frame_sz, &rq->xdp_rxq);
+    xdp_prepare_buff(xdp, buf - VIRTIO_XDP_HEADROOM,
+             VIRTIO_XDP_HEADROOM + vi->hdr_len, len - vi->hdr_len, true);

drivers/net/virtio_net.c +17 -0
+
+    if (*num_buf > 1) {
+        shinfo = xdp_get_shared_info_from_buff(xdp);
+        shinfo->nr_frags = 0;
+        shinfo->xdp_frags_size = 0;
+    }
+
+    if ((*num_buf - 1) > MAX_SKB_FRAGS)
+        return -EINVAL;
+
+    while ((--*num_buf) >= 1) {
+        buf = virtqueue_get_buf_ctx(rq->vq, &len, &ctx);
+        if (unlikely(!buf)) {
+            pr_debug("%s: rx error: %d buffers out of %d missing\n",
+                 dev->name, *num_buf,
+                 virtio16_to_cpu(vi->vdev, hdr->num_buffers));
+            dev->stats.rx_length_errors++;
Quoted 8 lines
Quoted 5 lines

+            return -EINVAL;
+        }
+
+        if (!xdp_buff_has_frags(xdp))
+            xdp_buff_set_frags_flag(xdp);

Any reason to put this inside the loop?

I'll move it outside the loop in the next version.

Quoted 15 lines
Quoted 10 lines
drivers/net/virtio_net.c +8 -0
+
+        stats->bytes += len;
+        page = virt_to_head_page(buf);
+        offset = buf - page_address(page);
+        truesize = mergeable_ctx_to_truesize(ctx);
+        headroom = mergeable_ctx_to_headroom(ctx);
+
+        cur_frag_size = truesize + (headroom ? (headroom + tailroom) 
Quoted 2 lines

: 0);
+        xdp_frags_truesz += cur_frag_size;

Not related to this patch, but it would easily confuse the future readers that the we need another math for truesize. I think at least we need some comments for this or

Yes it might, I'll add more comments on this.

Quoted 16 lines

I guess the root cause is in get_mergeable_buf_len:

static unsigned int get_mergeable_buf_len(struct receive_queue *rq, struct ewma_pkt_len *avg_pkt_len, unsigned int room) {
        struct virtnet_info *vi = rq->vq->vdev->priv;
        const size_t hdr_len = vi->hdr_len;
        unsigned int len;

        if (room)
        return PAGE_SIZE - room;

And we do

Quoted 8 lines

    len = get_mergeable_buf_len(rq, &rq->mrg_avg_pkt_len, room);

    ...

    ctx = mergeable_len_to_ctx(len, headroom);

I wonder if it's better to pack the real truesize (PAGE_SIZE) here. This may ease a lot of things.

I don't know the historical reason for not packing, but I guess this is for the convenience of comparing the actual length len given by the device with truesize. Therefore, I think it would be better to keep the above practice for now? Perhaps, I can add more explanation to the above code to help future readers try not to get confused.

Thanks.

Quoted 28 lines

Thanks

Quoted 24 lines

+        if (unlikely(len > truesize || cur_frag_size > PAGE_SIZE)) { +            pr_debug("%s: rx error: len %u exceeds truesize %lu\n", +                 dev->name, len, (unsigned long)ctx);
+            dev->stats.rx_length_errors++;
+            return -EINVAL;
+        }

drivers/net/virtio_net.c +15 -0
+
+        frag = &shinfo->frags[shinfo->nr_frags++];
+        __skb_frag_set_page(frag, page);
+        skb_frag_off_set(frag, offset);
+        skb_frag_size_set(frag, len);
+        if (page_is_pfmemalloc(page))
+            xdp_buff_set_frag_pfmemalloc(xdp);
+
+        shinfo->xdp_frags_size += len;
+    }
+
+    *xdp_frags_truesize = xdp_frags_truesz;
+    return 0;
+}
+

  static struct sk_buff *receive_mergeable(struct net_device *dev, struct virtnet_info *vi,
                       struct receive_queue *rq,

#15 lore raw

Re: [PATCH v2 4/9] virtio_net: build xdp_buff with multi buffers

Quoted 120 lines · Heng Qi
Quoted 68 lines · Jason Wang
Quoted 66 lines · Heng Qi

Support xdp for multi buffer packets in mergeable mode.

Putting the first buffer as the linear part for xdp_buff, and the rest of the buffers as non-linear fragments to struct skb_shared_info in the tailroom belonging to xdp_buff.

---
  drivers/net/virtio_net.c | 78
++++++++++++++++++++++++++++++++++++++++

  1 file changed, 78 insertions(+)
drivers/net/virtio_net.c +0 -0
index 08f209d7b0bf..8fc3b1841d92 100644
@@ -931,6 +931,84 @@ static struct sk_buff *receive_big(struct 
 net_device *dev,
       return NULL;
   }

  +/* TODO: build xdp in big mode */

drivers/net/virtio_net.c +12 -0
+static int virtnet_build_xdp_buff_mrg(struct net_device *dev,
+                      struct virtnet_info *vi,
+                      struct receive_queue *rq,
+                      struct xdp_buff *xdp,
+                      void *buf,
+                      unsigned int len,
+                      unsigned int frame_sz,
+                      u16 *num_buf,
+                      unsigned int *xdp_frags_truesize,
+                      struct virtnet_rq_stats *stats)
+{
+    unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct 

skb_shared_info));
+    struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
+    unsigned int truesize, headroom;
+    struct skb_shared_info *shinfo;
+    unsigned int xdp_frags_truesz = 0;
+    unsigned int cur_frag_size;
+    struct page *page;
+    skb_frag_t *frag;
+    int offset;
+    void *ctx;
+
+    xdp_init_buff(xdp, frame_sz, &rq->xdp_rxq);
+    xdp_prepare_buff(xdp, buf - VIRTIO_XDP_HEADROOM,
+             VIRTIO_XDP_HEADROOM + vi->hdr_len, len - vi->hdr_len, true);

drivers/net/virtio_net.c +17 -0
+
+    if (*num_buf > 1) {
+        shinfo = xdp_get_shared_info_from_buff(xdp);
+        shinfo->nr_frags = 0;
+        shinfo->xdp_frags_size = 0;
+    }
+
+    if ((*num_buf - 1) > MAX_SKB_FRAGS)
+        return -EINVAL;
+
+    while ((--*num_buf) >= 1) {
+        buf = virtqueue_get_buf_ctx(rq->vq, &len, &ctx);
+        if (unlikely(!buf)) {
+            pr_debug("%s: rx error: %d buffers out of %d missing\n",
+                 dev->name, *num_buf,
+                 virtio16_to_cpu(vi->vdev, hdr->num_buffers));
+            dev->stats.rx_length_errors++;
Quoted 8 lines
Quoted 5 lines

+            return -EINVAL;
+        }
+
+        if (!xdp_buff_has_frags(xdp))
+            xdp_buff_set_frags_flag(xdp);

Any reason to put this inside the loop?

I'll move it outside the loop in the next version.

Quoted 15 lines
Quoted 10 lines
drivers/net/virtio_net.c +8 -0
+
+        stats->bytes += len;
+        page = virt_to_head_page(buf);
+        offset = buf - page_address(page);
+        truesize = mergeable_ctx_to_truesize(ctx);
+        headroom = mergeable_ctx_to_headroom(ctx);
+
+        cur_frag_size = truesize + (headroom ? (headroom + 
Quoted 2 lines

tailroom) : 0);
+        xdp_frags_truesz += cur_frag_size;

Not related to this patch, but it would easily confuse the future readers that the we need another math for truesize. I think at least we need some comments for this or

Yes it might, I'll add more comments on this.

Quoted 20 lines

I guess the root cause is in get_mergeable_buf_len:

static unsigned int get_mergeable_buf_len(struct receive_queue *rq, struct ewma_pkt_len *avg_pkt_len, unsigned int room) {
        struct virtnet_info *vi = rq->vq->vdev->priv;
        const size_t hdr_len = vi->hdr_len;
        unsigned int len;

        if (room)
        return PAGE_SIZE - room;

And we do

    len = get_mergeable_buf_len(rq, &rq->mrg_avg_pkt_len, room);

    ...

Quoted 8 lines
Quoted 4 lines

    ctx = mergeable_len_to_ctx(len, headroom);

I wonder if it's better to pack the real truesize (PAGE_SIZE) here. This may ease a lot of things.

I don't know the historical reason for not packing, but I guess this is for the convenience of comparing the actual length len given by the device with truesize. Therefore, I think it would be better to keep the above practice for now?

The problem is:

We had

static unsigned int mergeable_ctx_to_truesize(void *mrg_ctx)

It means the truesize should be calculated before packing into the context. So having another math to get truesize is very confusing, I think it's better to fix that. Otherwise we may end up code that is hard to be reviewed even with the help of comments.

Thanks

Quoted 34 lines

Perhaps, I can add more explanation to the
above code to help future readers try not to get confused.

Thanks.

Quoted 28 lines

Thanks

Quoted 24 lines

+        if (unlikely(len > truesize || cur_frag_size > PAGE_SIZE)) { +            pr_debug("%s: rx error: len %u exceeds truesize %lu\n", +                 dev->name, len, (unsigned long)ctx);
+            dev->stats.rx_length_errors++;
+            return -EINVAL;
+        }

drivers/net/virtio_net.c +15 -0
+
+        frag = &shinfo->frags[shinfo->nr_frags++];
+        __skb_frag_set_page(frag, page);
+        skb_frag_off_set(frag, offset);
+        skb_frag_size_set(frag, len);
+        if (page_is_pfmemalloc(page))
+            xdp_buff_set_frag_pfmemalloc(xdp);
+
+        shinfo->xdp_frags_size += len;
+    }
+
+    *xdp_frags_truesize = xdp_frags_truesz;
+    return 0;
+}
+

  static struct sk_buff *receive_mergeable(struct net_device *dev, struct virtnet_info *vi,
                       struct receive_queue *rq,

author
#16 lore raw

Re: [PATCH v2 4/9] virtio_net: build xdp_buff with multi buffers

Quoted 132 lines · Jason Wang
Quoted 120 lines · Heng Qi
Quoted 68 lines · Jason Wang
Quoted 66 lines · Heng Qi

Support xdp for multi buffer packets in mergeable mode.

Putting the first buffer as the linear part for xdp_buff, and the rest of the buffers as non-linear fragments to struct skb_shared_info in the tailroom belonging to xdp_buff.

---
  drivers/net/virtio_net.c | 78
++++++++++++++++++++++++++++++++++++++++

  1 file changed, 78 insertions(+)
drivers/net/virtio_net.c +0 -0
index 08f209d7b0bf..8fc3b1841d92 100644
@@ -931,6 +931,84 @@ static struct sk_buff *receive_big(struct 
 net_device *dev,
       return NULL;
   }

  +/* TODO: build xdp in big mode */

drivers/net/virtio_net.c +12 -0
+static int virtnet_build_xdp_buff_mrg(struct net_device *dev,
+                      struct virtnet_info *vi,
+                      struct receive_queue *rq,
+                      struct xdp_buff *xdp,
+                      void *buf,
+                      unsigned int len,
+                      unsigned int frame_sz,
+                      u16 *num_buf,
+                      unsigned int *xdp_frags_truesize,
+                      struct virtnet_rq_stats *stats)
+{
+    unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct 

skb_shared_info));
+    struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
+    unsigned int truesize, headroom;
+    struct skb_shared_info *shinfo;
+    unsigned int xdp_frags_truesz = 0;
+    unsigned int cur_frag_size;
+    struct page *page;
+    skb_frag_t *frag;
+    int offset;
+    void *ctx;
+
+    xdp_init_buff(xdp, frame_sz, &rq->xdp_rxq);
+    xdp_prepare_buff(xdp, buf - VIRTIO_XDP_HEADROOM,
+             VIRTIO_XDP_HEADROOM + vi->hdr_len, len - vi->hdr_len, true);

drivers/net/virtio_net.c +17 -0
+
+    if (*num_buf > 1) {
+        shinfo = xdp_get_shared_info_from_buff(xdp);
+        shinfo->nr_frags = 0;
+        shinfo->xdp_frags_size = 0;
+    }
+
+    if ((*num_buf - 1) > MAX_SKB_FRAGS)
+        return -EINVAL;
+
+    while ((--*num_buf) >= 1) {
+        buf = virtqueue_get_buf_ctx(rq->vq, &len, &ctx);
+        if (unlikely(!buf)) {
+            pr_debug("%s: rx error: %d buffers out of %d missing\n",
+                 dev->name, *num_buf,
+                 virtio16_to_cpu(vi->vdev, hdr->num_buffers));
+            dev->stats.rx_length_errors++;
Quoted 8 lines
Quoted 5 lines

+            return -EINVAL;
+        }
+
+        if (!xdp_buff_has_frags(xdp))
+            xdp_buff_set_frags_flag(xdp);

Any reason to put this inside the loop?

I'll move it outside the loop in the next version.

Quoted 15 lines
Quoted 10 lines
drivers/net/virtio_net.c +8 -0
+
+        stats->bytes += len;
+        page = virt_to_head_page(buf);
+        offset = buf - page_address(page);
+        truesize = mergeable_ctx_to_truesize(ctx);
+        headroom = mergeable_ctx_to_headroom(ctx);
+
+        cur_frag_size = truesize + (headroom ? (headroom + 
Quoted 2 lines

tailroom) : 0);
+        xdp_frags_truesz += cur_frag_size;

Not related to this patch, but it would easily confuse the future readers that the we need another math for truesize. I think at least we need some comments for this or

Yes it might, I'll add more comments on this.

Quoted 20 lines

I guess the root cause is in get_mergeable_buf_len:

static unsigned int get_mergeable_buf_len(struct receive_queue *rq, struct ewma_pkt_len *avg_pkt_len, unsigned int room) {
        struct virtnet_info *vi = rq->vq->vdev->priv;
        const size_t hdr_len = vi->hdr_len;
        unsigned int len;

        if (room)
        return PAGE_SIZE - room;

And we do

    len = get_mergeable_buf_len(rq, &rq->mrg_avg_pkt_len, room);

    ...

Quoted 8 lines
Quoted 4 lines

    ctx = mergeable_len_to_ctx(len, headroom);

I wonder if it's better to pack the real truesize (PAGE_SIZE) here. This may ease a lot of things.

I don't know the historical reason for not packing, but I guess this is for the convenience of comparing the actual length len given by the device with truesize. Therefore, I think it would be better to keep the above practice for now?

Quoted 7 lines

The problem is:

We had

static unsigned int mergeable_ctx_to_truesize(void *mrg_ctx)

It means the truesize should be calculated before packing into the context. So having another math to get truesize is very confusing, I think it's better to fix that. Otherwise we may end up code that is hard to be reviewed even with the help of comments.

It is reasonable to let truesize return to its literal meaning, which includes the length of headroom and tailroom, and I will modify this in the next version.

Thanks.

Quoted 38 lines

Thanks

Quoted 34 lines

Perhaps, I can add more explanation to the
above code to help future readers try not to get confused.

Thanks.

Quoted 28 lines

Thanks

Quoted 24 lines

+        if (unlikely(len > truesize || cur_frag_size > PAGE_SIZE)) { +            pr_debug("%s: rx error: len %u exceeds truesize %lu\n", +                 dev->name, len, (unsigned long)ctx);
+            dev->stats.rx_length_errors++;
+            return -EINVAL;
+        }

drivers/net/virtio_net.c +15 -0
+
+        frag = &shinfo->frags[shinfo->nr_frags++];
+        __skb_frag_set_page(frag, page);
+        skb_frag_off_set(frag, offset);
+        skb_frag_size_set(frag, len);
+        if (page_is_pfmemalloc(page))
+            xdp_buff_set_frag_pfmemalloc(xdp);
+
+        shinfo->xdp_frags_size += len;
+    }
+
+    *xdp_frags_truesize = xdp_frags_truesz;
+    return 0;
+}
+

  static struct sk_buff *receive_mergeable(struct net_device *dev, struct virtnet_info *vi,
                       struct receive_queue *rq,

author
#17 lore raw

PATCH v2 5/9 virtio_net: construct multi-buffer xdp in mergeable

In reply to Heng Qi

Build multi-buffer xdp using virtnet_build_xdp_buff_mrg().

For the prefilled buffer before xdp is set, we will probably use vq reset in the future. At the same time, virtio net currently uses comp pages, and bpf_xdp_frags_increase_tail() needs to calculate the tailroom of the last frag, which will involve the offset of the corresponding page and cause a negative value, so we disable tail increase by not setting xdp_rxq->frag_size.

---
 drivers/net/virtio_net.c | 60 +++++++++++++++++++++++++++++-----------
 1 file changed, 44 insertions(+), 16 deletions(-)
drivers/net/virtio_net.c +44 -16
index 8fc3b1841d92..40bc58fa57f5 100644
@@ -1018,6 +1018,7 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 					 unsigned int *xdp_xmit,
 					 struct virtnet_rq_stats *stats)
 {
+	unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
 	struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
 	u16 num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers);
 	struct page *page = virt_to_head_page(buf);
@@ -1048,11 +1049,14 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 	rcu_read_lock();
 	xdp_prog = rcu_dereference(rq->xdp_prog);
 	if (xdp_prog) {
+		unsigned int xdp_frags_truesz = 0;
+		struct skb_shared_info *shinfo;
 		struct xdp_frame *xdpf;
 		struct page *xdp_page;
 		struct xdp_buff xdp;
 		void *data;
 		u32 act;
+		int i;
 
 		/* Transient failure which in theory could occur if
 		 * in-flight packets from before XDP was enabled reach
@@ -1061,19 +1065,23 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 		if (unlikely(hdr->hdr.gso_type))
 			goto err_xdp;
 
-		/* Buffers with headroom use PAGE_SIZE as alloc size,
-		 * see add_recvbuf_mergeable() + get_mergeable_buf_len()
+		/* Now XDP core assumes frag size is PAGE_SIZE, but buffers
+		 * with headroom may add hole in truesize, which
+		 * make their length exceed PAGE_SIZE. So we disabled the
+		 * hole mechanism for xdp. See add_recvbuf_mergeable().
 		 */
 		frame_sz = headroom ? PAGE_SIZE : truesize;
 
-		/* This happens when rx buffer size is underestimated
-		 * or headroom is not enough because of the buffer
-		 * was refilled before XDP is set. This should only
-		 * happen for the first several packets, so we don't
-		 * care much about its performance.
+		/* This happens when headroom is not enough because
+		 * of the buffer was prefilled before XDP is set.
+		 * This should only happen for the first several packets.
+		 * In fact, vq reset can be used here to help us clean up
+		 * the prefilled buffers, but many existing devices do not
+		 * support it, and we don't want to bother users who are
+		 * using xdp normally.
 		 */
-		if (unlikely(num_buf > 1 ||
-			     headroom < virtnet_get_headroom(vi))) {
+		if (!xdp_prog->aux->xdp_has_frags &&
+		    (num_buf > 1 || headroom < virtnet_get_headroom(vi))) {
 			/* linearize data for XDP */
 			xdp_page = xdp_linearize_page(rq, &num_buf,
 						      page, offset,
@@ -1084,17 +1092,26 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 			if (!xdp_page)
 				goto err_xdp;
 			offset = VIRTIO_XDP_HEADROOM;
+		} else if (unlikely(headroom < virtnet_get_headroom(vi))) {
+			if ((VIRTIO_XDP_HEADROOM + len + tailroom) > PAGE_SIZE)
+				goto err_xdp;
+
+			xdp_page = alloc_page(GFP_ATOMIC);
+			if (!xdp_page)
+				goto err_xdp;
+
+			memcpy(page_address(xdp_page) + VIRTIO_XDP_HEADROOM,
+			       page_address(page) + offset, len);
+			frame_sz = PAGE_SIZE;
+			offset = VIRTIO_XDP_HEADROOM;
 		} else {
 			xdp_page = page;
 		}
-
-		/* Allow consuming headroom but reserve enough space to push
-		 * the descriptor on if we get an XDP_TX return code.
-		 */
 		data = page_address(xdp_page) + offset;
-		xdp_init_buff(&xdp, frame_sz - vi->hdr_len, &rq->xdp_rxq);
-		xdp_prepare_buff(&xdp, data - VIRTIO_XDP_HEADROOM + vi->hdr_len,
-				 VIRTIO_XDP_HEADROOM, len - vi->hdr_len, true);
+		err = virtnet_build_xdp_buff_mrg(dev, vi, rq, &xdp, data, len, frame_sz,
+						 &num_buf, &xdp_frags_truesz, stats);
+		if (unlikely(err))
+			goto err_xdp_frags;
 
 		act = bpf_prog_run_xdp(xdp_prog, &xdp);
 		stats->xdp_packets++;
@@ -1190,6 +1207,17 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 				__free_pages(xdp_page, 0);
 			goto err_xdp;
 		}
+err_xdp_frags:
+		shinfo = xdp_get_shared_info_from_buff(&xdp);
+
+		if (unlikely(xdp_page != page))
+			__free_pages(xdp_page, 0);
+
+		for (i = 0; i < shinfo->nr_frags; i++) {
+			xdp_page = skb_frag_page(&shinfo->frags[i]);
+			put_page(xdp_page);
+		}
+		goto err_xdp;
 	}
 	rcu_read_unlock();
 
Signature · 1 line
2.19.1.6.gb485710b
#18 lore raw

Re: [PATCH v2 5/9] virtio_net: construct multi-buffer xdp in mergeable

Quoted 76 lines · Heng Qi

Build multi-buffer xdp using virtnet_build_xdp_buff_mrg().

For the prefilled buffer before xdp is set, we will probably use vq reset in the future. At the same time, virtio net currently uses comp pages, and bpf_xdp_frags_increase_tail() needs to calculate the tailroom of the last frag, which will involve the offset of the corresponding page and cause a negative value, so we disable tail increase by not setting xdp_rxq->frag_size.

---
  drivers/net/virtio_net.c | 60 +++++++++++++++++++++++++++++-----------
  1 file changed, 44 insertions(+), 16 deletions(-)
drivers/net/virtio_net.c +17 -9
index 8fc3b1841d92..40bc58fa57f5 100644
@@ -1018,6 +1018,7 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
  					 unsigned int *xdp_xmit,
  					 struct virtnet_rq_stats *stats)
  {
+	unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
  	struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
  	u16 num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers);
  	struct page *page = virt_to_head_page(buf);
@@ -1048,11 +1049,14 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
  	rcu_read_lock();
  	xdp_prog = rcu_dereference(rq->xdp_prog);
  	if (xdp_prog) {
+		unsigned int xdp_frags_truesz = 0;
+		struct skb_shared_info *shinfo;
  		struct xdp_frame *xdpf;
  		struct page *xdp_page;
  		struct xdp_buff xdp;
  		void *data;
  		u32 act;
+		int i;
 
  		/* Transient failure which in theory could occur if
  		 * in-flight packets from before XDP was enabled reach
@@ -1061,19 +1065,23 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
  		if (unlikely(hdr->hdr.gso_type))
  			goto err_xdp;
 
-		/* Buffers with headroom use PAGE_SIZE as alloc size,
-		 * see add_recvbuf_mergeable() + get_mergeable_buf_len()
+		/* Now XDP core assumes frag size is PAGE_SIZE, but buffers
+		 * with headroom may add hole in truesize, which
+		 * make their length exceed PAGE_SIZE. So we disabled the
+		 * hole mechanism for xdp. See add_recvbuf_mergeable().
  		 */
  		frame_sz = headroom ? PAGE_SIZE : truesize;
 
-		/* This happens when rx buffer size is underestimated
-		 * or headroom is not enough because of the buffer
-		 * was refilled before XDP is set. This should only
-		 * happen for the first several packets, so we don't
-		 * care much about its performance.
+		/* This happens when headroom is not enough because
+		 * of the buffer was prefilled before XDP is set.
+		 * This should only happen for the first several packets.
+		 * In fact, vq reset can be used here to help us clean up
+		 * the prefilled buffers, but many existing devices do not
+		 * support it, and we don't want to bother users who are
+		 * using xdp normally.
  		 */
-		if (unlikely(num_buf > 1 ||
-			     headroom < virtnet_get_headroom(vi))) {
+		if (!xdp_prog->aux->xdp_has_frags &&
+		    (num_buf > 1 || headroom < virtnet_get_headroom(vi))) {
  			/* linearize data for XDP */
  			xdp_page = xdp_linearize_page(rq, &num_buf,
  						      page, offset,

drivers/net/virtio_net.c static struct sk_buff *receive_mergeable(struct net_device *dev, · line 1095

Quoted 4 lines
drivers/net/virtio_net.c +1 -0
@@ -1084,17 +1092,26 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
  			if (!xdp_page)
  				goto err_xdp;
  			offset = VIRTIO_XDP_HEADROOM;
+		} else if (unlikely(headroom < virtnet_get_headroom(vi))) {

I believe we need to check xdp_prog->aux->xdp_has_frags at least since this may not work if it needs more than one frags?

Btw, I don't see a reason why we can't reuse xdp_linearize_page(), (we probably don't need error is the buffer exceeds PAGE_SIZE).

Other looks good.

Thanks

Quoted 47 lines
drivers/net/virtio_net.c +26 -7
+			if ((VIRTIO_XDP_HEADROOM + len + tailroom) > PAGE_SIZE)
+				goto err_xdp;
+
+			xdp_page = alloc_page(GFP_ATOMIC);
+			if (!xdp_page)
+				goto err_xdp;
+
+			memcpy(page_address(xdp_page) + VIRTIO_XDP_HEADROOM,
+			       page_address(page) + offset, len);
+			frame_sz = PAGE_SIZE;
+			offset = VIRTIO_XDP_HEADROOM;
  		} else {
  			xdp_page = page;
  		}
-
-		/* Allow consuming headroom but reserve enough space to push
-		 * the descriptor on if we get an XDP_TX return code.
-		 */
  		data = page_address(xdp_page) + offset;
-		xdp_init_buff(&xdp, frame_sz - vi->hdr_len, &rq->xdp_rxq);
-		xdp_prepare_buff(&xdp, data - VIRTIO_XDP_HEADROOM + vi->hdr_len,
-				 VIRTIO_XDP_HEADROOM, len - vi->hdr_len, true);
+		err = virtnet_build_xdp_buff_mrg(dev, vi, rq, &xdp, data, len, frame_sz,
+						 &num_buf, &xdp_frags_truesz, stats);
+		if (unlikely(err))
+			goto err_xdp_frags;
 
  		act = bpf_prog_run_xdp(xdp_prog, &xdp);
  		stats->xdp_packets++;
@@ -1190,6 +1207,17 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
  				__free_pages(xdp_page, 0);
  			goto err_xdp;
  		}
+err_xdp_frags:
+		shinfo = xdp_get_shared_info_from_buff(&xdp);
+
+		if (unlikely(xdp_page != page))
+			__free_pages(xdp_page, 0);
+
+		for (i = 0; i < shinfo->nr_frags; i++) {
+			xdp_page = skb_frag_page(&shinfo->frags[i]);
+			put_page(xdp_page);
+		}
+		goto err_xdp;
  	}
  	rcu_read_unlock();
 
author
#19 lore raw

Re: [PATCH v2 5/9] virtio_net: construct multi-buffer xdp in mergeable

Quoted 79 lines · Jason Wang
Quoted 77 lines · Heng Qi

Build multi-buffer xdp using virtnet_build_xdp_buff_mrg().

For the prefilled buffer before xdp is set, we will probably use vq reset in the future. At the same time, virtio net currently uses comp pages, and bpf_xdp_frags_increase_tail() needs to calculate the tailroom of the last frag, which will involve the offset of the corresponding page and cause a negative value, so we disable tail increase by not setting xdp_rxq->frag_size.

---
  drivers/net/virtio_net.c | 60 +++++++++++++++++++++++++++++-----------

  1 file changed, 44 insertions(+), 16 deletions(-)
drivers/net/virtio_net.c +1 -0
index 8fc3b1841d92..40bc58fa57f5 100644
@@ -1018,6 +1018,7 @@ static struct sk_buff *receive_mergeable(struct 
 net_device *dev,
                        unsigned int *xdp_xmit,
                        struct virtnet_rq_stats *stats)
   {
+    unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct 
 skb_shared_info));
       struct virtio_net_hdr_mrg_rxbuf *hdr = buf;

      u16 num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers); struct page *page = virt_to_head_page(buf);

drivers/net/virtio_net.c +7 -1
@@ -1048,11 +1049,14 @@ static struct sk_buff 
 *receive_mergeable(struct net_device *dev,
       rcu_read_lock();
       xdp_prog = rcu_dereference(rq->xdp_prog);
       if (xdp_prog) {
+        unsigned int xdp_frags_truesz = 0;
+        struct skb_shared_info *shinfo;
           struct xdp_frame *xdpf;
           struct page *xdp_page;
           struct xdp_buff xdp;
           void *data;
           u32 act;
+        int i;
             /* Transient failure which in theory could occur if
            * in-flight packets from before XDP was enabled reach
@@ -1061,19 +1065,23 @@ static struct sk_buff 
 *receive_mergeable(struct net_device *dev,
           if (unlikely(hdr->hdr.gso_type))
               goto err_xdp;
   -        /* Buffers with headroom use PAGE_SIZE as alloc size,
-         * see add_recvbuf_mergeable() + get_mergeable_buf_len()
+        /* Now XDP core assumes frag size is PAGE_SIZE, but buffers
+         * with headroom may add hole in truesize, which
+         * make their length exceed PAGE_SIZE. So we disabled the
+         * hole mechanism for xdp. See add_recvbuf_mergeable().
            */
           frame_sz = headroom ? PAGE_SIZE : truesize;

  -        /* This happens when rx buffer size is underestimated -         * or headroom is not enough because of the buffer -         * was refilled before XDP is set. This should only -         * happen for the first several packets, so we don't -         * care much about its performance.
+        /* This happens when headroom is not enough because +         * of the buffer was prefilled before XDP is set. +         * This should only happen for the first several packets. +         * In fact, vq reset can be used here to help us clean up +         * the prefilled buffers, but many existing devices do not +         * support it, and we don't want to bother users who are +         * using xdp normally.
           */
-        if (unlikely(num_buf > 1 ||
-                 headroom < virtnet_get_headroom(vi))) { +        if (!xdp_prog->aux->xdp_has_frags &&
+            (num_buf > 1 || headroom < virtnet_get_headroom(vi))) { /* linearize data for XDP */
              xdp_page = xdp_linearize_page(rq, &num_buf, page, offset,

drivers/net/virtio_net.c +0 -0
@@ -1084,17 +1092,26 @@ static struct sk_buff 
Quoted 8 lines

drivers/net/virtio_net.c static struct sk_buff *receive_mergeable(struct net_device *dev, · line 1095

Quoted 5 lines
drivers/net/virtio_net.c +1 -0
 *receive_mergeable(struct net_device *dev,
               if (!xdp_page)
                   goto err_xdp;
               offset = VIRTIO_XDP_HEADROOM;
+        } else if (unlikely(headroom < virtnet_get_headroom(vi))) {

I believe we need to check xdp_prog->aux->xdp_has_frags at least since this may not work if it needs more than one frags?

Sorry Jason, I didn't understand you, I'll try to answer. For multi-buffer xdp programs, if the first buffer is a pre-filled buffer (no headroom), we need to copy it out and use the subsequent buffers of this packet as its frags (this is done in virtnet_build_xdp_buff_mrg()), therefore, it seems that there is no need to check 'xdp_prog->aux->xdp_has_frags' to mark multi-buffer xdp (of course I can add it),

+ } else if (unlikely(headroom < virtnet_get_headroom(vi))) {

Because the linearization of single-buffer xdp has all been done before, the subsequent situation can only be applied to multi-buffer xdp:
+ if (!xdp_prog->aux->xdp_has_frags &&
+ (num_buf > 1 || headroom < virtnet_get_headroom(vi))) {

Quoted 2 lines

Btw, I don't see a reason why we can't reuse xdp_linearize_page(), (we probably don't need error is the buffer exceeds PAGE_SIZE).

For multi-buffer xdp, we only need to copy out the pre-filled first buffer, and use the remaining buffers of this packet as frags in virtnet_build_xdp_buff_mrg().

Thanks.

Quoted 54 lines

Other looks good.

Thanks

Quoted 48 lines
drivers/net/virtio_net.c +15 -7
+            if ((VIRTIO_XDP_HEADROOM + len + tailroom) > PAGE_SIZE)
+                goto err_xdp;
+
+            xdp_page = alloc_page(GFP_ATOMIC);
+            if (!xdp_page)
+                goto err_xdp;
+
+            memcpy(page_address(xdp_page) + VIRTIO_XDP_HEADROOM,
+                   page_address(page) + offset, len);
+            frame_sz = PAGE_SIZE;
+            offset = VIRTIO_XDP_HEADROOM;
           } else {
               xdp_page = page;
           }
-
-        /* Allow consuming headroom but reserve enough space to push
-         * the descriptor on if we get an XDP_TX return code.
-         */
           data = page_address(xdp_page) + offset;
-        xdp_init_buff(&xdp, frame_sz - vi->hdr_len, &rq->xdp_rxq);
-        xdp_prepare_buff(&xdp, data - VIRTIO_XDP_HEADROOM + 
 vi->hdr_len,
-                 VIRTIO_XDP_HEADROOM, len - vi->hdr_len, true);
+        err = virtnet_build_xdp_buff_mrg(dev, vi, rq, &xdp, data, 
 len, frame_sz,
+                         &num_buf, &xdp_frags_truesz, stats);
+        if (unlikely(err))
+            goto err_xdp_frags;

            act = bpf_prog_run_xdp(xdp_prog, &xdp); stats->xdp_packets++;

drivers/net/virtio_net.c +11 -0
@@ -1190,6 +1207,17 @@ static struct sk_buff 
 *receive_mergeable(struct net_device *dev,
                   __free_pages(xdp_page, 0);
               goto err_xdp;
           }
+err_xdp_frags:
+        shinfo = xdp_get_shared_info_from_buff(&xdp);
+
+        if (unlikely(xdp_page != page))
+            __free_pages(xdp_page, 0);
+
+        for (i = 0; i < shinfo->nr_frags; i++) {
+            xdp_page = skb_frag_page(&shinfo->frags[i]);
+            put_page(xdp_page);
+        }
+        goto err_xdp;
       }
       rcu_read_unlock();
#20 lore raw

Re: [PATCH v2 5/9] virtio_net: construct multi-buffer xdp in mergeable

Quoted 96 lines · Heng Qi
Quoted 80 lines · Jason Wang
Quoted 78 lines · Heng Qi

Build multi-buffer xdp using virtnet_build_xdp_buff_mrg().

For the prefilled buffer before xdp is set, we will probably use vq reset in the future. At the same time, virtio net currently uses comp pages, and bpf_xdp_frags_increase_tail() needs to calculate the tailroom of the last frag, which will involve the offset of the corresponding page and cause a negative value, so we disable tail increase by not setting xdp_rxq->frag_size.

---
  drivers/net/virtio_net.c | 60
+++++++++++++++++++++++++++++-----------

  1 file changed, 44 insertions(+), 16 deletions(-)
drivers/net/virtio_net.c +1 -0
index 8fc3b1841d92..40bc58fa57f5 100644
@@ -1018,6 +1018,7 @@ static struct sk_buff 
 *receive_mergeable(struct net_device *dev,
                        unsigned int *xdp_xmit,
                        struct virtnet_rq_stats *stats)
   {
+    unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct 
 skb_shared_info));
       struct virtio_net_hdr_mrg_rxbuf *hdr = buf;

      u16 num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers); struct page *page = virt_to_head_page(buf);

drivers/net/virtio_net.c +7 -1
@@ -1048,11 +1049,14 @@ static struct sk_buff 
 *receive_mergeable(struct net_device *dev,
       rcu_read_lock();
       xdp_prog = rcu_dereference(rq->xdp_prog);
       if (xdp_prog) {
+        unsigned int xdp_frags_truesz = 0;
+        struct skb_shared_info *shinfo;
           struct xdp_frame *xdpf;
           struct page *xdp_page;
           struct xdp_buff xdp;
           void *data;
           u32 act;
+        int i;
             /* Transient failure which in theory could occur if
            * in-flight packets from before XDP was enabled reach
@@ -1061,19 +1065,23 @@ static struct sk_buff 
 *receive_mergeable(struct net_device *dev,
           if (unlikely(hdr->hdr.gso_type))
               goto err_xdp;
   -        /* Buffers with headroom use PAGE_SIZE as alloc size,
-         * see add_recvbuf_mergeable() + get_mergeable_buf_len()
+        /* Now XDP core assumes frag size is PAGE_SIZE, but buffers
+         * with headroom may add hole in truesize, which
+         * make their length exceed PAGE_SIZE. So we disabled the
+         * hole mechanism for xdp. See add_recvbuf_mergeable().
            */
           frame_sz = headroom ? PAGE_SIZE : truesize;

  -        /* This happens when rx buffer size is underestimated -         * or headroom is not enough because of the buffer -         * was refilled before XDP is set. This should only -         * happen for the first several packets, so we don't -         * care much about its performance.
+        /* This happens when headroom is not enough because +         * of the buffer was prefilled before XDP is set. +         * This should only happen for the first several packets. +         * In fact, vq reset can be used here to help us clean up +         * the prefilled buffers, but many existing devices do not +         * support it, and we don't want to bother users who are +         * using xdp normally.
           */
-        if (unlikely(num_buf > 1 ||
-                 headroom < virtnet_get_headroom(vi))) { +        if (!xdp_prog->aux->xdp_has_frags &&
+            (num_buf > 1 || headroom < virtnet_get_headroom(vi))) { /* linearize data for XDP */
              xdp_page = xdp_linearize_page(rq, &num_buf, page, offset,

drivers/net/virtio_net.c +0 -0
@@ -1084,17 +1092,26 @@ static struct sk_buff 
Quoted 8 lines

drivers/net/virtio_net.c static struct sk_buff *receive_mergeable(struct net_device *dev, · line 1095

Quoted 5 lines
drivers/net/virtio_net.c +1 -0
 *receive_mergeable(struct net_device *dev,
               if (!xdp_page)
                   goto err_xdp;
               offset = VIRTIO_XDP_HEADROOM;
+        } else if (unlikely(headroom < virtnet_get_headroom(vi))) {

I believe we need to check xdp_prog->aux->xdp_has_frags at least since this may not work if it needs more than one frags?

Sorry Jason, I didn't understand you, I'll try to answer. For multi-buffer xdp programs, if the first buffer is a pre-filled buffer (no headroom), we need to copy it out and use the subsequent buffers of this packet as its frags (this is done in virtnet_build_xdp_buff_mrg()), therefore, it seems that there is no need to check 'xdp_prog->aux->xdp_has_frags' to mark multi-buffer xdp (of course I can add it),

Quoted 5 lines

+ } else if (unlikely(headroom < virtnet_get_headroom(vi))) {

Because the linearization of single-buffer xdp has all been done before, the subsequent situation can only be applied to multi-buffer xdp:
+ if (!xdp_prog->aux->xdp_has_frags &&
+ (num_buf > 1 || headroom < virtnet_get_headroom(vi))) {

I basically meant what happens if

!xdp_prog->aux->xdp_has_frags && num_buf > 2 && headroom < virtnet_get_headroom(vi)

In this case the current code seems to leave the second buffer in the frags. This is the case of the buffer size underestimation that is mentioned in the comment before (I'd like to keep that).

(And that's why I'm asking to use linearizge_page())

Thanks

Quoted 63 lines
Quoted 3 lines

Btw, I don't see a reason why we can't reuse xdp_linearize_page(), (we probably don't need error is the buffer exceeds PAGE_SIZE).

For multi-buffer xdp, we only need to copy out the pre-filled first buffer, and use the remaining buffers of this packet as frags in virtnet_build_xdp_buff_mrg().

Thanks.

Quoted 54 lines

Other looks good.

Thanks

Quoted 48 lines
drivers/net/virtio_net.c +15 -7
+            if ((VIRTIO_XDP_HEADROOM + len + tailroom) > PAGE_SIZE)
+                goto err_xdp;
+
+            xdp_page = alloc_page(GFP_ATOMIC);
+            if (!xdp_page)
+                goto err_xdp;
+
+            memcpy(page_address(xdp_page) + VIRTIO_XDP_HEADROOM,
+                   page_address(page) + offset, len);
+            frame_sz = PAGE_SIZE;
+            offset = VIRTIO_XDP_HEADROOM;
           } else {
               xdp_page = page;
           }
-
-        /* Allow consuming headroom but reserve enough space to push
-         * the descriptor on if we get an XDP_TX return code.
-         */
           data = page_address(xdp_page) + offset;
-        xdp_init_buff(&xdp, frame_sz - vi->hdr_len, &rq->xdp_rxq);
-        xdp_prepare_buff(&xdp, data - VIRTIO_XDP_HEADROOM + 
 vi->hdr_len,
-                 VIRTIO_XDP_HEADROOM, len - vi->hdr_len, true);
+        err = virtnet_build_xdp_buff_mrg(dev, vi, rq, &xdp, data, 
 len, frame_sz,
+                         &num_buf, &xdp_frags_truesz, stats);
+        if (unlikely(err))
+            goto err_xdp_frags;

            act = bpf_prog_run_xdp(xdp_prog, &xdp); stats->xdp_packets++;

drivers/net/virtio_net.c +11 -0
@@ -1190,6 +1207,17 @@ static struct sk_buff 
 *receive_mergeable(struct net_device *dev,
                   __free_pages(xdp_page, 0);
               goto err_xdp;
           }
+err_xdp_frags:
+        shinfo = xdp_get_shared_info_from_buff(&xdp);
+
+        if (unlikely(xdp_page != page))
+            __free_pages(xdp_page, 0);
+
+        for (i = 0; i < shinfo->nr_frags; i++) {
+            xdp_page = skb_frag_page(&shinfo->frags[i]);
+            put_page(xdp_page);
+        }
+        goto err_xdp;
       }
       rcu_read_unlock();
author
#21 lore raw

Re: [PATCH v2 5/9] virtio_net: construct multi-buffer xdp in mergeable

Quoted 114 lines · Jason Wang
Quoted 103 lines · Heng Qi
Quoted 81 lines · Jason Wang
Quoted 79 lines · Heng Qi

Build multi-buffer xdp using virtnet_build_xdp_buff_mrg().

For the prefilled buffer before xdp is set, we will probably use vq reset in the future. At the same time, virtio net currently uses comp pages, and bpf_xdp_frags_increase_tail() needs to calculate the tailroom of the last frag, which will involve the offset of the corresponding page and cause a negative value, so we disable tail increase by not setting xdp_rxq->frag_size.

---
  drivers/net/virtio_net.c | 60
+++++++++++++++++++++++++++++-----------

  1 file changed, 44 insertions(+), 16 deletions(-)
drivers/net/virtio_net.c +1 -0
index 8fc3b1841d92..40bc58fa57f5 100644
@@ -1018,6 +1018,7 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
                        unsigned int *xdp_xmit,
                        struct virtnet_rq_stats *stats)
   {
+    unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct
 skb_shared_info));
       struct virtio_net_hdr_mrg_rxbuf *hdr = buf;

      u16 num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers); struct page *page = virt_to_head_page(buf);

drivers/net/virtio_net.c +7 -1
@@ -1048,11 +1049,14 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
       rcu_read_lock();
       xdp_prog = rcu_dereference(rq->xdp_prog);
       if (xdp_prog) {
+        unsigned int xdp_frags_truesz = 0;
+        struct skb_shared_info *shinfo;
           struct xdp_frame *xdpf;
           struct page *xdp_page;
           struct xdp_buff xdp;
           void *data;
           u32 act;
+        int i;
             /* Transient failure which in theory could occur if
            * in-flight packets from before XDP was enabled reach
@@ -1061,19 +1065,23 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
           if (unlikely(hdr->hdr.gso_type))
               goto err_xdp;
   -        /* Buffers with headroom use PAGE_SIZE as alloc size,
-         * see add_recvbuf_mergeable() + get_mergeable_buf_len()
+        /* Now XDP core assumes frag size is PAGE_SIZE, but buffers
+         * with headroom may add hole in truesize, which
+         * make their length exceed PAGE_SIZE. So we disabled the
+         * hole mechanism for xdp. See add_recvbuf_mergeable().
            */
           frame_sz = headroom ? PAGE_SIZE : truesize;

  -        /* This happens when rx buffer size is underestimated -         * or headroom is not enough because of the buffer -         * was refilled before XDP is set. This should only -         * happen for the first several packets, so we don't -         * care much about its performance.
+        /* This happens when headroom is not enough because +         * of the buffer was prefilled before XDP is set. +         * This should only happen for the first several packets. +         * In fact, vq reset can be used here to help us clean up +         * the prefilled buffers, but many existing devices do not +         * support it, and we don't want to bother users who are +         * using xdp normally.
           */
-        if (unlikely(num_buf > 1 ||
-                 headroom < virtnet_get_headroom(vi))) { +        if (!xdp_prog->aux->xdp_has_frags &&
+            (num_buf > 1 || headroom < virtnet_get_headroom(vi))) { /* linearize data for XDP */
              xdp_page = xdp_linearize_page(rq, &num_buf, page, offset,

drivers/net/virtio_net.c +0 -0
@@ -1084,17 +1092,26 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
Quoted 8 lines

drivers/net/virtio_net.c static struct sk_buff *receive_mergeable(struct net_device *dev, · line 1095

Quoted 4 lines
drivers/net/virtio_net.c +1 -0
               if (!xdp_page)
                   goto err_xdp;
               offset = VIRTIO_XDP_HEADROOM;
+        } else if (unlikely(headroom < virtnet_get_headroom(vi))) {

I believe we need to check xdp_prog->aux->xdp_has_frags at least since this may not work if it needs more than one frags?

Sorry Jason, I didn't understand you, I'll try to answer. For multi-buffer xdp programs, if the first buffer is a pre-filled buffer (no headroom),
we need to copy it out and use the subsequent buffers of this packet as its frags (this is done in virtnet_build_xdp_buff_mrg()), therefore,
it seems that there is no need to check 'xdp_prog->aux->xdp_has_frags' to mark multi-buffer xdp (of course I can add it),

Quoted 7 lines

+ } else if (unlikely(headroom < virtnet_get_headroom(vi))) {

Because the linearization of single-buffer xdp has all been done before, the subsequent situation can only be applied to multi-buffer xdp:
+ if (!xdp_prog->aux->xdp_has_frags &&
+ (num_buf > 1 || headroom < virtnet_get_headroom(vi))) {

Quoted 8 lines

I basically meant what happens if

!xdp_prog->aux->xdp_has_frags && num_buf > 2 && headroom < virtnet_get_headroom(vi)

In this case the current code seems to leave the second buffer in the frags. This is the case of the buffer size underestimation that is mentioned in the comment before (I'd like to keep that).

If I'm not wrong, this case is still directly into the first 'if' loop.

- if (unlikely(num_buf > 1 ||
- headroom < virtnet_get_headroom(vi))) {
+ if (!xdp_prog->aux->xdp_has_frags &&
+ (num_buf > 1 || headroom < virtnet_get_headroom(vi))) {

Thanks.

Quoted 73 lines

(And that's why I'm asking to use linearizge_page())

Thanks

Quoted 67 lines
Quoted 5 lines

Btw, I don't see a reason why we can't reuse xdp_linearize_page(), (we probably don't need error is the buffer exceeds PAGE_SIZE).

For multi-buffer xdp, we only need to copy out the pre-filled first buffer, and use the remaining buffers of this packet as frags in virtnet_build_xdp_buff_mrg().

Thanks.

Quoted 54 lines

Other looks good.

Thanks

Quoted 48 lines
drivers/net/virtio_net.c +15 -7
+            if ((VIRTIO_XDP_HEADROOM + len + tailroom) > PAGE_SIZE)
+                goto err_xdp;
+
+            xdp_page = alloc_page(GFP_ATOMIC);
+            if (!xdp_page)
+                goto err_xdp;
+
+            memcpy(page_address(xdp_page) + VIRTIO_XDP_HEADROOM,
+                   page_address(page) + offset, len);
+            frame_sz = PAGE_SIZE;
+            offset = VIRTIO_XDP_HEADROOM;
           } else {
               xdp_page = page;
           }
-
-        /* Allow consuming headroom but reserve enough space to push
-         * the descriptor on if we get an XDP_TX return code.
-         */
           data = page_address(xdp_page) + offset;
-        xdp_init_buff(&xdp, frame_sz - vi->hdr_len, &rq->xdp_rxq);
-        xdp_prepare_buff(&xdp, data - VIRTIO_XDP_HEADROOM +
 vi->hdr_len,
-                 VIRTIO_XDP_HEADROOM, len - vi->hdr_len, true);
+        err = virtnet_build_xdp_buff_mrg(dev, vi, rq, &xdp,
 data, len, frame_sz,
+                         &num_buf, &xdp_frags_truesz, stats);
+        if (unlikely(err))
+            goto err_xdp_frags;

            act = bpf_prog_run_xdp(xdp_prog, &xdp); stats->xdp_packets++;

drivers/net/virtio_net.c +11 -0
@@ -1190,6 +1207,17 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
                   __free_pages(xdp_page, 0);
               goto err_xdp;
           }
+err_xdp_frags:
+        shinfo = xdp_get_shared_info_from_buff(&xdp);
+
+        if (unlikely(xdp_page != page))
+            __free_pages(xdp_page, 0);
+
+        for (i = 0; i < shinfo->nr_frags; i++) {
+            xdp_page = skb_frag_page(&shinfo->frags[i]);
+            put_page(xdp_page);
+        }
+        goto err_xdp;
       }
       rcu_read_unlock();
#22 lore raw

Re: [PATCH v2 5/9] virtio_net: construct multi-buffer xdp in mergeable

Quoted 118 lines · Heng Qi
Quoted 116 lines · Jason Wang
Quoted 103 lines · Heng Qi
Quoted 81 lines · Jason Wang
Quoted 79 lines · Heng Qi

Build multi-buffer xdp using virtnet_build_xdp_buff_mrg().

For the prefilled buffer before xdp is set, we will probably use vq reset in the future. At the same time, virtio net currently uses comp pages, and bpf_xdp_frags_increase_tail() needs to calculate the tailroom of the last frag, which will involve the offset of the corresponding page and cause a negative value, so we disable tail increase by not setting xdp_rxq->frag_size.

---
 drivers/net/virtio_net.c | 60
+++++++++++++++++++++++++++++-----------
 1 file changed, 44 insertions(+), 16 deletions(-)
drivers/net/virtio_net.c +1 -0
index 8fc3b1841d92..40bc58fa57f5 100644
@@ -1018,6 +1018,7 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
                      unsigned int *xdp_xmit,
                      struct virtnet_rq_stats *stats)
 {
+    unsigned int tailroom = SKB_DATA_ALIGN(sizeof(struct
 skb_shared_info));
     struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
     u16 num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers);
     struct page *page = virt_to_head_page(buf);
drivers/net/virtio_net.c +16 -7
@@ -1048,11 +1049,14 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
     rcu_read_lock();
     xdp_prog = rcu_dereference(rq->xdp_prog);
     if (xdp_prog) {
+        unsigned int xdp_frags_truesz = 0;
+        struct skb_shared_info *shinfo;
         struct xdp_frame *xdpf;
         struct page *xdp_page;
         struct xdp_buff xdp;
         void *data;
         u32 act;
+        int i;
           /* Transient failure which in theory could occur if
          * in-flight packets from before XDP was enabled reach
@@ -1061,19 +1065,23 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
         if (unlikely(hdr->hdr.gso_type))
             goto err_xdp;
 -        /* Buffers with headroom use PAGE_SIZE as alloc size,
-         * see add_recvbuf_mergeable() + get_mergeable_buf_len()
+        /* Now XDP core assumes frag size is PAGE_SIZE, but buffers
+         * with headroom may add hole in truesize, which
+         * make their length exceed PAGE_SIZE. So we disabled the
+         * hole mechanism for xdp. See add_recvbuf_mergeable().
          */
         frame_sz = headroom ? PAGE_SIZE : truesize;
 -        /* This happens when rx buffer size is underestimated
-         * or headroom is not enough because of the buffer
-         * was refilled before XDP is set. This should only
-         * happen for the first several packets, so we don't
-         * care much about its performance.
+        /* This happens when headroom is not enough because
+         * of the buffer was prefilled before XDP is set.
+         * This should only happen for the first several packets.
+         * In fact, vq reset can be used here to help us clean up
+         * the prefilled buffers, but many existing devices do not
+         * support it, and we don't want to bother users who are
+         * using xdp normally.
          */
-        if (unlikely(num_buf > 1 ||
-                 headroom < virtnet_get_headroom(vi))) {
+        if (!xdp_prog->aux->xdp_has_frags &&
+            (num_buf > 1 || headroom < virtnet_get_headroom(vi))) {
             /* linearize data for XDP */
             xdp_page = xdp_linearize_page(rq, &num_buf,
                               page, offset,
@@ -1084,17 +1092,26 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
Quoted 8 lines

drivers/net/virtio_net.c static struct sk_buff *receive_mergeable(struct net_device *dev, · line 1095

Quoted 4 lines
drivers/net/virtio_net.c +1 -0
             if (!xdp_page)
                 goto err_xdp;
             offset = VIRTIO_XDP_HEADROOM;
+        } else if (unlikely(headroom < virtnet_get_headroom(vi))) {

I believe we need to check xdp_prog->aux->xdp_has_frags at least since this may not work if it needs more than one frags?

Sorry Jason, I didn't understand you, I'll try to answer. For multi-buffer xdp programs, if the first buffer is a pre-filled buffer (no headroom),
we need to copy it out and use the subsequent buffers of this packet as its frags (this is done in virtnet_build_xdp_buff_mrg()), therefore,
it seems that there is no need to check 'xdp_prog->aux->xdp_has_frags' to mark multi-buffer xdp (of course I can add it),

Quoted 7 lines

+ } else if (unlikely(headroom < virtnet_get_headroom(vi))) {

Because the linearization of single-buffer xdp has all been done before, the subsequent situation can only be applied to multi-buffer xdp:
+ if (!xdp_prog->aux->xdp_has_frags &&
+ (num_buf > 1 || headroom < virtnet_get_headroom(vi))) {

I basically meant what happens if

Quoted 8 lines
Quoted 6 lines

!xdp_prog->aux->xdp_has_frags && num_buf > 2 && headroom < virtnet_get_headroom(vi)

In this case the current code seems to leave the second buffer in the frags. This is the case of the buffer size underestimation that is mentioned in the comment before (I'd like to keep that).

If I'm not wrong, this case is still directly into the first 'if' loop.

My fault, you're right.

Thanks

Quoted 82 lines
drivers/net/virtio_net.c +2 -2
-               if (unlikely(num_buf > 1 ||
-                            headroom < virtnet_get_headroom(vi))) {
+               if (!xdp_prog->aux->xdp_has_frags &&
+                   (num_buf > 1 || headroom < virtnet_get_headroom(vi))) {

Thanks.

Quoted 73 lines

(And that's why I'm asking to use linearizge_page())

Thanks

Quoted 67 lines
Quoted 5 lines

Btw, I don't see a reason why we can't reuse xdp_linearize_page(), (we probably don't need error is the buffer exceeds PAGE_SIZE).

For multi-buffer xdp, we only need to copy out the pre-filled first buffer, and use the remaining buffers of this packet as frags in virtnet_build_xdp_buff_mrg().

Thanks.

Quoted 54 lines

Other looks good.

Thanks

Quoted 48 lines
drivers/net/virtio_net.c +15 -7
+            if ((VIRTIO_XDP_HEADROOM + len + tailroom) > PAGE_SIZE)
+                goto err_xdp;
+
+            xdp_page = alloc_page(GFP_ATOMIC);
+            if (!xdp_page)
+                goto err_xdp;
+
+            memcpy(page_address(xdp_page) + VIRTIO_XDP_HEADROOM,
+                   page_address(page) + offset, len);
+            frame_sz = PAGE_SIZE;
+            offset = VIRTIO_XDP_HEADROOM;
         } else {
             xdp_page = page;
         }
-
-        /* Allow consuming headroom but reserve enough space to push
-         * the descriptor on if we get an XDP_TX return code.
-         */
         data = page_address(xdp_page) + offset;
-        xdp_init_buff(&xdp, frame_sz - vi->hdr_len, &rq->xdp_rxq);
-        xdp_prepare_buff(&xdp, data - VIRTIO_XDP_HEADROOM +
 vi->hdr_len,
-                 VIRTIO_XDP_HEADROOM, len - vi->hdr_len, true);
+        err = virtnet_build_xdp_buff_mrg(dev, vi, rq, &xdp,
 data, len, frame_sz,
+                         &num_buf, &xdp_frags_truesz, stats);
+        if (unlikely(err))
+            goto err_xdp_frags;
           act = bpf_prog_run_xdp(xdp_prog, &xdp);
         stats->xdp_packets++;
drivers/net/virtio_net.c +11 -0
@@ -1190,6 +1207,17 @@ static struct sk_buff
 *receive_mergeable(struct net_device *dev,
                 __free_pages(xdp_page, 0);
             goto err_xdp;
         }
+err_xdp_frags:
+        shinfo = xdp_get_shared_info_from_buff(&xdp);
+
+        if (unlikely(xdp_page != page))
+            __free_pages(xdp_page, 0);
+
+        for (i = 0; i < shinfo->nr_frags; i++) {
+            xdp_page = skb_frag_page(&shinfo->frags[i]);
+            put_page(xdp_page);
+        }
+        goto err_xdp;
     }
     rcu_read_unlock();
author
#23 lore raw

PATCH v2 6/9 virtio_net: transmit the multi-buffer xdp

In reply to Heng Qi

This serves as the basis for XDP_TX and XDP_REDIRECT to send a multi-buffer xdp_frame.

---
 drivers/net/virtio_net.c | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)
drivers/net/virtio_net.c +22 -5
index 40bc58fa57f5..9f31bfa7f9a6 100644
@@ -563,22 +563,39 @@ static int __virtnet_xdp_xmit_one(struct virtnet_info *vi,
 				   struct xdp_frame *xdpf)
 {
 	struct virtio_net_hdr_mrg_rxbuf *hdr;
-	int err;
+	struct skb_shared_info *shinfo;
+	u8 nr_frags = 0;
+	int err, i;
 
 	if (unlikely(xdpf->headroom < vi->hdr_len))
 		return -EOVERFLOW;
 
-	/* Make room for virtqueue hdr (also change xdpf->headroom?) */
+	if (unlikely(xdp_frame_has_frags(xdpf))) {
+		shinfo = xdp_get_shared_info_from_frame(xdpf);
+		nr_frags = shinfo->nr_frags;
+	}
+
+	/* Need to adjust this to calculate the correct postion
+	 * for shinfo of the xdpf.
+	 */
+	xdpf->headroom -= vi->hdr_len;
 	xdpf->data -= vi->hdr_len;
 	/* Zero header and leave csum up to XDP layers */
 	hdr = xdpf->data;
 	memset(hdr, 0, vi->hdr_len);
 	xdpf->len   += vi->hdr_len;
 
-	sg_init_one(sq->sg, xdpf->data, xdpf->len);
+	sg_init_table(sq->sg, nr_frags + 1);
+	sg_set_buf(sq->sg, xdpf->data, xdpf->len);
+	for (i = 0; i < nr_frags; i++) {
+		skb_frag_t *frag = &shinfo->frags[i];
+
+		sg_set_page(&sq->sg[i + 1], skb_frag_page(frag),
+			    skb_frag_size(frag), skb_frag_off(frag));
+	}
 
-	err = virtqueue_add_outbuf(sq->vq, sq->sg, 1, xdp_to_ptr(xdpf),
-				   GFP_ATOMIC);
+	err = virtqueue_add_outbuf(sq->vq, sq->sg, nr_frags + 1,
+				   xdp_to_ptr(xdpf), GFP_ATOMIC);
 	if (unlikely(err))
 		return -ENOSPC; /* Caller handle free/refcnt */
 
Signature · 1 line
2.19.1.6.gb485710b
#24 lore raw

Re: [PATCH v2 6/9] virtio_net: transmit the multi-buffer xdp

Quoted 25 lines · Heng Qi

This serves as the basis for XDP_TX and XDP_REDIRECT to send a multi-buffer xdp_frame.

---
  drivers/net/virtio_net.c | 27 ++++++++++++++++++++++-----
  1 file changed, 22 insertions(+), 5 deletions(-)
drivers/net/virtio_net.c +3 -1
index 40bc58fa57f5..9f31bfa7f9a6 100644
@@ -563,22 +563,39 @@ static int __virtnet_xdp_xmit_one(struct virtnet_info *vi,
  				   struct xdp_frame *xdpf)
  {
  	struct virtio_net_hdr_mrg_rxbuf *hdr;
-	int err;
+	struct skb_shared_info *shinfo;
+	u8 nr_frags = 0;
+	int err, i;
 
  	if (unlikely(xdpf->headroom < vi->hdr_len))
  		return -EOVERFLOW;
 

drivers/net/virtio_net.c static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, · line 581

Quoted 10 lines
drivers/net/virtio_net.c +9 -1
-	/* Make room for virtqueue hdr (also change xdpf->headroom?) */
+	if (unlikely(xdp_frame_has_frags(xdpf))) {
+		shinfo = xdp_get_shared_info_from_frame(xdpf);
+		nr_frags = shinfo->nr_frags;
+	}
+
+	/* Need to adjust this to calculate the correct postion
+	 * for shinfo of the xdpf.
+	 */
+	xdpf->headroom -= vi->hdr_len;

Any reason we need to do this here? (Or if it is, is it only needed for multibuffer XDP?)

Other looks good.

Thanks

Quoted 23 lines
drivers/net/virtio_net.c +10 -3
  	xdpf->data -= vi->hdr_len;
  	/* Zero header and leave csum up to XDP layers */
  	hdr = xdpf->data;
  	memset(hdr, 0, vi->hdr_len);
  	xdpf->len   += vi->hdr_len;
 
-	sg_init_one(sq->sg, xdpf->data, xdpf->len);
+	sg_init_table(sq->sg, nr_frags + 1);
+	sg_set_buf(sq->sg, xdpf->data, xdpf->len);
+	for (i = 0; i < nr_frags; i++) {
+		skb_frag_t *frag = &shinfo->frags[i];
+
+		sg_set_page(&sq->sg[i + 1], skb_frag_page(frag),
+			    skb_frag_size(frag), skb_frag_off(frag));
+	}
 
-	err = virtqueue_add_outbuf(sq->vq, sq->sg, 1, xdp_to_ptr(xdpf),
-				   GFP_ATOMIC);
+	err = virtqueue_add_outbuf(sq->vq, sq->sg, nr_frags + 1,
+				   xdp_to_ptr(xdpf), GFP_ATOMIC);
  	if (unlikely(err))
  		return -ENOSPC; /* Caller handle free/refcnt */
 
author
#25 lore raw

Re: [PATCH v2 6/9] virtio_net: transmit the multi-buffer xdp

Quoted 31 lines · Jason Wang
Quoted 29 lines · Heng Qi

This serves as the basis for XDP_TX and XDP_REDIRECT to send a multi-buffer xdp_frame.

---
  drivers/net/virtio_net.c | 27 ++++++++++++++++++++++-----

  1 file changed, 22 insertions(+), 5 deletions(-)
drivers/net/virtio_net.c +3 -1
index 40bc58fa57f5..9f31bfa7f9a6 100644
@@ -563,22 +563,39 @@ static int __virtnet_xdp_xmit_one(struct 
 virtnet_info *vi,
                      struct xdp_frame *xdpf)
   {
       struct virtio_net_hdr_mrg_rxbuf *hdr;
-    int err;
+    struct skb_shared_info *shinfo;
+    u8 nr_frags = 0;
+    int err, i;
         if (unlikely(xdpf->headroom < vi->hdr_len))
           return -EOVERFLOW;

  -    /* Make room for virtqueue hdr (also change xdpf->headroom?) */ +    if (unlikely(xdp_frame_has_frags(xdpf))) {
+        shinfo = xdp_get_shared_info_from_frame(xdpf);
+        nr_frags = shinfo->nr_frags;
+    }

Quoted 8 lines
Quoted 5 lines

+
+    /* Need to adjust this to calculate the correct postion +     * for shinfo of the xdpf.
+     */
+    xdpf->headroom -= vi->hdr_len;

Any reason we need to do this here? (Or if it is, is it only needed for multibuffer XDP?)

Going back to its wrapping function virtnet_xdp_xmit(), we need to free up the pending old buffers. If the "is_xdp_frame(ptr)" condition is met, then we need to calculate the position of skb_shared_info in xdp_get_frame_len() and xdp_return_frame(), which will involve to xdpf->data and xdpf->headroom. Therefore, we need to update the value of headroom synchronously here.

Also, it's not necessary for single-buffer xdp, but we need to keep it because it's harmless and as it should be.

Thanks.

Quoted 26 lines

Other looks good.

Thanks

Quoted 20 lines

      xdpf->data -= vi->hdr_len;
      /* Zero header and leave csum up to XDP layers */
      hdr = xdpf->data;
      memset(hdr, 0, vi->hdr_len);
      xdpf->len   += vi->hdr_len;
  -    sg_init_one(sq->sg, xdpf->data, xdpf->len);
+    sg_init_table(sq->sg, nr_frags + 1);
+    sg_set_buf(sq->sg, xdpf->data, xdpf->len);
+    for (i = 0; i < nr_frags; i++) {
+        skb_frag_t *frag = &shinfo->frags[i];
+
+        sg_set_page(&sq->sg[i + 1], skb_frag_page(frag),
+                skb_frag_size(frag), skb_frag_off(frag)); +    }
  -    err = virtqueue_add_outbuf(sq->vq, sq->sg, 1, xdp_to_ptr(xdpf), -                   GFP_ATOMIC);
+    err = virtqueue_add_outbuf(sq->vq, sq->sg, nr_frags + 1, +                   xdp_to_ptr(xdpf), GFP_ATOMIC);
      if (unlikely(err))
          return -ENOSPC; /* Caller handle free/refcnt */

Acked-by
#26 lore raw

Re: [PATCH v2 6/9] virtio_net: transmit the multi-buffer xdp

Quoted 43 lines · Heng Qi
Quoted 31 lines · Jason Wang
Quoted 29 lines · Heng Qi

This serves as the basis for XDP_TX and XDP_REDIRECT to send a multi-buffer xdp_frame.

---
  drivers/net/virtio_net.c | 27 ++++++++++++++++++++++-----

  1 file changed, 22 insertions(+), 5 deletions(-)
drivers/net/virtio_net.c +3 -1
index 40bc58fa57f5..9f31bfa7f9a6 100644
@@ -563,22 +563,39 @@ static int __virtnet_xdp_xmit_one(struct 
 virtnet_info *vi,
                      struct xdp_frame *xdpf)
   {
       struct virtio_net_hdr_mrg_rxbuf *hdr;
-    int err;
+    struct skb_shared_info *shinfo;
+    u8 nr_frags = 0;
+    int err, i;
         if (unlikely(xdpf->headroom < vi->hdr_len))
           return -EOVERFLOW;

  -    /* Make room for virtqueue hdr (also change xdpf->headroom?) */ +    if (unlikely(xdp_frame_has_frags(xdpf))) {
+        shinfo = xdp_get_shared_info_from_frame(xdpf);
+        nr_frags = shinfo->nr_frags;
+    }

Quoted 8 lines
Quoted 5 lines

+
+    /* Need to adjust this to calculate the correct postion +     * for shinfo of the xdpf.
+     */
+    xdpf->headroom -= vi->hdr_len;

Any reason we need to do this here? (Or if it is, is it only needed for multibuffer XDP?)

Quoted 4 lines

Going back to its wrapping function virtnet_xdp_xmit(), we need to free up the pending old buffers. If the "is_xdp_frame(ptr)" condition is met, then we need to calculate the position of skb_shared_info in xdp_get_frame_len() and xdp_return_frame(), which will involve to xdpf->data and xdpf->headroom. Therefore, we need to update the value of headroom synchronously here.

Let's tweak the comment above to something like this.

With that fixed,

Thanks

Quoted 32 lines

Also, it's not necessary for single-buffer xdp, but we need to keep it because it's harmless and as it should be.

Thanks.

Quoted 26 lines

Other looks good.

Thanks

Quoted 20 lines

      xdpf->data -= vi->hdr_len;
      /* Zero header and leave csum up to XDP layers */
      hdr = xdpf->data;
      memset(hdr, 0, vi->hdr_len);
      xdpf->len   += vi->hdr_len;
  -    sg_init_one(sq->sg, xdpf->data, xdpf->len);
+    sg_init_table(sq->sg, nr_frags + 1);
+    sg_set_buf(sq->sg, xdpf->data, xdpf->len);
+    for (i = 0; i < nr_frags; i++) {
+        skb_frag_t *frag = &shinfo->frags[i];
+
+        sg_set_page(&sq->sg[i + 1], skb_frag_page(frag),
+                skb_frag_size(frag), skb_frag_off(frag)); +    }
  -    err = virtqueue_add_outbuf(sq->vq, sq->sg, 1, xdp_to_ptr(xdpf), -                   GFP_ATOMIC);
+    err = virtqueue_add_outbuf(sq->vq, sq->sg, nr_frags + 1, +                   xdp_to_ptr(xdpf), GFP_ATOMIC);
      if (unlikely(err))
          return -ENOSPC; /* Caller handle free/refcnt */

author
#27 lore raw

Re: [PATCH v2 6/9] virtio_net: transmit the multi-buffer xdp

Quoted 45 lines · Jason Wang
Quoted 43 lines · Heng Qi
Quoted 39 lines · Jason Wang
Quoted 26 lines · Heng Qi

This serves as the basis for XDP_TX and XDP_REDIRECT to send a multi-buffer xdp_frame.

---
  drivers/net/virtio_net.c | 27 ++++++++++++++++++++++-----

  1 file changed, 22 insertions(+), 5 deletions(-)
drivers/net/virtio_net.c +3 -1
index 40bc58fa57f5..9f31bfa7f9a6 100644
@@ -563,22 +563,39 @@ static int __virtnet_xdp_xmit_one(struct 
 virtnet_info *vi,
                      struct xdp_frame *xdpf)
   {
       struct virtio_net_hdr_mrg_rxbuf *hdr;
-    int err;
+    struct skb_shared_info *shinfo;
+    u8 nr_frags = 0;
+    int err, i;
         if (unlikely(xdpf->headroom < vi->hdr_len))
           return -EOVERFLOW;

  -    /* Make room for virtqueue hdr (also change xdpf->headroom?) */ +    if (unlikely(xdp_frame_has_frags(xdpf))) {

Quoted 8 lines

+        shinfo = xdp_get_shared_info_from_frame(xdpf); +        nr_frags = shinfo->nr_frags;
+    }
+
+    /* Need to adjust this to calculate the correct postion +     * for shinfo of the xdpf.
+     */
+    xdpf->headroom -= vi->hdr_len;

Any reason we need to do this here? (Or if it is, is it only needed for multibuffer XDP?)

Quoted 7 lines
Quoted 4 lines

Going back to its wrapping function virtnet_xdp_xmit(), we need to free up the pending old buffers. If the "is_xdp_frame(ptr)" condition is met, then we need to calculate the position of skb_shared_info in xdp_get_frame_len() and xdp_return_frame(), which will involve to xdpf->data and xdpf->headroom. Therefore, we need to update the value of headroom synchronously here.

Let's tweak the comment above to something like this.

Ok, thanks for your energy.

Quoted 40 lines

With that fixed,

Thanks

Quoted 32 lines

Also, it's not necessary for single-buffer xdp, but we need to keep it because it's harmless and as it should be.

Thanks.

Quoted 26 lines

Other looks good.

Thanks

Quoted 20 lines

      xdpf->data -= vi->hdr_len;
      /* Zero header and leave csum up to XDP layers */
      hdr = xdpf->data;
      memset(hdr, 0, vi->hdr_len);
      xdpf->len   += vi->hdr_len;
  -    sg_init_one(sq->sg, xdpf->data, xdpf->len);
+    sg_init_table(sq->sg, nr_frags + 1);
+    sg_set_buf(sq->sg, xdpf->data, xdpf->len);
+    for (i = 0; i < nr_frags; i++) {
+        skb_frag_t *frag = &shinfo->frags[i];
+
+        sg_set_page(&sq->sg[i + 1], skb_frag_page(frag),
+                skb_frag_size(frag), skb_frag_off(frag)); +    }
  -    err = virtqueue_add_outbuf(sq->vq, sq->sg, 1, xdp_to_ptr(xdpf), -                   GFP_ATOMIC);
+    err = virtqueue_add_outbuf(sq->vq, sq->sg, nr_frags + 1, +                   xdp_to_ptr(xdpf), GFP_ATOMIC);
      if (unlikely(err))
          return -ENOSPC; /* Caller handle free/refcnt */

author
#28 lore raw

PATCH v2 7/9 virtio_net: build skb from multi-buffer xdp

In reply to Heng Qi

This converts the xdp_buff directly to a skb, including multi-buffer and single buffer xdp. We'll isolate the construction of skb based on xdp from page_to_skb().

---
 drivers/net/virtio_net.c | 50 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
drivers/net/virtio_net.c +50 -0
index 9f31bfa7f9a6..4e12196fcfd4 100644
@@ -948,6 +948,56 @@ static struct sk_buff *receive_big(struct net_device *dev,
 	return NULL;
 }
 
+/* Why not use xdp_build_skb_from_frame() ?
+ * XDP core assumes that xdp frags are PAGE_SIZE in length, while in
+ * virtio-net there are 2 points that do not match its requirements:
+ *  1. The size of the prefilled buffer is not fixed before xdp is set.
+ *  2. When xdp is loaded, virtio-net has a hole mechanism (refer to
+ *     add_recvbuf_mergeable()), which will make the size of a buffer
+ *     exceed PAGE_SIZE.
+ */
+static struct sk_buff *build_skb_from_xdp_buff(struct net_device *dev,
+					       struct virtnet_info *vi,
+					       struct xdp_buff *xdp,
+					       unsigned int xdp_frags_truesz)
+{
+	struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp);
+	unsigned int headroom, data_len;
+	struct sk_buff *skb;
+	int metasize;
+	u8 nr_frags;
+
+	if (unlikely(xdp->data_end > xdp_data_hard_end(xdp))) {
+		pr_debug("Error building skb as missing reserved tailroom for xdp");
+		return NULL;
+	}
+
+	if (unlikely(xdp_buff_has_frags(xdp)))
+		nr_frags = sinfo->nr_frags;
+
+	skb = build_skb(xdp->data_hard_start, xdp->frame_sz);
+	if (unlikely(!skb))
+		return NULL;
+
+	headroom = xdp->data - xdp->data_hard_start;
+	data_len = xdp->data_end - xdp->data;
+	skb_reserve(skb, headroom);
+	__skb_put(skb, data_len);
+
+	metasize = xdp->data - xdp->data_meta;
+	metasize = metasize > 0 ? metasize : 0;
+	if (metasize)
+		skb_metadata_set(skb, metasize);
+
+	if (unlikely(xdp_buff_has_frags(xdp)))
+		xdp_update_skb_shared_info(skb, nr_frags,
+					   sinfo->xdp_frags_size,
+					   xdp_frags_truesz,
+					   xdp_buff_is_frag_pfmemalloc(xdp));
+
+	return skb;
+}
+
 /* TODO: build xdp in big mode */
 static int virtnet_build_xdp_buff_mrg(struct net_device *dev,
 				      struct virtnet_info *vi,
Signature · 1 line
2.19.1.6.gb485710b
Acked-by on v2 7/9
#29 lore raw

Re: [PATCH v2 7/9] virtio_net: build skb from multi-buffer xdp

Quoted 15 lines · Heng Qi

This converts the xdp_buff directly to a skb, including multi-buffer and single buffer xdp. We'll isolate the construction of skb based on xdp from page_to_skb().

---
  drivers/net/virtio_net.c | 50 ++++++++++++++++++++++++++++++++++++++++
  1 file changed, 50 insertions(+)
drivers/net/virtio_net.c +0 -0
index 9f31bfa7f9a6..4e12196fcfd4 100644

drivers/net/virtio_net.c static struct sk_buff *receive_big(struct net_device *dev, · line 957

Quoted 10 lines
drivers/net/virtio_net.c +7 -0
@@ -948,6 +948,56 @@ static struct sk_buff *receive_big(struct net_device *dev,
  	return NULL;
  }
 
+/* Why not use xdp_build_skb_from_frame() ?
+ * XDP core assumes that xdp frags are PAGE_SIZE in length, while in
+ * virtio-net there are 2 points that do not match its requirements:
+ *  1. The size of the prefilled buffer is not fixed before xdp is set.
+ *  2. When xdp is loaded, virtio-net has a hole mechanism (refer to
+ *     add_recvbuf_mergeable()), which will make the size of a buffer
+ *     exceed PAGE_SIZE.

Is point 2 still valid after patch 1?

Other than this:

Thanks

Quoted 46 lines
drivers/net/virtio_net.c +43 -0
+ */
+static struct sk_buff *build_skb_from_xdp_buff(struct net_device *dev,
+					       struct virtnet_info *vi,
+					       struct xdp_buff *xdp,
+					       unsigned int xdp_frags_truesz)
+{
+	struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp);
+	unsigned int headroom, data_len;
+	struct sk_buff *skb;
+	int metasize;
+	u8 nr_frags;
+
+	if (unlikely(xdp->data_end > xdp_data_hard_end(xdp))) {
+		pr_debug("Error building skb as missing reserved tailroom for xdp");
+		return NULL;
+	}
+
+	if (unlikely(xdp_buff_has_frags(xdp)))
+		nr_frags = sinfo->nr_frags;
+
+	skb = build_skb(xdp->data_hard_start, xdp->frame_sz);
+	if (unlikely(!skb))
+		return NULL;
+
+	headroom = xdp->data - xdp->data_hard_start;
+	data_len = xdp->data_end - xdp->data;
+	skb_reserve(skb, headroom);
+	__skb_put(skb, data_len);
+
+	metasize = xdp->data - xdp->data_meta;
+	metasize = metasize > 0 ? metasize : 0;
+	if (metasize)
+		skb_metadata_set(skb, metasize);
+
+	if (unlikely(xdp_buff_has_frags(xdp)))
+		xdp_update_skb_shared_info(skb, nr_frags,
+					   sinfo->xdp_frags_size,
+					   xdp_frags_truesz,
+					   xdp_buff_is_frag_pfmemalloc(xdp));
+
+	return skb;
+}
+
  /* TODO: build xdp in big mode */
  static int virtnet_build_xdp_buff_mrg(struct net_device *dev,
  				      struct virtnet_info *vi,
author
#30 lore raw

Re: [PATCH v2 7/9] virtio_net: build skb from multi-buffer xdp

Quoted 22 lines · Jason Wang
Quoted 20 lines · Heng Qi

This converts the xdp_buff directly to a skb, including multi-buffer and single buffer xdp. We'll isolate the construction of skb based on xdp from page_to_skb().

---
  drivers/net/virtio_net.c | 50 ++++++++++++++++++++++++++++++++++++++++

  1 file changed, 50 insertions(+)
drivers/net/virtio_net.c +0 -0
index 9f31bfa7f9a6..4e12196fcfd4 100644
@@ -948,6 +948,56 @@ static struct sk_buff *receive_big(struct 
 net_device *dev,
       return NULL;
   }

  +/* Why not use xdp_build_skb_from_frame() ?
+ * XDP core assumes that xdp frags are PAGE_SIZE in length, while in

Quoted 8 lines
Quoted 5 lines

+ * virtio-net there are 2 points that do not match its requirements:
+ *  1. The size of the prefilled buffer is not fixed before xdp is set.
+ *  2. When xdp is loaded, virtio-net has a hole mechanism (refer to
+ *     add_recvbuf_mergeable()), which will make the size of a buffer
+ *     exceed PAGE_SIZE.

Is point 2 still valid after patch 1?

Yes, it is invalid anymore, I'll correct that, and there's a little more reason that xdp_build_skb_from_frame() does more checks that we don't need, like eth_type_trans() (which virtio-net does in receive_buf()).

Quoted 4 lines

Other than this:

Thanks for your energy.

Quoted 51 lines

Thanks

Quoted 47 lines

+ */

drivers/net/virtio_net.c +13 -0
+static struct sk_buff *build_skb_from_xdp_buff(struct net_device *dev,
+                           struct virtnet_info *vi,
+                           struct xdp_buff *xdp,
+                           unsigned int xdp_frags_truesz)
+{
+    struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp);
+    unsigned int headroom, data_len;
+    struct sk_buff *skb;
+    int metasize;
+    u8 nr_frags;
+
+    if (unlikely(xdp->data_end > xdp_data_hard_end(xdp))) {
+        pr_debug("Error building skb as missing reserved tailroom 

for xdp");
+        return NULL;
+    }

drivers/net/virtio_net.c +27 -0
+
+    if (unlikely(xdp_buff_has_frags(xdp)))
+        nr_frags = sinfo->nr_frags;
+
+    skb = build_skb(xdp->data_hard_start, xdp->frame_sz);
+    if (unlikely(!skb))
+        return NULL;
+
+    headroom = xdp->data - xdp->data_hard_start;
+    data_len = xdp->data_end - xdp->data;
+    skb_reserve(skb, headroom);
+    __skb_put(skb, data_len);
+
+    metasize = xdp->data - xdp->data_meta;
+    metasize = metasize > 0 ? metasize : 0;
+    if (metasize)
+        skb_metadata_set(skb, metasize);
+
+    if (unlikely(xdp_buff_has_frags(xdp)))
+        xdp_update_skb_shared_info(skb, nr_frags,
+                       sinfo->xdp_frags_size,
+                       xdp_frags_truesz,
+                       xdp_buff_is_frag_pfmemalloc(xdp));
+
+    return skb;
+}
+

  /* TODO: build xdp in big mode */
  static int virtnet_build_xdp_buff_mrg(struct net_device *dev, struct virtnet_info *vi,

author
#31 lore raw

PATCH v2 8/9 virtio_net: remove xdp related info from page_to_skb()

In reply to Heng Qi

For the clear construction of xdp_buff, we remove the xdp processing interleaved with page_to_skb(). Now, the logic of xdp and building skb from xdp are separate and independent.

---
 drivers/net/virtio_net.c | 41 +++++++++-------------------------------
 1 file changed, 9 insertions(+), 32 deletions(-)
drivers/net/virtio_net.c +9 -32
index 4e12196fcfd4..398ffe2a5084 100644
@@ -439,9 +439,7 @@ static unsigned int mergeable_ctx_to_truesize(void *mrg_ctx)
 static struct sk_buff *page_to_skb(struct virtnet_info *vi,
 				   struct receive_queue *rq,
 				   struct page *page, unsigned int offset,
-				   unsigned int len, unsigned int truesize,
-				   bool hdr_valid, unsigned int metasize,
-				   unsigned int headroom)
+				   unsigned int len, unsigned int truesize)
 {
 	struct sk_buff *skb;
 	struct virtio_net_hdr_mrg_rxbuf *hdr;
@@ -459,21 +457,11 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi,
 	else
 		hdr_padded_len = sizeof(struct padded_vnet_hdr);
 
-	/* If headroom is not 0, there is an offset between the beginning of the
-	 * data and the allocated space, otherwise the data and the allocated
-	 * space are aligned.
-	 *
-	 * Buffers with headroom use PAGE_SIZE as alloc size, see
-	 * add_recvbuf_mergeable() + get_mergeable_buf_len()
-	 */
-	truesize = headroom ? PAGE_SIZE : truesize;
-	tailroom = truesize - headroom;
-	buf = p - headroom;
-
+	buf = p;
 	len -= hdr_len;
 	offset += hdr_padded_len;
 	p += hdr_padded_len;
-	tailroom -= hdr_padded_len + len;
+	tailroom = truesize - hdr_padded_len - len;
 
 	shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
 
@@ -503,7 +491,7 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi,
 	if (len <= skb_tailroom(skb))
 		copy = len;
 	else
-		copy = ETH_HLEN + metasize;
+		copy = ETH_HLEN;
 	skb_put_data(skb, p, copy);
 
 	len -= copy;
@@ -542,19 +530,11 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi,
 		give_pages(rq, page);
 
 ok:
-	/* hdr_valid means no XDP, so we can copy the vnet header */
-	if (hdr_valid) {
-		hdr = skb_vnet_hdr(skb);
-		memcpy(hdr, hdr_p, hdr_len);
-	}
+	hdr = skb_vnet_hdr(skb);
+	memcpy(hdr, hdr_p, hdr_len);
 	if (page_to_free)
 		put_page(page_to_free);
 
-	if (metasize) {
-		__skb_pull(skb, metasize);
-		skb_metadata_set(skb, metasize);
-	}
-
 	return skb;
 }
 
@@ -934,7 +914,7 @@ static struct sk_buff *receive_big(struct net_device *dev,
 {
 	struct page *page = buf;
 	struct sk_buff *skb =
-		page_to_skb(vi, rq, page, 0, len, PAGE_SIZE, true, 0, 0);
+		page_to_skb(vi, rq, page, 0, len, PAGE_SIZE);
 
 	stats->bytes += len - vi->hdr_len;
 	if (unlikely(!skb))
@@ -1222,9 +1202,7 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 				rcu_read_unlock();
 				put_page(page);
 				head_skb = page_to_skb(vi, rq, xdp_page, offset,
-						       len, PAGE_SIZE, false,
-						       metasize,
-						       headroom);
+						       len, PAGE_SIZE);
 				return head_skb;
 			}
 			break;
@@ -1289,8 +1267,7 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 	rcu_read_unlock();
 
 skip_xdp:
-	head_skb = page_to_skb(vi, rq, page, offset, len, truesize, !xdp_prog,
-			       metasize, headroom);
+	head_skb = page_to_skb(vi, rq, page, offset, len, truesize);
 	curr_skb = head_skb;
 
 	if (unlikely(!curr_skb))
Signature · 1 line
2.19.1.6.gb485710b
Acked-by on v2 8/9
#32 lore raw

Re: [PATCH v2 8/9] virtio_net: remove xdp related info from page_to_skb()

Quoted 6 lines · Heng Qi

For the clear construction of xdp_buff, we remove the xdp processing interleaved with page_to_skb(). Now, the logic of xdp and building skb from xdp are separate and independent.

Thanks

Quoted 104 lines
---
  drivers/net/virtio_net.c | 41 +++++++++-------------------------------
  1 file changed, 9 insertions(+), 32 deletions(-)
drivers/net/virtio_net.c +9 -32
index 4e12196fcfd4..398ffe2a5084 100644
@@ -439,9 +439,7 @@ static unsigned int mergeable_ctx_to_truesize(void *mrg_ctx)
  static struct sk_buff *page_to_skb(struct virtnet_info *vi,
  				   struct receive_queue *rq,
  				   struct page *page, unsigned int offset,
-				   unsigned int len, unsigned int truesize,
-				   bool hdr_valid, unsigned int metasize,
-				   unsigned int headroom)
+				   unsigned int len, unsigned int truesize)
  {
  	struct sk_buff *skb;
  	struct virtio_net_hdr_mrg_rxbuf *hdr;
@@ -459,21 +457,11 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi,
  	else
  		hdr_padded_len = sizeof(struct padded_vnet_hdr);
 
-	/* If headroom is not 0, there is an offset between the beginning of the
-	 * data and the allocated space, otherwise the data and the allocated
-	 * space are aligned.
-	 *
-	 * Buffers with headroom use PAGE_SIZE as alloc size, see
-	 * add_recvbuf_mergeable() + get_mergeable_buf_len()
-	 */
-	truesize = headroom ? PAGE_SIZE : truesize;
-	tailroom = truesize - headroom;
-	buf = p - headroom;
-
+	buf = p;
  	len -= hdr_len;
  	offset += hdr_padded_len;
  	p += hdr_padded_len;
-	tailroom -= hdr_padded_len + len;
+	tailroom = truesize - hdr_padded_len - len;
 
  	shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
 
@@ -503,7 +491,7 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi,
  	if (len <= skb_tailroom(skb))
  		copy = len;
  	else
-		copy = ETH_HLEN + metasize;
+		copy = ETH_HLEN;
  	skb_put_data(skb, p, copy);
 
  	len -= copy;
@@ -542,19 +530,11 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi,
  		give_pages(rq, page);
 
  ok:
-	/* hdr_valid means no XDP, so we can copy the vnet header */
-	if (hdr_valid) {
-		hdr = skb_vnet_hdr(skb);
-		memcpy(hdr, hdr_p, hdr_len);
-	}
+	hdr = skb_vnet_hdr(skb);
+	memcpy(hdr, hdr_p, hdr_len);
  	if (page_to_free)
  		put_page(page_to_free);
 
-	if (metasize) {
-		__skb_pull(skb, metasize);
-		skb_metadata_set(skb, metasize);
-	}
-
  	return skb;
  }
 
@@ -934,7 +914,7 @@ static struct sk_buff *receive_big(struct net_device *dev,
  {
  	struct page *page = buf;
  	struct sk_buff *skb =
-		page_to_skb(vi, rq, page, 0, len, PAGE_SIZE, true, 0, 0);
+		page_to_skb(vi, rq, page, 0, len, PAGE_SIZE);
 
  	stats->bytes += len - vi->hdr_len;
  	if (unlikely(!skb))
@@ -1222,9 +1202,7 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
  				rcu_read_unlock();
  				put_page(page);
  				head_skb = page_to_skb(vi, rq, xdp_page, offset,
-						       len, PAGE_SIZE, false,
-						       metasize,
-						       headroom);
+						       len, PAGE_SIZE);
  				return head_skb;
  			}
  			break;
@@ -1289,8 +1267,7 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
  	rcu_read_unlock();
 
  skip_xdp:
-	head_skb = page_to_skb(vi, rq, page, offset, len, truesize, !xdp_prog,
-			       metasize, headroom);
+	head_skb = page_to_skb(vi, rq, page, offset, len, truesize);
  	curr_skb = head_skb;
 
  	if (unlikely(!curr_skb))
author
#33 lore raw

Re: [PATCH v2 8/9] virtio_net: remove xdp related info from page_to_skb()

Quoted 11 lines · Jason Wang
Quoted 6 lines · Heng Qi

For the clear construction of xdp_buff, we remove the xdp processing interleaved with page_to_skb(). Now, the logic of xdp and building skb from xdp are separate and independent.

Thanks for your energy.

Quoted 105 lines

Thanks

Quoted 101 lines

---
  drivers/net/virtio_net.c | 41 +++++++++-------------------------------

  1 file changed, 9 insertions(+), 32 deletions(-)
drivers/net/virtio_net.c +1 -3
index 4e12196fcfd4..398ffe2a5084 100644
@@ -439,9 +439,7 @@ static unsigned int 
 mergeable_ctx_to_truesize(void *mrg_ctx)
   static struct sk_buff *page_to_skb(struct virtnet_info *vi,
                      struct receive_queue *rq,
                      struct page *page, unsigned int offset,
-                   unsigned int len, unsigned int truesize,
-                   bool hdr_valid, unsigned int metasize,
-                   unsigned int headroom)
+                   unsigned int len, unsigned int truesize)
   {
       struct sk_buff *skb;

      struct virtio_net_hdr_mrg_rxbuf *hdr;

drivers/net/virtio_net.c +0 -0
@@ -459,21 +457,11 @@ static struct sk_buff *page_to_skb(struct 
 virtnet_info *vi,

      else
          hdr_padded_len = sizeof(struct padded_vnet_hdr);
  -    /* If headroom is not 0, there is an offset between the beginning of the -     * data and the allocated space, otherwise the data and the allocated
-     * space are aligned.
-     *
-     * Buffers with headroom use PAGE_SIZE as alloc size, see
-     * add_recvbuf_mergeable() + get_mergeable_buf_len()
-     */
-    truesize = headroom ? PAGE_SIZE : truesize;
-    tailroom = truesize - headroom;
-    buf = p - headroom;
-
+    buf = p;
      len -= hdr_len;
      offset += hdr_padded_len;
      p += hdr_padded_len;
-    tailroom -= hdr_padded_len + len;
+    tailroom = truesize - hdr_padded_len - len;
        shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); @@ -503,7 +491,7 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi, if (len <= skb_tailroom(skb))
          copy = len;
      else
-        copy = ETH_HLEN + metasize;
+        copy = ETH_HLEN;
      skb_put_data(skb, p, copy);
        len -= copy;

drivers/net/virtio_net.c +0 -0
@@ -542,19 +530,11 @@ static struct sk_buff *page_to_skb(struct 
 virtnet_info *vi,
           give_pages(rq, page);

    ok:
-    /* hdr_valid means no XDP, so we can copy the vnet header */
-    if (hdr_valid) {
-        hdr = skb_vnet_hdr(skb);
-        memcpy(hdr, hdr_p, hdr_len);
-    }
+    hdr = skb_vnet_hdr(skb);
+    memcpy(hdr, hdr_p, hdr_len);
      if (page_to_free)
          put_page(page_to_free);
  -    if (metasize) {
-        __skb_pull(skb, metasize);
-        skb_metadata_set(skb, metasize);
-    }
-
      return skb;
  }
  @@ -934,7 +914,7 @@ static struct sk_buff *receive_big(struct net_device *dev, {
      struct page *page = buf;
      struct sk_buff *skb =
-        page_to_skb(vi, rq, page, 0, len, PAGE_SIZE, true, 0, 0); +        page_to_skb(vi, rq, page, 0, len, PAGE_SIZE);
        stats->bytes += len - vi->hdr_len;
      if (unlikely(!skb))

drivers/net/virtio_net.c +1 -3
@@ -1222,9 +1202,7 @@ static struct sk_buff *receive_mergeable(struct 
 net_device *dev,
                   rcu_read_unlock();
                   put_page(page);
                   head_skb = page_to_skb(vi, rq, xdp_page, offset,
-                               len, PAGE_SIZE, false,
-                               metasize,
-                               headroom);
+                               len, PAGE_SIZE);
                   return head_skb;
               }

              break;

drivers/net/virtio_net.c +0 -0
@@ -1289,8 +1267,7 @@ static struct sk_buff *receive_mergeable(struct 
 net_device *dev,
       rcu_read_unlock();

    skip_xdp:
-    head_skb = page_to_skb(vi, rq, page, offset, len, truesize, !xdp_prog,
-                   metasize, headroom);
+    head_skb = page_to_skb(vi, rq, page, offset, len, truesize); curr_skb = head_skb;
        if (unlikely(!curr_skb))

author
#34 lore raw

PATCH v2 9/9 virtio_net: support multi-buffer xdp

In reply to Heng Qi

Driver can pass the skb to stack by build_skb_from_xdp_buff().

Driver forwards multi-buffer packets using the send queue when XDP_TX and XDP_REDIRECT, and clears the reference of multi pages when XDP_DROP.

---
 drivers/net/virtio_net.c | 65 ++++++----------------------------------
 1 file changed, 9 insertions(+), 56 deletions(-)
drivers/net/virtio_net.c +9 -56
index 398ffe2a5084..daa380b9d1cc 100644
@@ -1074,7 +1074,6 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 	struct bpf_prog *xdp_prog;
 	unsigned int truesize = mergeable_ctx_to_truesize(ctx);
 	unsigned int headroom = mergeable_ctx_to_headroom(ctx);
-	unsigned int metasize = 0;
 	unsigned int frame_sz;
 	int err;
 
@@ -1165,63 +1164,22 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 
 		switch (act) {
 		case XDP_PASS:
-			metasize = xdp.data - xdp.data_meta;
-
-			/* recalculate offset to account for any header
-			 * adjustments and minus the metasize to copy the
-			 * metadata in page_to_skb(). Note other cases do not
-			 * build an skb and avoid using offset
-			 */
-			offset = xdp.data - page_address(xdp_page) -
-				 vi->hdr_len - metasize;
-
-			/* recalculate len if xdp.data, xdp.data_end or
-			 * xdp.data_meta were adjusted
-			 */
-			len = xdp.data_end - xdp.data + vi->hdr_len + metasize;
-
-			/* recalculate headroom if xdp.data or xdp_data_meta
-			 * were adjusted, note that offset should always point
-			 * to the start of the reserved bytes for virtio_net
-			 * header which are followed by xdp.data, that means
-			 * that offset is equal to the headroom (when buf is
-			 * starting at the beginning of the page, otherwise
-			 * there is a base offset inside the page) but it's used
-			 * with a different starting point (buf start) than
-			 * xdp.data (buf start + vnet hdr size). If xdp.data or
-			 * data_meta were adjusted by the xdp prog then the
-			 * headroom size has changed and so has the offset, we
-			 * can use data_hard_start, which points at buf start +
-			 * vnet hdr size, to calculate the new headroom and use
-			 * it later to compute buf start in page_to_skb()
-			 */
-			headroom = xdp.data - xdp.data_hard_start - metasize;
-
-			/* We can only create skb based on xdp_page. */
-			if (unlikely(xdp_page != page)) {
-				rcu_read_unlock();
-				put_page(page);
-				head_skb = page_to_skb(vi, rq, xdp_page, offset,
-						       len, PAGE_SIZE);
-				return head_skb;
-			}
-			break;
+			head_skb = build_skb_from_xdp_buff(dev, vi, &xdp, xdp_frags_truesz);
+			rcu_read_unlock();
+			return head_skb;
 		case XDP_TX:
 			stats->xdp_tx++;
 			xdpf = xdp_convert_buff_to_frame(&xdp);
 			if (unlikely(!xdpf)) {
-				if (unlikely(xdp_page != page))
-					put_page(xdp_page);
-				goto err_xdp;
+				netdev_dbg(dev, "convert buff to frame failed for xdp\n");
+				goto err_xdp_frags;
 			}
 			err = virtnet_xdp_xmit(dev, 1, &xdpf, 0);
 			if (unlikely(!err)) {
 				xdp_return_frame_rx_napi(xdpf);
 			} else if (unlikely(err < 0)) {
 				trace_xdp_exception(vi->dev, xdp_prog, act);
-				if (unlikely(xdp_page != page))
-					put_page(xdp_page);
-				goto err_xdp;
+				goto err_xdp_frags;
 			}
 			*xdp_xmit |= VIRTIO_XDP_TX;
 			if (unlikely(xdp_page != page))
@@ -1231,11 +1189,8 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 		case XDP_REDIRECT:
 			stats->xdp_redirects++;
 			err = xdp_do_redirect(dev, &xdp, xdp_prog);
-			if (err) {
-				if (unlikely(xdp_page != page))
-					put_page(xdp_page);
-				goto err_xdp;
-			}
+			if (err)
+				goto err_xdp_frags;
 			*xdp_xmit |= VIRTIO_XDP_REDIR;
 			if (unlikely(xdp_page != page))
 				put_page(page);
@@ -1248,9 +1203,7 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 			trace_xdp_exception(vi->dev, xdp_prog, act);
 			fallthrough;
 		case XDP_DROP:
-			if (unlikely(xdp_page != page))
-				__free_pages(xdp_page, 0);
-			goto err_xdp;
+			goto err_xdp_frags;
 		}
 err_xdp_frags:
 		shinfo = xdp_get_shared_info_from_buff(&xdp);
Signature · 1 line
2.19.1.6.gb485710b
Acked-by on v2 9/9
#35 lore raw

Re: [PATCH v2 9/9] virtio_net: support multi-buffer xdp

Quoted 9 lines · Heng Qi

Driver can pass the skb to stack by build_skb_from_xdp_buff().

Driver forwards multi-buffer packets using the send queue when XDP_TX and XDP_REDIRECT, and clears the reference of multi pages when XDP_DROP.

Thanks

Quoted 114 lines
---
 drivers/net/virtio_net.c | 65 ++++++----------------------------------
 1 file changed, 9 insertions(+), 56 deletions(-)
drivers/net/virtio_net.c +9 -57
index 398ffe2a5084..daa380b9d1cc 100644
@@ -1074,7 +1074,6 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
        struct bpf_prog *xdp_prog;
        unsigned int truesize = mergeable_ctx_to_truesize(ctx);
        unsigned int headroom = mergeable_ctx_to_headroom(ctx);
-       unsigned int metasize = 0;
        unsigned int frame_sz;
        int err;
 
@@ -1165,63 +1164,22 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 
                switch (act) {
                case XDP_PASS:
-                       metasize = xdp.data - xdp.data_meta;
-
-                       /* recalculate offset to account for any header
-                        * adjustments and minus the metasize to copy the
-                        * metadata in page_to_skb(). Note other cases do not
-                        * build an skb and avoid using offset
-                        */
-                       offset = xdp.data - page_address(xdp_page) -
-                                vi->hdr_len - metasize;
-
-                       /* recalculate len if xdp.data, xdp.data_end or
-                        * xdp.data_meta were adjusted
-                        */
-                       len = xdp.data_end - xdp.data + vi->hdr_len + metasize;
-
-                       /* recalculate headroom if xdp.data or xdp_data_meta
-                        * were adjusted, note that offset should always point
-                        * to the start of the reserved bytes for virtio_net
-                        * header which are followed by xdp.data, that means
-                        * that offset is equal to the headroom (when buf is
-                        * starting at the beginning of the page, otherwise
-                        * there is a base offset inside the page) but it's used
-                        * with a different starting point (buf start) than
-                        * xdp.data (buf start + vnet hdr size). If xdp.data or
-                        * data_meta were adjusted by the xdp prog then the
-                        * headroom size has changed and so has the offset, we
-                        * can use data_hard_start, which points at buf start +
-                        * vnet hdr size, to calculate the new headroom and use
-                        * it later to compute buf start in page_to_skb()
-                        */
-                       headroom = xdp.data - xdp.data_hard_start - metasize;
-
-                       /* We can only create skb based on xdp_page. */
-                       if (unlikely(xdp_page != page)) {
-                               rcu_read_unlock();
-                               put_page(page);
-                               head_skb = page_to_skb(vi, rq, xdp_page, offset,
-                                                      len, PAGE_SIZE);
-                               return head_skb;
-                       }
-                       break;
+                       head_skb = build_skb_from_xdp_buff(dev, vi, &xdp, xdp_frags_truesz);
+                       rcu_read_unlock();
+                       return head_skb;
                case XDP_TX:
                        stats->xdp_tx++;
                        xdpf = xdp_convert_buff_to_frame(&xdp);
                        if (unlikely(!xdpf)) {
-                               if (unlikely(xdp_page != page))
-                                       put_page(xdp_page);
-                               goto err_xdp;
+                               netdev_dbg(dev, "convert buff to frame failed for xdp\n");
+                               goto err_xdp_frags;
                        }
                        err = virtnet_xdp_xmit(dev, 1, &xdpf, 0);
                        if (unlikely(!err)) {
                                xdp_return_frame_rx_napi(xdpf);
                        } else if (unlikely(err < 0)) {
                                trace_xdp_exception(vi->dev, xdp_prog, act);
-                               if (unlikely(xdp_page != page))
-                                       put_page(xdp_page);
-                               goto err_xdp;
+                               goto err_xdp_frags;
                        }
                        *xdp_xmit |= VIRTIO_XDP_TX;
                        if (unlikely(xdp_page != page))
@@ -1231,11 +1189,8 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
                case XDP_REDIRECT:
                        stats->xdp_redirects++;
                        err = xdp_do_redirect(dev, &xdp, xdp_prog);
-                       if (err) {
-                               if (unlikely(xdp_page != page))
-                                       put_page(xdp_page);
-                               goto err_xdp;
-                       }
+                       if (err)
+                               goto err_xdp_frags;
                        *xdp_xmit |= VIRTIO_XDP_REDIR;
                        if (unlikely(xdp_page != page))
                                put_page(page);
@@ -1248,9 +1203,7 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
                        trace_xdp_exception(vi->dev, xdp_prog, act);
                        fallthrough;
                case XDP_DROP:
-                       if (unlikely(xdp_page != page))
-                               __free_pages(xdp_page, 0);
-                       goto err_xdp;
+                       goto err_xdp_frags;
                }
 err_xdp_frags:
                shinfo = xdp_get_shared_info_from_buff(&xdp);
--

2.19.1.6.gb485710b

author
#36 lore raw

Re: [PATCH v2 9/9] virtio_net: support multi-buffer xdp

Quoted 10 lines · Jason Wang
Quoted 8 lines · Heng Qi

Driver can pass the skb to stack by build_skb_from_xdp_buff().

Driver forwards multi-buffer packets using the send queue when XDP_TX and XDP_REDIRECT, and clears the reference of multi pages when XDP_DROP.

Thanks for your energy.

Quoted 117 lines

Thanks

Quoted 114 lines
---
  drivers/net/virtio_net.c | 65 ++++++----------------------------------
  1 file changed, 9 insertions(+), 56 deletions(-)
drivers/net/virtio_net.c +9 -57
index 398ffe2a5084..daa380b9d1cc 100644
@@ -1074,7 +1074,6 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
         struct bpf_prog *xdp_prog;
         unsigned int truesize = mergeable_ctx_to_truesize(ctx);
         unsigned int headroom = mergeable_ctx_to_headroom(ctx);
-       unsigned int metasize = 0;
         unsigned int frame_sz;
         int err;
 
@@ -1165,63 +1164,22 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 
                 switch (act) {
                 case XDP_PASS:
-                       metasize = xdp.data - xdp.data_meta;
-
-                       /* recalculate offset to account for any header
-                        * adjustments and minus the metasize to copy the
-                        * metadata in page_to_skb(). Note other cases do not
-                        * build an skb and avoid using offset
-                        */
-                       offset = xdp.data - page_address(xdp_page) -
-                                vi->hdr_len - metasize;
-
-                       /* recalculate len if xdp.data, xdp.data_end or
-                        * xdp.data_meta were adjusted
-                        */
-                       len = xdp.data_end - xdp.data + vi->hdr_len + metasize;
-
-                       /* recalculate headroom if xdp.data or xdp_data_meta
-                        * were adjusted, note that offset should always point
-                        * to the start of the reserved bytes for virtio_net
-                        * header which are followed by xdp.data, that means
-                        * that offset is equal to the headroom (when buf is
-                        * starting at the beginning of the page, otherwise
-                        * there is a base offset inside the page) but it's used
-                        * with a different starting point (buf start) than
-                        * xdp.data (buf start + vnet hdr size). If xdp.data or
-                        * data_meta were adjusted by the xdp prog then the
-                        * headroom size has changed and so has the offset, we
-                        * can use data_hard_start, which points at buf start +
-                        * vnet hdr size, to calculate the new headroom and use
-                        * it later to compute buf start in page_to_skb()
-                        */
-                       headroom = xdp.data - xdp.data_hard_start - metasize;
-
-                       /* We can only create skb based on xdp_page. */
-                       if (unlikely(xdp_page != page)) {
-                               rcu_read_unlock();
-                               put_page(page);
-                               head_skb = page_to_skb(vi, rq, xdp_page, offset,
-                                                      len, PAGE_SIZE);
-                               return head_skb;
-                       }
-                       break;
+                       head_skb = build_skb_from_xdp_buff(dev, vi, &xdp, xdp_frags_truesz);
+                       rcu_read_unlock();
+                       return head_skb;
                 case XDP_TX:
                         stats->xdp_tx++;
                         xdpf = xdp_convert_buff_to_frame(&xdp);
                         if (unlikely(!xdpf)) {
-                               if (unlikely(xdp_page != page))
-                                       put_page(xdp_page);
-                               goto err_xdp;
+                               netdev_dbg(dev, "convert buff to frame failed for xdp\n");
+                               goto err_xdp_frags;
                         }
                         err = virtnet_xdp_xmit(dev, 1, &xdpf, 0);
                         if (unlikely(!err)) {
                                 xdp_return_frame_rx_napi(xdpf);
                         } else if (unlikely(err < 0)) {
                                 trace_xdp_exception(vi->dev, xdp_prog, act);
-                               if (unlikely(xdp_page != page))
-                                       put_page(xdp_page);
-                               goto err_xdp;
+                               goto err_xdp_frags;
                         }
                         *xdp_xmit |= VIRTIO_XDP_TX;
                         if (unlikely(xdp_page != page))
@@ -1231,11 +1189,8 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
                 case XDP_REDIRECT:
                         stats->xdp_redirects++;
                         err = xdp_do_redirect(dev, &xdp, xdp_prog);
-                       if (err) {
-                               if (unlikely(xdp_page != page))
-                                       put_page(xdp_page);
-                               goto err_xdp;
-                       }
+                       if (err)
+                               goto err_xdp_frags;
                         *xdp_xmit |= VIRTIO_XDP_REDIR;
                         if (unlikely(xdp_page != page))
                                 put_page(page);
@@ -1248,9 +1203,7 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
                         trace_xdp_exception(vi->dev, xdp_prog, act);
                         fallthrough;
                 case XDP_DROP:
-                       if (unlikely(xdp_page != page))
-                               __free_pages(xdp_page, 0);
-                       goto err_xdp;
+                       goto err_xdp_frags;
                 }
  err_xdp_frags:
                 shinfo = xdp_get_shared_info_from_buff(&xdp);
--

2.19.1.6.gb485710b

maintainer
#37 lore raw
In reply to Heng Qi
Quoted 7 lines · On Tue, 20 Dec 2022 22:14:40 +0800 Heng Qi wrote:

Changes since RFC:
- Using headroom instead of vi->xdp_enabled to avoid re-reading in add_recvbuf_mergeable();
- Disable GRO_HW and keep linearization for single buffer xdp;
- Renamed to virtnet_build_xdp_buff_mrg();
- pr_debug() to netdev_dbg();
- Adjusted the order of the patch series.

# Form letter - net-next is closed

We have already submitted the networking pull request to Linus for v6.2 and therefore net-next is closed for new drivers, features, code refactoring and optimizations. We are currently accepting bug fixes only.

Please repost when net-next reopens after Jan 2nd.

RFC patches sent for review only are obviously welcome at any time.

author
#38 lore raw
Quoted 10 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 20 Dec 2022 22:14:40 +0800 Heng Qi wrote:

Changes since RFC:
- Using headroom instead of vi->xdp_enabled to avoid re-reading in add_recvbuf_mergeable();
- Disable GRO_HW and keep linearization for single buffer xdp;
- Renamed to virtnet_build_xdp_buff_mrg();
- pr_debug() to netdev_dbg();
- Adjusted the order of the patch series.

# Form letter - net-next is closed

Quoted 8 lines

We have already submitted the networking pull request to Linus for v6.2 and therefore net-next is closed for new drivers, features, code refactoring and optimizations. We are currently accepting bug fixes only.

Please repost when net-next reopens after Jan 2nd.

RFC patches sent for review only are obviously welcome at any time.

Yes, I understand, we can also review this patch series first.

Thanks.

author
#39 lore raw
In reply to Heng Qi

Hi Jason, do you have any comments on this?

Thanks.

Quoted 39 lines · Heng Qi

Changes since RFC:
- Using headroom instead of vi->xdp_enabled to avoid re-reading in add_recvbuf_mergeable();
- Disable GRO_HW and keep linearization for single buffer xdp;
- Renamed to virtnet_build_xdp_buff_mrg();
- pr_debug() to netdev_dbg();
- Adjusted the order of the patch series.

Currently, virtio net only supports xdp for single-buffer packets or linearized multi-buffer packets. This patchset supports xdp for multi-buffer packets, then larger MTU can be used if xdp sets the xdp.frags. This does not affect single buffer handling.

In order to build multi-buffer xdp neatly, we integrated the code into virtnet_build_xdp_buff_mrg() for xdp. The first buffer is used for prepared xdp buff, and the rest of the buffers are added to its skb_shared_info structure. This structure can also be conveniently converted during XDP_PASS to get the corresponding skb.

Since virtio net uses comp pages, and bpf_xdp_frags_increase_tail() is based on the assumption of the page pool, (rxq->frag_size - skb_frag_size(frag) - skb_frag_off(frag)) is negative in most cases. So we didn't set xdp_rxq->frag_size in virtnet_open() to disable the tail increase.

Heng Qi (9):
virtio_net: disable the hole mechanism for xdp
virtio_net: set up xdp for multi buffer packets
virtio_net: update bytes calculation for xdp_frame
virtio_net: build xdp_buff with multi buffers
virtio_net: construct multi-buffer xdp in mergeable
virtio_net: transmit the multi-buffer xdp
virtio_net: build skb from multi-buffer xdp
virtio_net: remove xdp related info from page_to_skb()
virtio_net: support multi-buffer xdp

  drivers/net/virtio_net.c | 332 ++++++++++++++++++++++++++-------------
  1 file changed, 219 insertions(+), 113 deletions(-)
#40 lore raw
Quoted 3 lines · Heng Qi

Hi Jason, do you have any comments on this?

I will review this no later than the end of this week.

Thanks

Quoted 44 lines

Thanks.

Quoted 39 lines · Heng Qi

Changes since RFC:
- Using headroom instead of vi->xdp_enabled to avoid re-reading in add_recvbuf_mergeable();
- Disable GRO_HW and keep linearization for single buffer xdp;
- Renamed to virtnet_build_xdp_buff_mrg();
- pr_debug() to netdev_dbg();
- Adjusted the order of the patch series.

Currently, virtio net only supports xdp for single-buffer packets or linearized multi-buffer packets. This patchset supports xdp for multi-buffer packets, then larger MTU can be used if xdp sets the xdp.frags. This does not affect single buffer handling.

In order to build multi-buffer xdp neatly, we integrated the code into virtnet_build_xdp_buff_mrg() for xdp. The first buffer is used for prepared xdp buff, and the rest of the buffers are added to its skb_shared_info structure. This structure can also be conveniently converted during XDP_PASS to get the corresponding skb.

Since virtio net uses comp pages, and bpf_xdp_frags_increase_tail() is based on the assumption of the page pool, (rxq->frag_size - skb_frag_size(frag) - skb_frag_off(frag)) is negative in most cases. So we didn't set xdp_rxq->frag_size in virtnet_open() to disable the tail increase.

Heng Qi (9):
virtio_net: disable the hole mechanism for xdp
virtio_net: set up xdp for multi buffer packets
virtio_net: update bytes calculation for xdp_frame
virtio_net: build xdp_buff with multi buffers
virtio_net: construct multi-buffer xdp in mergeable
virtio_net: transmit the multi-buffer xdp
virtio_net: build skb from multi-buffer xdp
virtio_net: remove xdp related info from page_to_skb()
virtio_net: support multi-buffer xdp

  drivers/net/virtio_net.c | 332 ++++++++++++++++++++++++++-------------
  1 file changed, 219 insertions(+), 113 deletions(-)