add multi-buff support for xdp running in generic mode

Allocate percpu page_pools in softnet_data.
Moreover add cpuid filed in page_pool struct in order to recycle the page in the page_pool "hot" cache if napi_pp_put_page() is running on the same cpu.
This is a preliminary patch to add xdp multi-buff support for xdp running in generic mode.

Signed-off-by Lorenzo Bianconi <[email protected]>

---
 include/linux/netdevice.h       |  1 +
 include/net/page_pool/helpers.h |  5 +++++
 include/net/page_pool/types.h   |  1 +
 net/core/dev.c                  | 39 ++++++++++++++++++++++++++++++++-
 net/core/page_pool.c            |  5 +++++
 net/core/skbuff.c               |  5 +++--
 6 files changed, 53 insertions(+), 3 deletions(-)

Quoted 8 lines · Lorenzo Bianconi

Allocate percpu page_pools in softnet_data.
Moreover add cpuid filed in page_pool struct in order to recycle the page in the page_pool "hot" cache if napi_pp_put_page() is running on the same cpu.
This is a preliminary patch to add xdp multi-buff support for xdp running in generic mode.

Signed-off-by Lorenzo Bianconi <[email protected]>

Quoted 8 lines

---
  include/linux/netdevice.h       |  1 +
  include/net/page_pool/helpers.h |  5 +++++
  include/net/page_pool/types.h   |  1 +
  net/core/dev.c                  | 39 ++++++++++++++++++++++++++++++++-
  net/core/page_pool.c            |  5 +++++
  net/core/skbuff.c               |  5 +++--
  6 files changed, 53 insertions(+), 3 deletions(-)

@Jesper, @Ilias: could you please have a look at the pp bits?

Quoted 18 lines · Lorenzo Bianconi

Quoted 8 lines

Allocate percpu page_pools in softnet_data.
Moreover add cpuid filed in page_pool struct in order to recycle the page in the page_pool "hot" cache if napi_pp_put_page() is running on the same cpu.
This is a preliminary patch to add xdp multi-buff support for xdp running in generic mode.

Signed-off-by Lorenzo Bianconi <[email protected]>

Quoted 8 lines

---
  include/linux/netdevice.h       |  1 +
  include/net/page_pool/helpers.h |  5 +++++
  include/net/page_pool/types.h   |  1 +
  net/core/dev.c                  | 39 ++++++++++++++++++++++++++++++++-
  net/core/page_pool.c            |  5 +++++
  net/core/skbuff.c               |  5 +++--
  6 files changed, 53 insertions(+), 3 deletions(-)

Quoted 2 lines

@Jesper, @Ilias: could you please have a look at the pp bits?

I have some concerns... I'm still entertaining the idea, but we need to be aware of the tradeoffs we are making.

(1)
Adding PP to softnet_data means per CPU caching 256 pages in the
ptr_ring (plus likely 64 in the alloc-cache). Fortunately, PP starts
out empty, so as long as this PP isn't used they don't get cached. But if used, then PP don't have a MM shrinker that removes these cached pages, in case system is under MM pressure. I guess, you can argue that keeping this per netdev rx-queue would make memory usage even higher. This is a tradeoff, we are trading memory (waste) for speed.

(2) (Question to Jakub I guess)
How does this connect with Jakub's PP netlink stats interface? E.g. I find it very practical that this allow us get PP stats per netdev, but in this case there isn't a netdev.

(3) (Implicit locking)
PP have lockless "alloc" because it it relies on drivers NAPI context. The places where netstack access softnet_data provide similar protection that we can rely on for PP, so this is likely correct implementation wise. But it will give people like Sebastian (Cc) more gray hair when figuring out how PREEMPT_RT handle these cases.

(4)
The optimization is needed for the case where we need to re-allocate and copy SKB fragments. I think we should focus on avoiding this code path, instead of optimizing it. For UDP it should be fairly easy, but for TCP this is harder.

--Jesper

I would resume this activity and it seems to me there is no a clear direction about where we should add the page_pool (in a per_cpu pointer or in netdev_rx_queue struct) or if we can rely on page_frag_cache instead.

@Jakub: what do you think? Should we add a page_pool in a per_cpu pointer?

Quoted 5 lines

I would resume this activity and it seems to me there is no a clear direction about where we should add the page_pool (in a per_cpu pointer or in netdev_rx_queue struct) or if we can rely on page_frag_cache instead.

@Jakub: what do you think? Should we add a page_pool in a per_cpu pointer?

Let's try to summarize. We want skb reallocation without linearization for XDP generic. We need some fast-ish way to get pages for the payload.

First, options for placing the allocator:
- struct netdev_rx_queue
- per-CPU

IMO per-CPU has better scaling properties - you're less likely to increase the CPU count to infinity than spawn extra netdev queues.

The second question is:
- page_frag_cache
- page_pool

I like the page pool because we have an increasing amount of infra for it, and page pool is already used in veth, which we can hopefully also de-duplicate if we have a per-CPU one, one day. But I do agree that it's not a perfect fit.

To answer Jesper's questions:
ad1. cache size - we can lower the cache to match page_frag_cache,
so I think 8 entries? page frag cache can give us bigger frags and therefore lower frag count, so that's a minus for using page pool
ad2. nl API - we can extend netlink to dump unbound page pools fairly
easily, I didn't want to do it without a clear use case, but I don't think there are any blockers
ad3. locking - a bit independent of allocator but fair point, we assume
XDP generic or Rx path for now, so sirq context / bh locked out
ad4. right, well, right, IDK what real workloads need, and whether
XDP generic should be optimized at all.. I personally lean towards "no"

Sorry if I haven't helped much to clarify the direction :) I have no strong preference on question #2, I would prefer to not add per-queue state for something that's in no way tied to the device (question #1 -> per-CPU).

You did good perf analysis of the options, could you share it here again?

Allocate percpu page_pools in softnet_data.
Moreover add cpuid filed in page_pool struct in order to recycle the page in the page_pool "hot" cache if napi_pp_put_page() is running on the same cpu.
This is a preliminary patch to add xdp multi-buff support for xdp running in generic mode.

Signed-off-by Lorenzo Bianconi <[email protected]>

---
 include/linux/netdevice.h       |  1 +
 include/net/page_pool/helpers.h |  5 +++++
 include/net/page_pool/types.h   |  1 +
 net/core/dev.c                  | 39 ++++++++++++++++++++++++++++++++-
 net/core/page_pool.c            |  5 +++++
 net/core/skbuff.c               |  5 +++--
 6 files changed, 53 insertions(+), 3 deletions(-)

include/linux/netdevice.h +0 -0

index 1b935ee341b4..30b6a3f601fe 100644

Quoted 18 lines

Allocate percpu page_pools in softnet_data.
Moreover add cpuid filed in page_pool struct in order to recycle the page in the page_pool "hot" cache if napi_pp_put_page() is running on the same cpu.
This is a preliminary patch to add xdp multi-buff support for xdp running in generic mode.

Signed-off-by Lorenzo Bianconi <[email protected]>

---
 include/linux/netdevice.h       |  1 +
 include/net/page_pool/helpers.h |  5 +++++
 include/net/page_pool/types.h   |  1 +
 net/core/dev.c                  | 39 ++++++++++++++++++++++++++++++++-
 net/core/page_pool.c            |  5 +++++
 net/core/skbuff.c               |  5 +++--
 6 files changed, 53 insertions(+), 3 deletions(-)

include/linux/netdevice.h struct softnet_data { · line 3324

Quoted 11 lines

include/linux/netdevice.h +1 -0

index 1b935ee341b4..30b6a3f601fe 100644
@@ -3319,6 +3319,7 @@ struct softnet_data {
        int                     defer_count;
        int                     defer_ipi_scheduled;
        struct sk_buff          *defer_list;
+       struct page_pool        *page_pool;
        call_single_data_t      defer_csd;
 };

This field should be put elsewhere, not in this contended cache line.

Similar to native xdp, do not always linearize the skb in netif_receive_generic_xdp routine but create a non-linear xdp_buff to be processed by the eBPF program. This allow to add multi-buffer support for xdp running in generic mode.

Signed-off-by Lorenzo Bianconi <[email protected]>

---
  net/core/dev.c | 153 +++++++++++++++++++++++++++++++++++++++++++------
  1 file changed, 134 insertions(+), 19 deletions(-)

net/core/dev.c +14 -0

index d7857de03dba..47164acc3268 100644
@@ -4854,6 +4854,12 @@ u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
  	xdp_init_buff(xdp, frame_sz, &rxqueue->xdp_rxq);
  	xdp_prepare_buff(xdp, hard_start, skb_headroom(skb) - mac_len,
  			 skb_headlen(skb) + mac_len, true);
+	if (skb_is_nonlinear(skb)) {
+		skb_shinfo(skb)->xdp_frags_size = skb->data_len;
+		xdp_buff_set_frags_flag(xdp);
+	} else {
+		xdp_buff_clear_frags_flag(xdp);
+	}
 
  	orig_data_end = xdp->data_end;
  	orig_data = xdp->data;

@@ -4883,6 +4889,14 @@ u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
  		skb->len += off; /* positive on grow, negative on shrink */
  	}
 
+	/* XDP frag metadata (e.g. nr_frags) are updated in eBPF helpers
+	 * (e.g. bpf_xdp_adjust_tail), we need to update data_len here.
+	 */
+	if (xdp_buff_has_frags(xdp))
+		skb->data_len = skb_shinfo(skb)->xdp_frags_size;
+	else
+		skb->data_len = 0;
+
  	/* check if XDP changed eth hdr such SKB needs update */
  	eth = (struct ethhdr *)xdp->data;
  	if ((orig_eth_type != eth->h_proto) ||

net/core/dev.c +26 -0

+				     struct bpf_prog *prog)
+{
+#if IS_ENABLED(CONFIG_PAGE_POOL)
+	struct softnet_data *sd = this_cpu_ptr(&softnet_data);
+	u32 size, truesize, len, max_head_size, off;
+	struct sk_buff *skb = *pskb, *nskb;
+	int err, i, head_off;
+	void *data;
+
+	/* XDP does not support fraglist so we need to linearize
+	 * the skb.
+	 */
+	if (skb_has_frag_list(skb) || !prog->aux->xdp_has_frags)
+		return -EOPNOTSUPP;
+
+	max_head_size = SKB_WITH_OVERHEAD(PAGE_SIZE - XDP_PACKET_HEADROOM);
+	if (skb->len > max_head_size + MAX_SKB_FRAGS * PAGE_SIZE)
+		return -ENOMEM;
+
+	size = min_t(u32, skb->len, max_head_size);
+	truesize = SKB_HEAD_ALIGN(size) + XDP_PACKET_HEADROOM;
+	data = page_pool_dev_alloc_va(sd->page_pool, &truesize);
+	if (!data)
+		return -ENOMEM;
+
+	nskb = napi_build_skb(data, truesize);

net/core/dev.c +18 -0

+	if (err) {
+		consume_skb(nskb);
+		return err;
+	}
+	skb_put(nskb, size);
+
+	head_off = skb_headroom(nskb) - skb_headroom(skb);
+	skb_headers_offset_update(nskb, head_off);
+
+	off = size;
+	len = skb->len - off;
+	for (i = 0; i < MAX_SKB_FRAGS && off < skb->len; i++) {
+		struct page *page;
+		u32 page_off;
+
+		size = min_t(u32, len, PAGE_SIZE);
+		truesize = size;
+

net/core/dev.c +16 -0

+		if (err) {
+			consume_skb(nskb);
+			return err;
+		}
+
+		len -= size;
+		off += size;
+	}
+
+	consume_skb(skb);
+	*pskb = nskb;
+
+	return 0;
+#else
+	return -EOPNOTSUPP;
+#endif

net/core/dev.c +16 -1

+
+	/* In case we have to go down the path and also linearize,
+	 * then lets do the pskb_expand_head() work just once here.
+	 */
+	hroom = XDP_PACKET_HEADROOM - skb_headroom(skb);
+	troom = skb->tail + skb->data_len - skb->end;
+	err = pskb_expand_head(skb,
+			       hroom > 0 ? ALIGN(hroom, NET_SKB_PAD) : 0,
+			       troom > 0 ? troom + 128 : 0, GFP_ATOMIC);
+	if (err)
+		return err;
+
+	return skb_linearize(skb);
+}
+
  static u32 netif_receive_generic_xdp(struct sk_buff **pskb,
  				     struct xdp_buff *xdp,
  				     struct bpf_prog *xdp_prog)
  {
  	struct sk_buff *skb = *pskb;
-	u32 act = XDP_DROP;
+	u32 mac_len, act = XDP_DROP;
 
  	/* Reinjected packets coming from act_mirred or similar should
  	 * not get XDP generic processing.

net/core/dev.c +10 -15

  	 */
+	mac_len = skb->data - skb_mac_header(skb);
+	__skb_push(skb, mac_len);
+
  	if (skb_cloned(skb) || skb_is_nonlinear(skb) ||
  	    skb_headroom(skb) < XDP_PACKET_HEADROOM) {
-		int hroom = XDP_PACKET_HEADROOM - skb_headroom(skb);
-		int troom = skb->tail + skb->data_len - skb->end;
-
-		/* In case we have to go down the path and also linearize,
-		 * then lets do the pskb_expand_head() work just once here.
-		 */
-		if (pskb_expand_head(skb,
-				     hroom > 0 ? ALIGN(hroom, NET_SKB_PAD) : 0,
-				     troom > 0 ? troom + 128 : 0, GFP_ATOMIC))
-			goto do_drop;
-		if (skb_linearize(skb))
+		if (netif_skb_check_for_xdp(pskb, xdp_prog))
  			goto do_drop;
  	}
 
-	act = bpf_prog_run_generic_xdp(skb, xdp, xdp_prog);
+	__skb_pull(*pskb, mac_len);
+
+	act = bpf_prog_run_generic_xdp(*pskb, xdp, xdp_prog);
  	switch (act) {
  	case XDP_REDIRECT:
  	case XDP_TX:
  	case XDP_PASS:
  		break;
  	default:
-		bpf_warn_invalid_xdp_action(skb->dev, xdp_prog, act);
+		bpf_warn_invalid_xdp_action((*pskb)->dev, xdp_prog, act);
  		fallthrough;
  	case XDP_ABORTED:
-		trace_xdp_exception(skb->dev, xdp_prog, act);
+		trace_xdp_exception((*pskb)->dev, xdp_prog, act);
  		fallthrough;
  	case XDP_DROP:
  	do_drop:
-		kfree_skb(skb);
+		kfree_skb(*pskb);
  		break;
  	}
 

Quoted 40 lines

Similar to native xdp, do not always linearize the skb in netif_receive_generic_xdp routine but create a non-linear xdp_buff to be processed by the eBPF program. This allow to add multi-buffer support for xdp running in generic mode.

Signed-off-by Lorenzo Bianconi <[email protected]>

---
  net/core/dev.c | 153 +++++++++++++++++++++++++++++++++++++++++++------
  1 file changed, 134 insertions(+), 19 deletions(-)

net/core/dev.c +14 -0

index d7857de03dba..47164acc3268 100644
@@ -4854,6 +4854,12 @@ u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
  	xdp_init_buff(xdp, frame_sz, &rxqueue->xdp_rxq);
  	xdp_prepare_buff(xdp, hard_start, skb_headroom(skb) - mac_len,
  			 skb_headlen(skb) + mac_len, true);
+	if (skb_is_nonlinear(skb)) {
+		skb_shinfo(skb)->xdp_frags_size = skb->data_len;
+		xdp_buff_set_frags_flag(xdp);
+	} else {
+		xdp_buff_clear_frags_flag(xdp);
+	}
  	orig_data_end = xdp->data_end;
  	orig_data = xdp->data;

@@ -4883,6 +4889,14 @@ u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
  		skb->len += off; /* positive on grow, negative on shrink */
  	}
+	/* XDP frag metadata (e.g. nr_frags) are updated in eBPF helpers
+	 * (e.g. bpf_xdp_adjust_tail), we need to update data_len here.
+	 */
+	if (xdp_buff_has_frags(xdp))
+		skb->data_len = skb_shinfo(skb)->xdp_frags_size;
+	else
+		skb->data_len = 0;
+
  	/* check if XDP changed eth hdr such SKB needs update */
  	eth = (struct ethhdr *)xdp->data;
  	if ((orig_eth_type != eth->h_proto) ||

net/core/dev.c u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp, · line 4932

Quoted 4 lines

net/core/dev.c +1 -0

@@ -4916,12 +4930,118 @@ u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
  	return act;
  }
+static int netif_skb_segment_for_xdp(struct sk_buff **pskb,

This function "...segment_for_xdp" always reallocate SKB and copies all bits over.
Should it have been named "skb_realloc_for_xdp" ?

I was really hopeing we can find a design to avoid doing this realloc.

If the BPF-prog doesn't write into any of the fragments, then we can avoid this realloc (+copy) dance. We designed XDP multi-buff to have exactly the same layout+location as SKB in skb_shared_info, exactly to avoid having to reallocated.

More comments inline below...

Quoted 26 lines

net/core/dev.c +26 -0

+				     struct bpf_prog *prog)
+{
+#if IS_ENABLED(CONFIG_PAGE_POOL)
+	struct softnet_data *sd = this_cpu_ptr(&softnet_data);
+	u32 size, truesize, len, max_head_size, off;
+	struct sk_buff *skb = *pskb, *nskb;
+	int err, i, head_off;
+	void *data;
+
+	/* XDP does not support fraglist so we need to linearize
+	 * the skb.
+	 */
+	if (skb_has_frag_list(skb) || !prog->aux->xdp_has_frags)
+		return -EOPNOTSUPP;
+
+	max_head_size = SKB_WITH_OVERHEAD(PAGE_SIZE - XDP_PACKET_HEADROOM);
+	if (skb->len > max_head_size + MAX_SKB_FRAGS * PAGE_SIZE)
+		return -ENOMEM;
+
+	size = min_t(u32, skb->len, max_head_size);
+	truesize = SKB_HEAD_ALIGN(size) + XDP_PACKET_HEADROOM;
+	data = page_pool_dev_alloc_va(sd->page_pool, &truesize);
+	if (!data)
+		return -ENOMEM;
+
+	nskb = napi_build_skb(data, truesize);

net/core/dev.c u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp, · line 4969

Quoted 10 lines

net/core/dev.c +10 -0

+	if (!nskb) {
+		page_pool_free_va(sd->page_pool, data, true);
+		return -ENOMEM;
+	}
+
+	skb_reserve(nskb, XDP_PACKET_HEADROOM);
+	skb_copy_header(nskb, skb);
+	skb_mark_for_recycle(nskb);
+
+	err = skb_copy_bits(skb, 0, nskb->data, size);

This will likely copy part of the "frags" into the SKB "head" area.

Especially for netstack generated TCP packets, this will change the segmentation layout significantly. I wonder what (performance) effects this will have on further handling of these SKBs.

Quoted 26 lines

net/core/dev.c +23 -0

+	if (err) {
+		consume_skb(nskb);
+		return err;
+	}
+	skb_put(nskb, size);
+
+	head_off = skb_headroom(nskb) - skb_headroom(skb);
+	skb_headers_offset_update(nskb, head_off);
+
+	off = size;
+	len = skb->len - off;
+	for (i = 0; i < MAX_SKB_FRAGS && off < skb->len; i++) {
+		struct page *page;
+		u32 page_off;
+
+		size = min_t(u32, len, PAGE_SIZE);
+		truesize = size;
+
+		page = page_pool_dev_alloc(sd->page_pool, &page_off,
+					   &truesize);
+		if (!data) {
+			consume_skb(nskb);
+			return -ENOMEM;

net/core/dev.c u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp, · line 4997

Quoted 5 lines

net/core/dev.c +5 -0

+		}
+
+		skb_add_rx_frag(nskb, i, page, page_off, size, truesize);
+		err = skb_copy_bits(skb, off, page_address(page) + page_off,
+				    size);

I think it is correct, but we can easily endup with the new SKB (nskb) having a different nskb->nr_frags.

Quoted 24 lines

net/core/dev.c +22 -0

+		if (err) {
+			consume_skb(nskb);
+			return err;
+		}
+
+		len -= size;
+		off += size;
+	}
+
+	consume_skb(skb);
+	*pskb = nskb;
+
+	return 0;
+#else
+	return -EOPNOTSUPP;
+#endif
+}
+
+static int netif_skb_check_for_xdp(struct sk_buff **pskb,
+				   struct bpf_prog *prog)
+{
+	struct sk_buff *skb = *pskb;

net/core/dev.c u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp, · line 5023

Quoted 4 lines

net/core/dev.c +4 -0

+	int err, hroom, troom;
+
+	if (!netif_skb_segment_for_xdp(pskb, prog))
+		return 0;

IMHO the code call logic, does not make it easy to add cases where we can avoid the realloc. With this patch, it feels like the realloc+copy code path is the "main" code path for XDP-generic.

Our goal should be to avoid realloc.

My goal for XDP multi-buff was/is that it can co-exist with GSO/GRO packets. This patchset is a step in the direction of enabling GRO on devices with XDP (generic) loaded. And I was really excited about this, but the overhead is going to be massive compared to normal GRO (without realloc+copy) that XDP end-users are going to be disappointed.

Quoted 29 lines

net/core/dev.c +16 -3

+
+	/* In case we have to go down the path and also linearize,
+	 * then lets do the pskb_expand_head() work just once here.
+	 */
+	hroom = XDP_PACKET_HEADROOM - skb_headroom(skb);
+	troom = skb->tail + skb->data_len - skb->end;
+	err = pskb_expand_head(skb,
+			       hroom > 0 ? ALIGN(hroom, NET_SKB_PAD) : 0,
+			       troom > 0 ? troom + 128 : 0, GFP_ATOMIC);
+	if (err)
+		return err;
+
+	return skb_linearize(skb);
+}
+
  static u32 netif_receive_generic_xdp(struct sk_buff **pskb,
  				     struct xdp_buff *xdp,
  				     struct bpf_prog *xdp_prog)
  {
  	struct sk_buff *skb = *pskb;
-	u32 act = XDP_DROP;
+	u32 mac_len, act = XDP_DROP;
  	/* Reinjected packets coming from act_mirred or similar should
  	 * not get XDP generic processing.

@@ -4929,41 +5049,36 @@ static u32 netif_receive_generic_xdp(struct sk_buff **pskb,
  	if (skb_is_redirected(skb))
  		return XDP_PASS;
-	/* XDP packets must be linear and must have sufficient headroom
-	 * of XDP_PACKET_HEADROOM bytes. This is the guarantee that also

net/core/dev.c static u32 netif_receive_generic_xdp(struct sk_buff **pskb, · line 5054

Quoted 4 lines

net/core/dev.c +3 -1

-	 * native XDP provides, thus we need to do it here as well.
+	/* XDP packets must have sufficient headroom of XDP_PACKET_HEADROOM
+	 * bytes. This is the guarantee that also native XDP provides,
+	 * thus we need to do it here as well.

Some "native" XDP provider only have 192 bytes as HEADROOM and XDP code can this not being static (256 bytes). So, perhaps it is time to allow XDP generic to only require 192 bytes?

Quoted 44 lines

net/core/dev.c +10 -15

  	 */
+	mac_len = skb->data - skb_mac_header(skb);
+	__skb_push(skb, mac_len);
+
  	if (skb_cloned(skb) || skb_is_nonlinear(skb) ||
  	    skb_headroom(skb) < XDP_PACKET_HEADROOM) {
-		int hroom = XDP_PACKET_HEADROOM - skb_headroom(skb);
-		int troom = skb->tail + skb->data_len - skb->end;
-
-		/* In case we have to go down the path and also linearize,
-		 * then lets do the pskb_expand_head() work just once here.
-		 */
-		if (pskb_expand_head(skb,
-				     hroom > 0 ? ALIGN(hroom, NET_SKB_PAD) : 0,
-				     troom > 0 ? troom + 128 : 0, GFP_ATOMIC))
-			goto do_drop;
-		if (skb_linearize(skb))
+		if (netif_skb_check_for_xdp(pskb, xdp_prog))
  			goto do_drop;
  	}
-	act = bpf_prog_run_generic_xdp(skb, xdp, xdp_prog);
+	__skb_pull(*pskb, mac_len);
+
+	act = bpf_prog_run_generic_xdp(*pskb, xdp, xdp_prog);
  	switch (act) {
  	case XDP_REDIRECT:
  	case XDP_TX:
  	case XDP_PASS:
  		break;
  	default:
-		bpf_warn_invalid_xdp_action(skb->dev, xdp_prog, act);
+		bpf_warn_invalid_xdp_action((*pskb)->dev, xdp_prog, act);
  		fallthrough;
  	case XDP_ABORTED:
-		trace_xdp_exception(skb->dev, xdp_prog, act);
+		trace_xdp_exception((*pskb)->dev, xdp_prog, act);
  		fallthrough;
  	case XDP_DROP:
  	do_drop:
-		kfree_skb(skb);
+		kfree_skb(*pskb);
  		break;
  	}