Skip to contentSkip to conversation

page_pool: keep frag_offset aligned for odd-sized requests

PATCH Applied Florian Schauer 6 participants 6 messages Aug 28 – Sep 1

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 page_pool: keep frag_offset aligned for odd-sized requests

page_pool_alloc_frag_netmem() rounds the requested fragment size with

	size = ALIGN(size, dma_get_cache_alignment());

dma_get_cache_alignment() returns 1 unless the architecture defines ARCH_DMA_MINALIGN, which DMA-coherent architectures such as x86 do not. There the ALIGN() is a no-op and pool->frag_offset advances by the raw, unrounded size.

A single caller asking for an odd size then leaves frag_offset misaligned for every fragment carved out of that page afterwards. The pool is shared, so the damage is not confined to the caller that caused it.

The per-cpu system_page_pool used by generic XDP hits this. skb_pp_cow_data() allocates its fragments with the raw packet length:

	size = min_t(u32, len, PAGE_SIZE);
	truesize = size;
	page = page_pool_dev_alloc(pool, &page_off, &truesize);

leaving frag_offset odd for whatever is carved out of that page next. Its own head allocation is already aligned -- SKB_HEAD_ALIGN(size) plus the XDP_PACKET_HEADROOM its callers pass -- so it is a later user of the shared pool that pays: page_pool_dev_alloc_va() returns a misaligned buffer, napi_build_skb() installs it as skb->head, and skb_shinfo(skb) == skb->head + skb->end is misaligned with it.

skb_shinfo()->dataref is a 4-byte atomic_t at offset 0x20, so the atomic_inc() in __skb_clone() straddles a cache line. On x86 with split lock detection -- fatal for kernel split locks by default -- this panics the machine:

  Oops: Split lock detected
  RIP: 0010:skb_clone+0x154/0x1e0
  Call Trace:
   <IRQ>
   raw_local_deliver+0x1ed/0x2c0
   ip_protocol_deliver_rcu+0x54/0x1c0
   ip_local_deliver_finish+0x85/0x100
   ip_local_deliver+0x67/0x100
   __netif_receive_skb_one_core+0x85/0xa0
   process_backlog+0x87/0x130

Reproduced by attaching any generic-mode XDP program to loopback and opening a RAW IPPROTO_UDP socket, which makes raw_local_deliver() clone every locally delivered UDP packet; ordinary DNS traffic then triggers it, roughly once per 2500 clones. Observed on 6.12.101 and 7.1.8.

Tracing page_pool_alloc_frag_netmem() over one such run shows the amplification -- two odd-sized requests, nine misaligned offsets:

  requested size & 7:   0: 17035    5: 1    7: 1
  frag_offset & 7:      0: 17028    3: 1    4: 1    5: 1    6: 1    7: 5

and skb_pp_cow_data() returning heads that were aligned on entry:

head 0xffff8f4c86aeac00 -> 0xffff8f4c53a9a9c4 (&7=4) head 0xffff8f4d6a8a42c0 -> 0xffff8f4c4f7b7a45 (&7=5)

Round the fragment size up to at least the alignment struct skb_shared_info requires, so fragments are always suitably aligned for the objects callers build on them. Architectures needing a larger DMA alignment keep it.

This also makes the remainder computed in page_pool_alloc_netmem(),

	*size = max_size - *offset;

aligned, since max_size is a power of two -- which fixes the matching misalignment of skb->end.

Verified with a controlled A/B under QEMU/KVM: same tree, same config, same compiler, same rootfs and identical traffic, differing only by this patch. A SEC("xdp.frags") XDP_PASS program on lo plus UDP datagrams larger than max_head_size drives skb_pp_cow_data()'s fragment loop, which passes raw packet lengths to the pool. Measured at the return of skb_pp_cow_data():

unpatched patched
skb_pp_cow_data calls 40800 40800
misaligned skb->head 1120 0
dataref at line offset >60 80 0

The last row counts the accesses that actually fault:
skb_shinfo()->dataref sits at head+end+0x20 and is a 4-byte atomic, so lock incl splits a 64-byte cache line only when that address lands at offset 61..63. All 80 occurrences were at offset 61; the panic reported above was at offset 62. Eliminating the misalignment removes every one of them.

Same class of bug as commit 3bed3cc4156e ("net: Do not allocate page fragments that are not skb aligned"), which fixed the older netdev_alloc_frag()/napi_alloc_frag() allocators.

Fixes 53e0961da1c7 ("page_pool: add frag page recycling support in page pool")

---
v2:
- express the minimum alignment as __alignof__(struct skb_shared_info) instead of sizeof(long), and drop the explanatory comment the previous version carried, since the expression now states the requirement directly (Eric Dumazet)
- no functional change vs v1 on 64-bit: the emitted code is identical
- Cc the maintainers and the blamed author that v1 missed
- correct two statements in the commit message: dataref is at offset 0x20 in struct skb_shared_info, not its first member, and skb_pp_cow_data() allocates its head before the fragment loop, so the misaligned head comes from an earlier user of the shared pool
v1: https://lore.kernel.org/netdev/[email protected]/

 net/core/page_pool.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/core/page_pool.c +2 -1
index 8f8956fb0..08d7f35cf 100644
@@ -1073,7 +1073,8 @@ netmem_ref page_pool_alloc_frag_netmem(struct page_pool *pool,
 	if (WARN_ON(size > max_size))
 		return 0;
 
-	size = ALIGN(size, dma_get_cache_alignment());
+	size = ALIGN(size, max_t(unsigned int, dma_get_cache_alignment(),
+				 __alignof__(struct skb_shared_info)));
 	*offset = pool->frag_offset;
 
 	if (netmem && *offset + size > max_size) {
Signature · 1 line
2.47.3
#1 lore raw
Quoted 5 lines · Florian Schauer

Tracing page_pool_alloc_frag_netmem() over one such run shows the amplification -- two odd-sized requests, nine misaligned offsets:

  requested size & 7:   0: 17035    5: 1    7: 1
  frag_offset & 7:      0: 17028    3: 1    4: 1    5: 1    6: 1    7: 5

Two things you may not have: a competing patch for this same defect, and a measurement of how far the amplification goes under a different load.

The competing patch fixes the caller instead of the allocator:

net: skbuff: keep the page_pool fragment offset aligned in skb_pp_cow_data()
Bruno Xavier <[email protected]>, 2026-08-27
https://lore.kernel.org/netdev/[email protected]/

Its Notes: section describes your change as "the alternative" and offers to send that version instead -- the two look to have been written independently, a day apart. Both are in state "new" with no review comments on either thread, so a maintainer opening one cannot see the other. I have copied Bruno here.

On the numbers: an independent reproduction on a third configuration. Fedora 44, kernels 7.1.8 / 7.1.9 / 7.1.10, Intel i5-13420H, NetBird v0.77.1, which attaches a SEC("xdp.frags") program to lo and holds an unbound raw IPPROTO_UDP socket. Eight panics, all skb_clone+0x159, six from raw_v4_input() and two from ipv6_raw_deliver(). Same tracing as yours, over a 12 s run of a reproducer generating UDP datagrams of 1400..63000 B on loopback:

total aligned misaligned
size requested from

    page_pool_alloc_frag_netmem()     1876609      23.7%         76.3%
  *offset returned by the pool        1876609      31.9%         68.1%
  skb->head from napi_build_skb()     2314689      91.8%          8.2%

Your trace shows 2 odd-sized requests out of 17037 (0.01%); driving skb_pp_cow_data()'s fragment loop with varied datagram sizes puts it at 76%. Once an odd-sized request has moved frag_offset off alignment, the allocations carved out of that page afterwards are misaligned too, until the accumulated sizes happen to land back on a multiple of 8 -- which is why the share of misaligned offsets (68%) is so much higher than the rate of odd requests alone would suggest. If the changelog needs an argument for the stable backport, this is one: the rate is workload-dependent, and it is not bounded by anything.

On coverage, which is the part that may bear on which fix is preferred:
skb_pp_cow_data() is not the only caller passing a raw length to the per-cpu system_page_pool. xdp_copy_frags_from_zc() does the same, at net/core/xdp.c:700:

	const skb_frag_t *frag = &xinfo->frags[i];
	u32 len = skb_frag_size(frag);
	u32 offset, truesize = len;
	struct page *page;

	page = page_pool_dev_alloc(pp, &offset, &truesize);

and its caller xdp_build_skb_from_zc() takes that pp from this_cpu_read(system_page_pool.pool) at xdp.c:753, then feeds page_pool_dev_alloc_va() at xdp.c:754 into napi_build_skb() at xdp.c:758. So that path both leaves odd frag_offsets behind and consumes the head allocations that follow them, on the same per-cpu pool. All three callers of page_pool_dev_alloc() in the tree pass an unrounded size -- enic_rq.c:277-291 asks for netdev->mtu + VLAN_ETH_HLEN, plus xdp.c:702 and skbuff.c:988 -- and
where a caller is safe it is because it rounds on its own beforehand, as virtio_net does with ALIGN(len, L1_CACHE_BYTES) at virtio_net.c:2710.

I measured the attribution rather than only arguing it, and it does not settle the question: over ~75 s and some 8.6 million fragment requests, the probe saw none originating outside skb_pp_cow_data(). That is what one should expect on this box, which drives neither the AF_XDP zero-copy path nor a page_pool-backed NIC driver, so no other producer was exercised. It says the caller-side fix would be enough for this workload, not that it is enough.

One observation for the changelog, if useful: the skbs that actually panic are small and linear (48..222 B, data_len == 0, ordinary DNS traffic). The large non-linear packets are what leave frag_offset odd; they are not the victims. That makes the failure look unrelated to the traffic causing it, and it is why reports of this are easy to misattribute to whatever process happened to be running the softirq.

We have not built and run the patch here; a Tested-by: will follow separately if we measure a patched kernel.

Thanks,
Fabricio Gava

#2 lore raw
In reply to Florian Schauer
Quoted 3 lines · Florian Schauer

Round the fragment size up to at least the alignment struct skb_shared_info requires, so fragments are always suitably aligned for the objects callers build on them.

I sent a caller-side fix for the same defect a day after your v1, without having seen it:

net: skbuff: keep the page_pool fragment offset aligned in skb_pp_cow_data()
https://lore.kernel.org/netdev/[email protected]/

Fabricio connected the threads. xdp_copy_frags_from_zc() at net/core/xdp.c:700 passes a raw length to the same per-cpu pool, so the caller-side fix is not enough. Yours is the right one and I have asked for mine to be dropped.

skb_pp_cow_data() is also called from veth, drivers/net/veth.c:762, so the fragment loop runs outside generic XDP mode as well. Good to flag that in the changelog.

The skbs that actually panic are small and linear, not the large packets that leave frag_offset odd. skb->end was 114 to 178 on my traces against 384 to 955 in ordinary traffic, because the tail fragment of a page is the one that gets an arbitrary size, page_pool_alloc_netmem() setting *size = max_size - *offset. Your patch aligns that remainder too, and it is the case that actually reaches cache-line offset 61 to 63.

Another configuration for the record. ThinkPad T14 Gen 6, Fedora 44, 7.1.9-200.fc44, netbird attaching a generic XDP program to lo and holding a raw IPv4 socket. Four panics, all skb_clone+0x159, split-lock detection in the sld_warn state that still dies on kernel split locks. Tracing napi_build_skb() on the same box puts the misaligned heads on skb_pp_cow_data() <- netif_receive_generic_xdp <- do_xdp_generic, and the clones that hit them on raw_v4_input().

Building your v2 here now, Tested-by to follow.

Thanks,
Bruno

maintainer Acked-by on v2 1/1
#3 lore raw
In reply to Florian Schauer
Quoted 118 lines · Florian Schauer

page_pool_alloc_frag_netmem() rounds the requested fragment size with

	size = ALIGN(size, dma_get_cache_alignment());

dma_get_cache_alignment() returns 1 unless the architecture defines ARCH_DMA_MINALIGN, which DMA-coherent architectures such as x86 do not. There the ALIGN() is a no-op and pool->frag_offset advances by the raw, unrounded size.

A single caller asking for an odd size then leaves frag_offset misaligned for every fragment carved out of that page afterwards. The pool is shared, so the damage is not confined to the caller that caused it.

The per-cpu system_page_pool used by generic XDP hits this. skb_pp_cow_data() allocates its fragments with the raw packet length:

	size = min_t(u32, len, PAGE_SIZE);
	truesize = size;
	page = page_pool_dev_alloc(pool, &page_off, &truesize);

leaving frag_offset odd for whatever is carved out of that page next. Its own head allocation is already aligned -- SKB_HEAD_ALIGN(size) plus the XDP_PACKET_HEADROOM its callers pass -- so it is a later user of the shared pool that pays: page_pool_dev_alloc_va() returns a misaligned buffer, napi_build_skb() installs it as skb->head, and skb_shinfo(skb) == skb->head + skb->end is misaligned with it.

skb_shinfo()->dataref is a 4-byte atomic_t at offset 0x20, so the atomic_inc() in __skb_clone() straddles a cache line. On x86 with split lock detection -- fatal for kernel split locks by default -- this panics the machine:

   Oops: Split lock detected
   RIP: 0010:skb_clone+0x154/0x1e0
   Call Trace:
    <IRQ>
    raw_local_deliver+0x1ed/0x2c0
    ip_protocol_deliver_rcu+0x54/0x1c0
    ip_local_deliver_finish+0x85/0x100
    ip_local_deliver+0x67/0x100
    __netif_receive_skb_one_core+0x85/0xa0
    process_backlog+0x87/0x130

Reproduced by attaching any generic-mode XDP program to loopback and opening a RAW IPPROTO_UDP socket, which makes raw_local_deliver() clone every locally delivered UDP packet; ordinary DNS traffic then triggers it, roughly once per 2500 clones. Observed on 6.12.101 and 7.1.8.

Tracing page_pool_alloc_frag_netmem() over one such run shows the amplification -- two odd-sized requests, nine misaligned offsets:

   requested size & 7:   0: 17035    5: 1    7: 1
   frag_offset & 7:      0: 17028    3: 1    4: 1    5: 1    6: 1    7: 5

and skb_pp_cow_data() returning heads that were aligned on entry:

   head 0xffff8f4c86aeac00 -> 0xffff8f4c53a9a9c4 (&7=4)
   head 0xffff8f4d6a8a42c0 -> 0xffff8f4c4f7b7a45 (&7=5)

Round the fragment size up to at least the alignment struct skb_shared_info requires, so fragments are always suitably aligned for the objects callers build on them. Architectures needing a larger DMA alignment keep it.

This also makes the remainder computed in page_pool_alloc_netmem(),

	*size = max_size - *offset;

aligned, since max_size is a power of two -- which fixes the matching misalignment of skb->end.

Verified with a controlled A/B under QEMU/KVM: same tree, same config, same compiler, same rootfs and identical traffic, differing only by this patch. A SEC("xdp.frags") XDP_PASS program on lo plus UDP datagrams larger than max_head_size drives skb_pp_cow_data()'s fragment loop, which passes raw packet lengths to the pool. Measured at the return of skb_pp_cow_data():

unpatched patched
skb_pp_cow_data calls 40800 40800
misaligned skb->head 1120 0
dataref at line offset >60 80 0

The last row counts the accesses that actually fault:
skb_shinfo()->dataref sits at head+end+0x20 and is a 4-byte atomic, so lock incl splits a 64-byte cache line only when that address lands at offset 61..63. All 80 occurrences were at offset 61; the panic reported above was at offset 62. Eliminating the misalignment removes every one of them.

Same class of bug as commit 3bed3cc4156e ("net: Do not allocate page fragments that are not skb aligned"), which fixed the older netdev_alloc_frag()/napi_alloc_frag() allocators.

Fixes 53e0961da1c7 ("page_pool: add frag page recycling support in page pool")

---
v2:
- express the minimum alignment as __alignof__(struct skb_shared_info) instead of sizeof(long), and drop the explanatory comment the previous version carried, since the expression now states the requirement directly (Eric Dumazet)
- no functional change vs v1 on 64-bit: the emitted code is identical
- Cc the maintainers and the blamed author that v1 missed
- correct two statements in the commit message: dataref is at offset 0x20 in struct skb_shared_info, not its first member, and skb_pp_cow_data() allocates its head before the fragment loop, so the misaligned head comes from an earlier user of the shared pool
v1: https://lore.kernel.org/netdev/[email protected]/

  net/core/page_pool.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
net/core/page_pool.c +0 -0
index 8f8956fb0..08d7f35cf 100644

net/core/page_pool.c netmem_ref page_pool_alloc_frag_netmem(struct page_pool *pool, · line 1080

Quoted 9 lines
net/core/page_pool.c +2 -1
@@ -1073,7 +1073,8 @@ netmem_ref page_pool_alloc_frag_netmem(struct page_pool *pool,
  	if (WARN_ON(size > max_size))
  		return 0;
 
-	size = ALIGN(size, dma_get_cache_alignment());
+	size = ALIGN(size, max_t(unsigned int, dma_get_cache_alignment(),
+				 __alignof__(struct skb_shared_info)));
  	*offset = pool->frag_offset;
 
  	if (netmem && *offset + size > max_size) {

On x86_64 __alignof__(struct skb_shared_info) returns 8. So, this should work, to avoid getting the lock in skb_shared_info split across cache-lines.

Thanks for finding and fixing this bug!
--Jesper

Reviewed-by
#4 lore raw
Quoted 93 lines · Jesper Dangaard Brouer
Quoted 89 lines · Florian Schauer

page_pool_alloc_frag_netmem() rounds the requested fragment size with

      size = ALIGN(size, dma_get_cache_alignment());

dma_get_cache_alignment() returns 1 unless the architecture defines ARCH_DMA_MINALIGN, which DMA-coherent architectures such as x86 do not. There the ALIGN() is a no-op and pool->frag_offset advances by the raw, unrounded size.

A single caller asking for an odd size then leaves frag_offset misaligned for every fragment carved out of that page afterwards. The pool is shared, so the damage is not confined to the caller that caused it.

The per-cpu system_page_pool used by generic XDP hits this. skb_pp_cow_data() allocates its fragments with the raw packet length:

      size = min_t(u32, len, PAGE_SIZE);
      truesize = size;
      page = page_pool_dev_alloc(pool, &page_off, &truesize);

leaving frag_offset odd for whatever is carved out of that page next. Its own head allocation is already aligned -- SKB_HEAD_ALIGN(size) plus the XDP_PACKET_HEADROOM its callers pass -- so it is a later user of the shared pool that pays: page_pool_dev_alloc_va() returns a misaligned buffer, napi_build_skb() installs it as skb->head, and skb_shinfo(skb) == skb->head + skb->end is misaligned with it.

skb_shinfo()->dataref is a 4-byte atomic_t at offset 0x20, so the atomic_inc() in __skb_clone() straddles a cache line. On x86 with split lock detection -- fatal for kernel split locks by default -- this panics the machine:

   Oops: Split lock detected
   RIP: 0010:skb_clone+0x154/0x1e0
   Call Trace:
    <IRQ>
    raw_local_deliver+0x1ed/0x2c0
    ip_protocol_deliver_rcu+0x54/0x1c0
    ip_local_deliver_finish+0x85/0x100
    ip_local_deliver+0x67/0x100
    __netif_receive_skb_one_core+0x85/0xa0
    process_backlog+0x87/0x130

Reproduced by attaching any generic-mode XDP program to loopback and opening a RAW IPPROTO_UDP socket, which makes raw_local_deliver() clone every locally delivered UDP packet; ordinary DNS traffic then triggers it, roughly once per 2500 clones. Observed on 6.12.101 and 7.1.8.

Tracing page_pool_alloc_frag_netmem() over one such run shows the amplification -- two odd-sized requests, nine misaligned offsets:

   requested size & 7:   0: 17035    5: 1    7: 1
   frag_offset & 7:      0: 17028    3: 1    4: 1    5: 1    6: 1    7: 5

and skb_pp_cow_data() returning heads that were aligned on entry:

   head 0xffff8f4c86aeac00 -> 0xffff8f4c53a9a9c4 (&7=4)
   head 0xffff8f4d6a8a42c0 -> 0xffff8f4c4f7b7a45 (&7=5)

Round the fragment size up to at least the alignment struct skb_shared_info requires, so fragments are always suitably aligned for the objects callers build on them. Architectures needing a larger DMA alignment keep it.

This also makes the remainder computed in page_pool_alloc_netmem(),

      *size = max_size - *offset;

aligned, since max_size is a power of two -- which fixes the matching misalignment of skb->end.

Verified with a controlled A/B under QEMU/KVM: same tree, same config, same compiler, same rootfs and identical traffic, differing only by this patch. A SEC("xdp.frags") XDP_PASS program on lo plus UDP datagrams larger than max_head_size drives skb_pp_cow_data()'s fragment loop, which passes raw packet lengths to the pool. Measured at the return of skb_pp_cow_data():

unpatched patched
skb_pp_cow_data calls 40800 40800
misaligned skb->head 1120 0
dataref at line offset >60 80 0

The last row counts the accesses that actually fault:
skb_shinfo()->dataref sits at head+end+0x20 and is a 4-byte atomic, so lock incl splits a 64-byte cache line only when that address lands at offset 61..63. All 80 occurrences were at offset 61; the panic reported above was at offset 62. Eliminating the misalignment removes every one of them.

Quoted 7 lines
Quoted 7 lines

Same class of bug as commit 3bed3cc4156e ("net: Do not allocate page fragments that are not skb aligned"), which fixed the older netdev_alloc_frag()/napi_alloc_frag() allocators.

Fixes 53e0961da1c7 ("page_pool: add frag page recycling support in page pool")

Thanks.

#5 lore raw
In reply to Florian Schauer
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Fri, 28 Aug 2026 08:08:22 +0200 you wrote:

page_pool_alloc_frag_netmem() rounds the requested fragment size with

	size = ALIGN(size, dma_get_cache_alignment());

dma_get_cache_alignment() returns 1 unless the architecture defines ARCH_DMA_MINALIGN, which DMA-coherent architectures such as x86 do not. There the ALIGN() is a no-op and pool->frag_offset advances by the raw, unrounded size.

[...]

Here is the summary with links:
- [net,v2] page_pool: keep frag_offset aligned for odd-sized requests

    https://git.kernel.org/netdev/net/c/dc0df5a0c62c

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html