Skip to contentSkip to conversation

io_uring/net: don't charge the folio offset to truesize for managed frags

PATCH In review Louis Thiery 2 participants 3 messages Nov 19 – Nov 20

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 io_uring/net: don't charge the folio offset to truesize for managed frags

Registered buffers backed by huge pages are coalesced into one bvec per folio, so bv_offset is relative to the folio, not to a 4K page. io_sg_from_iter() adds PAGE_ALIGN(bv_len + bv_offset) to the skb truesize, which charges the socket for the whole run-up inside the folio. A 64K send from 1.5MB into a 2MB THP folio adds about 1.6MB of truesize, and sk_wmem_queued fills long before any real memory is used.

The visible effect is a socket that looks full: with a 2MB send buffer only 4 of 128 queued 64K sends complete before the sender blocks, versus 30 of 128 with the same buffer registered as 4K pages, and most pipelined send_zc requests come back short.

Only the in-page part of the offset matters for how much memory a fragment ties up, so use offset_in_page(). This matches what zerocopy_fill_skb_from_iter() charges for the same data.

Fixes 57bebf807e2a ("io_uring/rsrc: optimise registered huge pages")
---
 io_uring/net.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
io_uring/net.c +6 -1
index 533afd335..8afad1b2d 100644
@@ -1501,7 +1501,12 @@ static int io_sg_from_iter(struct sk_buff *skb,
 		struct bio_vec v = mp_bvec_iter_bvec(from->bvec, bi);
 
 		copied += v.bv_len;
-		truesize += PAGE_ALIGN(v.bv_len + v.bv_offset);
+		/*
+		 * bv_offset is relative to the folio for coalesced huge page
+		 * registrations, so only the in-page part of it matters for
+		 * how much memory this fragment ties up.
+		 */
+		truesize += PAGE_ALIGN(v.bv_len + offset_in_page(v.bv_offset));
 		__skb_fill_page_desc_noacc(shinfo, frag++, v.bv_page,
 					   v.bv_offset, v.bv_len);
 		bvec_iter_advance_single(from->bvec, &bi, v.bv_len);

base-commit: 0659e44ce739ecd856164c75a5325b7959a2921d

Signature · 1 line
2.43.0
Tested-by on v1 1/1
#1 lore raw
Quoted 2 lines · Louis Thiery

Only the in-page part of the offset matters for how much memory a fragment ties up, so use offset_in_page().

This is the change suggested in my report earlier today:

https://lore.kernel.org/io-uring/CAC+fX10wVVSqGnfCg+aazorsCxB34sUDV-XJiBPd6R3Kuqc-7Q@mail.gmail.com/

I applied this patch to io_uring-7.4 (0659e44ce739) and ran the reproducer from that report. With it, SEND_ZC from a registered huge-page buffer is charged the same as SEND_ZC without one on every row, for hugetlb and THP, including 8192 for a send that crosses a page boundary. On the stock kernel the same run shows the overcharge.

Jens, if you apply this, could you add:

#2 lore raw

Thanks for the corroboration, Brian. Been playing around with this for a few months and missed your report from this morning. Happy to have the Reported-by, Closes and Tested-by
added.
--Louis

Quoted 21 lines · Brian Martin
Quoted 2 lines · Louis Thiery

Only the in-page part of the offset matters for how much memory a fragment ties up, so use offset_in_page().

This is the change suggested in my report earlier today:

https://lore.kernel.org/io-uring/CAC+fX10wVVSqGnfCg+aazorsCxB34sUDV-XJiBPd6R3Kuqc-7Q@mail.gmail.com/

I applied this patch to io_uring-7.4 (0659e44ce739) and ran the reproducer from that report. With it, SEND_ZC from a registered huge-page buffer is charged the same as SEND_ZC without one on every row, for hugetlb and THP, including 8192 for a send that crosses a page boundary. On the stock kernel the same run shows the overcharge.

Jens, if you apply this, could you add: