io_uring/net: don't charge the folio offset to truesize for managed frags
Subsystems (2)
PATCH 1/1 io_uring/net: don't charge the folio offset to truesize for managed frags
Registered buffers backed by huge pages are coalesced into one bvec per folio, so bv_offset is relative to the folio, not to a 4K page. io_sg_from_iter() adds PAGE_ALIGN(bv_len + bv_offset) to the skb truesize, which charges the socket for the whole run-up inside the folio. A 64K send from 1.5MB into a 2MB THP folio adds about 1.6MB of truesize, and sk_wmem_queued fills long before any real memory is used.
The visible effect is a socket that looks full: with a 2MB send buffer only 4 of 128 queued 64K sends complete before the sender blocks, versus 30 of 128 with the same buffer registered as 4K pages, and most pipelined send_zc requests come back short.
Only the in-page part of the offset matters for how much memory a fragment ties up, so use offset_in_page(). This matches what zerocopy_fill_skb_from_iter() charges for the same data.
--- io_uring/net.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
io_uring/net.c +6 -1
@@ -1501,7 +1501,12 @@ static int io_sg_from_iter(struct sk_buff *skb,struct bio_vec v =mp_bvec_iter_bvec (from->bvec, bi); copied += v.bv_len; - truesize +=PAGE_ALIGN (v.bv_len + v.bv_offset); +/* +* bv_offset is relative to the folio for coalesced huge page +* registrations, so only the in-page part of it matters for +* how much memory this fragment ties up. +*/ + truesize +=PAGE_ALIGN (v.bv_len +offset_in_page (v.bv_offset));__skb_fill_page_desc_noacc (shinfo, frag++, v.bv_page, v.bv_offset, v.bv_len);bvec_iter_advance_single (from->bvec, &bi, v.bv_len);
base-commit: 0659e44ce739ecd856164c75a5325b7959a2921d
Signature · 1 line
2.43.0
Quoted 2 lines · Louis Thiery
Only the in-page part of the offset matters for how much memory a fragment ties up, so use
offset_in_page().
This is the change suggested in my report earlier today:
https://lore.kernel.org/io-uring/CAC+fX10wVVSqGnfCg+aazorsCxB34sUDV-XJiBPd6R3Kuqc-7Q@mail.gmail.com/
I applied this patch to io_uring-7.4 (0659e44ce739) and ran the reproducer from that report. With it, SEND_ZC from a registered huge-page buffer is charged the same as SEND_ZC without one on every row, for hugetlb and THP, including 8192 for a send that crosses a page boundary. On the stock kernel the same run shows the overcharge.
Jens, if you apply this, could you add:
Thanks for the corroboration, Brian. Been playing around with this for a few months and missed your report from this morning. Happy to have the Reported-by, Closes and Tested-by
added.
--Louis
Quoted 21 lines · Brian Martin
Quoted 2 lines · Louis Thiery
Only the in-page part of the offset matters for how much memory a fragment ties up, so use
offset_in_page().This is the change suggested in my report earlier today:
https://lore.kernel.org/io-uring/CAC+fX10wVVSqGnfCg+aazorsCxB34sUDV-XJiBPd6R3Kuqc-7Q@mail.gmail.com/
I applied this patch to io_uring-7.4 (
0659e44ce739) and ran the reproducer from that report. With it,SEND_ZCfrom a registered huge-page buffer is charged the same asSEND_ZCwithout one on every row, for hugetlb and THP, including 8192 for a send that crosses a page boundary. On the stock kernel the same run shows the overcharge.Jens, if you apply this, could you add:
Reported-by Brian Martin <[email protected]>Tested-by Brian Martin <[email protected]>