tcp: fix skb_copy_ubufs() vs BIG TCP
Subsystems (2)
PATCH v2 1/1 tcp: fix skb_copy_ubufs() vs BIG TCP
David Ahern reported crashes in skb_copy_ubufs() caused by TCP tx zerocopy using hugepages, and skb length bigger than ~68 KB.
skb_copy_ubufs() assumed it could copy all payload using up to MAX_SKB_FRAGS order-0 pages.
This assumption broke when BIG TCP was able to put up to 512 KB per skb.
We did not hit this bug at Google because we use CONFIG_MAX_SKB_FRAGS=45 and limit gso_max_size to 180000.
A solution is to use higher order pages if needed.
v2: add missing __GFP_COMP, or we leak memory.
--- net/core/skbuff.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
net/core/skbuff.c +14 -6
@@ -1758,7 +1758,7 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask) {int num_frags =skb_shinfo (skb)->nr_frags;struct page *page, *head =NULL ; -int i, new_frags; +int i, order, psize, new_frags;u32 d_off;if (skb_shared (skb) ||skb_unclone (skb, gfp_mask))
@@ -1767,9 +1767,17 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask)if (!num_frags) goto release; - new_frags = (__skb_pagelen (skb) +PAGE_SIZE -1 ) >>PAGE_SHIFT ; +/* We might have to allocate high order pages, so compute what minimum +* page order is needed. +*/ + order =0 ; +while ((PAGE_SIZE << order) *MAX_SKB_FRAGS <__skb_pagelen (skb)) + order++; + psize = (PAGE_SIZE << order); + + new_frags = (__skb_pagelen (skb) + psize -1 ) >> (PAGE_SHIFT + order);for (i =0 ; i < new_frags; i++) { - page =alloc_page (gfp_mask); + page =alloc_pages (gfp_mask | __GFP_COMP, order);if (!page) {while (head) {struct page *next = (struct page *)page_private (head);
@@ -1796,11 +1804,11 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask) vaddr =kmap_atomic (p);while (done < p_len) { -if (d_off ==PAGE_SIZE ) { +if (d_off == psize) { d_off =0 ; page = (struct page *)page_private (page); } - copy =min_t (u32,PAGE_SIZE - d_off, p_len - done); + copy =min_t (u32, psize - d_off, p_len - done);memcpy (page_address (page) + d_off, vaddr + p_off + done, copy); done += copy;
@@ -1816,7 +1824,7 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask)/* skb frags point to kernel buffers */ for (i =0 ; i < new_frags -1 ; i++) { -__skb_fill_page_desc (skb, i, head,0 ,PAGE_SIZE ); +__skb_fill_page_desc (skb, i, head,0 , psize); head = (struct page *)page_private (head); }__skb_fill_page_desc (skb, new_frags -1 , head,0 , d_off);
Signature · 1 line
2.40.1.495.gc816e09b53d-goog
This patch was applied to netdev/net.git (main) by David S. Miller <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by David S. Miller <[email protected]>:
Quoted 9 lines · On Fri, 28 Apr 2023 04:32:31 +0000 you wrote:
David Ahern reported crashes in
skb_copy_ubufs()caused by TCP tx zerocopy using hugepages, and skb length bigger than ~68 KB.
skb_copy_ubufs()assumed it could copy all payload using up toMAX_SKB_FRAGSorder-0 pages.This assumption broke when BIG TCP was able to put up to 512 KB per skb.
[...]
Here is the summary with links:
- [v2,net] tcp: fix skb_copy_ubufs() vs BIG TCP
https://git.kernel.org/netdev/net/c/7e692df39336
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
From: Eric Dumazet
Quoted 1 line
Sent: 28 April 2023 05:33
...
net/core/skbuff.c int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask) · line 1778
Quoted 10 lines
net/core/skbuff.c +9
-1- new_frags = (__skb_pagelen (skb) +PAGE_SIZE -1 ) >>PAGE_SHIFT ; +/* We might have to allocate high order pages, so compute what minimum +* page order is needed. +*/ + order =0 ; +while ((PAGE_SIZE << order) *MAX_SKB_FRAGS <__skb_pagelen (skb)) + order++; + psize = (PAGE_SIZE << order); + + new_frags = (__skb_pagelen (skb) + psize -1 ) >> (PAGE_SHIFT + order);
That looks like it will generate quite horrid code. Perhaps something like:
new_frags = (__skb_pagelen(skb) + PAGE_SIZE - 1) >> PAGE_SHIFT;
order = 0;
psize = PAGE_SIZE;
if (new_frags > MAX_SKB_FRAGS) {
/* Allocate high order pages to reduce the number of frags. */
order = ilog2(DIV_ROUNDUP(new_frags, MAX_SKB_FRAGS) - 1) + 1;
psize <<= order;
}There might be an 'off-by-one' in there somewhere though...
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)