net: ip: avoid OOM kills with large UDP sends over loopback
Subsystems (3)
Jakub Kicinski
maintainerPATCH v4 1/1 net: ip: avoid OOM kills with large UDP sends over loopback
Dave observed number of machines hitting OOM on the UDP send path. The workload seems to be sending large UDP packets over loopback. Since loopback has MTU of 64k kernel will try to allocate an skb with up to 64k of head space. This has a good chance of failing under memory pressure. What's worse if the message length is <32k the allocation may trigger an OOM killer.
This is entirely avoidable, we can use an skb with page frags.
af_unix solves a similar problem by limiting the head length to SKB_MAX_ALLOC. This seems like a good and simple approach. It means that UDP messages > 16kB will now use fragments if underlying device supports SG, if extra allocator pressure causes regressions in real workloads we can switch to trying the large allocation first and falling back.
v4: pre-calculate all the additions to alloclen so
we can be sure it won't go over order-2
--- net/ipv4/ip_output.c | 32 ++++++++++++++++++-------------- net/ipv6/ip6_output.c | 32 +++++++++++++++++--------------- 2 files changed, 35 insertions(+), 29 deletions(-)
net/ipv4/ip_output.c +18 -14
@@ -1054,7 +1054,7 @@ static int __ip_append_data(struct sock *sk,unsigned int datalen;unsigned int fraglen;unsigned int fraggap; -unsigned int alloclen; +unsigned int alloclen, alloc_extra;unsigned int pagedlen;struct sk_buff *skb_prev; alloc_new_skb:
@@ -1074,35 +1074,39 @@ static int __ip_append_data(struct sock *sk, fraglen = datalen + fragheaderlen; pagedlen =0 ; + alloc_extra = hh_len +15 ; + alloc_extra += exthdrlen; + +/* The last fragment gets additional space at tail. +* Note, with MSG_MORE we overallocate on fragments, +* because we have no idea what fragment will be +* the last. +*/ +if (datalen == length + fraggap) + alloc_extra += rt->dst.trailer_len; +if ((flags &MSG_MORE ) && !(rt->dst.dev->features&NETIF_F_SG )) alloclen = mtu; -else if (!paged) +else if (!paged && + (fraglen + alloc_extra <SKB_MAX_ALLOC || + !(rt->dst.dev->features &NETIF_F_SG ))) alloclen = fraglen;else { alloclen =min_t (int, fraglen,MAX_HEADER ); pagedlen = fraglen - alloclen; } - alloclen += exthdrlen; - -/* The last fragment gets additional space at tail. -* Note, with MSG_MORE we overallocate on fragments, -* because we have no idea what fragment will be -* the last. -*/ -if (datalen == length + fraggap) - alloclen += rt->dst.trailer_len; + alloclen += alloc_extra;if (transhdrlen) { - skb =sock_alloc_send_skb (sk, - alloclen + hh_len +15 , + skb =sock_alloc_send_skb (sk, alloclen, (flags &MSG_DONTWAIT ), &err); }else { skb =NULL ;if (refcount_read (&sk->sk_wmem_alloc) + wmem_alloc_delta <=2 * sk->sk_sndbuf) - skb =alloc_skb (alloclen + hh_len +15 , + skb =alloc_skb (alloclen, sk->sk_allocation);if (unlikely (!skb)) err = -ENOBUFS ;
net/ipv6/ip6_output.c +17 -15
@@ -1555,7 +1555,7 @@ static int __ip6_append_data(struct sock *sk,unsigned int datalen;unsigned int fraglen;unsigned int fraggap; -unsigned int alloclen; +unsigned int alloclen, alloc_extra;unsigned int pagedlen; alloc_new_skb:/* There's no room in the current skb */
@@ -1582,17 +1582,28 @@ static int __ip6_append_data(struct sock *sk, fraglen = datalen + fragheaderlen; pagedlen =0 ; + alloc_extra = hh_len; + alloc_extra += dst_exthdrlen; + alloc_extra += rt->dst.trailer_len; + +/* We just reserve space for fragment header. +* Note: this may be overallocation if the message +* (without MSG_MORE) fits into the MTU. +*/ + alloc_extra +=sizeof (struct frag_hdr ); +if ((flags &MSG_MORE ) && !(rt->dst.dev->features&NETIF_F_SG )) alloclen = mtu; -else if (!paged) +else if (!paged && + (fraglen + alloc_extra <SKB_MAX_ALLOC || + !(rt->dst.dev->features &NETIF_F_SG ))) alloclen = fraglen;else { alloclen =min_t (int, fraglen,MAX_HEADER ); pagedlen = fraglen - alloclen; } - - alloclen += dst_exthdrlen; + alloclen += alloc_extra;if (datalen != length + fraggap) {/*
@@ -1602,30 +1613,21 @@ static int __ip6_append_data(struct sock *sk, datalen += rt->dst.trailer_len; } - alloclen += rt->dst.trailer_len; fraglen = datalen + fragheaderlen; -/* -* We just reserve space for fragment header. -* Note: this may be overallocation if the message -* (without MSG_MORE) fits into the MTU. -*/ - alloclen +=sizeof (struct frag_hdr ); - copy = datalen - transhdrlen - fraggap - pagedlen;if (copy <0 ) { err = -EINVAL ; goto error; }if (transhdrlen) { - skb =sock_alloc_send_skb (sk, - alloclen + hh_len, + skb =sock_alloc_send_skb (sk, alloclen, (flags &MSG_DONTWAIT ), &err); }else { skb =NULL ;if (refcount_read (&sk->sk_wmem_alloc) + wmem_alloc_delta <=2 * sk->sk_sndbuf) - skb =alloc_skb (alloclen + hh_len, + skb =alloc_skb (alloclen, sk->sk_allocation);if (unlikely (!skb)) err = -ENOBUFS ;
Signature · 1 line
2.31.1
Quoted 34 lines · Jakub Kicinski
Dave observed number of machines hitting OOM on the UDP send path. The workload seems to be sending large UDP packets over loopback. Since loopback has MTU of 64k kernel will try to allocate an skb with up to 64k of head space. This has a good chance of failing under memory pressure. What's worse if the message length is <32k the allocation may trigger an OOM killer.
This is entirely avoidable, we can use an skb with page frags.
af_unixsolves a similar problem by limiting the head length toSKB_MAX_ALLOC. This seems like a good and simple approach. It means that UDP messages > 16kB will now use fragments if underlying device supports SG, if extra allocator pressure causes regressions in real workloads we can switch to trying the large allocation first and falling back.v4: pre-calculate all the additions to alloclen so
we can be sure it won't go over order-2Reported-by Dave Jones <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>--- net/ipv4/ip_output.c | 32 ++++++++++++++++++-------------- net/ipv6/ip6_output.c | 32 +++++++++++++++++--------------- 2 files changed, 35 insertions(+), 29 deletions(-)net/ipv4/ip_output.c +0
-0
net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1057
Quoted 5 lines
net/ipv4/ip_output.c +1
-1@@ -1054,7 +1054,7 @@ static int __ip_append_data(struct sock *sk,unsigned int datalen;unsigned int fraglen;unsigned int fraggap; -unsigned int alloclen; +unsigned int alloclen, alloc_extra;
Separate line?
Quoted 15 lines
net/ipv4/ip_output.c +8
-0unsigned int pagedlen;struct sk_buff *skb_prev; alloc_new_skb:@@ -1074,35 +1074,39 @@ static int __ip_append_data(struct sock *sk, fraglen = datalen + fragheaderlen; pagedlen =0 ; + alloc_extra = hh_len +15 ; + alloc_extra += exthdrlen; + +/* The last fragment gets additional space at tail. +* Note, with MSG_MORE we overallocate on fragments, +* because we have no idea what fragment will be +* the last. +*/
net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1093
Quoted 10 lines
net/ipv4/ip_output.c +6
-1+if (datalen == length + fraggap) + alloc_extra += rt->dst.trailer_len; +if ((flags &MSG_MORE ) && !(rt->dst.dev->features&NETIF_F_SG )) alloclen = mtu; -else if (!paged) +else if (!paged && + (fraglen + alloc_extra <SKB_MAX_ALLOC || + !(rt->dst.dev->features &NETIF_F_SG )))
This perhaps deserves a comment. Something like this?
/* avoid order-3 allocations where possible: replace with frags if
allowed (sg) */
Quoted 21 lines
net/ipv4/ip_output.c +2
-11alloclen = fraglen; else { alloclen =min_t (int, fraglen,MAX_HEADER ); pagedlen = fraglen - alloclen; } - alloclen += exthdrlen; - -/* The last fragment gets additional space at tail. -* Note, with MSG_MORE we overallocate on fragments, -* because we have no idea what fragment will be -* the last. -*/ -if (datalen == length + fraggap) - alloclen += rt->dst.trailer_len; + alloclen += alloc_extra;if (transhdrlen) { - skb = sock_alloc_send_skb(sk, - alloclen + hh_len + 15, + skb = sock_alloc_send_skb(sk, alloclen,
net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1112
Quoted 10 lines
net/ipv4/ip_output.c +1
-1(flags & MSG_DONTWAIT), &err); }else { skb =NULL ;if (refcount_read (&sk->sk_wmem_alloc) + wmem_alloc_delta <=2 * sk->sk_sndbuf) - skb =alloc_skb (alloclen + hh_len +15 , + skb =alloc_skb (alloclen, sk->sk_allocation);if (unlikely (!skb)) err = -ENOBUFS ;
Is there any risk of regressions? If so, would it be preferable to try regular alloc and only on failure, just below here, do the size and SG test and if permitted jump back to the last of the three alloc_len options?
Quoted 15 lines
Quoted 4 lines
if (transhdrlen) {net/ipv4/ip_output.c +1
-2- skb = sock_alloc_send_skb(sk, - alloclen + hh_len + 15, + skb = sock_alloc_send_skb(sk, alloclen,net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1112
Quoted 10 lines
net/ipv4/ip_output.c +1
-1(flags & MSG_DONTWAIT), &err); }else { skb =NULL ;if (refcount_read (&sk->sk_wmem_alloc) + wmem_alloc_delta <=2 * sk->sk_sndbuf) - skb =alloc_skb (alloclen + hh_len +15 , + skb =alloc_skb (alloclen, sk->sk_allocation);if (unlikely (!skb)) err = -ENOBUFS ;
Quoted 4 lines
Is there any risk of regressions? If so, would it be preferable to try regular alloc and only on failure, just below here, do the size and SG test and if permitted jump back to the last of the three
alloc_lenoptions?
sk_page_frag_refill when using frags will try also try large allocations first (SKB_FRAG_PAGE_ORDER == order-3) , but can degrade more gracefully under memory pressure than this header alloc. Which can only succeed or fail for the total size. So without memory pressure this only takes two extra allocations for a 64KB skb.
Jakub Kicinski
maintainerQuoted 30 lines · On Wed, 23 Jun 2021 22:21:11 -0400 Willem de Bruijn wrote:
Quoted 29 lines · Jakub Kicinski
Dave observed number of machines hitting OOM on the UDP send path. The workload seems to be sending large UDP packets over loopback. Since loopback has MTU of 64k kernel will try to allocate an skb with up to 64k of head space. This has a good chance of failing under memory pressure. What's worse if the message length is <32k the allocation may trigger an OOM killer.
This is entirely avoidable, we can use an skb with page frags.
af_unixsolves a similar problem by limiting the head length toSKB_MAX_ALLOC. This seems like a good and simple approach. It means that UDP messages > 16kB will now use fragments if underlying device supports SG, if extra allocator pressure causes regressions in real workloads we can switch to trying the large allocation first and falling back.v4: pre-calculate all the additions to alloclen so
we can be sure it won't go over order-2Reported-by Dave Jones <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>--- net/ipv4/ip_output.c | 32 ++++++++++++++++++-------------- net/ipv6/ip6_output.c | 32 +++++++++++++++++--------------- 2 files changed, 35 insertions(+), 29 deletions(-)
Quoted 12 lines
net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1057
Quoted 10 lines
net/ipv4/ip_output.c +1
-1@@ -1054,7 +1054,7 @@ static int __ip_append_data(struct sock *sk,unsigned int datalen;unsigned int fraglen;unsigned int fraggap; -unsigned int alloclen; +unsigned int alloclen, alloc_extra;Separate line?
But why? What makes it preferable to have logically connected variables declared on separate lines? The function is already 300 LoC. I've been meaning to ask someone about this preference for a while :)
Quoted 28 lines
Quoted 15 lines
net/ipv4/ip_output.c +8
-0unsigned int pagedlen;struct sk_buff *skb_prev; alloc_new_skb:@@ -1074,35 +1074,39 @@ static int __ip_append_data(struct sock *sk, fraglen = datalen + fragheaderlen; pagedlen =0 ; + alloc_extra = hh_len +15 ; + alloc_extra += exthdrlen; + +/* The last fragment gets additional space at tail. +* Note, with MSG_MORE we overallocate on fragments, +* because we have no idea what fragment will be +* the last. +*/ net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1093
Quoted 10 lines
net/ipv4/ip_output.c +6
-1+if (datalen == length + fraggap) + alloc_extra += rt->dst.trailer_len; +if ((flags &MSG_MORE ) && !(rt->dst.dev->features&NETIF_F_SG )) alloclen = mtu; -else if (!paged) +else if (!paged && + (fraglen + alloc_extra <SKB_MAX_ALLOC || + !(rt->dst.dev->features &NETIF_F_SG )))This perhaps deserves a comment. Something like this?
Quoted 2 lines
/* avoid order-3 allocations where possible: replace with frags if
allowed (sg) */
Here I thought comparing skb alloc size to SKB_MAX_ALLOC is explanatory enough ;)
In the middle of the test, like this, right?
else if (!paged &&
/* avoid order-3 allocations if device
* can handle skb frags (sg)
*/
(fraglen + alloc_extra < SKB_MAX_ALLOC ||
!(rt->dst.dev->features & NETIF_F_SG))) I should make it less-equal while at it.
Quoted 32 lines
Quoted 21 lines
net/ipv4/ip_output.c +2
-11alloclen = fraglen; else { alloclen =min_t (int, fraglen,MAX_HEADER ); pagedlen = fraglen - alloclen; } - alloclen += exthdrlen; - -/* The last fragment gets additional space at tail. -* Note, with MSG_MORE we overallocate on fragments, -* because we have no idea what fragment will be -* the last. -*/ -if (datalen == length + fraggap) - alloclen += rt->dst.trailer_len; + alloclen += alloc_extra;if (transhdrlen) { - skb = sock_alloc_send_skb(sk, - alloclen + hh_len + 15, + skb = sock_alloc_send_skb(sk, alloclen,net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1112
Quoted 10 lines
net/ipv4/ip_output.c +1
-1(flags & MSG_DONTWAIT), &err); }else { skb =NULL ;if (refcount_read (&sk->sk_wmem_alloc) + wmem_alloc_delta <=2 * sk->sk_sndbuf) - skb =alloc_skb (alloclen + hh_len +15 , + skb =alloc_skb (alloclen, sk->sk_allocation);if (unlikely (!skb)) err = -ENOBUFS ;
Quoted 4 lines
Is there any risk of regressions? If so, would it be preferable to try regular alloc and only on failure, just below here, do the size and SG test and if permitted jump back to the last of the three
alloc_lenoptions?
There is, that's what I tried in v1, Eric pointed out that we can't modify sk->sk_allocation here because UDP fast path doesn't take the lock, and pointed out that UNIX code has to handle similar problem. So I decided to just copy what AF_UNIX does. In practical terms MTU > 16k is highly unlikely on physical devices (AFAIK) and with messages that large hopefully the trip thru the memory allocator won't be all that noticeable? If we were capping at one page that'd be a problem, but my gut feeling was that order-2 cap is unlikely to hurt.
But I can go back, I'd have to refactor sock_alloc_send_pskb() to pass gfp_t explicitly. Probably by creating another layer of helpers (__sock_alloc_send_pskb()?). sock_alloc_send_pskb() already takes 6 params so I was also thinking of converting it to ERR_PTR() return (instead of passing the error pointer) (6 is max for register passing).
Should I go back to retry?
Quoted 45 lines · Jakub Kicinski
Quoted 30 lines · On Wed, 23 Jun 2021 22:21:11 -0400 Willem de Bruijn wrote:
Quoted 29 lines · Jakub Kicinski
Dave observed number of machines hitting OOM on the UDP send path. The workload seems to be sending large UDP packets over loopback. Since loopback has MTU of 64k kernel will try to allocate an skb with up to 64k of head space. This has a good chance of failing under memory pressure. What's worse if the message length is <32k the allocation may trigger an OOM killer.
This is entirely avoidable, we can use an skb with page frags.
af_unixsolves a similar problem by limiting the head length toSKB_MAX_ALLOC. This seems like a good and simple approach. It means that UDP messages > 16kB will now use fragments if underlying device supports SG, if extra allocator pressure causes regressions in real workloads we can switch to trying the large allocation first and falling back.v4: pre-calculate all the additions to alloclen so
we can be sure it won't go over order-2Reported-by Dave Jones <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>--- net/ipv4/ip_output.c | 32 ++++++++++++++++++-------------- net/ipv6/ip6_output.c | 32 +++++++++++++++++--------------- 2 files changed, 35 insertions(+), 29 deletions(-)Quoted 12 lines
net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1057
Quoted 10 lines
net/ipv4/ip_output.c +1
-1@@ -1054,7 +1054,7 @@ static int __ip_append_data(struct sock *sk,unsigned int datalen;unsigned int fraglen;unsigned int fraggap; -unsigned int alloclen; +unsigned int alloclen, alloc_extra;Separate line?
Quoted 3 lines
But why? What makes it preferable to have logically connected variables declared on separate lines? The function is already 300 LoC. I've been meaning to ask someone about this preference for a while :)
Reverse christmas tree is the norm in netdev. Pointing out for consistency only. I have no particular opinion on the rule.
Agreed that in this function, multiple entries per line would be preferable.
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
net/ipv4/ip_output.c +8
-0unsigned int pagedlen;struct sk_buff *skb_prev; alloc_new_skb:@@ -1074,35 +1074,39 @@ static int __ip_append_data(struct sock *sk, fraglen = datalen + fragheaderlen; pagedlen =0 ; + alloc_extra = hh_len +15 ; + alloc_extra += exthdrlen; + +/* The last fragment gets additional space at tail. +* Note, with MSG_MORE we overallocate on fragments, +* because we have no idea what fragment will be +* the last. +*/ net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1093
Quoted 10 lines
net/ipv4/ip_output.c +6
-1+if (datalen == length + fraggap) + alloc_extra += rt->dst.trailer_len; +if ((flags &MSG_MORE ) && !(rt->dst.dev->features&NETIF_F_SG )) alloclen = mtu; -else if (!paged) +else if (!paged && + (fraglen + alloc_extra <SKB_MAX_ALLOC || + !(rt->dst.dev->features &NETIF_F_SG )))This perhaps deserves a comment. Something like this?
Quoted 5 lines
Quoted 2 lines
/* avoid order-3 allocations where possible: replace with frags if
allowed (sg) */Here I thought comparing skb alloc size to
SKB_MAX_ALLOCis explanatory enough ;)
Yeah, I guess you're right. The comment only rewords *what* the code does, so not super informative. Never mind that suggestion.
Quoted 57 lines
In the middle of the test, like this, right?
else if (!paged && /* avoid order-3 allocations if device * can handle skb frags (sg) */ (fraglen + alloc_extra < SKB_MAX_ALLOC || !(rt->dst.dev->features & NETIF_F_SG)))I should make it less-equal while at it.
Quoted 32 lines
Quoted 21 lines
net/ipv4/ip_output.c +2
-11alloclen = fraglen; else { alloclen =min_t (int, fraglen,MAX_HEADER ); pagedlen = fraglen - alloclen; } - alloclen += exthdrlen; - -/* The last fragment gets additional space at tail. -* Note, with MSG_MORE we overallocate on fragments, -* because we have no idea what fragment will be -* the last. -*/ -if (datalen == length + fraggap) - alloclen += rt->dst.trailer_len; + alloclen += alloc_extra;if (transhdrlen) { - skb = sock_alloc_send_skb(sk, - alloclen + hh_len + 15, + skb = sock_alloc_send_skb(sk, alloclen,net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1112
Quoted 10 lines
net/ipv4/ip_output.c +1
-1(flags & MSG_DONTWAIT), &err); }else { skb =NULL ;if (refcount_read (&sk->sk_wmem_alloc) + wmem_alloc_delta <=2 * sk->sk_sndbuf) - skb =alloc_skb (alloclen + hh_len +15 , + skb =alloc_skb (alloclen, sk->sk_allocation);if (unlikely (!skb)) err = -ENOBUFS ;Quoted 4 lines
Is there any risk of regressions? If so, would it be preferable to try regular alloc and only on failure, just below here, do the size and SG test and if permitted jump back to the last of the three
alloc_lenoptions?There is, that's what I tried in v1, Eric pointed out that we can't modify
sk->sk_allocationhere because UDP fast path doesn't take the lock, and pointed out that UNIX code has to handle similar problem. So I decided to just copy whatAF_UNIXdoes. In practical terms MTU > 16k is highly unlikely on physical devices (AFAIK) and with messages that large hopefully the trip thru the memory allocator won't be all that noticeable? If we were capping at one page that'd be a problem, but my gut feeling was that order-2 cap is unlikely to hurt.
Quoted 7 lines
But I can go back, I'd have to refactor
sock_alloc_send_pskb()to passgfp_texplicitly. Probably by creating another layer of helpers (__sock_alloc_send_pskb()?).sock_alloc_send_pskb()already takes 6 params so I was also thinking of converting it toERR_PTR()return (instead of passing the error pointer) (6 is max for register passing).Should I go back to retry?
For __GFP_NOWARN? Sorry, I missed that.
Okay, then I understand why this approach is preferable. And LGTM. Thanks!
This patch was applied to netdev/net-next.git (refs/heads/master):
Hello:
This patch was applied to netdev/net-next.git (refs/heads/master):
Quoted 9 lines · On Wed, 23 Jun 2021 14:44:38 -0700 you wrote:
Dave observed number of machines hitting OOM on the UDP send path. The workload seems to be sending large UDP packets over loopback. Since loopback has MTU of 64k kernel will try to allocate an skb with up to 64k of head space. This has a good chance of failing under memory pressure. What's worse if the message length is <32k the allocation may trigger an OOM killer.
[...]
Here is the summary with links:
- [net-next,v4] net: ip: avoid OOM kills with large UDP sends over loopback
https://git.kernel.org/netdev/net-next/c/6d123b81ac61
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html