Skip to contentSkip to conversation

skbuff: Hide GFP_ATOMIC page allocation failures for dropped packets

PATCH Superseded [email protected] 6 participants 9 messages May 26, 2013 – May 29, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 skbuff: Hide GFP_ATOMIC page allocation failures for dropped packets

From: Aaron Tomlin <[email protected]>

Since v1:
- Removed unnecessary parentheses

---8<---

Failed GFP_ATOMIC allocations by the network stack result in dropped packets, which will be received on a subsequent retransmit, and an unnecessary, noisy warning with a kernel backtrace.

These warnings are harmless, but they still cause users to panic and file bug reports over dropped packets. It would be better to hide the failed allocation warnings and backtraces, and let retransmits handle dropped packets quietly.

---
 net/core/skbuff.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/skbuff.c +1 -1
index 02139d6..84aa870 100644
@@ -236,7 +236,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
 		? skbuff_fclone_cache : skbuff_head_cache;
 
 	if (sk_memalloc_socks() && (flags & SKB_ALLOC_RX))
-		gfp_mask |= (__GFP_MEMALLOC|__GFP_NOWARN);
+		gfp_mask |= __GFP_MEMALLOC | __GFP_NOWARN;
 
 	/* Get the HEAD */
 	skb = kmem_cache_alloc_node(cache, gfp_mask & ~__GFP_DMA, node);
Signature · 1 line
1.8.1.4
Reviewed-by on v2 1/1
#1 lore raw
Quoted 7 lines · On 05/26/2013 04:19 PM, [email protected] wrote:

From: Aaron Tomlin <[email protected]>

Since v1:
- Removed unnecessary parentheses

---8<---

Quoted 8 lines

Failed GFP_ATOMIC allocations by the network stack result in dropped packets, which will be received on a subsequent retransmit, and an unnecessary, noisy warning with a kernel backtrace.

These warnings are harmless, but they still cause users to panic and file bug reports over dropped packets. It would be better to hide the failed allocation warnings and backtraces, and let retransmits handle dropped packets quietly.

Yes please. Getting memory management bug reports for dropped network packets got old years ago. Lets get rid of those messages.

Quoted 1 line
#2 lore raw
Quoted 11 lines · Rik van Riel
Quoted 10 lines · On 05/26/2013 04:19 PM, [email protected] wrote:

From: Aaron Tomlin <[email protected]>

Since v1:
- Removed unnecessary parentheses

---8<---

Failed GFP_ATOMIC allocations by the network stack result in dropped packets, which will be received on a subsequent retransmit, and an unnecessary, noisy warning with a kernel backtrace.

This claim is wrong, only some protocols deal with retransmits.

Quoted 9 lines
Quoted 5 lines

These warnings are harmless, but they still cause users to panic and file bug reports over dropped packets. It would be better to hide the failed allocation warnings and backtraces, and let retransmits handle dropped packets quietly.

Yes please. Getting memory management bug reports for dropped network packets got old years ago. Lets get rid of those messages.

I am only wondering why this path has anything needing special attention, over thousands of kmalloc() like call sites in the kernel.

If mm allocation warnings are useless, just make __GFP_NOWARN the default, and save us thousand of patches (adding the __GFP_NOWARN everywhere)

Truth is : some network drivers don't deal very well with allocation errors. mlx4 for example absolutely wants order-2 pages in RX path, with no fallback to order-0 pages.

So I am not against this patch, but I can not really acknowledge it, sorry.

#3 lore raw
Quoted 13 lines · Eric Dumazet
Quoted 4 lines · Rik van Riel
Quoted 3 lines · On 05/26/2013 04:19 PM, [email protected] wrote:

Failed GFP_ATOMIC allocations by the network stack result in dropped packets, which will be received on a subsequent retransmit, and an unnecessary, noisy warning with a kernel backtrace.

This claim is wrong, only some protocols deal with retransmits.

Quoted 5 lines
Quoted 4 lines

These warnings are harmless, but they still cause users to panic and file bug reports over dropped packets. It would be better to hide the failed allocation warnings and backtraces, and let retransmits handle dropped packets quietly.

Quoted 6 lines
Quoted 3 lines

Yes please. Getting memory management bug reports for dropped network packets got old years ago. Lets get rid of those messages.

I am only wondering why this path has anything needing special attention, over thousands of kmalloc() like call sites in the kernel.

I don't think this site is particularly special.

Quoted 10 lines

If mm allocation warnings are useless, just make __GFP_NOWARN the default, and save us thousand of patches (adding the __GFP_NOWARN everywhere)

Truth is : some network drivers don't deal very well with allocation errors. mlx4 for example absolutely wants order-2 pages in RX path, with no fallback to order-0 pages.

So I am not against this patch, but I can not really acknowledge it, sorry.

I think the __alloc_skb alloc failure message is ok, but maybe there shouldn't be something "scary" like a dump_stack.

Maybe this site should use a trivial debug error
message like below instead.

---
 net/core/skbuff.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
net/core/skbuff.c +9 -2
index d629891..0154803 100644
@@ -231,17 +231,24 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
 	struct sk_buff *skb;
 	u8 *data;
 	bool pfmemalloc;
+	bool warn_no_skb = !(gfp_mask & __GFP_NOWARN);
 
 	cache = (flags & SKB_ALLOC_FCLONE)
 		? skbuff_fclone_cache : skbuff_head_cache;
 
 	if (sk_memalloc_socks() && (flags & SKB_ALLOC_RX))
-		gfp_mask |= __GFP_MEMALLOC;
+		gfp_mask |= __GFP_MEMALLOC | __GFP_NOWARN;
 
 	/* Get the HEAD */
 	skb = kmem_cache_alloc_node(cache, gfp_mask & ~__GFP_DMA, node);
-	if (!skb)
+	if (!skb) {
+		if (warn_no_skb)
+			printk_ratelimited(KERN_DEBUG "%s: OOM from %pF via %pF\n",
+					   __func__,
+					   __builtin_return_address(0),
+					   __builtin_return_address(1));
 		goto out;
+	}
 	prefetchw(skb);
 
 	/* We do our best to align skb_shared_info on a separate cache
#4 lore raw
Quoted 7 lines · Joe Perches

I think the __alloc_skb alloc failure message is ok, but maybe there shouldn't be something "scary" like a dump_stack.

Maybe this site should use a trivial debug error
message like below instead.
---

Oh well.

If dump_stack are scary, they are scary for every k[mz]alloc() users, not only __alloc_skb_alloc()

I just said : Please do not add GFP_NOWARN to thousand of call sites, and you suggest adding more code in network fast path. (???)

This is not a trivial code, we are speaking of a very sensitive one.

Let mm guys explain in what cases a full stack trace is nice to have, and in what cases its useless. An heuristic should be defined in mm tree for that, and not spread everywhere.

There must be a reason GFP_NOWARN is seldom used in the kernel, even if most callers are able to recover properly from a failed memory allocation.

#5 lore raw
Quoted 2 lines · Eric Dumazet

If dump_stack are scary, they are scary for every k[mz]alloc() users, not only __alloc_skb_alloc()

True, but perhaps they are relatively rarer though for non net use cases.

#6 lore raw
In reply to Joe Perches
Quoted 7 lines · Joe Perches

I think the __alloc_skb alloc failure message is ok, but maybe there shouldn't be something "scary" like a dump_stack.

Maybe this site should use a trivial debug error
message like below instead.

The stack trace may or may not be important for debugging, however what is important is that we know how often this is happening. That implies that there should be a percpu counter of allocation failures here, regardless of a dump_stack, rate limited or not. I would suppose it's ok for no stack by default as long as there is a counter, for this specific call only.

There are some scary ways this code is called, for example from tcp_send_fin(). It does yield() in that loop, but otherwise it just spins.

-Deb

#7 lore raw
In reply to Eric Dumazet
Quoted 9 lines · Eric Dumazet
Quoted 3 lines · Rik van Riel

Yes please. Getting memory management bug reports for dropped network packets got old years ago. Lets get rid of those messages.

I am only wondering why this path has anything needing special attention, over thousands of kmalloc() like call sites in the kernel.

There are a few special things about the network code:

1) network packets can arrive extremely fast, in large batches
2) the network code cannot wait for the VM to free memory (GFP_ATOMIC)

Other allocations tend to be done less at a time, and/or allow the VM to free up memory before proceeding.

Quoted 7 lines

If mm allocation warnings are useless, just make __GFP_NOWARN the default, and save us thousand of patches (adding the __GFP_NOWARN everywhere)

Truth is : some network drivers don't deal very well with allocation errors. mlx4 for example absolutely wants order-2 pages in RX path, with no fallback to order-0 pages.

Network protocols and network applications tend to deal with packet loss by retransmitting data, though.

Also, once all the data from one of those order-2 page buffers has been delivered or forwarded, that buffer becomes available to subsequent network packets.

Other allocations tend not to free & reuse their memory as quickly as the network stack.

Quoted 5 lines

So I am not against this patch, but I can not really acknowledge it, sorry.

#8 lore raw
In reply to [email protected]

Hello.

Quoted 4 lines · On 27-05-2013 0:19, [email protected] wrote:

From: Aaron Tomlin <[email protected]>

Since v1:
- Removed unnecessary parentheses

The "changes since version X" section should typically go after ---
tearline, else the mainatiner will have to edit your patch before applying.

Quoted 6 lines

---8<---

Failed GFP_ATOMIC allocations by the network stack result in dropped packets, which will be received on a subsequent retransmit, and an unnecessary, noisy warning with a kernel backtrace.

Quoted 6 lines

These warnings are harmless, but they still cause users to panic and file bug reports over dropped packets. It would be better to hide the failed allocation warnings and backtraces, and let retransmits handle dropped packets quietly.

WBR, Sergei