Change in alloc_skb() behavior in 3.2+ kernels?
Subsystems (2)
I'm tracking down a problem that appears to be caused by a change in the behavior of alloc_skb() introduced in kernel version 3.2. In kernel versions prior to 3.2, calling alloc_skb(1350), returned an sk_buff with a tailroom of around 1400 bytes (safely below the default Ethernet frame size limit of 1500).
In 3.2 and later, calling alloc_skb(1350) returns an sk_buff with a tailroom of about 1850.
Why has the "extra" space increased from 60 bytes to 500 bytes?
[It's always possible that I've unintentionally changed something in the kernel configs that causes this, but I've tried to build the kernels as identically as possible.]
The kernel module that's started failing fills the allocated sk_buff until tailroom() indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).
I've found man pages for alloc_skb() from a few years ago that state explicitly that alloc_skb(_size_) will allocate a new sk_buff with no headroom and a tail room of _size_ bytes. This doesn't seem to be the case for recent kernels. Is there any documentation stating what the current behavior is supposed to be?
Are callers to alloc_skb() supposed to check the tailroom and reserve() an appropriate number of bytes such that the tailroom is correct?
Is the tailroom of the allocated sk_buff guaranteed to be at least as large as the requested size, or does application code also have to check for tailroom less than the requested size?
The ultimate question I'm trying to answer is what is the "right" way to allocate an sk_buff that has a size appropriate for an Ethernet frame assuming an MTU of 1500?
Signature · 3 lines
Grant Edwards grant.b.edwards Yow! Let's send the
at Russians defective
gmail.com lifestyle accessories!Quoted 11 lines · Grant Edwards
I'm tracking down a problem that appears to be caused by a change in the behavior of
alloc_skb()introduced in kernel version 3.2. In kernel versions prior to 3.2, callingalloc_skb(1350), returned ansk_buffwith a tailroom of around 1400 bytes (safely below the default Ethernet frame size limit of 1500).In 3.2 and later, calling
alloc_skb(1350)returns ansk_buffwith a tailroom of about 1850.Why has the "extra" space increased from 60 bytes to 500 bytes?
Because of kmalloc-2048 being used. Previous kernels were losing this space. We are now able to expand some packets without extra re-allocation/copy.
Quoted 8 lines
[It's always possible that I've unintentionally changed something in the kernel configs that causes this, but I've tried to build the kernels as identically as possible.]
The kernel module that's started failing fills the allocated
sk_buffuntiltailroom()indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).
This code seems buggy.
Quoted 11 lines
I've found man pages for
alloc_skb()from a few years ago that state explicitly thatalloc_skb(_size_)will allocate a newsk_buffwith no headroom and a tail room of _size_ bytes. This doesn't seem to be the case for recent kernels. Is there any documentation stating what the current behavior is supposed to be?Are callers to
alloc_skb()supposed to check the tailroom andreserve()an appropriate number of bytes such that the tailroom is correct?
If you allocate skbs with 1500 bytes, you probably should check skb->len more than tailroom...
Quoted 8 lines
Is the tailroom of the allocated
sk_buffguaranteed to be at least as large as the requested size, or does application code also have to check for tailroom less than the requested size?The ultimate question I'm trying to answer is what is the "right" way to allocate an
sk_buffthat has a size appropriate for an Ethernet frame assuming an MTU of 1500?
I dont know what to answer. Could you point the code in question ?
David Miller
maintainerFrom: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:32:57 +0000 (UTC)
Quoted 4 lines
The kernel module that's started failing fills the allocated
sk_buffuntiltailroom()indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).
The amount of tailroom an SKB has is implementation dependent.
It's incredibly poor form to rely upon it to determine whether a fully sized frame has been constructed or not.
Please fix the code that does this.
Quoted 21 lines · Eric Dumazet
Quoted 10 lines · Grant Edwards
I'm tracking down a problem that appears to be caused by a change in the behavior of
alloc_skb()introduced in kernel version 3.2. In kernel versions prior to 3.2, callingalloc_skb(1350), returned ansk_buffwith a tailroom of around 1400 bytes (safely below the default Ethernet frame size limit of 1500).In 3.2 and later, calling
alloc_skb(1350)returns ansk_buffwith a tailroom of about 1850.Why has the "extra" space increased from 60 bytes to 500 bytes?
Because of kmalloc-2048 being used. Previous kernels were losing this space. We are now able to expand some packets without extra re-allocation/copy.
Quoted 4 lines
[It's always possible that I've unintentionally changed something in the kernel configs that causes this, but I've tried to build the kernels as identically as possible.]
Quoted 6 lines
Quoted 4 lines
The kernel module that's started failing fills the allocated
sk_buffuntiltailroom()indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).This code seems buggy.
It is with today's alloc_skb().
At the time it was written (probably 10+ years ago) it was relying on the documented API for alloc_skb() that stated alloc_skb() either returned an sk_buff of the requested size or it failed.
Quoted 6 lines
Quoted 6 lines
I've found man pages for
alloc_skb()from a few years ago that state explicitly thatalloc_skb(_size_)will allocate a newsk_buffwith no headroom and a tail room of _size_ bytes. This doesn't seem to be the case for recent kernels. Is there any documentation stating what the current behavior is supposed to be?
Quoted 6 lines
Quoted 3 lines
Are callers to
alloc_skb()supposed to check the tailroom andreserve()an appropriate number of bytes such that the tailroom is correct?If you allocate skbs with 1500 bytes, you probably should check
skb->lenmore than tailroom...
I can do that -- assuming it's backwards compatible with older kernels as well (let's say as far back as 2.6 -- we stopped trying to support 2.4 kernels last year).
Quoted 9 lines
Quoted 7 lines
Is the tailroom of the allocated
sk_buffguaranteed to be at least as large as the requested size, or does application code also have to check for tailroom less than the requested size?The ultimate question I'm trying to answer is what is the "right" way to allocate an
sk_buffthat has a size appropriate for an Ethernet frame assuming an MTU of 1500?I dont know what to answer. Could you point the code in question?
Sure:
alloc_skb(dev->hard_header_len + 1340)
The intent was to allocate a frame that we can guarantee we can send out via the Ethernet device 'dev'. Sometime 12-14 years ago, somebody pulled the number "1340" out of the air under the assumption that the resulting sk_buff would always be safely under the Ethernet limit of 1500 bytes.
That worked until kernel 3.2 came out, at which time we started ending up with 1800 byte frames.
Signature · 3 lines
Grant Edwards grant.b.edwards Yow! I've read SEVEN
at MILLION books!!
gmail.com David Miller
maintainerFrom: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:59:19 +0000 (UTC)
Quoted 3 lines
At the time it was written (probably 10+ years ago) it was relying on the documented API for
alloc_skb()that statedalloc_skb()either returned ansk_buffof the requested size or it failed.
It was never a formal API that we would only allocate 'size' amount of tailroom.
Quoted 9 lines · David Miller
From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:32:57 +0000 (UTC)Quoted 4 lines
The kernel module that's started failing fills the allocated
sk_buffuntiltailroom()indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).The amount of tailroom an SKB has is implementation dependent.
Today, that's apparently the case, but that's not what the man page for alloc_skb said when the code was written.
Quoted 4 lines
It's incredibly poor form to rely upon it to determine whether a fully sized frame has been constructed or not.
Please fix the code that does this.
That's what I'll do as soon as I can find a definition of what the API for alloc_skb() actually _is_. It has clearly changed in the past few years.
Signature · 3 lines
Grant Edwards grant.b.edwards Yow! I'm also against
at BODY-SURFING!!
gmail.com Quoted 8 lines · David Miller
From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:32:57 +0000 (UTC)Quoted 4 lines
The kernel module that's started failing fills the allocated
sk_buffuntiltailroom()indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).
Quoted 6 lines
The amount of tailroom an SKB has is implementation dependent.
It's incredibly poor form to rely upon it to determine whether a fully sized frame has been constructed or not.
Please fix the code that does this.
By the way, we had a similar problem, and the fix was :
Grant, depending on the context, you might use skb->avail_size and skb_availroom() as well.
Beware skb->avail_size is unioned with skb->{mark|dropcount}
David Miller
maintainerFrom: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 19:01:40 +0000 (UTC)
Quoted 3 lines
That's what I'll do as soon as I can find a definition of what the API for
alloc_skb()actually _is_. It has clearly changed in the past few years.
And it will continue to change. There are no stable APIs inside of the kernel, none.
Quoted 9 lines · David Miller
From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:59:19 +0000 (UTC)Quoted 3 lines
At the time it was written (probably 10+ years ago) it was relying on the documented API for
alloc_skb()that statedalloc_skb()either returned ansk_buffof the requested size or it failed.It was never a formal API that we would only allocate 'size' amount of tailroom.
Well, somebody forgot to tell whoever wrote the man page for alloc_skb(). :)
Signature · 3 lines
Grant Edwards grant.b.edwards Yow! I'm ZIPPY the PINHEAD
at and I'm totally committed
gmail.com to the festive mode.Quoted 9 lines · David Miller
From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 19:01:40 +0000 (UTC)Quoted 3 lines
That's what I'll do as soon as I can find a definition of what the API for
alloc_skb()actually _is_. It has clearly changed in the past few years.And it will continue to change. There are no stable APIs inside of the kernel, none.
Oh, I know (as does anybody who has maintained a driver for more than a few months). Right now, I'm just trying to find out what the current API for alloc_skb() is.
Is skb_tailroom() guaranteed to be >= the requested size?
Signature · 3 lines
Grant Edwards grant.b.edwards Yow! I'm wet! I'm wild!
at
gmail.com Quoted 16 lines · Eric Dumazet
Quoted 8 lines · David Miller
From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:32:57 +0000 (UTC)Quoted 4 lines
The kernel module that's started failing fills the allocated
sk_buffuntiltailroom()indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).Quoted 6 lines
The amount of tailroom an SKB has is implementation dependent.
It's incredibly poor form to rely upon it to determine whether a fully sized frame has been constructed or not.
Please fix the code that does this.
Quoted 8 lines
By the way, we had a similar problem, and the fix was :
Grant, depending on the context, you might use
skb->avail_sizeandskb_availroom()as well.Beware
skb->avail_sizeis unioned with skb->{mark|dropcount}
Thanks for the pointer.
Signature · 4 lines
Grant Edwards grant.b.edwards Yow! ANN JILLIAN'S HAIR
at makes LONI ANDERSON'S
gmail.com HAIR look like RICARDO
MONTALBAN'S HAIR!Quoted 2 lines · Grant Edwards
Is
skb_tailroom()guaranteed to be >= the requested size?
Of course, but only right after alloc_skb().
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Grant Edwards
Is
skb_tailroom()guaranteed to be >= the requested size?Of course, but only right after
alloc_skb().
That's good enough.
Signature · 4 lines
Grant Edwards grant.b.edwards Yow! Okay ... I'm going
at home to write the "I HATE
gmail.com RUBIK's CUBE HANDBOOK FOR
DEAD CAT LOVERS" ...PATCH 1/1 net: Update kernel-doc for __alloc_skb()
In reply to Grant Edwards__alloc_skb() now extends tailroom to allow the use of padding added by the heap allocator.
--- net/core/skbuff.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
net/core/skbuff.c +2 -2
@@ -160,8 +160,8 @@ static void skb_under_panic(struct sk_buff *skb, int sz, void *here)* @node: numa node to allocate memory on * * Allocate a new &sk_buff. The returned buffer has no headroom and a -* tail room of size bytes. The object has a reference count of one. -* The return is the buffer. On a failure the return is %NULL. +* tail room of at least size bytes. The object has a reference count +* of one. The return is the buffer. On a failure the return is %NULL. * * Buffers may only be allocated from interrupts using a @gfp_mask of * %GFP_ATOMIC.
Signature · 7 lines
1.7.7.6 -- Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Quoted 9 lines · David Miller
From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:59:19 +0000 (UTC)Quoted 3 lines
At the time it was written (probably 10+ years ago) it was relying on the documented API for
alloc_skb()that statedalloc_skb()either returned ansk_buffof the requested size or it failed.It was never a formal API that we would only allocate 'size' amount of tailroom.
How can you say that?
Quoted 1 line
From
skbuff.c:
/**
*__alloc_skb-allocate a network buffer
*@size: size to allocate
*@gfp_mask: allocation mask
*@fclone: allocate from fclone cache instead of head cache *and allocate a cloned (child) skb
*@node: numa node to allocate memory on
*
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
*Allocate a new
&sk_buff. The returned buffer has no headroom and a *tail room of size bytes. The object has a reference count of one.
*The return is the buffer. On a failure the return is %NULL.
*
*Buffers may only be allocated from interrupts using a @gfp_mask of
*%GFP_ATOMIC.
*/Signature · 3 lines
Grant Edwards grant.b.edwards Yow! Did you move a lot of
at KOREAN STEAK KNIVES this
gmail.com trip, Dingy?Quoted 4 lines · Grant Edwards
Quoted 3 lines · David Miller
From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:59:19 +0000 (UTC)
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
At the time it was written (probably 10+ years ago) it was relying on the documented API for
alloc_skb()that statedalloc_skb()either returned ansk_buffof the requested size or it failed.It was never a formal API that we would only allocate 'size' amount of tailroom.
How can you say that?
Documentation was stale, so what ?
kmalloc(99) doesnt allocate 99 bytes but 128, so what ?
Grant, what about you fix your code ?
Quoted 9 lines · Eric Dumazet
Quoted 6 lines · Grant Edwards
Quoted 2 lines · David Miller
It was never a formal API that we would only allocate 'size' amount of tailroom.
How can you say that?
Documentation was stale, so what ?
So there _was_ a formal API that said you would only allocate 'size' amount of tailroom. That's what.
Quoted 1 line
kmalloc(99) doesnt allocate 99 bytes but 128, so what?
Doing so violated the documented API.
You said there was never any API definition that said tailroom() == requested size, and implied that it was stupid to write code that expected tailroom() == requested size.
I was merely pointing out that the API was indeed documented that way.
Quoted 1 line
Grant, what about you fix your code ?
I did.
And the API documentation has now been fixed as well, but don't try to tell me that the API documentation didn't promise to work the way my code expected it to work.
Signature · 4 lines
Grant Edwards grant.b.edwards Yow! Youth of today!
at Join me in a mass rally
gmail.com for traditional mental
attitudes!Quoted 1 line · Grant Edwards
I was merely pointing out that the API was indeed documented that way.
Good, you are right and we were wrong.
Hopefully you can still use linux-2
Re: [PATCH net-next] net: Update kernel-doc for __alloc_skb()
In reply to Ben HutchingsFrom: Ben Hutchings <[email protected]>
Date: Thu, 7 Jun 2012 02:23:37 +0100
Quoted 4 lines
__alloc_skb()now extends tailroom to allow the use of padding added by the heap allocator.Signed-off-by Ben Hutchings <[email protected]>
Applied.