Skip to contentSkip to conversation

Change in alloc_skb() behavior in 3.2+ kernels?

PATCH Applied Grant Edwards 4 participants 19 messages Jun 6, 2012 – Jun 7, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I'm tracking down a problem that appears to be caused by a change in the behavior of alloc_skb() introduced in kernel version 3.2. In kernel versions prior to 3.2, calling alloc_skb(1350), returned an sk_buff with a tailroom of around 1400 bytes (safely below the default Ethernet frame size limit of 1500).

In 3.2 and later, calling alloc_skb(1350) returns an sk_buff with a tailroom of about 1850.

Why has the "extra" space increased from 60 bytes to 500 bytes?

[It's always possible that I've unintentionally changed something in the kernel configs that causes this, but I've tried to build the kernels as identically as possible.]

The kernel module that's started failing fills the allocated sk_buff until tailroom() indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).

I've found man pages for alloc_skb() from a few years ago that state explicitly that alloc_skb(_size_) will allocate a new sk_buff with no headroom and a tail room of _size_ bytes. This doesn't seem to be the case for recent kernels. Is there any documentation stating what the current behavior is supposed to be?

Are callers to alloc_skb() supposed to check the tailroom and reserve() an appropriate number of bytes such that the tailroom is correct?

Is the tailroom of the allocated sk_buff guaranteed to be at least as large as the requested size, or does application code also have to check for tailroom less than the requested size?

The ultimate question I'm trying to answer is what is the "right" way to allocate an sk_buff that has a size appropriate for an Ethernet frame assuming an MTU of 1500?

Signature · 3 lines
Grant Edwards               grant.b.edwards        Yow! Let's send the
                                  at               Russians defective
                              gmail.com            lifestyle accessories!
#1 lore raw
Quoted 11 lines · Grant Edwards

I'm tracking down a problem that appears to be caused by a change in the behavior of alloc_skb() introduced in kernel version 3.2. In kernel versions prior to 3.2, calling alloc_skb(1350), returned an sk_buff with a tailroom of around 1400 bytes (safely below the default Ethernet frame size limit of 1500).

In 3.2 and later, calling alloc_skb(1350) returns an sk_buff with a tailroom of about 1850.

Why has the "extra" space increased from 60 bytes to 500 bytes?

Because of kmalloc-2048 being used. Previous kernels were losing this space. We are now able to expand some packets without extra re-allocation/copy.

Quoted 8 lines

[It's always possible that I've unintentionally changed something in the kernel configs that causes this, but I've tried to build the kernels as identically as possible.]

The kernel module that's started failing fills the allocated sk_buff until tailroom() indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).

This code seems buggy.

Quoted 11 lines

I've found man pages for alloc_skb() from a few years ago that state explicitly that alloc_skb(_size_) will allocate a new sk_buff with no headroom and a tail room of _size_ bytes. This doesn't seem to be the case for recent kernels. Is there any documentation stating what the current behavior is supposed to be?

Are callers to alloc_skb() supposed to check the tailroom and reserve() an appropriate number of bytes such that the tailroom is correct?

If you allocate skbs with 1500 bytes, you probably should check skb->len more than tailroom...

Quoted 8 lines

Is the tailroom of the allocated sk_buff guaranteed to be at least as large as the requested size, or does application code also have to check for tailroom less than the requested size?

The ultimate question I'm trying to answer is what is the "right" way to allocate an sk_buff that has a size appropriate for an Ethernet frame assuming an MTU of 1500?

I dont know what to answer. Could you point the code in question ?

maintainer
#11 lore raw
In reply to Grant Edwards

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:32:57 +0000 (UTC)

Quoted 4 lines

The kernel module that's started failing fills the allocated sk_buff until tailroom() indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).

The amount of tailroom an SKB has is implementation dependent.

It's incredibly poor form to rely upon it to determine whether a fully sized frame has been constructed or not.

Please fix the code that does this.

#2 lore raw
In reply to Eric Dumazet
Quoted 21 lines · Eric Dumazet
Quoted 10 lines · Grant Edwards

I'm tracking down a problem that appears to be caused by a change in the behavior of alloc_skb() introduced in kernel version 3.2. In kernel versions prior to 3.2, calling alloc_skb(1350), returned an sk_buff with a tailroom of around 1400 bytes (safely below the default Ethernet frame size limit of 1500).

In 3.2 and later, calling alloc_skb(1350) returns an sk_buff with a tailroom of about 1850.

Why has the "extra" space increased from 60 bytes to 500 bytes?

Because of kmalloc-2048 being used. Previous kernels were losing this space. We are now able to expand some packets without extra re-allocation/copy.

Quoted 4 lines

[It's always possible that I've unintentionally changed something in the kernel configs that causes this, but I've tried to build the kernels as identically as possible.]

Quoted 6 lines
Quoted 4 lines

The kernel module that's started failing fills the allocated sk_buff until tailroom() indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).

This code seems buggy.

It is with today's alloc_skb().

At the time it was written (probably 10+ years ago) it was relying on the documented API for alloc_skb() that stated alloc_skb() either returned an sk_buff of the requested size or it failed.

Quoted 6 lines
Quoted 6 lines

I've found man pages for alloc_skb() from a few years ago that state explicitly that alloc_skb(_size_) will allocate a new sk_buff with no headroom and a tail room of _size_ bytes. This doesn't seem to be the case for recent kernels. Is there any documentation stating what the current behavior is supposed to be?

Quoted 6 lines
Quoted 3 lines

Are callers to alloc_skb() supposed to check the tailroom and reserve() an appropriate number of bytes such that the tailroom is correct?

If you allocate skbs with 1500 bytes, you probably should check skb->len more than tailroom...

I can do that -- assuming it's backwards compatible with older kernels as well (let's say as far back as 2.6 -- we stopped trying to support 2.4 kernels last year).

Quoted 9 lines
Quoted 7 lines

Is the tailroom of the allocated sk_buff guaranteed to be at least as large as the requested size, or does application code also have to check for tailroom less than the requested size?

The ultimate question I'm trying to answer is what is the "right" way to allocate an sk_buff that has a size appropriate for an Ethernet frame assuming an MTU of 1500?

I dont know what to answer. Could you point the code in question?

Sure:

alloc_skb(dev->hard_header_len + 1340)

The intent was to allocate a frame that we can guarantee we can send out via the Ethernet device 'dev'. Sometime 12-14 years ago, somebody pulled the number "1340" out of the air under the assumption that the resulting sk_buff would always be safely under the Ethernet limit of 1500 bytes.

That worked until kernel 3.2 came out, at which time we started ending up with 1800 byte frames.

Signature · 3 lines
Grant Edwards               grant.b.edwards        Yow! I've read SEVEN
                                  at               MILLION books!!
                              gmail.com            
maintainer
#3 lore raw

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:59:19 +0000 (UTC)

Quoted 3 lines

At the time it was written (probably 10+ years ago) it was relying on the documented API for alloc_skb() that stated alloc_skb() either returned an sk_buff of the requested size or it failed.

It was never a formal API that we would only allocate 'size' amount of tailroom.

#12 lore raw
In reply to David Miller
Quoted 9 lines · David Miller

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:32:57 +0000 (UTC)

Quoted 4 lines

The kernel module that's started failing fills the allocated sk_buff until tailroom() indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).

The amount of tailroom an SKB has is implementation dependent.

Today, that's apparently the case, but that's not what the man page for alloc_skb said when the code was written.

Quoted 4 lines

It's incredibly poor form to rely upon it to determine whether a fully sized frame has been constructed or not.

Please fix the code that does this.

That's what I'll do as soon as I can find a definition of what the API for alloc_skb() actually _is_. It has clearly changed in the past few years.

Signature · 3 lines
Grant Edwards               grant.b.edwards        Yow! I'm also against
                                  at               BODY-SURFING!!
                              gmail.com            
#17 lore raw
In reply to David Miller
Quoted 8 lines · David Miller

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:32:57 +0000 (UTC)

Quoted 4 lines

The kernel module that's started failing fills the allocated sk_buff until tailroom() indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).

Quoted 6 lines

The amount of tailroom an SKB has is implementation dependent.

It's incredibly poor form to rely upon it to determine whether a fully sized frame has been constructed or not.

Please fix the code that does this.

By the way, we had a similar problem, and the fix was :

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=a21d45726acacc963d8baddf74607d9b74e2b723

Grant, depending on the context, you might use skb->avail_size and skb_availroom() as well.

Beware skb->avail_size is unioned with skb->{mark|dropcount}

maintainer
#13 lore raw
In reply to Grant Edwards

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 19:01:40 +0000 (UTC)

Quoted 3 lines

That's what I'll do as soon as I can find a definition of what the API for alloc_skb() actually _is_. It has clearly changed in the past few years.

And it will continue to change. There are no stable APIs inside of the kernel, none.

#4 lore raw
In reply to David Miller
Quoted 9 lines · David Miller

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:59:19 +0000 (UTC)

Quoted 3 lines

At the time it was written (probably 10+ years ago) it was relying on the documented API for alloc_skb() that stated alloc_skb() either returned an sk_buff of the requested size or it failed.

It was never a formal API that we would only allocate 'size' amount of tailroom.

Well, somebody forgot to tell whoever wrote the man page for alloc_skb(). :)

Signature · 3 lines
Grant Edwards               grant.b.edwards        Yow! I'm ZIPPY the PINHEAD
                                  at               and I'm totally committed
                              gmail.com            to the festive mode.
#14 lore raw
In reply to David Miller
Quoted 9 lines · David Miller

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 19:01:40 +0000 (UTC)

Quoted 3 lines

That's what I'll do as soon as I can find a definition of what the API for alloc_skb() actually _is_. It has clearly changed in the past few years.

And it will continue to change. There are no stable APIs inside of the kernel, none.

Oh, I know (as does anybody who has maintained a driver for more than a few months). Right now, I'm just trying to find out what the current API for alloc_skb() is.

Is skb_tailroom() guaranteed to be >= the requested size?

Signature · 3 lines
Grant Edwards               grant.b.edwards        Yow! I'm wet!  I'm wild!
                                  at               
                              gmail.com            
#18 lore raw
In reply to Eric Dumazet
Quoted 16 lines · Eric Dumazet
Quoted 8 lines · David Miller

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:32:57 +0000 (UTC)

Quoted 4 lines

The kernel module that's started failing fills the allocated sk_buff until tailroom() indicates it is full and then sends it. The problem is that sending a packet with a length of 1850 won't work (it's a MAC-layer Ethernet packet).

Quoted 6 lines

The amount of tailroom an SKB has is implementation dependent.

It's incredibly poor form to rely upon it to determine whether a fully sized frame has been constructed or not.

Please fix the code that does this.

Quoted 8 lines

By the way, we had a similar problem, and the fix was :

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=a21d45726acacc963d8baddf74607d9b74e2b723

Grant, depending on the context, you might use skb->avail_size and skb_availroom() as well.

Beware skb->avail_size is unioned with skb->{mark|dropcount}

Thanks for the pointer.

Signature · 4 lines
Grant Edwards               grant.b.edwards        Yow! ANN JILLIAN'S HAIR
                                  at               makes LONI ANDERSON'S
                              gmail.com            HAIR look like RICARDO
                                                   MONTALBAN'S HAIR!
#16 lore raw
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Grant Edwards

Is skb_tailroom() guaranteed to be >= the requested size?

Of course, but only right after alloc_skb().

That's good enough.

Signature · 4 lines
Grant Edwards               grant.b.edwards        Yow! Okay ... I'm going
                                  at               home to write the "I HATE
                              gmail.com            RUBIK's CUBE HANDBOOK FOR
                                                   DEAD CAT LOVERS" ...
#5 lore raw

PATCH 1/1 net: Update kernel-doc for __alloc_skb()

In reply to Grant Edwards

__alloc_skb() now extends tailroom to allow the use of padding added by the heap allocator.

---
 net/core/skbuff.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
net/core/skbuff.c +2 -2
index 016694d..1d74cea 100644
@@ -160,8 +160,8 @@ static void skb_under_panic(struct sk_buff *skb, int sz, void *here)
  *	@node: numa node to allocate memory on
  *
  *	Allocate a new &sk_buff. The returned buffer has no headroom and a
- *	tail room of size bytes. The object has a reference count of one.
- *	The return is the buffer. On a failure the return is %NULL.
+ *	tail room of at least size bytes. The object has a reference count
+ *	of one. The return is the buffer. On a failure the return is %NULL.
  *
  *	Buffers may only be allocated from interrupts using a @gfp_mask of
  *	%GFP_ATOMIC.
Signature · 7 lines
1.7.7.6


-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#7 lore raw
In reply to David Miller
Quoted 9 lines · David Miller

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:59:19 +0000 (UTC)

Quoted 3 lines

At the time it was written (probably 10+ years ago) it was relying on the documented API for alloc_skb() that stated alloc_skb() either returned an sk_buff of the requested size or it failed.

It was never a formal API that we would only allocate 'size' amount of tailroom.

How can you say that?

Quoted 1 line

From skbuff.c:

/**
*__alloc_skb-allocate a network buffer
*@size: size to allocate
*@gfp_mask: allocation mask
*@fclone: allocate from fclone cache instead of head cache *and allocate a cloned (child) skb
*@node: numa node to allocate memory on
*

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

*Allocate a new &sk_buff. The returned buffer has no headroom and a *tail room of size bytes. The object has a reference count of one.

    *The return is the buffer. On a failure the return is %NULL.
    *
    *Buffers may only be allocated from interrupts using a @gfp_mask of
    *%GFP_ATOMIC.
    */
Signature · 3 lines
Grant Edwards               grant.b.edwards        Yow! Did you move a lot of
                                  at               KOREAN STEAK KNIVES this
                              gmail.com            trip, Dingy?
#8 lore raw
Quoted 4 lines · Grant Edwards
Quoted 3 lines · David Miller

From: Grant Edwards <[email protected]>
Date: Wed, 6 Jun 2012 18:59:19 +0000 (UTC)

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

At the time it was written (probably 10+ years ago) it was relying on the documented API for alloc_skb() that stated alloc_skb() either returned an sk_buff of the requested size or it failed.

It was never a formal API that we would only allocate 'size' amount of tailroom.

How can you say that?

Documentation was stale, so what ?

kmalloc(99) doesnt allocate 99 bytes but 128, so what ?

Grant, what about you fix your code ?

#9 lore raw
Quoted 9 lines · Eric Dumazet
Quoted 6 lines · Grant Edwards
Quoted 2 lines · David Miller

It was never a formal API that we would only allocate 'size' amount of tailroom.

How can you say that?

Documentation was stale, so what ?

So there _was_ a formal API that said you would only allocate 'size' amount of tailroom. That's what.

Quoted 1 line

kmalloc(99) doesnt allocate 99 bytes but 128, so what?

Doing so violated the documented API.

You said there was never any API definition that said tailroom() == requested size, and implied that it was stupid to write code that expected tailroom() == requested size.

I was merely pointing out that the API was indeed documented that way.

Quoted 1 line

Grant, what about you fix your code ?

I did.

And the API documentation has now been fixed as well, but don't try to tell me that the API documentation didn't promise to work the way my code expected it to work.

Signature · 4 lines
Grant Edwards               grant.b.edwards        Yow! Youth of today!
                                  at               Join me in a mass rally
                              gmail.com            for traditional mental
                                                   attitudes!
#10 lore raw
Quoted 1 line · Grant Edwards

I was merely pointing out that the API was indeed documented that way.

Good, you are right and we were wrong.

Hopefully you can still use linux-2