Skip to contentSkip to conversation

kswapd/tg3 issue

DISCUSSION Yaroslav Halchenko 5 participants 10 messages Nov 30, 2006 – Dec 1, 2006

lore
Share & tools
lore
#0 lore raw

Dear Kernel People,

Just got a logwatch daily mail which revealed a problem:

[2024412.788680] kswapd1: page allocation failure. order:2, mode:0x20

and a lengthy backtrace with head

,------------------------------------------------------------------------

| [2024412.795212] Call Trace:
| [2024412.799768]  <IRQ> [<ffffffff8020c852>] __alloc_pages+0x27a/0x291
| [2024412.806452]  [<ffffffff802a08e3>] kmem_getpages+0x5e/0xd8
| [2024412.812370]  [<ffffffff80212c68>] cache_grow+0xd0/0x185
| [2024412.818064]  [<ffffffff80245c4f>] cache_alloc_refill+0x18c/0x1da
| [2024412.824625]  [<ffffffff802a1979>] __kmalloc+0x93/0xa3
| [2024412.830145]  [<ffffffff80222e9e>] __alloc_skb+0x54/0x117
| [2024412.835958]  [<ffffffff803b8a55>] __netdev_alloc_skb+0x12/0x2d
| [2024412.842347]  [<ffffffff80370292>] tg3_alloc_rx_skb+0xbb/0x146

`---
full dmesg is at
http://www.onerussian.com/Linux/bugs/bug.kswapd/dmesg

is that critical? seems to behave ok but...

More details on the system and error in particular can be
http://www.onerussian.com/Linux/bugs/bug.kswapd/

Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik        
#1 lore raw
Quoted 16 lines · On Thu, 30 Nov 2006 09:43:55 -0500 Yaroslav Halchenko…

Dear Kernel People,

Just got a logwatch daily mail which revealed a problem:

[2024412.788680] kswapd1: page allocation failure. order:2, mode:0x20

and a lengthy backtrace with head

,------------------------------------------------------------------------

| [2024412.795212] Call Trace:
| [2024412.799768]  <IRQ> [<ffffffff8020c852>] __alloc_pages+0x27a/0x291
| [2024412.806452]  [<ffffffff802a08e3>] kmem_getpages+0x5e/0xd8
| [2024412.812370]  [<ffffffff80212c68>] cache_grow+0xd0/0x185
| [2024412.818064]  [<ffffffff80245c4f>] cache_alloc_refill+0x18c/0x1da
| [2024412.824625]  [<ffffffff802a1979>] __kmalloc+0x93/0xa3
| [2024412.830145]  [<ffffffff80222e9e>] __alloc_skb+0x54/0x117
| [2024412.835958]  [<ffffffff803b8a55>] __netdev_alloc_skb+0x12/0x2d
| [2024412.842347]  [<ffffffff80370292>] tg3_alloc_rx_skb+0xbb/0x146
Quoted 5 lines

`---
full dmesg is at
http://www.onerussian.com/Linux/bugs/bug.kswapd/dmesg

is that critical? seems to behave ok but...

Its tell us that the machine got very very tight on memory, far tighter than it probably ever should in normal situations. It is harmless of itself and if you only get the odd one is not a worry.

#2 lore raw

Thank you Alan

Ok - I am adding more memory in my purchasing plan ;-) For now I guess adding swap space should help, right?

Quoted 4 lines
Quoted 1 line
Quoted 1 line

...<

Its tell us that the machine got very very tight on memory, far tighter than it probably ever should in normal situations. It is harmless of itself and if you only get the odd one is not a worry.

Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik        
#3 lore raw
Quoted 4 lines · Yaroslav Halchenko

Thank you Alan

Ok - I am adding more memory in my purchasing plan ;-) For now I guess adding swap space should help, right?

actually since this was networking...
you probably should bump the value in

/proc/sys/vm/min_free_kbytes

a bit (like by 50%); that makes the kernel keep a bigger pool free for emergencies/spikes...

That might be enough already if your system isn't swapping a whole lot.

Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
#4 lore raw

Thank you Arjan for advice

I had 5746, made it 8619.

Is that a good practice in general to have that value higher for a server with lots of I/O including networking? (there is a RAID on that system and 2 bonded gigabit interfaces) Is there any heuristic to decide on that value ?

Quoted 8 lines · Arjan van de Ven
Quoted 1 line

...<

actually since this was networking...
you probably should bump the value in /proc/sys/vm/min_free_kbytes
a bit (like by 50%); that makes the kernel keep a bigger pool free for emergencies/spikes...
That might be enough already if your system isn't swapping a whole lot.

Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik        
#5 lore raw
In reply to Yaroslav Halchenko
Quoted 4 lines · On Thu, 30 Nov 2006 10:10:03 -0500 Yaroslav Halchenko…

Thank you Alan

Ok - I am adding more memory in my purchasing plan ;-) For now I guess adding swap space should help, right?

Under heavy network or I/O pressure it may not have time to swap to get the memory. Thus adding swap won't usually help. Adding RAM may do but its often not the best answer. Arjan's suggestion should sort it, and - yes typically boxes with very high I/O and network load need more of a pool of memory free for immediate use than other systems.

#6 lore raw
Quoted 6 lines · Alan

Under heavy network or I/O pressure it may not have time to swap to get the memory. Thus adding swap won't usually help. Adding RAM may do but its often not the best answer. Arjan's suggestion should sort it, and - yes typically boxes with very high I/O and network load need more of a pool of memory free for immediate use than other systems.

It could be nice if the kernel could autotune this, for example by raising the free memory goal when memory shortage is detected, and lowering it gradually when not.

The sysctl could be a minimum from which this is calculated.

Signature · 1 line
error compiling committee.c: too many arguments to function
#7 lore raw

On another box which has 4 times more RAM or a bit more than twice total memory, it had twice as high vm.min_free_kbytes
on another node with even more RAM it is 13821..

hm - so what is the algorithm which sets it? percent of available RAM?

For now I am adjusting it on that server to be twice from default.

Thanks everyone for your help

Quoted 12 lines · Avi Kivity
Quoted 5 lines · Alan

Under heavy network or I/O pressure it may not have time to swap to get the memory. Thus adding swap won't usually help. Adding RAM may do but its often not the best answer. Arjan's suggestion should sort it, and - yes typically boxes with very high I/O and network load need more of a pool of memory free for immediate use than other systems.

It could be nice if the kernel could autotune this, for example by raising the free memory goal when memory shortage is detected, and lowering it gradually when not.

The sysctl could be a minimum from which this is calculated.

Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik        
#8 lore raw
In reply to Alan
Quoted 19 lines · Alan
Quoted 19 lines

Just got a logwatch daily mail which revealed a problem:

[2024412.788680] kswapd1: page allocation failure. order:2, mode:0x20

and a lengthy backtrace with head

,------------------------------------------------------------------------

| [2024412.795212] Call Trace:
| [2024412.799768]  <IRQ> [<ffffffff8020c852>] __alloc_pages+0x27a/0x291
| [2024412.806452]  [<ffffffff802a08e3>] kmem_getpages+0x5e/0xd8
| [2024412.812370]  [<ffffffff80212c68>] cache_grow+0xd0/0x185
| [2024412.818064]  [<ffffffff80245c4f>] cache_alloc_refill+0x18c/0x1da
| [2024412.824625]  [<ffffffff802a1979>] __kmalloc+0x93/0xa3
| [2024412.830145]  [<ffffffff80222e9e>] __alloc_skb+0x54/0x117
| [2024412.835958]  [<ffffffff803b8a55>] __netdev_alloc_skb+0x12/0x2d
| [2024412.842347]  [<ffffffff80370292>] tg3_alloc_rx_skb+0xbb/0x146

`---
full dmesg is at
http://www.onerussian.com/Linux/bugs/bug.kswapd/dmesg

Quoted 5 lines
Quoted 1 line

is that critical? seems to behave ok but...

Its tell us that the machine got very very tight on memory, far tighter than it probably ever should in normal situations. It is harmless of itself and if you only get the odd one is not a worry.

Since this is a 2nd order allocation, it could also be that you have memory but it's fragmented. If you aren't using jumbograms you can try disabling that.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#9 lore raw
Quoted 2 lines

Since this is a 2nd order allocation, it could also be that you have memory but it's fragmented.

Thanks for the info!

Quoted 2 lines

If you aren't using jumbograms you can
try disabling that.

disabling 2nd order allocation?
and I do use jumbos on that box (it is an NFS server so jumbo frames -- MTU 9000 -- presumable help a bit on CPU utilization)

Quoted 1 line

Cheers,

Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik