kswapd/tg3 issue
Dear Kernel People,
Just got a logwatch daily mail which revealed a problem:
[2024412.788680] kswapd1: page allocation failure. order:2, mode:0x20
and a lengthy backtrace with head
,------------------------------------------------------------------------
| [2024412.795212] Call Trace: | [2024412.799768] <IRQ> [<ffffffff8020c852>] __alloc_pages+0x27a/0x291 | [2024412.806452] [<ffffffff802a08e3>] kmem_getpages+0x5e/0xd8 | [2024412.812370] [<ffffffff80212c68>] cache_grow+0xd0/0x185 | [2024412.818064] [<ffffffff80245c4f>] cache_alloc_refill+0x18c/0x1da | [2024412.824625] [<ffffffff802a1979>] __kmalloc+0x93/0xa3 | [2024412.830145] [<ffffffff80222e9e>] __alloc_skb+0x54/0x117 | [2024412.835958] [<ffffffff803b8a55>] __netdev_alloc_skb+0x12/0x2d | [2024412.842347] [<ffffffff80370292>] tg3_alloc_rx_skb+0xbb/0x146
`---
full dmesg is at
http://www.onerussian.com/Linux/bugs/bug.kswapd/dmesg
is that critical? seems to behave ok but...
More details on the system and error in particular can be
http://www.onerussian.com/Linux/bugs/bug.kswapd/
Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW: http://www.linkedin.com/in/yarik Quoted 16 lines · On Thu, 30 Nov 2006 09:43:55 -0500 Yaroslav Halchenko…
Dear Kernel People,
Just got a logwatch daily mail which revealed a problem:
[2024412.788680] kswapd1: page allocation failure. order:2, mode:0x20and a lengthy backtrace with head
,------------------------------------------------------------------------
| [2024412.795212] Call Trace: | [2024412.799768] <IRQ> [<ffffffff8020c852>] __alloc_pages+0x27a/0x291 | [2024412.806452] [<ffffffff802a08e3>] kmem_getpages+0x5e/0xd8 | [2024412.812370] [<ffffffff80212c68>] cache_grow+0xd0/0x185 | [2024412.818064] [<ffffffff80245c4f>] cache_alloc_refill+0x18c/0x1da | [2024412.824625] [<ffffffff802a1979>] __kmalloc+0x93/0xa3 | [2024412.830145] [<ffffffff80222e9e>] __alloc_skb+0x54/0x117 | [2024412.835958] [<ffffffff803b8a55>] __netdev_alloc_skb+0x12/0x2d | [2024412.842347] [<ffffffff80370292>] tg3_alloc_rx_skb+0xbb/0x146
Quoted 5 lines
`---
full dmesg is at
http://www.onerussian.com/Linux/bugs/bug.kswapd/dmesgis that critical? seems to behave ok but...
Its tell us that the machine got very very tight on memory, far tighter than it probably ever should in normal situations. It is harmless of itself and if you only get the odd one is not a worry.
Thank you Alan
Ok - I am adding more memory in my purchasing plan ;-) For now I guess adding swap space should help, right?
Quoted 4 lines
Quoted 1 line
Quoted 1 line
...<
Its tell us that the machine got very very tight on memory, far tighter than it probably ever should in normal situations. It is harmless of itself and if you only get the odd one is not a worry.
Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW: http://www.linkedin.com/in/yarik Quoted 4 lines · Yaroslav Halchenko
Thank you Alan
Ok - I am adding more memory in my purchasing plan ;-) For now I guess adding swap space should help, right?
actually since this was networking...
you probably should bump the value in
/proc/sys/vm/min_free_kbytes
a bit (like by 50%); that makes the kernel keep a bigger pool free for emergencies/spikes...
That might be enough already if your system isn't swapping a whole lot.
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Thank you Arjan for advice
I had 5746, made it 8619.
Is that a good practice in general to have that value higher for a server with lots of I/O including networking? (there is a RAID on that system and 2 bonded gigabit interfaces) Is there any heuristic to decide on that value ?
Quoted 8 lines · Arjan van de Ven
Quoted 1 line
...<
actually since this was networking...
you probably should bump the value in/proc/sys/vm/min_free_kbytes
a bit (like by 50%); that makes the kernel keep a bigger pool free for emergencies/spikes...
That might be enough already if your system isn't swapping a whole lot.
Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW: http://www.linkedin.com/in/yarik Quoted 4 lines · On Thu, 30 Nov 2006 10:10:03 -0500 Yaroslav Halchenko…
Thank you Alan
Ok - I am adding more memory in my purchasing plan ;-) For now I guess adding swap space should help, right?
Under heavy network or I/O pressure it may not have time to swap to get the memory. Thus adding swap won't usually help. Adding RAM may do but its often not the best answer. Arjan's suggestion should sort it, and - yes typically boxes with very high I/O and network load need more of a pool of memory free for immediate use than other systems.
Quoted 6 lines · Alan
Under heavy network or I/O pressure it may not have time to swap to get the memory. Thus adding swap won't usually help. Adding RAM may do but its often not the best answer. Arjan's suggestion should sort it, and - yes typically boxes with very high I/O and network load need more of a pool of memory free for immediate use than other systems.
It could be nice if the kernel could autotune this, for example by raising the free memory goal when memory shortage is detected, and lowering it gradually when not.
The sysctl could be a minimum from which this is calculated.
Signature · 1 line
error compiling committee.c: too many arguments to function
On another box which has 4 times more RAM or a bit more than twice total memory, it had twice as high vm.min_free_kbytes
on another node with even more RAM it is 13821..
hm - so what is the algorithm which sets it? percent of available RAM?
For now I am adjusting it on that server to be twice from default.
Thanks everyone for your help
Quoted 12 lines · Avi Kivity
Quoted 5 lines · Alan
Under heavy network or I/O pressure it may not have time to swap to get the memory. Thus adding swap won't usually help. Adding RAM may do but its often not the best answer. Arjan's suggestion should sort it, and - yes typically boxes with very high I/O and network load need more of a pool of memory free for immediate use than other systems.
It could be nice if the kernel could autotune this, for example by raising the free memory goal when memory shortage is detected, and lowering it gradually when not.
The sysctl could be a minimum from which this is calculated.
Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW: http://www.linkedin.com/in/yarik Quoted 19 lines · Alan
Quoted 19 lines
Just got a logwatch daily mail which revealed a problem:
[2024412.788680] kswapd1: page allocation failure. order:2, mode:0x20and a lengthy backtrace with head
,------------------------------------------------------------------------
| [2024412.795212] Call Trace: | [2024412.799768] <IRQ> [<ffffffff8020c852>] __alloc_pages+0x27a/0x291 | [2024412.806452] [<ffffffff802a08e3>] kmem_getpages+0x5e/0xd8 | [2024412.812370] [<ffffffff80212c68>] cache_grow+0xd0/0x185 | [2024412.818064] [<ffffffff80245c4f>] cache_alloc_refill+0x18c/0x1da | [2024412.824625] [<ffffffff802a1979>] __kmalloc+0x93/0xa3 | [2024412.830145] [<ffffffff80222e9e>] __alloc_skb+0x54/0x117 | [2024412.835958] [<ffffffff803b8a55>] __netdev_alloc_skb+0x12/0x2d | [2024412.842347] [<ffffffff80370292>] tg3_alloc_rx_skb+0xbb/0x146`---
full dmesg is at
http://www.onerussian.com/Linux/bugs/bug.kswapd/dmesg
Quoted 5 lines
Quoted 1 line
is that critical? seems to behave ok but...
Its tell us that the machine got very very tight on memory, far tighter than it probably ever should in normal situations. It is harmless of itself and if you only get the odd one is not a worry.
Since this is a 2nd order allocation, it could also be that you have memory but it's fragmented. If you aren't using jumbograms you can try disabling that.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 2 lines
Since this is a 2nd order allocation, it could also be that you have memory but it's fragmented.
Thanks for the info!
Quoted 2 lines
If you aren't using jumbograms you can
try disabling that.
disabling 2nd order allocation?
and I do use jumbos on that box (it is an NFS server so jumbo frames -- MTU 9000 -- presumable help a bit on CPU utilization)
Quoted 1 line
Cheers,
Signature · 6 lines
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW: http://www.linkedin.com/in/yarik