2.6.9 - e1000 - page allocation failed
Subsystems (2)
Hello,
I'm getting this message in kernel log:
swapper: page allocation failure. order:0, mode:0x20
[__alloc_pages+441/862] __alloc_pages+0x1b9/0x35e [__get_free_pages+37/63] __get_free_pages+0x25/0x3f [kmem_getpages+33/201] kmem_getpages+0x21/0xc9 [cache_grow+171/333] cache_grow+0xab/0x14d [cache_alloc_refill+372/537] cache_alloc_refill+0x174/0x219 [__kmalloc+133/140] __kmalloc+0x85/0x8c [alloc_skb+71/224] alloc_skb+0x47/0xe0 [e1000_alloc_rx_buffers+68/227] e1000_alloc_rx_buffers+0x44/0xe3 [e1000_clean_rx_irq+398/1095] e1000_clean_rx_irq+0x18e/0x447 [__kfree_skb+131/253] __kfree_skb+0x83/0xfd [e1000_clean+81/202] e1000_clean+0x51/0xca [net_rx_action+119/246] net_rx_action+0x77/0xf6 [__do_softirq+183/198] __do_softirq+0xb7/0xc6 [do_softirq+45/47] do_softirq+0x2d/0x2f [do_IRQ+274/304] do_IRQ+0x112/0x130 [common_interrupt+24/32] common_interrupt+0x18/0x20 [default_idle+0/44] default_idle+0x0/0x2c [default_idle+41/44] default_idle+0x29/0x2c [cpu_idle+63/88] cpu_idle+0x3f/0x58 [start_kernel+361/388] start_kernel+0x169/0x184 [unknown_bootoption+0/348] unknown_bootoption+0x0/0x15c
That machine has 300MB of free memory of 1GB total and 4GB of free swap. So what is wrong?
Signature · 1 line
Lukáš Hejtmánek
Lukas Hejtmanek <[email protected]> :
[page allocation failure with e1000]
If you are using TSO, try patch below by Herbert Xu (available from http://marc.theaimsgroup.com/?l=linux-netdev&m=109799935603132&w=3)
net/ipv4/tcp_output.c +5 -1
@@ -455,8 +455,12 @@ {struct tcp_opt *tp =tcp_sk (sk);struct sk_buff *buff; -int nsize = skb->len - len; +int nsize;u16 flags; + + nsize =skb_headlen (skb) - len; +if (nsize <0 ) + nsize =0 ; if (skb_cloned(skb) && skb_is_nonlinear(skb) &&
Quoted 13 lines · Francois Romieu
Lukas Hejtmanek <[email protected]> :
[page allocation failure with e1000]If you are using TSO, try patch below by Herbert Xu (available from http://marc.theaimsgroup.com/?l=linux-netdev&m=109799935603132&w=3)
net/ipv4/tcp_output.c +0
-0@@ -455,8 +455,12 @@ {struct tcp_opt *tp =tcp_sk (sk);struct sk_buff *buff;
net/ipv4/tcp_output.c line 466
Quoted 10 lines
net/ipv4/tcp_output.c +5
-1-int nsize = skb->len - len; +int nsize;u16 flags; + + nsize =skb_headlen (skb) - len; +if (nsize <0 ) + nsize =0 ; if (skb_cloned(skb) && skb_is_nonlinear(skb) &&
I'd be interested in knowing if this fixes it - I don't expect it will, because that's a zero-order allocation failure. He's really out of memory.
The e1000 driver has a default rx ring size of 256 which seems a bit nutty:
a back-to-back GFP_ATOMIC allocation of 256 skbs could easily exhaust the page allocator pools.
Probably this machine needs to increase /proc/sys/vm/min_free_kbytes.
Quoted 8 lines · Andrew Morton
I'd be interested in knowing if this fixes it - I don't expect it will, because that's a zero-order allocation failure. He's really out of memory.
The e1000 driver has a default rx ring size of 256 which seems a bit nutty:
a back-to-backGFP_ATOMICallocation of 256 skbs could easily exhaust the page allocator pools.Probably this machine needs to increase
/proc/sys/vm/min_free_kbytes.
It did not help.
However I tweak network stack a little bit:/sbin/sysctl -w net/core/rmem_max=8388608 /sbin/sysctl -w net/core/wmem_max=8388608 /sbin/sysctl -w net/core/rmem_default=1048576 /sbin/sysctl -w net/core/wmem_default=1048576 /sbin/sysctl -w net/ipv4/tcp_window_scaling=1 /sbin/sysctl -w net/ipv4/tcp_rmem="4096 1048576 8388608" /sbin/sysctl -w net/ipv4/tcp_wmem="4096 1048576 8388608" /sbin/ifconfig eth0 txqueuelen 1000
this is /proc/meminfo:
cat /proc/meminfo
MemTotal: 1035116 kB MemFree: 447028 kB Buffers: 0 kB Cached: 522444 kB SwapCached: 0 kB Active: 98092 kB Inactive: 460408 kB HighTotal: 131008 kB HighFree: 308 kB LowTotal: 904108 kB LowFree: 446720 kB SwapTotal: 4008208 kB SwapFree: 4008204 kB Dirty: 0 kB Writeback: 0 kB Mapped: 45804 kB Slab: 21112 kB Committed_AS: 128148 kB PageTables: 1528 kB VmallocTotal: 114680 kB VmallocUsed: 2964 kB VmallocChunk: 111700 kB
Note that there is 400MB of _free_ memory.
This is from slabinfo (hope that it is relevant info..)
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0 size-131072 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0 size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0 size-65536 1 1 65536 1 16 : tunables 8 4 0 : slabdata 1 1 0 size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0 size-32768 16 16 32768 1 8 : tunables 8 4 0 : slabdata 16 16 0 size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0 size-16384 2 2 16384 1 4 : tunables 8 4 0 : slabdata 2 2 0 size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0 size-8192 120 120 8192 1 2 : tunables 8 4 0 : slabdata 120 120 0 size-4096(DMA) 0 0 4096 1 1 : tunables 24 12
8 : slabdata 0 0 0
size-4096 300 300 4096 1 1 : tunables 24 12
8 : slabdata 300 300 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12
8 : slabdata 0 0 0
size-2048 106 106 2048 2 1 : tunables 24 12
8 : slabdata 53 53 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27
8 : slabdata 0 0 0
size-1024 1052 1052 1024 4 1 : tunables 54 27
8 : slabdata 263 263 0
size-512(DMA) 0 0 512 8 1 : tunables 54 27
8 : slabdata 0 0 0
size-512 294 1240 512 8 1 : tunables 54 27
8 : slabdata 155 155 0
size-256(DMA) 0 0 256 15 1 : tunables 120 60
8 : slabdata 0 0 0
size-256 211 1170 256 15 1 : tunables 120 60
8 : slabdata 78 78 0
size-128(DMA) 0 0 128 31 1 : tunables 120 60
8 : slabdata 0 0 0
size-128 1688 2728 128 31 1 : tunables 120 60
8 : slabdata 88 88 0
size-64(DMA) 0 0 64 61 1 : tunables 120 60
8 : slabdata 0 0 0
size-64 610 2318 64 61 1 : tunables 120 60
8 : slabdata 38 38 0
size-32(DMA) 0 0 32 119 1 : tunables 120 60
8 : slabdata 0 0 0
size-32 1381 4641 32 119 1 : tunables 120 60
8 : slabdata 39 39 15kmem_cache 165 165 256 15 1 : tunables 120 60
8 : slabdata 11 11 0
Signature · 1 line
Lukáš Hejtmánek
Quoted 5 lines · Lukas Hejtmanek
Quoted 3 lines · Andrew Morton
I'd be interested in knowing if this fixes it - I don't expect it will, because that's a zero-order allocation failure. He's really out of memory.
Quoted 9 lines
Quoted 5 lines
The e1000 driver has a default rx ring size of 256 which seems a bit nutty:
a back-to-backGFP_ATOMICallocation of 256 skbs could easily exhaust the page allocator pools.Probably this machine needs to increase
/proc/sys/vm/min_free_kbytes.It did not help.
What did you increase it to? What was the allocation failure message?
Quoted 10 lines · Nick Piggin
Quoted 8 lines
Quoted 4 lines
a back-to-back
GFP_ATOMICallocation of 256 skbs could easily exhaust the page allocator pools.Probably this machine needs to increase
/proc/sys/vm/min_free_kbytes.It did not help.
What did you increase it to? What was the allocation failure message?
Sorry for late answer I missed this mail. I increased it to 10MB and failuer message was the same...
Signature · 1 line
Lukáš Hejtmánek
Quoted 8 lines · Andrew Morton
I'd be interested in knowing if this fixes it - I don't expect it will, because that's a zero-order allocation failure. He's really out of memory.
The e1000 driver has a default rx ring size of 256 which seems a bit nutty:
a back-to-backGFP_ATOMICallocation of 256 skbs could easily exhaust the page allocator pools.Probably this machine needs to increase
/proc/sys/vm/min_free_kbytes.
Well is there some possiblility to solve it? I have lots of free memory (300MB) and it still complains about not enough free memory.
I'm using this:/sbin/sysctl -w net/core/rmem_max=8388608 /sbin/sysctl -w net/core/wmem_max=8388608 /sbin/sysctl -w net/core/rmem_default=1048576 /sbin/sysctl -w net/core/wmem_default=1048576 /sbin/sysctl -w net/ipv4/tcp_window_scaling=1 /sbin/sysctl -w net/ipv4/tcp_rmem="4096 1048576 8388608" /sbin/sysctl -w net/ipv4/tcp_wmem="4096 1048576 8388608" /sbin/ifconfig eth0 txqueuelen 1000
Signature · 1 line
Lukáš Hejtmánek
Quoted 8 lines · Andrew Morton
I'd be interested in knowing if this fixes it - I don't expect it will, because that's a zero-order allocation failure. He's really out of memory.
The e1000 driver has a default rx ring size of 256 which seems a bit nutty:
a back-to-backGFP_ATOMICallocation of 256 skbs could easily exhaust the page allocator pools.Probably this machine needs to increase
/proc/sys/vm/min_free_kbytes.
However. I have two machines very similar and one opses and one does not. I wondered why. I've found out that if I turn whole netfilter to module and disable:CONFIG_NET_IPGRE=yCONFIG_IP_MROUTE=yCONFIG_IP_PIMSM_V1=yCONFIG_IP_PIMSM_V2=y
Then problem seems to disappeare on both...
(I do not use iptables on any machine)
Signature · 1 line
Lukáš Hejtmánek