Skip to contentSkip to conversation

E1000 - page allocation failure - saga continues :(

DISCUSSION Lukas Hejtmanek 4 participants 14 messages Apr 14, 2005 – Apr 20, 2005

lore
Share & tools
lore
#0 lore raw

Hello,

today I tried 2.6.11.7 kernel with hoping that allocation failures disappear. Unfortunately they did not.

Default min_free_kb is 3200kB.

Here is stack trace:

swapper: page allocation failure. order:0, mode:0x20

 [<c0139783>] __alloc_pages+0x2b3/0x420
 [<c013c4f1>] kmem_getpages+0x31/0xa0
 [<c013d22e>] cache_grow+0xae/0x160
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c013d45b>] cache_alloc_refill+0x17b/0x230
 [<c013d7d8>] __kmalloc+0x88/0xa0
 [<c0327ce7>] alloc_skb+0x47/0xf0
 [<c02bff97>] e1000_alloc_rx_buffers+0x57/0x100
 [<c02bfc3f>] e1000_clean_rx_irq+0x1bf/0x4c0
 [<c02bf99e>] e1000_clean_tx_irq+0x14e/0x230
 [<c02bf79d>] e1000_clean+0x4d/0x100
 [<c032e6e1>] net_rx_action+0x81/0x110
 [<c011d5aa>] __do_softirq+0xba/0xd0
 [<c011d5ed>] do_softirq+0x2d/0x30
 [<c011d6b9>] irq_exit+0x39/0x40
 [<c01030dc>] apic_timer_interrupt+0x1c/0x24
 [<c0100513>] default_idle+0x23/0x30
 [<c01005bf>] cpu_idle+0x5f/0x70
 [<c04dc988>] start_kernel+0x158/0x180
 [<c04dc390>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:0, mode:0x20
 [<c0139783>] __alloc_pages+0x2b3/0x420
 [<c013c4f1>] kmem_getpages+0x31/0xa0
 [<c013d22e>] cache_grow+0xae/0x160
 [<c02b80d8>] as_next_request+0x38/0x50
 [<c013d45b>] cache_alloc_refill+0x17b/0x230
 [<c02df727>] scsi_put_command+0x77/0xb0
 [<c013d7d8>] __kmalloc+0x88/0xa0
 [<c0327ce7>] alloc_skb+0x47/0xf0
 [<c02bff97>] e1000_alloc_rx_buffers+0x57/0x100
 [<c02bfc3f>] e1000_clean_rx_irq+0x1bf/0x4c0
 [<c02bf99e>] e1000_clean_tx_irq+0x14e/0x230
 [<c02bf79d>] e1000_clean+0x4d/0x100
 [<c032e6e1>] net_rx_action+0x81/0x110
 [<c011d5aa>] __do_softirq+0xba/0xd0
 [<c011d5ed>] do_softirq+0x2d/0x30
 [<c011d6b9>] irq_exit+0x39/0x40
 [<c01030dc>] apic_timer_interrupt+0x1c/0x24
 [<c0100513>] default_idle+0x23/0x30
 [<c01005bf>] cpu_idle+0x5f/0x70
 [<c04dc988>] start_kernel+0x158/0x180
 [<c04dc390>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:1, mode:0x20
 [<c0139783>] __alloc_pages+0x2b3/0x420
 [<c013c4f1>] kmem_getpages+0x31/0xa0
 [<c013d077>] alloc_slabmgmt+0x57/0x70
 [<c013d22e>] cache_grow+0xae/0x160
 [<c013d45b>] cache_alloc_refill+0x17b/0x230
 [<c013d7d8>] __kmalloc+0x88/0xa0
 [<c0327ce7>] alloc_skb+0x47/0xf0
 [<c0354541>] tcp_collapse+0xf1/0x390
 [<c0354904>] tcp_prune_queue+0x94/0x1e0
 [<c0353cd4>] tcp_data_queue+0x3e4/0xb60
 [<c0352b73>] tcp_ack+0x4b3/0x590
 [<c03552df>] tcp_rcv_established+0x22f/0x8d0
 [<c010cbf9>] mark_offset_tsc+0x1d9/0x2d0
 [<c035e46b>] tcp_v4_do_rcv+0x12b/0x130
 [<c035eb61>] tcp_v4_rcv+0x6f1/0x950
 [<c03848f5>] ip_nat_fn+0x75/0x1d0
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c03429ff>] ip_local_deliver_finish+0xcf/0x220
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c03392a1>] nf_hook_slow+0xf1/0x130
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c0342410>] ip_local_deliver+0x280/0x2b0
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c0342d49>] ip_rcv_finish+0x1f9/0x280
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c03392a1>] nf_hook_slow+0xf1/0x130
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c034284e>] ip_rcv+0x40e/0x4f0
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c032e4b8>] netif_receive_skb+0x148/0x1d0
 [<c02bfbdc>] e1000_clean_rx_irq+0x15c/0x4c0
 [<c02bf99e>] e1000_clean_tx_irq+0x14e/0x230
 [<c02bf79d>] e1000_clean+0x4d/0x100
 [<c032e6e1>] net_rx_action+0x81/0x110
 [<c011d5aa>] __do_softirq+0xba/0xd0
 [<c011d5ed>] do_softirq+0x2d/0x30
 [<c011d6b9>] irq_exit+0x39/0x40
 [<c0104b0e>] do_IRQ+0x1e/0x30
 [<c010304e>] common_interrupt+0x1a/0x20
 [<c0100513>] default_idle+0x23/0x30
 [<c01005bf>] cpu_idle+0x5f/0x70
 [<c04dc988>] start_kernel+0x158/0x180
 [<c04dc390>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:1, mode:0x20
 [<c0139783>] __alloc_pages+0x2b3/0x420
 [<c013c4f1>] kmem_getpages+0x31/0xa0
 [<c013d077>] alloc_slabmgmt+0x57/0x70
 [<c013d22e>] cache_grow+0xae/0x160
 [<c013d45b>] cache_alloc_refill+0x17b/0x230
 [<c013d7d8>] __kmalloc+0x88/0xa0
 [<c0327ce7>] alloc_skb+0x47/0xf0
 [<c0354541>] tcp_collapse+0xf1/0x390
 [<c0354904>] tcp_prune_queue+0x94/0x1e0
 [<c0353cd4>] tcp_data_queue+0x3e4/0xb60
 [<c0352b73>] tcp_ack+0x4b3/0x590
 [<c03552df>] tcp_rcv_established+0x22f/0x8d0
 [<c035e46b>] tcp_v4_do_rcv+0x12b/0x130
 [<c035eb61>] tcp_v4_rcv+0x6f1/0x950
 [<c03848f5>] ip_nat_fn+0x75/0x1d0
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c03429ff>] ip_local_deliver_finish+0xcf/0x220
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c03392a1>] nf_hook_slow+0xf1/0x130
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c0342410>] ip_local_deliver+0x280/0x2b0
 [<c0342930>] ip_local_deliver_finish+0x0/0x220
 [<c0342d49>] ip_rcv_finish+0x1f9/0x280
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c03392a1>] nf_hook_slow+0xf1/0x130
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c034284e>] ip_rcv+0x40e/0x4f0
 [<c0342b50>] ip_rcv_finish+0x0/0x280
 [<c032e4b8>] netif_receive_skb+0x148/0x1d0
 [<c02bfbdc>] e1000_clean_rx_irq+0x15c/0x4c0
 [<c02bf99e>] e1000_clean_tx_irq+0x14e/0x230
 [<c02bf79d>] e1000_clean+0x4d/0x100
 [<c032e6e1>] net_rx_action+0x81/0x110
 [<c011d5aa>] __do_softirq+0xba/0xd0
 [<c011d5ed>] do_softirq+0x2d/0x30
 [<c011d6b9>] irq_exit+0x39/0x40
 [<c0104b0e>] do_IRQ+0x1e/0x30
 [<c010304e>] common_interrupt+0x1a/0x20
 [<c0100513>] default_idle+0x23/0x30
 [<c01005bf>] cpu_idle+0x5f/0x70
 [<c04dc988>] start_kernel+0x158/0x180
 [<c04dc390>] unknown_bootoption+0x0/0x1e0
Signature · 1 line
Lukáš Hejtmánek
#1 lore raw
Quoted 10 lines · Lukas Hejtmanek

Hello,

today I tried 2.6.11.7 kernel with hoping that allocation failures disappear. Unfortunately they did not.

Default min_free_kb is 3200kB.

Here is stack trace:

swapper: page allocation failure. order:0, mode:0x20

Quoted 11 lines
[<c0139783>] __alloc_pages+0x2b3/0x420
[<c013c4f1>] kmem_getpages+0x31/0xa0
[<c013d22e>] cache_grow+0xae/0x160
[<c0342b50>] ip_rcv_finish+0x0/0x280
[<c013d45b>] cache_alloc_refill+0x17b/0x230
[<c013d7d8>] __kmalloc+0x88/0xa0
[<c0327ce7>] alloc_skb+0x47/0xf0
[<c02bff97>] e1000_alloc_rx_buffers+0x57/0x100
[<c02bfc3f>] e1000_clean_rx_irq+0x1bf/0x4c0

I have those problems too. The (temporary ?) fix is to raise the min_free_kb to an higher value.
echo 65535 > /proc/sys/vm/min_free_kbytes

Maybe such an high value is totally silly, but at least I don't have those messages.

Sincerely yours,

Signature · 2 lines
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [email protected]
#2 lore raw
Quoted 6 lines · Yann Dupont

I have those problems too. The (temporary ?) fix is to raise the min_free_kb to an higher value.
echo 65535 > /proc/sys/vm/min_free_kbytes

Maybe such an high value is totally silly, but at least I don't have those messages.

I know that kernel 2.6.6-bk4 works. So were there some memory manager changes since 2.6.6? If so it looks like there are some bugs.
On the other hand, ethernet driver should not allocate much memory but rather drop packets.

Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.

Signature · 1 line
Lukáš Hejtmánek
#3 lore raw
Quoted 12 lines · Lukas Hejtmanek
Quoted 8 lines · Yann Dupont

I have those problems too. The (temporary ?) fix is to raise the min_free_kb to an higher value.
echo 65535 > /proc/sys/vm/min_free_kbytes

Maybe such an high value is totally silly, but at least I don't have those messages.

Quoted 9 lines

I know that kernel 2.6.6-bk4 works. So were there some memory manager changes since 2.6.6? If so it looks like there are some bugs.
On the other hand, ethernet driver should not allocate much memory but rather drop packets.

Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.

No tweaking at all. No jumbo frames.

Signature · 2 lines
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [email protected]
#4 lore raw
Quoted 8 lines · Yann Dupont
Quoted 7 lines

I know that kernel 2.6.6-bk4 works. So were there some memory manager changes since 2.6.6? If so it looks like there are some bugs.
On the other hand, ethernet driver should not allocate much memory but rather drop packets.

Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.

No tweaking at all. No jumbo frames.

There were assumptions that it is XFS related. Are you using XFS on that box?

I'm able to deterministically produce this error:
on XFS partition store a file from network using multiple threads. If file size is bigger then total memory, then it fails after major part of memory is used for a file cache.

Signature · 1 line
Lukáš Hejtmánek
#5 lore raw
Quoted 16 lines · Lukas Hejtmanek
Quoted 5 lines · Yann Dupont
Quoted 5 lines

I know that kernel 2.6.6-bk4 works. So were there some memory manager changes since 2.6.6? If so it looks like there are some bugs.
On the other hand, ethernet driver should not allocate much memory but rather drop packets.

Quoted 7 lines
Quoted 4 lines

Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.

No tweaking at all. No jumbo frames.

Quoted 9 lines

There were assumptions that it is XFS related. Are you using XFS on that box?

I'm able to deterministically produce this error:
on XFS partition store a file from network using multiple threads. If file size is bigger then total memory, then it fails after major part of memory is used for a file cache.

Ah yes, this is the case.
XFS all over ...

The server is quite heavily stressed, we have a bunch of servers rsyncing on a big SAN volume - formatted with XFS, that's right. (and, if that matters, XFS in on top of a EVMS volume (on top of a LVM2 region)...)

Signature · 2 lines
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [email protected]
#6 lore raw
In reply to Lukas Hejtmanek
Quoted 12 lines · Lukas Hejtmanek
Quoted 8 lines · Yann Dupont

I have those problems too. The (temporary ?) fix is to raise the min_free_kb to an higher value.
echo 65535 > /proc/sys/vm/min_free_kbytes

Maybe such an high value is totally silly, but at least I don't have those messages.

Quoted 9 lines

I know that kernel 2.6.6-bk4 works. So were there some memory manager changes since 2.6.6? If so it looks like there are some bugs.
On the other hand, ethernet driver should not allocate much memory but rather drop packets.

Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.

Do you have turned NAPI on ??? I tried without it off on e1000 and ... surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)

Signature · 2 lines
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [email protected]
#7 lore raw
Quoted 10 lines · Yann Dupont
Quoted 4 lines · Lukas Hejtmanek

Btw, are you using some TCP tweaks? E.g. I have default TCP window size 1MB.

Do you have turned NAPI on ??? I tried without it off on e1000 and ... surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)

Possibly kswapd might be unable to get enough CPU to free memory.

Signature · 1 line
SUSE Labs, Novell Inc.
#8 lore raw
Quoted 12 lines · Nick Piggin
Quoted 7 lines

Do you have turned NAPI on ??? I tried without it off on e1000 and ... surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)

Possibly kswapd might be unable to get enough CPU to free memory.

Ok, so what you're saying is that turning NAPI off is just slowing down things enough to not be hit by
this problem , right ?

Signature · 2 lines
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [email protected]
#9 lore raw
Quoted 10 lines · Yann Dupont
Quoted 8 lines · Nick Piggin
Quoted 7 lines

Do you have turned NAPI on ??? I tried without it off on e1000 and ... surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)

Quoted 8 lines
Quoted 4 lines

Possibly kswapd might be unable to get enough CPU to free memory.

Ok, so what you're saying is that turning NAPI off is just slowing down things enough to not be hit by
this problem , right ?

Perhaps, yes. Or that NAPI is using more CPU than non-NAPI (which I understand can happen in some corner cases).

If you have a multiprocessor (or even hyperthreading), I think you could test this by binding kswapd on cpu CPU, and put nic interrupts on the other - then test with and without NAPI.

That is, presuming you can reproduce the problem on your multiprocessor system in the first place.

Signature · 1 line
SUSE Labs, Novell Inc.
#10 lore raw
In reply to Yann Dupont
Quoted 1 line · Yann Dupont
Quoted 1 line

Possibly kswapd might be unable to get enough CPU to free memory.

I do not see why NIC rather does not drop packets instead of running out of memory.

I know that renicing kswapd helps. But still do not see why 2.6.6 kernel works.

Signature · 1 line
Lukáš Hejtmánek
#11 lore raw
In reply to Yann Dupont
Quoted 3 lines · Yann Dupont

Do you have turned NAPI on ??? I tried without it off on e1000 and ... surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)

I have NAPI on. I tried to turn it off but my test failed, I can see allocation failure again.

Signature · 1 line
Lukáš Hejtmánek
#12 lore raw
Quoted 10 lines · Lukas Hejtmanek
Quoted 3 lines · Yann Dupont

Do you have turned NAPI on ??? I tried without it off on e1000 and ... surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)

I have NAPI on. I tried to turn it off but my test failed, I can see allocation failure again.

Not sure if this was already sugested, but here it is anyway:
echo "vm.min_free_kbytes=16384" >> /etc/sysctl.conf

Regards,
Nuno Silva

#13 lore raw
In reply to Lukas Hejtmanek
Quoted 14 lines · Lukas Hejtmanek
Quoted 5 lines · Yann Dupont

Do you have turned NAPI on ??? I tried without it off on e1000 and ... surprise !
Don't have any messages since 12H now (usually I got those in less than 1H)

I have NAPI on. I tried to turn it off but my test failed, I can see allocation failure again.

Well. forgives me :)
I have re turned NAPI On and my box is still happy 19H later...

So it's obviously not napi.

The problem is beetween the 2 incarnations of kernel (2.6.11.7 with kswapd meesages on thoses who works well), I've changed some more options Not exactly the best way to track bugs :(

Anyway i'll try to catch THE option that make the kernel not so happy under heavy stress. Stay tuned,

Signature · 2 lines
Yann Dupont, Cri de l'université de Nantes
Tel: 02.51.12.53.91 - Fax: 02.51.12.58.60 - [email protected]