Skip to contentSkip to conversation

2.6.24 Page Allocation Failure

DISCUSSION AndrewL733 2 participants 3 messages Jan 31, 2008 – Feb 1, 2008

Subsystems (4)
lore
Share & tools
lore

AndrewL733

author
#0 lore raw

My setup is Mandriva 2007 64-bit:

IntelS5000PSLSATA motherboard
Single Dual Core 3 Ghz Xeon
2 GB ECC RAM
Myricom 10 GbE Network Adapter (using out of kernel 1.4.0 driver compiled with "MYRI10GE_ALLOC_ORDER=2")

I have two identical systems doing data transfer via Myricom cards connected directly to one another (in other words, not going through a switch).

No problems hammering Myricom with 350 MB/sec NFS traffic while running 2.6.20.15 and 2.6.22.9 (kernel.org "plain vanilla" kernels I compiled and run, also with updated Myricom drivers). However, I just compiled and installed 2.6.24 and get the following errors as soon as I begin doing NFS I/O over the Myricom card. Maybe I missed something in the kernel configuration? Throughput when running 2.6.24 is about 40 percent lower than when running 2.6.22.9-- surely due to these errors.

Jan 29 22:13:36 master kernel: nfsd: page allocation failure. order:2, mode:0x4020
Jan 29 22:13:36 master kernel: Pid: 6586, comm: nfsd Not tainted 2.6.24 #1
Jan 29 22:13:36 master kernel:
Jan 29 22:13:36 master kernel: Call Trace:
Jan 29 22:13:36 master kernel:  <IRQ>  [__alloc_pages+822/912] __alloc_pages+0x336/0x390
Jan 29 22:13:36 master kernel:  <IRQ>  [<ffffffff80279d66>] __alloc_pages+0x336/0x390
Jan 29 22:13:36 master kernel:  [ip_local_deliver+37/112] ip_local_deliver+0x25/0x70
Jan 29 22:13:36 master kernel:  [<ffffffff80417bb5>] ip_local_deliver+0x25/0x70
Jan 29 22:13:36 master kernel:  [_end+130301422/2130457992] :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:36 master kernel:  [<ffffffff88280866>] :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:36 master kernel:  [_end+130325174/2130457992] :myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:36 master kernel:  [<ffffffff8828652e>] :myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:36 master kernel:  [_spin_lock_bh+9/32] _spin_lock_bh+0x9/0x20
Jan 29 22:13:36 master kernel:  [<ffffffff80466819>] _spin_lock_bh+0x9/0x20
Jan 29 22:13:36 master kernel:  [_end+129719928/2130457992] :sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:36 master kernel:  [<ffffffff881f28f0>] :sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:36 master kernel:  [net_rx_action+153/448] net_rx_action+0x99/0x1c0
Jan 29 22:13:36 master kernel:  [<ffffffff803f6729>] net_rx_action+0x99/0x1c0
Jan 29 22:13:36 master kernel:  [__do_softirq+105/224] __do_softirq+0x69/0xe0
Jan 29 22:13:36 master kernel:  [<ffffffff80241db9>] __do_softirq+0x69/0xe0
Jan 29 22:13:36 master kernel:  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 29 22:13:36 master kernel:  [<ffffffff8020cd9c>] call_softirq+0x1c/0x30
Jan 29 22:13:36 master kernel:  <EOI>  [do_softirq+53/144] do_softirq+0x35/0x90
Jan 29 22:13:36 master kernel:  <EOI>  [<ffffffff8020eef5>] do_softirq+0x35/0x90
Jan 29 22:13:36 master kernel:  [local_bh_enable+91/160] local_bh_enable+0x5b/0xa0
Jan 29 22:13:36 master kernel:  [<ffffffff80241c6b>] local_bh_enable+0x5b/0xa0
Jan 29 22:13:36 master kernel:  [_end+129726255/2130457992] :sunrpc:svc_udp_recvfrom+0x407/0x430
Jan 29 22:13:36 master kernel:  [<ffffffff881f41a7>] :sunrpc:svc_udp_recvfrom+0x407/0x430
Jan 29 22:13:36 master kernel:  [_end+129730321/2130457992] :sunrpc:svc_recv+0x2a9/0x4e0
Jan 29 22:13:36 master kernel:  [<ffffffff881f5189>] :sunrpc:svc_recv+0x2a9/0x4e0
Jan 29 22:13:36 master kernel:  [default_wake_function+0/16] default_wake_function+0x0/0x10
Jan 29 22:13:36 master kernel:  [<ffffffff80233460>] default_wake_function+0x0/0x10
Jan 29 22:13:36 master kernel:  [__down_read+18/177] __down_read+0x12/0xb1
Jan 29 22:13:36 master kernel:  [<ffffffff80466242>] __down_read+0x12/0xb1
Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
an 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [_end+130772586/2130457992] :nfsd:nfsd+0xe2/0x2e0
Jan 29 22:13:36 master kernel:  [<ffffffff882f38e2>] :nfsd:nfsd+0xe2/0x2e0
Jan 29 22:13:36 master kernel:  [child_rip+10/18] child_rip+0xa/0x12
Jan 29 22:13:36 master kernel:  [<ffffffff8020ca28>] child_rip+0xa/0x12
Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [child_rip+0/18] child_rip+0x0/0x12
Jan 29 22:13:36 master kernel:  [<ffffffff8020ca1e>] child_rip+0x0/0x12
Jan 29 22:13:36 master kernel:
Jan 29 22:13:36 master kernel: Mem-info:
Jan 29 22:13:36 master kernel: DMA per-cpu:
Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:36 master kernel: DMA32 per-cpu:
Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 150   Cold: hi:   62, btch:  15 usd:  62
Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 105   Cold: hi:   62, btch:  15 usd:  52
Jan 29 22:13:36 master kernel: Active:41027 inactive:437314 dirty:29708 writeback:0 unstable:0
Jan 29 22:13:36 master kernel:  free:3892 slab:19405 mapped:11793 pagetables:1512 bounce:0
Jan 29 22:13:36 master kernel: DMA free:8008kB min:28kB low:32kB high:40kB active:0kB inactive:2384kB present:11100kB pages_scanned:0 all_unreclaimable? no
Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 1998 1998 1998
Jan 29 22:13:36 master kernel: DMA32 free:7560kB min:5704kB low:7128kB high:8556kB active:164108kB inactive:1746872kB present:2046800kB pages_scanned:32 all_unreclaimab
le? no
Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 0 0 0
Jan 29 22:13:36 master kernel: DMA: 4*4kB 5*8kB 6*16kB 6*32kB 4*64kB 2*128kB 4*256kB 4*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
Jan 29 22:13:36 master kernel: DMA32: 1365*4kB 11*8kB 38*16kB 18*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 7756kB
Jan 29 22:13:36 master kernel: Swap cache: add 43, delete 43, find 0/0, race 0+0
Jan 29 22:13:36 master kernel: Free swap  = 4088332kB
Jan 29 22:13:36 master kernel: Total swap = 4088500kB
Jan 29 22:13:36 master kernel: Free swap:       4088332kB
Jan 29 22:13:36 master kernel: 523264 pages of RAM
Jan 29 22:13:36 master kernel: 9510 reserved pages
Jan 29 22:13:36 master kernel: 492885 pages shared
Jan 29 22:13:36 master kernel: 0 pages swap cached
Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] nmbd/nmbd_packets.c:process_browse_packet(1061)
Jan 29 22:13:44 master nmbd[3438]:   process_browse_packet: Discarding datagram from IP 192.168.1.240. Source name master<00> is one of our names !
Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] nmbd/nmbd_packets.c:process_browse_packet(1061)
Jan 29 22:13:44 master nmbd[3438]:   process_browse_packet: Discarding datagram from IP 192.168.1.237. Source name master<00> is one of our names !
Jan 29 22:13:44 master kernel: nfsd: page allocation failure. order:2, mode:0x4020
Jan 29 22:13:44 master kernel: Pid: 6591, comm: nfsd Not tainted 2.6.24 #1
Jan 29 22:13:44 master kernel:
Jan 29 22:13:44 master kernel: Call Trace:
Jan 29 22:13:44 master kernel:  <IRQ>  [__alloc_pages+822/912] __alloc_pages+0x336/0x390
Jan 29 22:13:44 master kernel:  <IRQ>  [<ffffffff80279d66>] __alloc_pages+0x336/0x390
Jan 29 22:13:44 master kernel:  [_end+130301422/2130457992] :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:44 master kernel:  [<ffffffff88280866>] :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:44 master kernel:  [_end+130325174/2130457992] :myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:44 master kernel:  [<ffffffff8828652e>] :myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:44 master kernel:  [_spin_lock_bh+9/32] _spin_lock_bh+0x9/0x20
Jan 29 22:13:44 master kernel:  [<ffffffff80466819>] _spin_lock_bh+0x9/0x20
Jan 29 22:13:44 master kernel:  [_end+129719928/2130457992] :sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:44 master kernel:  [<ffffffff881f28f0>] :sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:44 master kernel:  [net_rx_action+153/448] net_rx_action+0x99/0x1c0
Jan 29 22:13:44 master kernel:  [<ffffffff803f6729>] net_rx_action+0x99/0x1c0
Jan 29 22:13:44 master kernel:  [__do_softirq+105/224] __do_softirq+0x69/0xe0
Jan 29 22:13:44 master kernel:  [<ffffffff80241db9>] __do_softirq+0x69/0xe0
Jan 29 22:13:44 master kernel:  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 29 22:13:44 master kernel:  [<ffffffff8020cd9c>] call_softirq+0x1c/0x30
Jan 29 22:13:44 master kernel:  [do_softirq+53/144] do_softirq+0x35/0x90
Jan 29 22:13:44 master kernel:  [<ffffffff8020eef5>] do_softirq+0x35/0x90
Jan 29 22:13:44 master kernel:  [do_IRQ+128/256] do_IRQ+0x80/0x100
Jan 29 22:13:44 master kernel:  [<ffffffff8020f170>] do_IRQ+0x80/0x100
Jan 29 22:13:44 master kernel:  [ret_from_intr+0/10] ret_from_intr+0x0/0xa
Jan 29 22:13:44 master kernel:  [<ffffffff8020c121>] ret_from_intr+0x0/0xa
Jan 29 22:13:44 master kernel:  <EOI>  [__copy_user_nocache+35/336] __copy_user_nocache+0x23/0x150
Jan 29 22:13:44 master kernel:  <EOI>  [<ffffffff803233b3>] __copy_user_nocache+0x23/0x150
Jan 29 22:13:44 master kernel:  [__iovec_copy_from_user_inatomic+67/128] __iovec_copy_from_user_inatomic+0x43/0x80
Jan 29 22:13:44 master kernel:  [<ffffffff80273ba3>] __iovec_copy_from_user_inatomic+0x43/0x80
Jan 29 22:13:44 master kernel:  [iov_iter_copy_from_user_atomic+163/176] iov_iter_copy_from_user_atomic+0xa3/0xb0
Jan 29 22:13:44 master kernel:  [<ffffffff80273d03>] iov_iter_copy_from_user_atomic+0xa3/0xb0
Jan 29 22:13:44 master kernel:  [generic_file_buffered_write+383/1728] generic_file_buffered_write+0x17f/0x6c0
Jan 29 22:13:44 master kernel:  [<ffffffff802756ff>] generic_file_buffered_write+0x17f/0x6c0
Jan 29 22:13:44 master kernel:  [__alloc_skb+133/336] __alloc_skb+0x85/0x150
Jan 29 22:13:44 master kernel:  [<ffffffff803eea15>] __alloc_skb+0x85/0x150
Jan 29 22:13:44 master kernel:  [_end+129525959/2130457992] :xfs:xfs_write+0x67f/0x8f0
Jan 29 22:13:44 master kernel:  [<ffffffff881c333f>] :xfs:xfs_write+0x67f/0x8f0
Jan 29 22:13:44 master kernel:  [_end+129343578/2130457992] :xfs:xfs_iget+0x142/0x160
Jan 29 22:13:44 master kernel:  [<ffffffff88196ad2>] :xfs:xfs_iget+0x142/0x160
Jan 29 22:13:44 master kernel:  [_end+129509416/2130457992] :xfs:xfs_file_aio_write+0x0/0x50
Jan 29 22:13:44 master kernel:  [<ffffffff881bf2a0>] :xfs:xfs_file_aio_write+0x0/0x50
Jan 29 22:13:44 master kernel:  [do_sync_readv_writev+203/272] do_sync_readv_writev+0xcb/0x110
Jan 29 22:13:44 master kernel:  [<ffffffff8029a46b>] do_sync_readv_writev+0xcb/0x110
Jan 29 22:13:44 master kernel:  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 29 22:13:44 master kernel:  [<ffffffff80251e10>] autoremove_wake_function+0x0/0x30
Jan 29 22:13:44 master kernel:  [_end+130243723/2130457992] :exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:44 master kernel:  [<ffffffff88272703>] :exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:44 master kernel:  [<ffffffff88272703>] :exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:44 master kernel:  [_end+130783832/2130457992] :nfsd:nfsd_acceptable+0x0/0x100
Jan 29 22:13:44 master kernel:  [<ffffffff882f64d0>] :nfsd:nfsd_acceptable+0x0/0x100
Jan 29 22:13:44 master kernel:  [_end+129748673/2130457992] :sunrpc:cache_check+0x49/0x490
Jan 29 22:13:44 master kernel:  [<ffffffff881f9939>] :sunrpc:cache_check+0x49/0x490
Jan 29 22:13:44 master kernel:  [rw_copy_check_uvector+157/304] rw_copy_check_uvector+0x9d/0x130
Jan 29 22:13:44 master kernel:  [<ffffffff8029a2cd>] rw_copy_check_uvector+0x9d/0x130
Jan 29 22:13:44 master kernel:  [do_readv_writev+226/528] do_readv_writev+0xe2/0x210
Jan 29 22:13:44 master kernel:  [<ffffffff8029abb2>] do_readv_writev+0xe2/0x210
Jan 29 22:13:44 master kernel:  [_end+130813642/2130457992] :nfsd:nfsd_setuser+0x82/0x210
Jan 29 22:13:44 master kernel:  [<ffffffff882fd942>] :nfsd:nfsd_setuser+0x82/0x210
Jan 29 22:13:44 master kernel:  [_end+130782532/2130457992] :nfsd:nfsd_setuser_and_check_port+0x5c/0x70
Jan 29 22:13:44 master kernel:  [<ffffffff882f5fbc>] :nfsd:nfsd_setuser_and_check_port+0x5c/0x70
Jan 29 22:13:44 master kernel:  [_end+130788665/2130457992] :nfsd:nfsd_vfs_write+0xf1/0x3a0
Jan 29 22:13:44 master kernel:  [<ffffffff882f77b1>] :nfsd:nfsd_vfs_write+0xf1/0x3a0
Jan 29 22:13:44 master kernel:  [__dentry_open+408/528] __dentry_open+0x198/0x210
Jan 29 22:13:44 master kernel:  [<ffffffff80298ab8>] __dentry_open+0x198/0x210
Jan 29 22:13:44 master kernel:  [_end+130790708/2130457992] :nfsd:nfsd_open+0x12c/0x1e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f7fac>] :nfsd:nfsd_open+0x12c/0x1e0
Jan 29 22:13:44 master kernel:  [_end+130791566/2130457992] :nfsd:nfsd_write+0x106/0x110
Jan 29 22:13:44 master kernel:  [<ffffffff882f8306>] :nfsd:nfsd_write+0x106/0x110
Jan 29 22:13:44 master kernel:  [_end+130825459/2130457992] :nfsd:nfsd3_proc_write+0x10b/0x160
Jan 29 22:13:44 master kernel:  [<ffffffff8830076b>] :nfsd:nfsd3_proc_write+0x10b/0x160
Jan 29 22:13:44 master kernel:  [_end+130770937/2130457992] :nfsd:nfsd_dispatch+0xb1/0x220
Jan 29 22:13:44 master kernel:  [<ffffffff882f3271>] :nfsd:nfsd_dispatch+0xb1/0x220
Jan 29 22:13:44 master kernel:  [_end+129716706/2130457992] :sunrpc:svc_process+0x47a/0x780
Jan 29 22:13:44 master kernel:  [<ffffffff881f1c5a>] :sunrpc:svc_process+0x47a/0x780
Jan 29 22:13:44 master kernel:  [__down_read+18/177] __down_read+0x12/0xb1
Jan 29 22:13:44 master kernel:  [<ffffffff80466242>] __down_read+0x12/0xb1
Jan 29 22:13:44 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [_end+130772771/2130457992] :nfsd:nfsd+0x19b/0x2e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f399b>] :nfsd:nfsd+0x19b/0x2e0
Jan 29 22:13:44 master kernel:  [child_rip+10/18] child_rip+0xa/0x12
Jan 29 22:13:44 master kernel:  [<ffffffff8020ca28>] child_rip+0xa/0x12
Jan 29 22:13:44 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [child_rip+0/18] child_rip+0x0/0x12
Jan 29 22:13:44 master kernel:  [<ffffffff8020ca1e>] child_rip+0x0/0x12
Jan 29 22:13:44 master kernel:
Jan 29 22:13:44 master kernel: Mem-info:
Jan 29 22:13:44 master kernel: DMA per-cpu:
Jan 29 22:13:44 master kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:44 master kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:44 master kernel: DMA32 per-cpu:
Jan 29 22:13:44 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd:  55   Cold: hi:   62, btch:  15 usd:  60
Jan 29 22:13:44 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 183   Cold: hi:   62, btch:  15 usd:  48
Jan 29 22:13:44 master kernel: Active:40766 inactive:437591 dirty:29984 writeback:1045 unstable:0
Jan 29 22:13:44 master kernel:  free:4000 slab:18952 mapped:11793 pagetables:1512 bounce:0
Jan 29 22:13:44 master kernel: DMA free:8008kB min:28kB low:32kB high:40kB active:0kB inactive:1676kB present:11100kB pages_scanned:0 all_unreclaimable? no
Jan 29 22:13:44 master kernel: lowmem_reserve[]: 0 1998 1998 1998
Jan 29 22:13:44 master kernel: DMA32 free:8076kB min:5704kB low:7128kB high:8556kB active:163064kB inactive:1748688kB present:2046800kB pages_scanned:0 all_unreclaimabl
e? no
Jan 29 22:13:44 master kernel: lowmem_reserve[]: 0 0 0 0
Jan 29 22:13:44 master kernel: DMA: 6*4kB 6*8kB 13*16kB 6*32kB 4*64kB 1*128kB 12*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
Jan 29 22:13:44 master kernel: DMA32: 1160*4kB 117*8kB 82*16kB 19*32kB 2*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 8136kB
Jan 29 22:13:44 master kernel: Swap cache: add 45, delete 44, find 0/0, race 0+0
Jan 29 22:13:44 master kernel: Free swap  = 4088324kB
Jan 29 22:13:44 master kernel: Total swap = 4088500kB
Jan 29 22:13:44 master kernel: Free swap:       4088324kB
Jan 29 22:13:44 master kernel: 523264 pages of RAM
Jan 29 22:13:44 master kernel: 9510 reserved pages
Jan 29 22:13:44 master kernel: 494203 pages shared
Jan 29 22:13:44 master kernel: 1 pages swap cached
Jan 29 22:13:45 master kernel: nfsd: page allocation failure. order:2, mode:0x4020
Jan 29 22:13:45 master kernel: Pid: 6595, comm: nfsd Not tainted 2.6.24 #1
Jan 29 22:13:45 master kernel:
Jan 29 22:13:45 master kernel: Call Trace:
Jan 29 22:13:45 master kernel:  <IRQ>  [__alloc_pages+822/912] __alloc_pages+0x336/0x390
Jan 29 22:13:45 master kernel:  <IRQ>  [<ffffffff80279d66>] __alloc_pages+0x336/0x390
Jan 29 22:13:45 master kernel:  [ip_local_deliver_finish+129/512] ip_local_deliver_finish+0x81/0x200
Jan 29 22:13:45 master kernel:  [<ffffffff80417611>] ip_local_deliver_finish+0x81/0x200
Jan 29 22:13:45 master kernel:  [_end+130301422/2130457992] :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:45 master kernel:  [<ffffffff88280866>] :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:45 master kernel:  [_end+130325174/2130457992] :myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:45 master kernel:  [<ffffffff8828652e>] :myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:45 master kernel:  [_spin_lock_bh+9/32] _spin_lock_bh+0x9/0x20
Jan 29 22:13:45 master kernel:  [<ffffffff80466819>] _spin_lock_bh+0x9/0x20
Jan 29 22:13:45 master kernel:  [_end+129719928/2130457992] :sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:45 master kernel:  [<ffffffff881f28f0>] :sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:45 master kernel:  [net_rx_action+153/448] net_rx_action+0x99/0x1c0
Jan 29 22:13:45 master kernel:  [<ffffffff803f6729>] net_rx_action+0x99/0x1c0
Jan 29 22:13:45 master kernel:  [__do_softirq+105/224] __do_softirq+0x69/0xe0
Jan 29 22:13:45 master kernel:  [<ffffffff80241db9>] __do_softirq+0x69/0xe0
Jan 29 22:13:45 master kernel:  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 29 22:13:45 master kernel:  [<ffffffff8020cd9c>] call_softirq+0x1c/0x30
Jan 29 22:13:45 master kernel:  [do_softirq+53/144] do_softirq+0x35/0x90
Jan 29 22:13:45 master kernel:  [<ffffffff8020eef5>] do_softirq+0x35/0x90
Jan 29 22:13:45 master kernel:  [do_IRQ+128/256] do_IRQ+0x80/0x100
Jan 29 22:13:45 master kernel:  [<ffffffff8020f170>] do_IRQ+0x80/0x100
Jan 29 22:13:45 master kernel:  [ret_from_intr+0/10] ret_from_intr+0x0/0xa
Jan 29 22:13:45 master kernel:  [<ffffffff8020c121>] ret_from_intr+0x0/0xa
Jan 29 22:13:45 master kernel:  <EOI>  [block_write_begin+21/224] block_write_begin+0x15/0xe0
Jan 29 22:13:45 master kernel:  <EOI>  [<ffffffff802c03b5>] block_write_begin+0x15/0xe0
Jan 29 22:13:45 master kernel:  [_end+129490730/2130457992] :xfs:xfs_vm_write_begin+0x22/0x30
Jan 29 22:13:45 master kernel:  [<ffffffff881ba9a2>] :xfs:xfs_vm_write_begin+0x22/0x30
Jan 29 22:13:45 master kernel:  [_end+129492696/2130457992] :xfs:xfs_get_blocks+0x0/0x10
Jan 29 22:13:45 master kernel:  [<ffffffff881bb150>] :xfs:xfs_get_blocks+0x0/0x10
Jan 29 22:13:45 master kernel:  [generic_file_buffered_write+328/1728] generic_file_buffered_write+0x148/0x6c0
Jan 29 22:13:45 master kernel:  [<ffffffff802756c8>] generic_file_buffered_write+0x148/0x6c0
Jan 29 22:13:45 master kernel:  [__alloc_skb+133/336] __alloc_skb+0x85/0x150
Jan 29 22:13:45 master kernel:  [<ffffffff803eea15>] __alloc_skb+0x85/0x150
Jan 29 22:13:45 master kernel:  [_end+129525959/2130457992] :xfs:xfs_write+0x67f/0x8f0
Jan 29 22:13:45 master kernel:  [<ffffffff881c333f>] :xfs:xfs_write+0x67f/0x8f0
Jan 29 22:13:45 master kernel:  [_end+129343578/2130457992] :xfs:xfs_iget+0x142/0x160
Jan 29 22:13:45 master kernel:  [<ffffffff88196ad2>] :xfs:xfs_iget+0x142/0x160
Jan 29 22:13:45 master kernel:  [_end+129509416/2130457992] :xfs:xfs_file_aio_write+0x0/0x50
Jan 29 22:13:45 master kernel:  [<ffffffff881bf2a0>] :xfs:xfs_file_aio_write+0x0/0x50
Jan 29 22:13:45 master kernel:  [do_sync_readv_writev+203/272] do_sync_readv_writev+0xcb/0x110
Jan 29 22:13:45 master kernel:  [<ffffffff8029a46b>] do_sync_readv_writev+0xcb/0x110
Jan 29 22:13:45 master kernel:  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 29 22:13:45 master kernel:  [<ffffffff80251e10>] autoremove_wake_function+0x0/0x30
Jan 29 22:13:45 master kernel:  [_end+130243723/2130457992] :exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:45 master kernel:  [<ffffffff88272703>] :exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:45 master kernel:  [_end+130783832/2130457992] :nfsd:nfsd_acceptable+0x0/0x100
Jan 29 22:13:45 master kernel:  [<ffffffff882f64d0>] :nfsd:nfsd_acceptable+0x0/0x100
Jan 29 22:13:45 master kernel:  [_end+129748673/2130457992] :sunrpc:cache_check+0x49/0x490
Jan 29 22:13:45 master kernel:  [<ffffffff881f9939>] :sunrpc:cache_check+0x49/0x490
Jan 29 22:13:45 master kernel:  [rw_copy_check_uvector+157/304] rw_copy_check_uvector+0x9d/0x130
Jan 29 22:13:45 master kernel:  [<ffffffff8029a2cd>] rw_copy_check_uvector+0x9d/0x130
Jan 29 22:13:45 master kernel:  [do_readv_writev+226/528] do_readv_writev+0xe2/0x210
Jan 29 22:13:45 master kernel:  [<ffffffff8029abb2>] do_readv_writev+0xe2/0x210
Jan 29 22:13:45 master kernel:  [_end+130813642/2130457992] :nfsd:nfsd_setuser+0x82/0x210
Jan 29 22:13:45 master kernel:  [<ffffffff882fd942>] :nfsd:nfsd_setuser+0x82/0x210
Jan 29 22:13:45 master kernel:  [_end+130782532/2130457992] :nfsd:nfsd_setuser_and_check_port+0x5c/0x70
Jan 29 22:13:45 master kernel:  [<ffffffff882f5fbc>] :nfsd:nfsd_setuser_and_check_port+0x5c/0x70
Jan 29 22:13:45 master kernel:  [_end+130788665/2130457992] :nfsd:nfsd_vfs_write+0xf1/0x3a0
Jan 29 22:13:45 master kernel:  [<ffffffff882f77b1>] :nfsd:nfsd_vfs_write+0xf1/0x3a0
Jan 29 22:13:45 master kernel:  [__dentry_open+408/528] __dentry_open+0x198/0x210
Jan 29 22:13:45 master kernel:  [<ffffffff80298ab8>] __dentry_open+0x198/0x210
Jan 29 22:13:45 master kernel:  [_end+130790708/2130457992] :nfsd:nfsd_open+0x12c/0x1e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f7fac>] :nfsd:nfsd_open+0x12c/0x1e0
Jan 29 22:13:45 master kernel:  [_end+130791566/2130457992] :nfsd:nfsd_write+0x106/0x110
Jan 29 22:13:45 master kernel:  [<ffffffff882f8306>] :nfsd:nfsd_write+0x106/0x110
Jan 29 22:13:45 master kernel:  [_end+130825459/2130457992] :nfsd:nfsd3_proc_write+0x10b/0x160
Jan 29 22:13:45 master kernel:  [<ffffffff8830076b>] :nfsd:nfsd3_proc_write+0x10b/0x160
Jan 29 22:13:45 master kernel:  [_end+130770937/2130457992] :nfsd:nfsd_dispatch+0xb1/0x220
Jan 29 22:13:45 master kernel:  [<ffffffff882f3271>] :nfsd:nfsd_dispatch+0xb1/0x220
Jan 29 22:13:45 master kernel:  [_end+129716706/2130457992] :sunrpc:svc_process+0x47a/0x780
Jan 29 22:13:45 master kernel:  [<ffffffff881f1c5a>] :sunrpc:svc_process+0x47a/0x780
Jan 29 22:13:45 master kernel:  [__down_read+18/177] __down_read+0x12/0xb1
Jan 29 22:13:45 master kernel:  [<ffffffff80466242>] __down_read+0x12/0xb1
Jan 29 22:13:45 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [_end+130772771/2130457992] :nfsd:nfsd+0x19b/0x2e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f399b>] :nfsd:nfsd+0x19b/0x2e0
Jan 29 22:13:45 master kernel:  [child_rip+10/18] child_rip+0xa/0x12
Jan 29 22:13:45 master kernel:  [<ffffffff8020ca28>] child_rip+0xa/0x12
Jan 29 22:13:45 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [child_rip+0/18] child_rip+0x0/0x12
Jan 29 22:13:45 master kernel:  [<ffffffff8020ca1e>] child_rip+0x0/0x12
Jan 29 22:13:45 master kernel:
Jan 29 22:13:45 master kernel: Mem-info:
Jan 29 22:13:45 master kernel: DMA per-cpu:
Jan 29 22:13:45 master kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:45 master kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:45 master kernel: DMA32 per-cpu:
Jan 29 22:13:45 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 156   Cold: hi:   62, btch:  15 usd:  48
Jan 29 22:13:45 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd:  73   Cold: hi:   62, btch:  15 usd:  56
Jan 29 22:13:45 master kernel: Active:40772 inactive:437908 dirty:24942 writeback:1051 unstable:0
Jan 29 22:13:45 master kernel:  free:3819 slab:18993 mapped:11793 pagetables:1512 bounce:0
Jan 29 22:13:45 master kernel: DMA free:8008kB min:28kB low:32kB high:40kB active:0kB inactive:400kB present:11100kB pages_scanned:0 all_unreclaimable? no
Jan 29 22:13:45 master kernel: lowmem_reserve[]: 0 1998 1998 1998
Jan 29 22:13:45 master kernel: DMA32 free:7268kB min:5704kB low:7128kB high:8556kB active:163088kB inactive:1751232kB present:2046800kB pages_scanned:0 all_unreclaimabl
e? no
Jan 29 22:13:45 master kernel: lowmem_reserve[]: 0 0 0 0
Jan 29 22:13:45 master kernel: DMA: 10*4kB 8*8kB 17*16kB 13*32kB 7*64kB 5*128kB 8*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
Jan 29 22:13:45 master kernel: DMA32: 1555*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 7228kB
Jan 29 22:13:45 master kernel: Swap cache: add 45, delete 44, find 0/0, race 0+0
Jan 29 22:13:45 master kernel: Free swap  = 4088324kB
Jan 29 22:13:45 master kernel: Total swap = 4088500kB
Jan 29 22:13:45 master kernel: Free swap:       4088324kB
Jan 29 22:13:45 master kernel: 523264 pages of RAM
author
#1 lore raw

The cause of this problem seems to be compiling the Myricom driver with the ALLOC_ORDER=2 option. When I use the in-kernel driver, (1.3.2) or recompile the Myricom 1.4.0 driver WITHOUT the option, the problem seems to go away even after heavy hammering of the system.

The ALLOC_ORDER=2 compiling option doesn't seem to cause any problem for the Myricom 1.4.0 driver in the 2.6.22 kernel but it does cause the problem when I run it in 2.6.24.

Maybe with 2.6.24 more RAM is required to use this option? I will contact Myricom to inform them.

Andrew

Quoted 88 lines · AndrewL733 wrote:

My setup is Mandriva 2007 64-bit:

IntelS5000PSLSATA motherboard
Single Dual Core 3 Ghz Xeon
2 GB ECC RAM
Myricom 10 GbE Network Adapter (using out of kernel 1.4.0 driver compiled with "MYRI10GE_ALLOC_ORDER=2")

I have two identical systems doing data transfer via Myricom cards connected directly to one another (in other words, not going through a switch).

No problems hammering Myricom with 350 MB/sec NFS traffic while running 2.6.20.15 and 2.6.22.9 (kernel.org "plain vanilla" kernels I compiled and run, also with updated Myricom drivers). However, I just compiled and installed 2.6.24 and get the following errors as soon as I begin doing NFS I/O over the Myricom card. Maybe I missed something in the kernel configuration? Throughput when running 2.6.24 is about 40 percent lower than when running 2.6.22.9-- surely due to these errors.

Jan 29 22:13:36 master kernel: nfsd: page allocation failure. order:2, mode:0x4020
Jan 29 22:13:36 master kernel: Pid: 6586, comm: nfsd Not tainted 2.6.24 #1
Jan 29 22:13:36 master kernel:
Jan 29 22:13:36 master kernel: Call Trace:
Jan 29 22:13:36 master kernel:  <IRQ>  [__alloc_pages+822/912] __alloc_pages+0x336/0x390
Jan 29 22:13:36 master kernel:  <IRQ>  [<ffffffff80279d66>] __alloc_pages+0x336/0x390
Jan 29 22:13:36 master kernel:  [ip_local_deliver+37/112] ip_local_deliver+0x25/0x70
Jan 29 22:13:36 master kernel:  [<ffffffff80417bb5>] ip_local_deliver+0x25/0x70
Jan 29 22:13:36 master kernel:  [_end+130301422/2130457992] :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:36 master kernel:  [<ffffffff88280866>] :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:36 master kernel:  [_end+130325174/2130457992] :myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:36 master kernel:  [<ffffffff8828652e>] :myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:36 master kernel:  [_spin_lock_bh+9/32] _spin_lock_bh+0x9/0x20
Jan 29 22:13:36 master kernel:  [<ffffffff80466819>] _spin_lock_bh+0x9/0x20
Jan 29 22:13:36 master kernel:  [_end+129719928/2130457992] :sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:36 master kernel:  [<ffffffff881f28f0>] :sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:36 master kernel:  [net_rx_action+153/448] net_rx_action+0x99/0x1c0
Jan 29 22:13:36 master kernel:  [<ffffffff803f6729>] net_rx_action+0x99/0x1c0
Jan 29 22:13:36 master kernel:  [__do_softirq+105/224] __do_softirq+0x69/0xe0
Jan 29 22:13:36 master kernel:  [<ffffffff80241db9>] __do_softirq+0x69/0xe0
Jan 29 22:13:36 master kernel:  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 29 22:13:36 master kernel:  [<ffffffff8020cd9c>] call_softirq+0x1c/0x30
Jan 29 22:13:36 master kernel:  <EOI>  [do_softirq+53/144] do_softirq+0x35/0x90
Jan 29 22:13:36 master kernel:  <EOI>  [<ffffffff8020eef5>] do_softirq+0x35/0x90
Jan 29 22:13:36 master kernel:  [local_bh_enable+91/160] local_bh_enable+0x5b/0xa0
Jan 29 22:13:36 master kernel:  [<ffffffff80241c6b>] local_bh_enable+0x5b/0xa0
Jan 29 22:13:36 master kernel:  [_end+129726255/2130457992] :sunrpc:svc_udp_recvfrom+0x407/0x430
Jan 29 22:13:36 master kernel:  [<ffffffff881f41a7>] :sunrpc:svc_udp_recvfrom+0x407/0x430
Jan 29 22:13:36 master kernel:  [_end+129730321/2130457992] :sunrpc:svc_recv+0x2a9/0x4e0
Jan 29 22:13:36 master kernel:  [<ffffffff881f5189>] :sunrpc:svc_recv+0x2a9/0x4e0
Jan 29 22:13:36 master kernel:  [default_wake_function+0/16] default_wake_function+0x0/0x10
Jan 29 22:13:36 master kernel:  [<ffffffff80233460>] default_wake_function+0x0/0x10
Jan 29 22:13:36 master kernel:  [__down_read+18/177] __down_read+0x12/0xb1
Jan 29 22:13:36 master kernel:  [<ffffffff80466242>] __down_read+0x12/0xb1
Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
an 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [_end+130772586/2130457992] :nfsd:nfsd+0xe2/0x2e0
Jan 29 22:13:36 master kernel:  [<ffffffff882f38e2>] :nfsd:nfsd+0xe2/0x2e0
Jan 29 22:13:36 master kernel:  [child_rip+10/18] child_rip+0xa/0x12
Jan 29 22:13:36 master kernel:  [<ffffffff8020ca28>] child_rip+0xa/0x12
Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [child_rip+0/18] child_rip+0x0/0x12
Jan 29 22:13:36 master kernel:  [<ffffffff8020ca1e>] child_rip+0x0/0x12
Jan 29 22:13:36 master kernel:
Jan 29 22:13:36 master kernel: Mem-info:
Jan 29 22:13:36 master kernel: DMA per-cpu:
Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:36 master kernel: DMA32 per-cpu:
Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 150   Cold: hi:   62, btch:  15 usd:  62
Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 105   Cold: hi:   62, btch:  15 usd:  52
Jan 29 22:13:36 master kernel: Active:41027 inactive:437314 dirty:29708 writeback:0 unstable:0
Jan 29 22:13:36 master kernel:  free:3892 slab:19405 mapped:11793 pagetables:1512 bounce:0
Jan 29 22:13:36 master kernel: DMA free:8008kB min:28kB low:32kB high:40kB active:0kB inactive:2384kB present:11100kB pages_scanned:0 all_unreclaimable? no
Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 1998 1998 1998
Jan 29 22:13:36 master kernel: DMA32 free:7560kB min:5704kB low:7128kB high:8556kB active:164108kB inactive:1746872kB present:2046800kB pages_scanned:32 all_unreclaimab
le? no
Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 0 0 0
Jan 29 22:13:36 master kernel: DMA: 4*4kB 5*8kB 6*16kB 6*32kB 4*64kB 2*128kB 4*256kB 4*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
Jan 29 22:13:36 master kernel: DMA32: 1365*4kB 11*8kB 38*16kB 18*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 7756kB
Jan 29 22:13:36 master kernel: Swap cache: add 43, delete 43, find 0/0, race 0+0
Jan 29 22:13:36 master kernel: Free swap  = 4088332kB
Jan 29 22:13:36 master kernel: Total swap = 4088500kB
Jan 29 22:13:36 master kernel: Free swap:       4088332kB
Jan 29 22:13:36 master kernel: 523264 pages of RAM
Jan 29 22:13:36 master kernel: 9510 reserved pages
Jan 29 22:13:36 master kernel: 492885 pages shared
Jan 29 22:13:36 master kernel: 0 pages swap cached
Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] nmbd/nmbd_packets.c:process_browse_packet(1061)
Jan 29 22:13:44 master nmbd[3438]:   process_browse_packet: Discarding datagram from IP 192.168.1.240. Source name master<00> is one of our names !
Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] nmbd/nmbd_packets.c:process_browse_packet(1061)
Jan 29 22:13:44 master nmbd[3438]:   process_browse_packet: Discarding datagram from IP 192.168.1.237. Source name master<00> is one of our names !
#2 lore raw
In reply to AndrewL733

Hi Andrew

On (31/01/08 10:49), AndrewL733 didst pronounce:

Quoted 13 lines

My setup is Mandriva 2007 64-bit:

IntelS5000PSLSATA motherboard
Single Dual Core 3 Ghz Xeon
2 GB ECC RAM
Myricom 10 GbE Network Adapter (using out of kernel 1.4.0 driver compiled with "MYRI10GE_ALLOC_ORDER=2")

I have two identical systems doing data transfer via Myricom cards connected directly to one another (in other words, not going through a switch).

Quoted 8 lines

No problems hammering Myricom with 350 MB/sec NFS traffic while running 2.6.20.15 and 2.6.22.9 (kernel.org "plain vanilla" kernels I compiled and run, also with updated Myricom drivers). However, I just compiled and installed 2.6.24 and get the following errors as soon as I begin doing NFS I/O over the Myricom card. Maybe I missed something in the kernel configuration? Throughput when running 2.6.24 is about 40 percent lower than when running 2.6.22.9-- surely due to these errors.

It's possible. Is there a stream of these errors coming through constantly?

Quoted 1 line
Jan 29 22:13:36 master kernel: nfsd: page allocation failure. order:2, mode:0x4020

So, that is a ZONE_NORMAL high-order atomic allocation (GFP_COMPOUND as well which is a little surprise but should make no difference). Not exactly recommended but there are a few drivers that do it and I guess yours is one.

Quoted 11 lines
Jan 29 22:13:36 master kernel: Pid: 6586, comm: nfsd Not tainted 2.6.24 #1
Jan 29 22:13:36 master kernel:
Jan 29 22:13:36 master kernel: Call Trace:
Jan 29 22:13:36 master kernel:  <IRQ>  [__alloc_pages+822/912] __alloc_pages+0x336/0x390
Jan 29 22:13:36 master kernel:  <IRQ>  [<ffffffff80279d66>]    __alloc_pages+0x336/0x390
Jan 29 22:13:36 master kernel:  [ip_local_deliver+37/112]      ip_local_deliver+0x25/0x70
Jan 29 22:13:36 master kernel:  [<ffffffff80417bb5>]           ip_local_deliver+0x25/0x70
Jan 29 22:13:36 master kernel:  [_end+130301422/2130457992]    :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:36 master kernel:  [<ffffffff88280866>]           :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:36 master kernel:  [_end+130325174/2130457992]    :myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:36 master kernel:  [<ffffffff8828652e>]           :myri10ge:myri10ge_poll+0x57e/0xae0

Coming from the driver in question so no suprise there

Quoted 14 lines

<SNIP>

Jan 29 22:13:36 master kernel: Mem-info:
Jan 29 22:13:36 master kernel: DMA per-cpu:
Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:36 master kernel: DMA32 per-cpu:
Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 150   Cold: hi:   62, btch:  15 usd:  62
Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 105   Cold: hi:   62, btch:  15 usd:  52
Jan 29 22:13:36 master kernel: Active:41027 inactive:437314 dirty:29708 writeback:0 unstable:0
Jan 29 22:13:36 master kernel:  free:3892 slab:19405 mapped:11793 pagetables:1512 bounce:0
Jan 29 22:13:36 master kernel: DMA free:8008kB min:28kB low:32kB high:40kB active:0kB inactive:2384kB present:11100kB pages_scanned:0 all_unreclaimable? no
Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 1998 1998 1998
Jan 29 22:13:36 master kernel: DMA32 free:7560kB min:5704kB low:7128kB high:8556kB active:164108kB inactive:1746872kB present:2046800kB pages_scanned:32 all_unreclaimable? no
Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 0 0 0

No apparent memory pressure there. We are not even below the low watermark for order-0 allocations and the lowmem_reserve is not a factor for DMA32 allocations.

Quoted 2 lines
Jan 29 22:13:36 master kernel: DMA: 4*4kB 5*8kB 6*16kB 6*32kB 4*64kB 2*128kB 4*256kB 4*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
Jan 29 22:13:36 master kernel: DMA32: 1365*4kB 11*8kB 38*16kB 18*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 7756kB

There are sufficient numbers of order-2 and higher pages free there. It is similar for the subsequent failures. This does not appear to be a fragmentation problem at least so lets see what watermarks look like just for zone->low_pages which should be relatively relaxed.

Zone watermarks in pages;

Low:                    1782
Free:                   1890
classzone_reserve:         0

Adjusted Low calculation

        low = 1782 - 4 + 1 == 1779
                ALLOC_HIGH for __GFP_HIGH so
        low -= (low / 2) == 890
                ALLOC_HARDER for !__GFP_WAIT so
        low -= (low / 4) == 668

Free pages at each order
DMA32: 1365*4kB 11*8kB 38*16kB 18*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 7756kB Watermark check

        order 0: is free_pages(1890) < low(668) + class_rsrv(0) -- no, ok
        order 1: is free_pages(525)  < low(334)                 -- no, ok
        order 2: is free_pages(503)  < low(167)                 -- no. ok

Err, so it looks like we are not hitting the watermarks either. This does not make sense, the allocation should not have failed. I have to be making a mistake with the zone watermark calculation, can anyone spot it?

Is this definetly 2.6.24 and otherwise unpatched?

Quoted 10 lines
Jan 29 22:13:36 master kernel: Swap cache: add 43, delete 43, find 0/0, race 0+0
Jan 29 22:13:36 master kernel: Free swap  = 4088332kB
Jan 29 22:13:36 master kernel: Total swap = 4088500kB
Jan 29 22:13:36 master kernel: Free swap:       4088332kB
Jan 29 22:13:36 master kernel: 523264 pages of RAM
Jan 29 22:13:36 master kernel: 9510 reserved pages
Jan 29 22:13:36 master kernel: 492885 pages shared
Jan 29 22:13:36 master kernel: 0 pages swap cached
Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] 

<SNIP>

Few questions to try pinning down what is going wrong.

1. Is the stream of errors constant once they start?
2. If you increase /proc/sys/vm/min_free_kbytes to 16384 or 64738, do the allocation errors still occur? If the errors no longer occur, is the performance restored?
3. If you compile with MYRI10GE_ALLOC_ORDER=0 on the old and new kernels, do the allocations errors occur? If not, how does the throughput compare between kernel versions?
(I am checking for changes in NFS/IO/writeback that causes a writeout performance regression. Such a slowdown may cause memory pressure later and then these allocation errors when watermarks are hit)
4. If you apply the patch below, do the allocation errors still occur? (The patch disables fragmentation avoidance. I don't think fragmentation is the problem here because ample memory is free and watermarks look ok)
5. Any chance you could test 2.6.23 in case the problem also happens there?

Thanks

include/linux/mmzone.h +2 -5
diff -rup -X /usr/src/patchset-0.6/bin//dontdiff linux-2.6.24-clean/include/linux/mmzone.h linux-2.6.24-disable-antifrag/include/linux/mmzone.h
@@ -45,14 +45,11 @@
 	for (order = 0; order < MAX_ORDER; order++) \
 		for (type = 0; type < MIGRATE_TYPES; type++)
 
-extern int page_group_by_mobility_disabled;
+#define page_group_by_mobility_disabled (0)
 
 static inline int get_pageblock_migratetype(struct page *page)
 {
-	if (unlikely(page_group_by_mobility_disabled))
-		return MIGRATE_UNMOVABLE;
-
-	return get_pageblock_flags_group(page, PB_migrate, PB_migrate_end);
+	return MIGRATE_UNMOVABLE;
 }
 
 struct free_area {
mm/page_alloc.c +1 -15
diff -rup -X /usr/src/patchset-0.6/bin//dontdiff linux-2.6.24-clean/mm/page_alloc.c linux-2.6.24-disable-antifrag/mm/page_alloc.c
@@ -164,12 +164,9 @@ int nr_node_ids __read_mostly = MAX_NUMN
 EXPORT_SYMBOL(nr_node_ids);
 #endif
 
-int page_group_by_mobility_disabled __read_mostly;
-
 static void set_pageblock_migratetype(struct page *page, int migratetype)
 {
-	set_pageblock_flags_group(page, (unsigned long)migratetype,
-					PB_migrate, PB_migrate_end);
+	return;
 }
 
 #ifdef CONFIG_DEBUG_VM
@@ -2354,17 +2351,6 @@ void build_all_zonelists(void)
 		/* cpuset refresh routine should be here */
 	}
 	vm_total_pages = nr_free_pagecache_pages();
-	/*
-	 * Disable grouping by mobility if the number of pages in the
-	 * system is too low to allow the mechanism to work. It would be
-	 * more accurate, but expensive to check per-zone. This check is
-	 * made on memory-hotadd so a system can start with mobility
-	 * disabled and enable it later
-	 */
-	if (vm_total_pages < (pageblock_nr_pages * MIGRATE_TYPES))
-		page_group_by_mobility_disabled = 1;
-	else
-		page_group_by_mobility_disabled = 0;
 
 	printk("Built %i zonelists in %s order, mobility grouping %s.  "
 		"Total pages: %ld\n",
Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab