TCP connection hang problem with 2.6.16.16, e1000.
Subsystems (4)
Kernel is 2.6.16.16 with my patches, including a patch to the e1000. I also rebuilt a fresh kernel with only the attached send-to-self patch. I see the hang, but there was no OOM messages, probably because the machine was freshly rebooted and had plenty of buffers available. I will try to reproduce this with some other NICs next...
I am running 2 TCP connections between two ports of a pro/1000 NIC with fiber interfaces. Both connections are trying to send 200Mbps in both directions. Each write to the socket is 5000 bytes. MTU is 1500.
After maybe 10 seconds, I see out-of-memory errors on the console and then the connections hang and send no more traffic. However, the NICs are still showing ~215kpps. According to ethereal, these are duplicate acks.
0.928361 172.1.5.169 -> 172.1.5.168 TCP [TCP Dup ACK 4#1389] 33020 > 33019
[ACK] Seq=4294967208 Ack=32 Win=23 Len=0 TSV=1278643133 TSER=1278356385
0.928618 172.1.5.169 -> 172.1.5.168 TCP [TCP Dup ACK 2#1300] 33018 > 33017
[ACK] Seq=0 Ack=0 Win=538 Len=0 TSV=1278643133 TSER=1278338974
0.928858 172.1.5.169 -> 172.1.5.168 TCP [TCP Dup ACK 4#1390] 33020 > 33019
[ACK] Seq=4294967208 Ack=32 Win=23 Len=0 TSV=1278643134 TSER=1278356385
0.929190 172.1.5.169 -> 172.1.5.168 TCP [TCP Dup ACK 2#1301] 33018 > 33017
[ACK] Seq=0 Ack=0 Win=538 Len=0 TSV=1278643134 TSER=1278338974
0.929461 172.1.5.169 -> 172.1.5.168 TCP [TCP Dup ACK 2#1302] 33018 > 33017
[ACK] Seq=0 Ack=0 Win=538 Len=0 TSV=1278643134 TSER=1278338974
0.929716 172.1.5.169 -> 172.1.5.168 TCP [TCP Dup ACK 4#1391] 33020 > 33019
[ACK] Seq=4294967208 Ack=32 Win=23 Len=0 TSV=1278643134 TSER=1278356385
107215 packets dropped
Stopping the TCP connection does not appear to immediately fix the problem:
The connections are in the FIN_WAIT1 state and the dup acks continue to flow.
ifdown/ifup after stopping the connections seems to clear the problem.
I saw no more OOM messages with TOE disabled, but the connection still hangs.
The OOM related console messages are attached.
The problem also happens when no OOM messages are printed, but it runs slightly longer (maybe 15-20 seconds).
I suspect that a dropped packet or packets is somehow the problem.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
May 28 04:02:06 xeon-dt syslogd 1.4.1: restart. May 30 11:54:40 xeon-dt kernel: e1000: eth4: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex May 30 11:54:43 xeon-dt kernel: e1000: eth3: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex May 30 12:33:54 xeon-dt kernel: e1000: eth4: e1000_watchdog_task: NIC Link is Down May 30 12:33:54 xeon-dt kernel: e1000: eth3: e1000_watchdog_task: NIC Link is Down May 30 12:33:58 xeon-dt kernel: e1000: eth3: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex May 30 12:33:58 xeon-dt kernel: e1000: eth4: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex May 30 12:52:57 xeon-dt kernel: e1000: eth4: e1000_watchdog_task: NIC Link is Down May 30 12:52:57 xeon-dt kernel: e1000: eth3: e1000_watchdog_task: NIC Link is Down May 31 08:07:00 xeon-dt kernel: irqbalance: page allocation failure. order:1, mode:0x20 May 31 08:07:00 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:07:00 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:07:00 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:07:00 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:07:00 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:07:00 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:07:00 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:07:00 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:07:00 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:07:00 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:07:00 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:07:01 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:07:01 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:07:01 xeon-dt kernel: [<78141615>] handle_IRQ_event+0x26/0x59 May 31 08:07:01 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:07:02 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:07:02 xeon-dt kernel: [<f898c6c2>] e1000_alloc_rx_buffers+0x22a/0x418 [e1000] May 31 08:07:02 xeon-dt kernel: [<782ee9bc>] tcp_v4_rcv+0x6ff/0x9b0 May 31 08:07:02 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:07:02 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:07:02 xeon-dt kernel: [<78125460>] getnstimeofday+0x10/0x2a May 31 08:07:02 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:07:02 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:07:03 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:07:03 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:07:03 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:07:04 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:07:04 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:07:04 xeon-dt kernel: [<781d007b>] as_get_io_context+0x15/0x33 May 31 08:07:04 xeon-dt kernel: [<7831adfb>] _spin_lock+0x4e/0x8e May 31 08:07:04 xeon-dt kernel: [<7817808f>] d_rehash+0x4b/0x7a May 31 08:07:05 xeon-dt kernel: [<78196900>] proc_lookup+0x83/0xb9 May 31 08:07:05 xeon-dt kernel: [<78193bea>] proc_root_lookup+0x19/0x47 May 31 08:07:05 xeon-dt kernel: [<7816db17>] real_lookup+0xb5/0xd1 May 31 08:07:05 xeon-dt kernel: [<7816dd93>] do_lookup+0x85/0x90 May 31 08:07:05 xeon-dt kernel: [<7816de86>] __link_path_walk+0xe8/0xc6f May 31 08:07:05 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:07:05 xeon-dt kernel: [<f898b750>] e1000_clean_tx_irq+0x123/0x32e [e1000] May 31 08:07:05 xeon-dt kernel: [<f898bba4>] e1000_clean_rx_irq+0x249/0x5c6 [e1000] May 31 08:07:05 xeon-dt kernel: [<7816ea54>] link_path_walk+0x47/0xc4 May 31 08:07:05 xeon-dt kernel: [<78155eb6>] free_pages_and_swap_cache+0x54/0x75 May 31 08:07:05 xeon-dt kernel: [<7816135b>] get_empty_filp+0x77/0x15d May 31 08:07:05 xeon-dt kernel: [<7816edf5>] do_path_lookup+0x124/0x26f May 31 08:07:05 xeon-dt kernel: [<7816efa8>] __path_lookup_intent_open+0x50/0x90 May 31 08:07:05 xeon-dt kernel: [<7816f84d>] open_namei+0xcb/0x631 May 31 08:07:05 xeon-dt kernel: [<7814e658>] unmap_vmas+0xff/0x245 May 31 08:07:05 xeon-dt kernel: [<7815f853>] do_filp_open+0x2e/0x4d May 31 08:07:06 xeon-dt kernel: [<78155eb6>] free_pages_and_swap_cache+0x54/0x75 May 31 08:07:06 xeon-dt kernel: [<7815fb65>] do_sys_open+0x3e/0xb2 May 31 08:07:06 xeon-dt kernel: [<7815fbf5>] sys_open+0x1c/0x20 May 31 08:07:06 xeon-dt kernel: [<78102c93>] sysenter_past_esp+0x54/0x75 May 31 08:07:06 xeon-dt kernel: Mem-info: May 31 08:07:06 xeon-dt kernel: DMA per-cpu: May 31 08:07:06 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:07:06 xeon-dt kernel: Normal per-cpu: May 31 08:07:06 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:183 May 31 08:07:06 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:47 May 31 08:07:06 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:51 May 31 08:07:06 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:33 May 31 08:07:06 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:167 May 31 08:07:06 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:52 May 31 08:07:06 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:23 May 31 08:07:06 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:14 May 31 08:07:06 xeon-dt kernel: HighMem per-cpu: empty May 31 08:07:06 xeon-dt kernel: Free pages: 39080kB (0kB HighMem) May 31 08:07:06 xeon-dt kernel: Active:141705 inactive:50502 dirty:96 writeback:0 unstable:0 free:9770 slab:311263 mapped:13115 pagetables:231 May 31 08:07:06 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:510 all_unreclaimable? yes May 31 08:07:06 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:06 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:06 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:06 xeon-dt kernel: Normal free:30936kB min:5744kB low:7180kB high:8616kB active:566472kB inactive:202008kB present:2080704kB pages_scanned:7 all_unreclaimable? no May 31 08:07:06 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:06 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:06 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:06 xeon-dt kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:07:06 xeon-dt kernel: DMA32: empty May 31 08:07:06 xeon-dt kernel: Normal: 6654*4kB 406*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 30936kB May 31 08:07:06 xeon-dt kernel: HighMem: empty May 31 08:07:06 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:07:06 xeon-dt kernel: Free swap = 2040244kB May 31 08:07:06 xeon-dt kernel: Total swap = 2040244kB May 31 08:07:06 xeon-dt kernel: Free swap: 2040244kB May 31 08:07:06 xeon-dt kernel: 524272 pages of RAM May 31 08:07:06 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:07:06 xeon-dt kernel: 7484 reserved pages May 31 08:07:06 xeon-dt kernel: 157201 pages shared May 31 08:07:06 xeon-dt kernel: 0 pages swap cached May 31 08:07:06 xeon-dt kernel: 96 pages dirty May 31 08:07:06 xeon-dt kernel: 0 pages writeback May 31 08:07:06 xeon-dt kernel: 13115 pages mapped May 31 08:07:06 xeon-dt kernel: 311263 pages slab May 31 08:07:06 xeon-dt kernel: 231 pages pagetables May 31 08:07:06 xeon-dt kernel: swapper: page allocation failure. order:1, mode:0x20 May 31 08:07:06 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:07:06 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:07:06 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:07:06 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:07:06 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:07:06 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:07:06 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:07:06 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:07:06 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:07:06 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:07:06 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:07:06 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:07:06 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:07:06 xeon-dt kernel: [<7815d312>] cache_grow+0x12c/0x14e May 31 08:07:06 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:07:06 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:07:06 xeon-dt kernel: [<f898c6c2>] e1000_alloc_rx_buffers+0x22a/0x418 [e1000] May 31 08:07:06 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:07:06 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:07:06 xeon-dt kernel: [<78125460>] getnstimeofday+0x10/0x2a May 31 08:07:06 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:07:06 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:07:06 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:07:06 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:07:06 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:07:06 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:07:06 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:07:06 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:07:06 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:07:06 xeon-dt kernel: [<78100cfd>] default_idle+0x33/0x52 May 31 08:07:06 xeon-dt kernel: [<78100d9e>] cpu_idle+0x6e/0x9f May 31 08:07:06 xeon-dt kernel: Mem-info: May 31 08:07:06 xeon-dt kernel: DMA per-cpu: May 31 08:07:06 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:07:06 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:07:06 xeon-dt kernel: Normal per-cpu: May 31 08:07:06 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:18 May 31 08:07:06 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:47 May 31 08:07:06 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:3 May 31 08:07:06 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:33 May 31 08:07:06 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:177 May 31 08:07:06 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:52 May 31 08:07:06 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:9 May 31 08:07:06 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:14 May 31 08:07:06 xeon-dt kernel: HighMem per-cpu: empty May 31 08:07:06 xeon-dt kernel: Free pages: 32536kB (0kB HighMem) May 31 08:07:06 xeon-dt kernel: Active:141711 inactive:50496 dirty:97 writeback:1 unstable:0 free:8134 slab:312584 mapped:13115 pagetables:231 May 31 08:07:06 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:510 all_unreclaimable? yes May 31 08:07:06 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:06 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:06 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:06 xeon-dt kernel: Normal free:24392kB min:5744kB low:7180kB high:8616kB active:566496kB inactive:201984kB present:2080704kB pages_scanned:0 all_unreclaimable? no May 31 08:07:06 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:06 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:06 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:06 xeon-dt kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:07:06 xeon-dt kernel: DMA32: empty May 31 08:07:06 xeon-dt kernel: Normal: 5828*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 24392kB May 31 08:07:06 xeon-dt kernel: HighMem: empty May 31 08:07:07 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:07:07 xeon-dt kernel: Free swap = 2040244kB May 31 08:07:07 xeon-dt kernel: Total swap = 2040244kB May 31 08:07:07 xeon-dt kernel: Free swap: 2040244kB May 31 08:07:07 xeon-dt kernel: 524272 pages of RAM May 31 08:07:07 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:07:07 xeon-dt kernel: 7484 reserved pages May 31 08:07:07 xeon-dt kernel: 157201 pages shared May 31 08:07:07 xeon-dt kernel: 0 pages swap cached May 31 08:07:07 xeon-dt kernel: 97 pages dirty May 31 08:07:07 xeon-dt kernel: 1 pages writeback May 31 08:07:07 xeon-dt kernel: 13115 pages mapped May 31 08:07:07 xeon-dt kernel: 312584 pages slab May 31 08:07:07 xeon-dt kernel: 231 pages pagetables May 31 08:07:07 xeon-dt kernel: swapper: page allocation failure. order:1, mode:0x20 May 31 08:07:07 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:07:07 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:07:07 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:07:07 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:07:07 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:07:07 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:07:07 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:07:07 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:07:07 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:07:07 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:07:07 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:07:07 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:07:07 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:07:07 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:07:07 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:07:07 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:07:07 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:07:07 xeon-dt kernel: [<f898b750>] e1000_clean_tx_irq+0x123/0x32e [e1000] May 31 08:07:07 xeon-dt kernel: [<f898bba4>] e1000_clean_rx_irq+0x249/0x5c6 [e1000] May 31 08:07:07 xeon-dt kernel: [<f898c6c2>] e1000_alloc_rx_buffers+0x22a/0x418 [e1000] May 31 08:07:07 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:07:07 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:07:07 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:07:07 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:07:07 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:07:07 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:07:07 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:07:07 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:07:07 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:07:07 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:07:07 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:07:07 xeon-dt kernel: [<78100cfd>] default_idle+0x33/0x52 May 31 08:07:07 xeon-dt kernel: [<78100d9e>] cpu_idle+0x6e/0x9f May 31 08:07:07 xeon-dt kernel: [<7841a7ed>] start_kernel+0x181/0x1bd May 31 08:07:07 xeon-dt kernel: [<7841a283>] unknown_bootoption+0x0/0x1a0 May 31 08:07:07 xeon-dt kernel: Mem-info: May 31 08:07:07 xeon-dt kernel: DMA per-cpu: May 31 08:07:07 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:07:07 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:07:07 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:07:07 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:07:07 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:07:07 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:07:07 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:07:07 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:07:07 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:07:07 xeon-dt kernel: Normal per-cpu: May 31 08:07:07 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:163 May 31 08:07:07 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:47 May 31 08:07:07 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:35 May 31 08:07:07 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:33 May 31 08:07:07 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:120 May 31 08:07:07 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:59 May 31 08:07:07 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:9 May 31 08:07:07 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:14 May 31 08:07:07 xeon-dt kernel: HighMem per-cpu: empty May 31 08:07:07 xeon-dt kernel: Free pages: 70496kB (0kB HighMem) May 31 08:07:07 xeon-dt kernel: Active:127541 inactive:54939 dirty:97 writeback:0 unstable:0 free:17624 slab:312881 mapped:13115 pagetables:231 May 31 08:07:07 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:680 all_unreclaimable? yes May 31 08:07:07 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:07 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:07 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:07 xeon-dt kernel: Normal free:62352kB min:5744kB low:7180kB high:8616kB active:509816kB inactive:219756kB present:2080704kB pages_scanned:0 all_unreclaimable? no May 31 08:07:07 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:07 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:07 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:07 xeon-dt kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:07:07 xeon-dt kernel: DMA32: empty May 31 08:07:07 xeon-dt kernel: Normal: 14506*4kB 389*8kB 10*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 62352kB May 31 08:07:07 xeon-dt kernel: HighMem: empty May 31 08:07:07 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:07:07 xeon-dt kernel: Free swap = 2040244kB May 31 08:07:07 xeon-dt kernel: Total swap = 2040244kB May 31 08:07:07 xeon-dt kernel: Free swap: 2040244kB May 31 08:07:07 xeon-dt kernel: 524272 pages of RAM May 31 08:07:07 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:07:07 xeon-dt kernel: 7484 reserved pages May 31 08:07:07 xeon-dt kernel: 155073 pages shared May 31 08:07:07 xeon-dt kernel: 0 pages swap cached May 31 08:07:07 xeon-dt kernel: 97 pages dirty May 31 08:07:07 xeon-dt kernel: 0 pages writeback May 31 08:07:07 xeon-dt kernel: 13115 pages mapped May 31 08:07:07 xeon-dt kernel: 312881 pages slab May 31 08:07:07 xeon-dt kernel: 231 pages pagetables May 31 08:07:07 xeon-dt kernel: kswapd0: page allocation failure. order:1, mode:0x20 May 31 08:07:07 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:07:07 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:07:07 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:07:07 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:07:07 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:07:07 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:07:07 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:07:07 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:07:07 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:07:07 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:07:07 xeon-dt kernel: [<782e6edf>] tcp_rcv_established+0x204/0x6cf May 31 08:07:07 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:07:07 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:07:07 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:07:07 xeon-dt kernel: [<782ee9bc>] tcp_v4_rcv+0x6ff/0x9b0 May 31 08:07:07 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:07:07 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:07:07 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:07:07 xeon-dt kernel: [<782b3eab>] __kfree_skb+0x4d/0x102 May 31 08:07:08 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:07:08 xeon-dt kernel: [<781060cf>] do_gettimeofday+0x1c/0xb8 May 31 08:07:08 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:07:08 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:07:08 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:07:08 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:07:08 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:07:08 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:07:08 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:07:08 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:07:08 xeon-dt kernel: [<7831007b>] skb_cow_data+0x8a/0x245 May 31 08:07:08 xeon-dt kernel: [<78176e61>] dentry_iput+0x48/0xb9 May 31 08:07:08 xeon-dt kernel: [<7831add1>] _spin_lock+0x24/0x8e May 31 08:07:08 xeon-dt kernel: [<78177376>] prune_dcache+0xc6/0x120 May 31 08:07:08 xeon-dt kernel: [<781776ee>] shrink_dcache_memory+0x14/0x37 May 31 08:07:08 xeon-dt kernel: [<7814ad72>] shrink_slab+0x16b/0x1c0 May 31 08:07:08 xeon-dt kernel: [<7814881d>] throttle_vm_writeout+0x1e/0x5d May 31 08:07:08 xeon-dt kernel: [<7814c05b>] balance_pgdat+0x2a1/0x375 May 31 08:07:08 xeon-dt kernel: [<7814c210>] kswapd+0xe1/0x100 May 31 08:07:08 xeon-dt kernel: [<7813346a>] autoremove_wake_function+0x0/0x43 May 31 08:07:08 xeon-dt kernel: [<78102bca>] ret_from_fork+0x6/0x14 May 31 08:07:08 xeon-dt kernel: [<7813346a>] autoremove_wake_function+0x0/0x43 May 31 08:07:08 xeon-dt kernel: [<7814c12f>] kswapd+0x0/0x100 May 31 08:07:08 xeon-dt kernel: [<78101109>] kernel_thread_helper+0x5/0xb May 31 08:07:08 xeon-dt kernel: Mem-info: May 31 08:07:08 xeon-dt kernel: DMA per-cpu: May 31 08:07:08 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:07:08 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:07:08 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:07:08 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:07:08 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:07:08 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:07:08 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:07:08 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:07:08 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:07:08 xeon-dt kernel: Normal per-cpu: May 31 08:07:08 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:100 May 31 08:07:08 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:47 May 31 08:07:08 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:30 May 31 08:07:08 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:33 May 31 08:07:08 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:166 May 31 08:07:08 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:52 May 31 08:07:08 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:3 May 31 08:07:08 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:14 May 31 08:07:08 xeon-dt kernel: HighMem per-cpu: empty May 31 08:07:08 xeon-dt kernel: Free pages: 69708kB (0kB HighMem) May 31 08:07:08 xeon-dt kernel: Active:127352 inactive:55000 dirty:97 writeback:0 unstable:0 free:17427 slab:312954 mapped:13115 pagetables:231 May 31 08:07:08 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:680 all_unreclaimable? yes May 31 08:07:08 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:08 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:08 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:08 xeon-dt kernel: Normal free:61564kB min:5744kB low:7180kB high:8616kB active:509060kB inactive:220000kB present:2080704kB pages_scanned:0 all_unreclaimable? no May 31 08:07:08 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:08 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:08 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:08 xeon-dt kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:07:08 xeon-dt kernel: DMA32: empty May 31 08:07:08 xeon-dt kernel: Normal: 15121*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 61564kB May 31 08:07:08 xeon-dt kernel: HighMem: empty May 31 08:07:08 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:07:08 xeon-dt kernel: Free swap = 2040244kB May 31 08:07:08 xeon-dt kernel: Total swap = 2040244kB May 31 08:07:08 xeon-dt kernel: Free swap: 2040244kB May 31 08:07:08 xeon-dt kernel: 524272 pages of RAM May 31 08:07:08 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:07:08 xeon-dt kernel: 7484 reserved pages May 31 08:07:08 xeon-dt kernel: 155088 pages shared May 31 08:07:08 xeon-dt kernel: 0 pages swap cached May 31 08:07:09 xeon-dt kernel: 97 pages dirty May 31 08:07:09 xeon-dt kernel: 0 pages writeback May 31 08:07:09 xeon-dt kernel: 13115 pages mapped May 31 08:07:09 xeon-dt kernel: 312954 pages slab May 31 08:07:09 xeon-dt kernel: 231 pages pagetables May 31 08:07:09 xeon-dt kernel: events/2: page allocation failure. order:1, mode:0x20 May 31 08:07:09 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:07:09 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:07:09 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:07:09 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:07:09 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:07:09 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:07:09 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:07:09 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:07:09 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:07:09 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:07:09 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:07:09 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:07:09 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:07:09 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:07:09 xeon-dt kernel: [<7815d455>] cache_alloc_refill+0x121/0x206 May 31 08:07:09 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:07:09 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:07:09 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:07:09 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:07:09 xeon-dt kernel: [<781060cf>] do_gettimeofday+0x1c/0xb8 May 31 08:07:09 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:07:09 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:07:09 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:07:09 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:07:09 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:07:09 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:07:09 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:07:09 xeon-dt kernel: [<7831b219>] _spin_unlock_irq+0xb/0x24 May 31 08:07:09 xeon-dt kernel: [<7815df70>] cache_reap+0x11c/0x1a0 May 31 08:07:09 xeon-dt kernel: [<7812fa8a>] run_workqueue+0x71/0xe5 May 31 08:07:09 xeon-dt kernel: [<7815de54>] cache_reap+0x0/0x1a0 May 31 08:07:09 xeon-dt kernel: [<7812fc12>] worker_thread+0x114/0x136 May 31 08:07:09 xeon-dt kernel: [<7811b16d>] default_wake_function+0x0/0xc May 31 08:07:09 xeon-dt kernel: [<7811b16d>] default_wake_function+0x0/0xc May 31 08:07:09 xeon-dt kernel: [<7812fafe>] worker_thread+0x0/0x136 May 31 08:07:09 xeon-dt kernel: [<78132f0e>] kthread+0x9c/0xa1 May 31 08:07:09 xeon-dt kernel: [<78132e72>] kthread+0x0/0xa1 May 31 08:07:09 xeon-dt kernel: [<78101109>] kernel_thread_helper+0x5/0xb May 31 08:07:09 xeon-dt kernel: Mem-info: May 31 08:07:09 xeon-dt kernel: DMA per-cpu: May 31 08:07:09 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:07:10 xeon-dt kernel: Normal per-cpu: May 31 08:07:10 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:34 May 31 08:07:10 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:47 May 31 08:07:10 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:5 May 31 08:07:10 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:33 May 31 08:07:10 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:69 May 31 08:07:10 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:52 May 31 08:07:10 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:180 May 31 08:07:10 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:55 May 31 08:07:10 xeon-dt kernel: HighMem per-cpu: empty May 31 08:07:10 xeon-dt kernel: Free pages: 90752kB (0kB HighMem) May 31 08:07:10 xeon-dt kernel: Active:119806 inactive:57132 dirty:48 writeback:0 unstable:0 free:22688 slab:312671 mapped:13115 pagetables:231 May 31 08:07:10 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:748 all_unreclaimable? yes May 31 08:07:10 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:10 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:10 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:10 xeon-dt kernel: Normal free:82608kB min:5744kB low:7180kB high:8616kB active:478876kB inactive:228528kB present:2080704kB pages_scanned:26 all_unreclaimable? no May 31 08:07:10 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:10 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:10 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:10 xeon-dt kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:07:10 xeon-dt kernel: DMA32: empty May 31 08:07:10 xeon-dt kernel: Normal: 19568*4kB 282*8kB 26*16kB 18*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 82608kB May 31 08:07:10 xeon-dt kernel: HighMem: empty May 31 08:07:10 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:07:10 xeon-dt kernel: Free swap = 2040244kB May 31 08:07:10 xeon-dt kernel: Total swap = 2040244kB May 31 08:07:10 xeon-dt kernel: Free swap: 2040244kB May 31 08:07:10 xeon-dt kernel: 524272 pages of RAM May 31 08:07:10 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:07:10 xeon-dt kernel: 7484 reserved pages May 31 08:07:10 xeon-dt kernel: 153733 pages shared May 31 08:07:10 xeon-dt kernel: 0 pages swap cached May 31 08:07:10 xeon-dt kernel: 48 pages dirty May 31 08:07:10 xeon-dt kernel: 0 pages writeback May 31 08:07:10 xeon-dt kernel: 13115 pages mapped May 31 08:07:10 xeon-dt kernel: 312671 pages slab May 31 08:07:10 xeon-dt kernel: 231 pages pagetables May 31 08:07:10 xeon-dt kernel: swapper: page allocation failure. order:1, mode:0x20 May 31 08:07:10 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:07:10 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:07:10 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:07:10 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:07:10 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:07:10 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:07:10 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:07:10 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:07:10 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:07:10 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:07:10 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:07:10 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:07:10 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:07:10 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:07:10 xeon-dt kernel: [<78141701>] __do_IRQ+0xb9/0xec May 31 08:07:10 xeon-dt kernel: [<78125842>] irq_exit+0xb/0x44 May 31 08:07:10 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:07:10 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:07:10 xeon-dt kernel: [<78125460>] getnstimeofday+0x10/0x2a May 31 08:07:10 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:07:10 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:07:10 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:07:10 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:07:10 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:07:10 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:07:10 xeon-dt kernel: [<78103748>] apic_timer_interrupt+0x1c/0x24 May 31 08:07:10 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:07:10 xeon-dt kernel: [<78100cfd>] default_idle+0x33/0x52 May 31 08:07:10 xeon-dt kernel: [<78100d9e>] cpu_idle+0x6e/0x9f May 31 08:07:10 xeon-dt kernel: Mem-info: May 31 08:07:10 xeon-dt kernel: DMA per-cpu: May 31 08:07:10 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:07:10 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:07:10 xeon-dt kernel: Normal per-cpu: May 31 08:07:10 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:15 May 31 08:07:10 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:46 May 31 08:07:10 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:35 May 31 08:07:10 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:54 May 31 08:07:10 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:81 May 31 08:07:10 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:52 May 31 08:07:10 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:158 May 31 08:07:10 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:54 May 31 08:07:10 xeon-dt kernel: HighMem per-cpu: empty May 31 08:07:10 xeon-dt kernel: Free pages: 116860kB (0kB HighMem) May 31 08:07:10 xeon-dt kernel: Active:112052 inactive:59078 dirty:49 writeback:0 unstable:0 free:29215 slab:310711 mapped:13154 pagetables:236 May 31 08:07:10 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:850 all_unreclaimable? yes May 31 08:07:10 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:10 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:10 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:07:10 xeon-dt kernel: Normal free:108716kB min:5744kB low:7180kB high:8616kB active:447860kB inactive:236312kB present:2080704kB pages_scanned:0 all_unreclaimable? no May 31 08:07:10 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:10 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:07:10 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:07:10 xeon-dt kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:07:10 xeon-dt kernel: DMA32: empty May 31 08:07:10 xeon-dt kernel: Normal: 26097*4kB 399*8kB 5*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 108716kB May 31 08:07:10 xeon-dt kernel: HighMem: empty May 31 08:07:10 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:07:10 xeon-dt kernel: Free swap = 2040244kB May 31 08:07:10 xeon-dt kernel: Total swap = 2040244kB May 31 08:07:10 xeon-dt kernel: Free swap: 2040244kB May 31 08:07:10 xeon-dt kernel: 524272 pages of RAM May 31 08:07:10 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:07:10 xeon-dt kernel: 7484 reserved pages May 31 08:07:10 xeon-dt kernel: 150664 pages shared May 31 08:07:10 xeon-dt kernel: 0 pages swap cached May 31 08:07:10 xeon-dt kernel: 49 pages dirty May 31 08:07:10 xeon-dt kernel: 0 pages writeback May 31 08:07:10 xeon-dt kernel: 13154 pages mapped May 31 08:07:10 xeon-dt kernel: 310711 pages slab May 31 08:07:10 xeon-dt kernel: 236 pages pagetables May 31 08:10:39 xeon-dt kernel: e1000: eth2: e1000_watchdog_task: NIC Link is Down May 31 08:10:39 xeon-dt kernel: e1000: eth2: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex May 31 08:10:39 xeon-dt kernel: e1000: eth1: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex May 31 08:10:39 xeon-dt kernel: e1000: eth1: e1000_watchdog_task: NIC Link is Down May 31 08:10:40 xeon-dt kernel: e1000: eth1: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex May 31 08:10:40 xeon-dt kernel: e1000: eth2: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex May 31 08:11:17 xeon-dt login(pam_unix)[4227]: authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost= user=root May 31 08:11:19 xeon-dt login[4227]: FAILED LOGIN 1 FROM (null) FOR root, Authentication failure May 31 08:11:24 xeon-dt login(pam_unix)[4227]: session opened for user root by LOGIN(uid=0) May 31 08:11:24 xeon-dt -- root[4227]: ROOT LOGIN ON tty1 May 31 08:12:35 xeon-dt kernel: device eth1 entered promiscuous mode May 31 08:12:37 xeon-dt kernel: device eth1 left promiscuous mode May 31 08:13:50 xeon-dt kernel: e1000: eth2: e1000_watchdog_task: NIC Link is Down May 31 08:13:50 xeon-dt kernel: e1000: eth2: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex May 31 08:13:50 xeon-dt kernel: e1000: eth1: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex May 31 08:13:50 xeon-dt kernel: e1000: eth1: e1000_watchdog_task: NIC Link is Down May 31 08:13:50 xeon-dt kernel: e1000: eth1: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex May 31 08:13:50 xeon-dt kernel: e1000: eth2: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex May 31 08:15:05 xeon-dt kernel: swapper: page allocation failure. order:1, mode:0x20 May 31 08:15:05 xeon-dt kernel: swapper: page allocation failure. order:1, mode:0x20 May 31 08:15:05 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:15:05 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:15:05 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:15:05 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:15:05 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:15:05 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:15:05 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:15:05 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:15:05 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:15:05 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:15:05 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:15:05 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:15:05 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:15:05 xeon-dt kernel: [<7815d312>] cache_grow+0x12c/0x14e May 31 08:15:05 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:15:05 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:15:05 xeon-dt kernel: [<f898c6c2>] e1000_alloc_rx_buffers+0x22a/0x418 [e1000] May 31 08:15:05 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:15:05 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:15:05 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:15:05 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:15:05 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:15:05 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:15:05 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:15:05 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:15:05 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:15:05 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:15:05 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:15:05 xeon-dt kernel: [<78100cfd>] default_idle+0x33/0x52 May 31 08:15:05 xeon-dt kernel: [<78100d9e>] cpu_idle+0x6e/0x9f May 31 08:15:06 xeon-dt kernel: Mem-info: May 31 08:15:06 xeon-dt kernel: DMA per-cpu: May 31 08:15:06 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:15:06 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:15:06 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:15:06 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:15:06 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:15:06 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:15:06 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:15:06 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:15:06 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:15:07 xeon-dt kernel: Normal per-cpu: May 31 08:15:07 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:18 May 31 08:15:07 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:37 May 31 08:15:07 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:178 May 31 08:15:08 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:12 May 31 08:15:08 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:180 May 31 08:15:08 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:48 May 31 08:15:08 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:30 May 31 08:15:08 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:8 May 31 08:15:08 xeon-dt kernel: HighMem per-cpu: empty May 31 08:15:08 xeon-dt kernel: Free pages: 86132kB (0kB HighMem) May 31 08:15:08 xeon-dt kernel: Active:113418 inactive:61465 dirty:94 writeback:0 unstable:0 free:21533 slab:316451 mapped:13243 pagetables:238 May 31 08:15:08 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? no May 31 08:15:08 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:08 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:08 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:08 xeon-dt kernel: Normal free:77988kB min:5744kB low:7180kB high:8616kB active:453324kB inactive:245860kB present:2080704kB pages_scanned:0 all_unreclaimable? no May 31 08:15:08 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:08 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:08 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:08 xeon-dt kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:15:08 xeon-dt kernel: DMA32: empty May 31 08:15:08 xeon-dt kernel: Normal: 19229*4kB 0*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 77988kB May 31 08:15:08 xeon-dt kernel: HighMem: empty May 31 08:15:08 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:15:08 xeon-dt kernel: Free swap = 2040244kB May 31 08:15:08 xeon-dt kernel: Total swap = 2040244kB May 31 08:15:08 xeon-dt kernel: Free swap: 2040244kB May 31 08:15:08 xeon-dt kernel: 524272 pages of RAM May 31 08:15:08 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:15:08 xeon-dt kernel: 7484 reserved pages May 31 08:15:08 xeon-dt kernel: 151287 pages shared May 31 08:15:08 xeon-dt kernel: 0 pages swap cached May 31 08:15:08 xeon-dt kernel: 94 pages dirty May 31 08:15:08 xeon-dt kernel: 0 pages writeback May 31 08:15:08 xeon-dt kernel: 13243 pages mapped May 31 08:15:08 xeon-dt kernel: 316451 pages slab May 31 08:15:08 xeon-dt kernel: 238 pages pagetables May 31 08:15:08 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:15:08 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:15:08 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:15:08 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:15:08 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:15:08 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:15:08 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:15:08 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:15:08 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:15:08 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:15:08 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:15:08 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:15:08 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:15:08 xeon-dt kernel: [<7815d312>] cache_grow+0x12c/0x14e May 31 08:15:08 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:15:08 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:15:08 xeon-dt kernel: [<f898c6c2>] e1000_alloc_rx_buffers+0x22a/0x418 [e1000] May 31 08:15:08 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:15:08 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:15:08 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:15:09 xeon-dt kernel: [<f898b750>] e1000_clean_tx_irq+0x123/0x32e [e1000] May 31 08:15:09 xeon-dt kernel: [<781060cf>] do_gettimeofday+0x1c/0xb8 May 31 08:15:09 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:15:09 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:15:09 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:15:09 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:15:09 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:15:09 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:15:09 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:15:09 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:15:09 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:15:09 xeon-dt kernel: [<78100cfd>] default_idle+0x33/0x52 May 31 08:15:09 xeon-dt kernel: [<78100d9e>] cpu_idle+0x6e/0x9f May 31 08:15:09 xeon-dt kernel: Mem-info: May 31 08:15:09 xeon-dt kernel: DMA per-cpu: May 31 08:15:09 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:15:09 xeon-dt kernel: Normal per-cpu: May 31 08:15:09 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:18 May 31 08:15:09 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:37 May 31 08:15:09 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:173 May 31 08:15:09 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:58 May 31 08:15:09 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:180 May 31 08:15:09 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:48 May 31 08:15:09 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:35 May 31 08:15:09 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:8 May 31 08:15:09 xeon-dt kernel: HighMem per-cpu: empty May 31 08:15:09 xeon-dt kernel: Free pages: 108132kB (0kB HighMem) May 31 08:15:09 xeon-dt kernel: Active:109890 inactive:62337 dirty:94 writeback:0 unstable:0 free:27033 slab:313563 mapped:13243 pagetables:238 May 31 08:15:09 xeon-dt kernel: DMA free:8296kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? no May 31 08:15:09 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:09 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:09 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:09 xeon-dt kernel: Normal free:99836kB min:5744kB low:7180kB high:8616kB active:439212kB inactive:249348kB present:2080704kB pages_scanned:0 all_unreclaimable? no May 31 08:15:09 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:09 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:09 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:09 xeon-dt kernel: DMA: 36*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8296kB May 31 08:15:09 xeon-dt kernel: DMA32: empty May 31 08:15:09 xeon-dt kernel: Normal: 23879*4kB 406*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 99836kB May 31 08:15:09 xeon-dt kernel: HighMem: empty May 31 08:15:09 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:15:09 xeon-dt kernel: Free swap = 2040244kB May 31 08:15:09 xeon-dt kernel: Total swap = 2040244kB May 31 08:15:09 xeon-dt kernel: Free swap: 2040244kB May 31 08:15:09 xeon-dt kernel: 524272 pages of RAM May 31 08:15:09 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:15:09 xeon-dt kernel: 7484 reserved pages May 31 08:15:09 xeon-dt kernel: 148633 pages shared May 31 08:15:09 xeon-dt kernel: 0 pages swap cached May 31 08:15:09 xeon-dt kernel: 94 pages dirty May 31 08:15:09 xeon-dt kernel: 0 pages writeback May 31 08:15:09 xeon-dt kernel: 13243 pages mapped May 31 08:15:09 xeon-dt kernel: 313558 pages slab May 31 08:15:09 xeon-dt kernel: 238 pages pagetables May 31 08:15:09 xeon-dt kernel: btserver: page allocation failure. order:1, mode:0x20 May 31 08:15:09 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:15:09 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:15:09 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:15:09 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:15:09 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:15:09 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:15:09 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:15:09 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:15:09 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:15:09 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:15:09 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:15:09 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:15:09 xeon-dt kernel: [<782b2927>] __release_sock+0x3e/0x5d May 31 08:15:09 xeon-dt kernel: [<782b30aa>] release_sock+0x6e/0x70 May 31 08:15:09 xeon-dt kernel: [<782df1a2>] tcp_sendmsg+0x326/0xad5 May 31 08:15:09 xeon-dt kernel: [<7814720d>] __alloc_pages+0x52/0x2b8 May 31 08:15:09 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:15:09 xeon-dt kernel: [<782fa5bd>] inet_sendmsg+0x47/0x5f May 31 08:15:09 xeon-dt kernel: [<782af085>] sock_sendmsg+0xfd/0xff May 31 08:15:09 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:15:09 xeon-dt kernel: [<f898c6c2>] e1000_alloc_rx_buffers+0x22a/0x418 [e1000] May 31 08:15:09 xeon-dt kernel: [<7813346a>] autoremove_wake_function+0x0/0x43 May 31 08:15:09 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:15:09 xeon-dt kernel: [<f898b750>] e1000_clean_tx_irq+0x123/0x32e [e1000] May 31 08:15:09 xeon-dt kernel: [<f898bba4>] e1000_clean_rx_irq+0x249/0x5c6 [e1000] May 31 08:15:09 xeon-dt kernel: [<78161670>] fget+0x5a/0x98 May 31 08:15:09 xeon-dt kernel: [<782b0779>] sys_sendto+0xc7/0xe2 May 31 08:15:09 xeon-dt kernel: [<782b9b40>] net_rx_action+0x147/0x1ae May 31 08:15:09 xeon-dt kernel: [<781256e2>] __do_softirq+0x6a/0xd0 May 31 08:15:09 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:15:09 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:15:09 xeon-dt kernel: [<782b07cb>] sys_send+0x37/0x3b May 31 08:15:09 xeon-dt kernel: [<782b0f48>] sys_socketcall+0x150/0x26d May 31 08:15:09 xeon-dt kernel: [<78124cf2>] sys_gettimeofday+0x2c/0x65 May 31 08:15:09 xeon-dt kernel: [<78102c93>] sysenter_past_esp+0x54/0x75 May 31 08:15:09 xeon-dt kernel: Mem-info: May 31 08:15:09 xeon-dt kernel: DMA per-cpu: May 31 08:15:09 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:15:09 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:15:09 xeon-dt kernel: Normal per-cpu: May 31 08:15:09 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:105 May 31 08:15:09 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:56 May 31 08:15:09 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:173 May 31 08:15:09 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:58 May 31 08:15:09 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:157 May 31 08:15:09 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:60 May 31 08:15:09 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:25 May 31 08:15:09 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:8 May 31 08:15:09 xeon-dt kernel: HighMem per-cpu: empty May 31 08:15:09 xeon-dt kernel: Free pages: 129132kB (0kB HighMem) May 31 08:15:09 xeon-dt kernel: Active:106610 inactive:63452 dirty:96 writeback:0 unstable:0 free:32283 slab:310207 mapped:13243 pagetables:238 May 31 08:15:09 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:136 all_unreclaimable? no May 31 08:15:09 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:09 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:09 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:09 xeon-dt kernel: Normal free:120988kB min:5744kB low:7180kB high:8616kB active:426092kB inactive:253808kB present:2080704kB pages_scanned:0 all_unreclaimable? no May 31 08:15:09 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:09 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:09 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:09 xeon-dt kernel: DMA: 36*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:15:09 xeon-dt kernel: DMA32: empty May 31 08:15:09 xeon-dt kernel: Normal: 29161*4kB 399*8kB 6*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 120988kB May 31 08:15:09 xeon-dt kernel: HighMem: empty May 31 08:15:09 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:15:09 xeon-dt kernel: Free swap = 2040244kB May 31 08:15:09 xeon-dt kernel: Total swap = 2040244kB May 31 08:15:09 xeon-dt kernel: Free swap: 2040244kB May 31 08:15:09 xeon-dt kernel: 524272 pages of RAM May 31 08:15:09 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:15:09 xeon-dt kernel: 7484 reserved pages May 31 08:15:09 xeon-dt kernel: 146453 pages shared May 31 08:15:09 xeon-dt kernel: 0 pages swap cached May 31 08:15:09 xeon-dt kernel: 96 pages dirty May 31 08:15:09 xeon-dt kernel: 0 pages writeback May 31 08:15:09 xeon-dt kernel: 13243 pages mapped May 31 08:15:09 xeon-dt kernel: 310207 pages slab May 31 08:15:09 xeon-dt kernel: 238 pages pagetables May 31 08:15:09 xeon-dt kernel: kswapd0: page allocation failure. order:1, mode:0x20 May 31 08:15:09 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:15:09 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:15:09 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:15:09 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:15:09 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:15:09 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:15:09 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:15:09 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:15:09 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:15:10 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:15:10 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:15:10 xeon-dt kernel: [<782e6ef8>] tcp_rcv_established+0x21d/0x6cf May 31 08:15:10 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:15:10 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:15:10 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:15:10 xeon-dt kernel: [<7815d312>] cache_grow+0x12c/0x14e May 31 08:15:10 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:15:10 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:15:10 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:15:10 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:15:10 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:15:10 xeon-dt kernel: [<f898b750>] e1000_clean_tx_irq+0x123/0x32e [e1000] May 31 08:15:10 xeon-dt kernel: [<f898bba4>] e1000_clean_rx_irq+0x249/0x5c6 [e1000] May 31 08:15:10 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:15:10 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:15:10 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:15:10 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:15:10 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:15:10 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:15:10 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:15:10 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:15:10 xeon-dt kernel: [<78176dd0>] d_callback+0x0/0x24 May 31 08:15:10 xeon-dt kernel: [<78130911>] call_rcu+0x1a/0x6e May 31 08:15:10 xeon-dt kernel: [<78177380>] prune_dcache+0xd0/0x120 May 31 08:15:10 xeon-dt kernel: [<781776ee>] shrink_dcache_memory+0x14/0x37 May 31 08:15:10 xeon-dt kernel: [<7814ad72>] shrink_slab+0x16b/0x1c0 May 31 08:15:10 xeon-dt kernel: [<7814881d>] throttle_vm_writeout+0x1e/0x5d May 31 08:15:10 xeon-dt kernel: [<7814c05b>] balance_pgdat+0x2a1/0x375 May 31 08:15:10 xeon-dt kernel: [<7814c210>] kswapd+0xe1/0x100 May 31 08:15:10 xeon-dt kernel: [<7813346a>] autoremove_wake_function+0x0/0x43 May 31 08:15:10 xeon-dt kernel: [<78102bca>] ret_from_fork+0x6/0x14 May 31 08:15:10 xeon-dt kernel: [<7813346a>] autoremove_wake_function+0x0/0x43 May 31 08:15:10 xeon-dt kernel: [<7814c12f>] kswapd+0x0/0x100 May 31 08:15:10 xeon-dt kernel: [<78101109>] kernel_thread_helper+0x5/0xb May 31 08:15:10 xeon-dt kernel: Mem-info: May 31 08:15:10 xeon-dt kernel: DMA per-cpu: May 31 08:15:10 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:15:10 xeon-dt kernel: Normal per-cpu: May 31 08:15:10 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:7 May 31 08:15:10 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:56 May 31 08:15:10 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:179 May 31 08:15:10 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:47 May 31 08:15:10 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:30 May 31 08:15:10 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:60 May 31 08:15:10 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:2 May 31 08:15:10 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:8 May 31 08:15:10 xeon-dt kernel: HighMem per-cpu: empty May 31 08:15:10 xeon-dt kernel: Free pages: 119148kB (0kB HighMem) May 31 08:15:10 xeon-dt kernel: Active:106530 inactive:63475 dirty:96 writeback:0 unstable:0 free:29787 slab:309881 mapped:13243 pagetables:238 May 31 08:15:10 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:136 all_unreclaimable? no May 31 08:15:10 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:10 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:10 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:10 xeon-dt kernel: Normal free:111004kB min:5744kB low:7180kB high:8616kB active:425772kB inactive:253900kB present:2080704kB pages_scanned:0 all_unreclaimable? no May 31 08:15:10 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:10 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:10 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:10 xeon-dt kernel: DMA: 36*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:15:10 xeon-dt kernel: DMA32: empty May 31 08:15:10 xeon-dt kernel: Normal: 27481*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 111004kB May 31 08:15:10 xeon-dt kernel: HighMem: empty May 31 08:15:10 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:15:10 xeon-dt kernel: Free swap = 2040244kB May 31 08:15:10 xeon-dt kernel: Total swap = 2040244kB May 31 08:15:10 xeon-dt kernel: Free swap: 2040244kB May 31 08:15:10 xeon-dt kernel: 524272 pages of RAM May 31 08:15:10 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:15:10 xeon-dt kernel: 7484 reserved pages May 31 08:15:10 xeon-dt kernel: 146589 pages shared May 31 08:15:10 xeon-dt kernel: 0 pages swap cached May 31 08:15:10 xeon-dt kernel: 96 pages dirty May 31 08:15:10 xeon-dt kernel: 0 pages writeback May 31 08:15:10 xeon-dt kernel: 13243 pages mapped May 31 08:15:10 xeon-dt kernel: 309881 pages slab May 31 08:15:10 xeon-dt kernel: 238 pages pagetables May 31 08:15:10 xeon-dt kernel: swapper: page allocation failure. order:1, mode:0x20 May 31 08:15:10 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:15:10 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:15:10 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:15:10 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:15:10 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:15:10 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:15:10 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:15:10 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:15:10 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:15:10 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:15:10 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:15:10 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:15:10 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:15:10 xeon-dt kernel: [<7815d312>] cache_grow+0x12c/0x14e May 31 08:15:10 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:15:10 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:15:10 xeon-dt kernel: [<f898c6c2>] e1000_alloc_rx_buffers+0x22a/0x418 [e1000] May 31 08:15:10 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:15:10 xeon-dt kernel: [<782b3eab>] __kfree_skb+0x4d/0x102 May 31 08:15:10 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:15:10 xeon-dt kernel: [<78125460>] getnstimeofday+0x10/0x2a May 31 08:15:10 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:15:10 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:15:10 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:15:10 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:15:10 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:15:10 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:15:10 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:15:10 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:15:10 xeon-dt kernel: [<78100cca>] default_idle+0x0/0x52 May 31 08:15:10 xeon-dt kernel: [<78100cfd>] default_idle+0x33/0x52 May 31 08:15:10 xeon-dt kernel: [<78100d9e>] cpu_idle+0x6e/0x9f May 31 08:15:10 xeon-dt kernel: Mem-info: May 31 08:15:10 xeon-dt kernel: DMA per-cpu: May 31 08:15:10 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:15:10 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:15:10 xeon-dt kernel: Normal per-cpu: May 31 08:15:10 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:155 May 31 08:15:10 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:51 May 31 08:15:10 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:141 May 31 08:15:10 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:47 May 31 08:15:10 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:160 May 31 08:15:10 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:60 May 31 08:15:10 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:1 May 31 08:15:10 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:8 May 31 08:15:10 xeon-dt kernel: HighMem per-cpu: empty May 31 08:15:10 xeon-dt kernel: Free pages: 140120kB (0kB HighMem) May 31 08:15:10 xeon-dt kernel: Active:104009 inactive:64399 dirty:96 writeback:0 unstable:0 free:35030 slab:306067 mapped:13243 pagetables:238 May 31 08:15:10 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:102 all_unreclaimable? no May 31 08:15:10 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:10 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:10 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:10 xeon-dt kernel: Normal free:131976kB min:5744kB low:7180kB high:8616kB active:415688kB inactive:257596kB present:2080704kB pages_scanned:0 all_unreclaimable? no May 31 08:15:10 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:10 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:10 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:10 xeon-dt kernel: DMA: 38*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:15:10 xeon-dt kernel: DMA32: empty May 31 08:15:10 xeon-dt kernel: Normal: 31904*4kB 407*8kB 3*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 131976kB May 31 08:15:10 xeon-dt kernel: HighMem: empty May 31 08:15:10 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:15:10 xeon-dt kernel: Free swap = 2040244kB May 31 08:15:10 xeon-dt kernel: Total swap = 2040244kB May 31 08:15:10 xeon-dt kernel: Free swap: 2040244kB May 31 08:15:10 xeon-dt kernel: 524272 pages of RAM May 31 08:15:10 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:15:10 xeon-dt kernel: 7484 reserved pages May 31 08:15:10 xeon-dt kernel: 144992 pages shared May 31 08:15:10 xeon-dt kernel: 0 pages swap cached May 31 08:15:10 xeon-dt kernel: 96 pages dirty May 31 08:15:10 xeon-dt kernel: 0 pages writeback May 31 08:15:10 xeon-dt kernel: 13243 pages mapped May 31 08:15:11 xeon-dt kernel: 306067 pages slab May 31 08:15:11 xeon-dt kernel: 238 pages pagetables May 31 08:15:52 xeon-dt kernel: find_dflt_route: page allocation failure. order:1, mode:0x20 May 31 08:15:52 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:15:52 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:15:52 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:15:52 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:15:52 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:15:52 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:15:52 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e May 31 08:15:52 xeon-dt kernel: [<782e6726>] tcp_prune_queue+0x97/0x1db May 31 08:15:52 xeon-dt kernel: [<782e5e5e>] tcp_data_queue+0x2ba/0x722 May 31 08:15:52 xeon-dt kernel: [<7831af17>] _read_lock+0x24/0x8c May 31 08:15:52 xeon-dt kernel: [<782e6eb7>] tcp_rcv_established+0x1dc/0x6cf May 31 08:15:52 xeon-dt kernel: [<782e7210>] tcp_rcv_established+0x535/0x6cf May 31 08:15:52 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:15:52 xeon-dt kernel: [<782ee2b8>] tcp_v4_do_rcv+0xe5/0xea May 31 08:15:52 xeon-dt kernel: [<782eea45>] tcp_v4_rcv+0x788/0x9b0 May 31 08:15:52 xeon-dt kernel: [<7831b18a>] _spin_unlock+0xf/0x23 May 31 08:15:52 xeon-dt kernel: [<7815d312>] cache_grow+0x12c/0x14e May 31 08:15:52 xeon-dt kernel: [<782d48fe>] ip_local_deliver+0x109/0x287 May 31 08:15:52 xeon-dt kernel: [<782d4d1f>] ip_rcv+0x2a3/0x577 May 31 08:15:52 xeon-dt kernel: [<f898c6c2>] e1000_alloc_rx_buffers+0x22a/0x418 [e1000] May 31 08:15:52 xeon-dt kernel: [<782b9841>] netif_receive_skb+0x2b8/0x36d May 31 08:15:52 xeon-dt kernel: [<782b3eab>] __kfree_skb+0x4d/0x102 May 31 08:15:52 xeon-dt kernel: [<f898bb26>] e1000_clean_rx_irq+0x1cb/0x5c6 [e1000] May 31 08:15:52 xeon-dt kernel: [<f898bba4>] e1000_clean_rx_irq+0x249/0x5c6 [e1000] May 31 08:15:52 xeon-dt kernel: [<f898b546>] e1000_clean+0xa6/0x18d [e1000] May 31 08:15:52 xeon-dt kernel: [<782b9a92>] net_rx_action+0x99/0x1ae May 31 08:15:52 xeon-dt kernel: [<78125733>] __do_softirq+0xbb/0xd0 May 31 08:15:52 xeon-dt kernel: [<78125790>] do_softirq+0x48/0x4a May 31 08:15:52 xeon-dt kernel: [<78125879>] irq_exit+0x42/0x44 May 31 08:15:52 xeon-dt kernel: [<781050e7>] do_IRQ+0x37/0x62 May 31 08:15:52 xeon-dt kernel: [<781036ba>] common_interrupt+0x1a/0x20 May 31 08:15:52 xeon-dt kernel: [<7814007b>] audit_filter_user_rules+0x89/0xb9 May 31 08:15:52 xeon-dt kernel: [<781db83c>] _raw_spin_trylock+0x7/0x28 May 31 08:15:52 xeon-dt kernel: [<7831add1>] _spin_lock+0x24/0x8e May 31 08:15:52 xeon-dt kernel: [<78150ec7>] unlink_file_vma+0x2b/0x62 May 31 08:15:52 xeon-dt kernel: [<7814dc6b>] free_pgtables+0x77/0x90 May 31 08:15:52 xeon-dt kernel: [<78152cec>] exit_mmap+0xa0/0x11d May 31 08:15:52 xeon-dt kernel: [<7811db29>] mmput+0x33/0x94 May 31 08:15:52 xeon-dt kernel: [<7816ac1b>] exec_mmap+0xf4/0x213 May 31 08:15:52 xeon-dt kernel: [<7816b489>] flush_old_exec+0x43/0x1f9 May 31 08:15:52 xeon-dt kernel: [<7816ab14>] kernel_read+0x40/0x53 May 31 08:15:52 xeon-dt kernel: [<7818b108>] load_elf_binary+0x2d4/0xc14 May 31 08:15:52 xeon-dt kernel: [<781daeaf>] copy_from_user+0x58/0x91 May 31 08:15:52 xeon-dt kernel: [<7818ae34>] load_elf_binary+0x0/0xc14 May 31 08:15:52 xeon-dt kernel: [<7816b9a7>] search_binary_handler+0x17d/0x2c9 May 31 08:15:52 xeon-dt kernel: [<7816bc58>] do_execve+0x165/0x203 May 31 08:15:52 xeon-dt kernel: [<78101a70>] sys_execve+0x39/0x85 May 31 08:15:52 xeon-dt kernel: [<78102c93>] sysenter_past_esp+0x54/0x75 May 31 08:15:52 xeon-dt kernel: Mem-info: May 31 08:15:52 xeon-dt kernel: DMA per-cpu: May 31 08:15:52 xeon-dt kernel: cpu 0 hot: high 0, batch 1 used:0 May 31 08:15:52 xeon-dt kernel: cpu 0 cold: high 0, batch 1 used:0 May 31 08:15:52 xeon-dt kernel: cpu 1 hot: high 0, batch 1 used:0 May 31 08:15:52 xeon-dt kernel: cpu 1 cold: high 0, batch 1 used:0 May 31 08:15:52 xeon-dt kernel: cpu 2 hot: high 0, batch 1 used:0 May 31 08:15:52 xeon-dt kernel: cpu 2 cold: high 0, batch 1 used:0 May 31 08:15:52 xeon-dt kernel: cpu 3 hot: high 0, batch 1 used:0 May 31 08:15:52 xeon-dt kernel: cpu 3 cold: high 0, batch 1 used:0 May 31 08:15:52 xeon-dt kernel: DMA32 per-cpu: empty May 31 08:15:52 xeon-dt kernel: Normal per-cpu: May 31 08:15:52 xeon-dt kernel: cpu 0 hot: high 186, batch 31 used:158 May 31 08:15:52 xeon-dt kernel: cpu 0 cold: high 62, batch 15 used:51 May 31 08:15:52 xeon-dt kernel: cpu 1 hot: high 186, batch 31 used:93 May 31 08:15:52 xeon-dt kernel: cpu 1 cold: high 62, batch 15 used:47 May 31 08:15:52 xeon-dt kernel: cpu 2 hot: high 186, batch 31 used:180 May 31 08:15:52 xeon-dt kernel: cpu 2 cold: high 62, batch 15 used:59 May 31 08:15:52 xeon-dt kernel: cpu 3 hot: high 186, batch 31 used:21 May 31 08:15:52 xeon-dt kernel: cpu 3 cold: high 62, batch 15 used:8 May 31 08:15:52 xeon-dt kernel: HighMem per-cpu: empty May 31 08:15:52 xeon-dt kernel: Free pages: 160616kB (0kB HighMem) May 31 08:15:52 xeon-dt kernel: Active:102218 inactive:65424 dirty:93 writeback:0 unstable:0 free:40154 slab:301671 mapped:13504 pagetables:253 May 31 08:15:52 xeon-dt kernel: DMA free:8144kB min:44kB low:52kB high:64kB active:348kB inactive:0kB present:16384kB pages_scanned:68 all_unreclaimable? no May 31 08:15:52 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:52 xeon-dt kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:52 xeon-dt kernel: lowmem_reserve[]: 0 0 2031 2031 May 31 08:15:52 xeon-dt kernel: Normal free:152472kB min:5744kB low:7180kB high:8616kB active:408524kB inactive:261696kB present:2080704kB pages_scanned:19 all_unreclaimable? no May 31 08:15:52 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:52 xeon-dt kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 31 08:15:53 xeon-dt kernel: lowmem_reserve[]: 0 0 0 0 May 31 08:15:53 xeon-dt kernel: DMA: 42*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8144kB May 31 08:15:53 xeon-dt kernel: DMA32: empty May 31 08:15:53 xeon-dt kernel: Normal: 37032*4kB 407*8kB 2*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 152472kB May 31 08:15:53 xeon-dt kernel: HighMem: empty May 31 08:15:53 xeon-dt kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0 May 31 08:15:53 xeon-dt kernel: Free swap = 2040244kB May 31 08:15:53 xeon-dt kernel: Total swap = 2040244kB May 31 08:15:53 xeon-dt kernel: Free swap: 2040244kB May 31 08:15:53 xeon-dt kernel: 524272 pages of RAM May 31 08:15:53 xeon-dt kernel: 0 pages of HIGHMEM May 31 08:15:53 xeon-dt kernel: 7484 reserved pages May 31 08:15:53 xeon-dt kernel: 144659 pages shared May 31 08:15:53 xeon-dt kernel: 0 pages swap cached May 31 08:15:53 xeon-dt kernel: 93 pages dirty May 31 08:15:53 xeon-dt kernel: 0 pages writeback May 31 08:15:53 xeon-dt kernel: 13504 pages mapped May 31 08:15:53 xeon-dt kernel: 301671 pages slab May 31 08:15:53 xeon-dt kernel: 253 pages pagetables
include/linux/if.h +5 -0
@@ -52,6 +52,11 @@/* Private (from user) interface flags (netdevice->priv_flags). */ #define IFF_802_1Q_VLAN 0x1/* 802.1Q VLAN device. */ #define IFF_EBRIDGE 0x2/* Ethernet bridging device. */ +#define IFF_ACCEPT_LOCAL_ADDRS 0x8/** Accept pkts even if they come from a local +* address. This lets use send pkts to ourselves +* over external interfaces (when used in conjunction +* with SO_BINDTODEVICE +*/ #define IF_GET_IFACE 0x0001/* for querying only */ #define IF_GET_PROTO 0x0002
include/linux/sockios.h +8 -0
@@ -122,6 +122,14 @@#define SIOCBRADDIF 0x89a2/* add interface to bridge */ #define SIOCBRDELIF 0x89a3/* remove interface from bridge */ +/* Ben's little hack land */ +#define SIOCSACCEPTLOCALADDRS 0x89ba/* Allow interfaces to accept pkts from +* local interfaces...use with SO_BINDTODEVICE +*/ +#define SIOCGACCEPTLOCALADDRS 0x89bb/* Allow interfaces to accept pkts from +* local interfaces...use with SO_BINDTODEVICE +*/ +/* Device private ioctl calls */ /*
net/core/dev.c +20 -0
@@ -2375,6 +2375,24 @@ static int dev_ifsioc(struct ifreq *ifr,case SIOCSIFNAME : ifr->ifr_newname[IFNAMSIZ -1 ] ='\0' ;return dev_change_name (dev, ifr->ifr_newname); + +case SIOCSACCEPTLOCALADDRS : +if (ifr->ifr_flags) { + dev->priv_flags |=IFF_ACCEPT_LOCAL_ADDRS ; + } +else { + dev->priv_flags &= ~IFF_ACCEPT_LOCAL_ADDRS ; + } +return 0 ; + +case SIOCGACCEPTLOCALADDRS : +if (dev->priv_flags &IFF_ACCEPT_LOCAL_ADDRS ) { + ifr->ifr_flags =1 ; + } +else { + ifr->ifr_flags =0 ; + } +return 0 ;/* * Unknown or private ioctl
@@ -2474,6 +2492,7 @@ int dev_ioctl(unsigned int cmd, void __ucase SIOCGIFMAP :case SIOCGIFINDEX :case SIOCGIFTXQLEN : +case SIOCGACCEPTLOCALADDRS :dev_load (ifr.ifr_name);read_lock (&dev_base_lock); ret =dev_ifsioc (&ifr, cmd);
@@ -2550,6 +2569,7 @@ int dev_ioctl(unsigned int cmd, void __ucase SIOCBONDCHANGEACTIVE :case SIOCBRADDIF :case SIOCBRDELIF : +case SIOCSACCEPTLOCALADDRS :if (!capable (CAP_NET_ADMIN ))return -EPERM ;dev_load (ifr.ifr_name);
net/ipv4/arp.c +51 -4
67 lines not on this page · raw message
net/ipv4/fib_frontend.c +10 -2
@@ -185,8 +185,16 @@ int fib_validate_source(u32 src, u32 dstif (fib_lookup (&fl, &res)) goto last_resort; -if (res.type !=RTN_UNICAST ) - goto e_inval_res; +if (res.type !=RTN_UNICAST ) { +if ((res.type ==RTN_LOCAL ) && + (dev->priv_flags &IFF_ACCEPT_LOCAL_ADDRS )) { +/* All is OK */ + } +else { + goto e_inval_res; + } + } + *spec_dst =FIB_RES_PREFSRC (res);fib_combine_itag (itag, &res);#ifdef CONFIG_IP_ROUTE_MULTIPATH
Quoted 6 lines · Ben Greear
Kernel is 2.6.16.16 with my patches, including a patch to the e1000. I also rebuilt a fresh kernel with only the attached send-to-self patch. I see the hang, but there was no OOM messages, probably because the machine was freshly rebooted and had plenty of buffers available. I will try to reproduce this with some other NICs next...
The same problem happens with a single TCP connection between two copper 10/100/1000 intel ports on the FC2 dual-xeon 32-bit machine.
I tried a dual-core AMD system running 64-bit kernel and FC5. This kernel is running my complete set of patches (against 2.6.16.16).
I can reproduce with Intel pro/1000 on this dual-core system when I up the rate to 400Mbps and do 50,000 byte writes (can only get 125Mbps with 5000 byte writes on this system, and it ran at that speed for several minutes w/out problem.)
I can also reproduce when using the tg3 adapters, but the tg3 can only run at about 220Mbps, even with the larger write size. This slower speed could be because of bus issues or the VLAN I used for one end of the connection. I could not see this problem with tg3 untill I yanked the cable a few times and then ran the connection through my network emulator (which will cause a bit of latency and some dropped packets). The tethereal output looks slightly different, but I think this is just because it is a different version from the older FC2 system from my previous email.
0.399107 192.168.1.3 -> 192.168.1.2 TCP 33020 > 33019 [ACK] Seq=4294967248 Ack=48 Win=11998 Len=0 TSV=943608 TSER=842823 0.400565 192.168.1.2 -> 192.168.1.3 TCP 33019 > 33020 [ACK] Seq=0 Ack=0 Win=11998 Len=0 TSV=943608 TSER=842824 0.400764 192.168.1.3 -> 192.168.1.2 TCP 33020 > 33019 [ACK] Seq=4294967248 Ack=48 Win=11998 Len=0 TSV=943608 TSER=842823 0.400948 192.168.1.2 -> 192.168.1.3 TCP 33019 > 33020 [ACK] Seq=0 Ack=0 Win=11998 Len=0 TSV=943608 TSER=842824 0.408851 192.168.1.3 -> 192.168.1.2 TCP 33020 > 33019 [ACK] Seq=4294967248 Ack=48 Win=11998 Len=0 TSV=943608 TSER=842823 0.410005 192.168.1.2 -> 192.168.1.3 TCP 33019 > 33020 [ACK] Seq=0 Ack=0 Win=11998 Len=0 TSV=943608 TSER=842824 0.411524 192.168.1.3 -> 192.168.1.2 TCP 33020 > 33019 [ACK] Seq=4294967248 Ack=48 Win=11998 Len=0 TSV=943608 TSER=842823 0.450702 192.168.1.2 -> 192.168.1.3 TCP 33019 > 33020 [ACK] Seq=0 Ack=0 Win=11998 Len=0 TSV=943608 TSER=842824
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 2 lines · Ben Greear
Kernel is 2.6.16.16 with my patches, including a patch to the e1000.
what patch? I don't see it anywhere.
Cheers,
Auke
Quoted 7 lines · Auke Kok
Quoted 2 lines · Ben Greear
Kernel is 2.6.16.16 with my patches, including a patch to the e1000.
what patch? I don't see it anywhere.
Well, it's reproducible with only the send-to-self patch, so I figured it wouldn't help to show the e1000 patches. Also, it is reproducible on tg3, so it is not e1000 specific, though it is slightly easier to trigger on e1000
for some reason (possibly higher throughput or something specific about my configuration.)
I haven't seen this problem on 2.6.13, so I'm now starting a manual bisect to see if I can narrow down where the problem appeared.
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 2 lines · Ben Greear
I haven't seen this problem on 2.6.13, so I'm now starting a manual bisect to see if I can narrow down where the problem appeared.
Turns out, I can reproduce it in 2.6.13, and 2.6.9. I haven't tried anything older.
I also tried to reproduce it using a simpler traffic generation tool, but could not reproduce the problem with it. That points to something wierd that my application is doing, but I can't imagine what user-space could do to screw up a TCP connection like this.
In all cases, there is a lot of data in the send-queue, but for whatever reason, the connection will not make progress. To user-space, it appears that poll returns neither readable nor writable for the sockets.
I notice that if I increase the send-buffer-size while the connection is in the hung state, my app will quickly fill the larger send buffer, but still receives nothing new.
Starting a new connection on the same interfaces works for a few seconds and then hangs as well, so the NICs can pass traffic.
Here is output from /proc/net/tcp and netstat from the 2.6.16.16 kernel.
netstat info:
tcp 0 5635368 172.1.5.169:33058 172.1.5.168:33057 ESTABLISHED tcp 0 5987504 172.1.5.168:33057 172.1.5.169:33058 ESTABLISHED
/proc/net/tcp:
20: A90501AC:8122 A80501AC:8121 01 0055FD28:00000000 01:00001A9F 0000000A
0 0 21309 2 f36d8580 120000 40 0 1 58
21: A80501AC:8121 A90501AC:8122 01 005B5CB0:00000000 01:00001C9D 0000000A
0 0 21226 3 ef7bfa80 120000 40 0 1 35
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: TCP connection hang problem with 2.6.16.16, e1000 (found work-around)
Ok, I think I found the problem. It appears to be at least partially self-inflicted. I was modifying the kernel buffer settings via /proc/* to improve performance. I noticed that when I used kernel-defaults, the TCP connection hangs stopped (and ran ~20% slower). My personal opinion is that this is still a bug somewhere in TCP, but at least I have a work-around.
Since these settings appear somewhat delicate, it might be nice if the kernel could warn about configurations that could cause TCP deadlocks and such.
My old settings that exhibitted the problem were:
my $netdev_max_backlog = 5000;
my $wmem_max = 4096000;
my $wmem_default = 128000;
my $rmem_max = 8096000;
my $rmem_default = 128000;
my $tcp_rmem_min = 4096;
my $tcp_rmem_default = 256000;
my $tcp_rmem_max = 30000000;
my $tcp_wmem_min = 4096;
my $tcp_wmem_default = 256000;
my $tcp_wmem_max = 30000000;
my $tcp_mem_lo = 20000000;
my $tcp_mem_pressure = 30000000;
my $tcp_mem_high = 60000000;
The new settings that appear to work around the TCP connection hang, and still give decent performance are below. Documentation being thin, I used the suggested values in Documentation/networking/ixgb.txt as hints:
my $netdev_max_backlog = 30000;
my $wmem_max = 524287;
my $wmem_default = 128000;
my $rmem_max = 524287;
my $rmem_default = 128000;
my $tcp_rmem_min = 4096;
my $tcp_rmem_default = 256000;
my $tcp_rmem_max = 5000000;
my $tcp_wmem_min = 4096;
my $tcp_wmem_default = 256000;
my $tcp_wmem_max = 5000000;
my $tcp_mem_lo = 10000000;
my $tcp_mem_pressure = 20000000;
my $tcp_mem_high = 30000000;
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: TCP connection hang problem with 2.6.16.16, e1000 (found work-around)
Quoted 5 lines · On Wed, 31 May 2006 14:45:13 -0700 Ben Greear <[email protected]> wrote:
Ok, I think I found the problem. It appears to be at least partially self-inflicted. I was modifying the kernel buffer settings via
/proc/*to improve performance. I noticed that when I used kernel-defaults, the TCP connection hangs stopped (and ran ~20% slower). My personal opinion is that this is still a bug somewhere in TCP, but at least I have a work-around.
If you set bogus values, you can end up consuming all the memory on your system.
Quoted 4 lines
Since these settings appear somewhat delicate, it might be nice if the kernel could warn about configurations that could cause TCP deadlocks and such.
There is little or no range checking on most all sysctl values. Whether or not that is a good thing is debatable.
Quoted 3 lines
My old settings that exhibitted the problem were:
my $netdev_max_backlog = 5000;
This only matters for non-NAPI devices.
Quoted 7 lines
my $wmem_max = 4096000;
my $wmem_default = 128000;
my $rmem_max = 8096000;
my $rmem_default = 128000;my $tcp_rmem_min = 4096;
my $tcp_rmem_default = 256000;
Receiver auto-tuning makes setting this unnecessary in most cases.
Quoted 4 lines
my $tcp_rmem_max = 30000000;
my $tcp_wmem_min = 4096;
my $tcp_wmem_default = 256000;
That's too big with current auto-tuning.
Quoted 11 lines
my $tcp_wmem_max = 30000000;
my $tcp_mem_lo = 20000000;
my $tcp_mem_pressure = 30000000;
my $tcp_mem_high = 60000000;The new settings that appear to work around the TCP connection hang, and still give decent performance are below. Documentation being thin, I used the suggested values in
Documentation/networking/ixgb.txtas hints:
Settings don't belong in this Documentation file because it a general thing not specific to that hardware.
Quoted 21 lines
my $netdev_max_backlog = 30000;
my $wmem_max = 524287;
my $wmem_default = 128000;
my $rmem_max = 524287;
my $rmem_default = 128000;my $tcp_rmem_min = 4096;
my $tcp_rmem_default = 256000;
my $tcp_rmem_max = 5000000;my $tcp_wmem_min = 4096;
my $tcp_wmem_default = 256000;
my $tcp_wmem_max = 5000000;my $tcp_mem_lo = 10000000;
my $tcp_mem_pressure = 20000000;
my $tcp_mem_high = 30000000;Thanks,
Ben
Re: TCP connection hang problem with 2.6.16.16, e1000 (found work-around)
Quoted 4 lines · Stephen Hemminger
On Wed, 31 May 2006 14:45:13 -0700
Ben Greear <[email protected]> wrote:
Quoted 8 lines
Quoted 5 lines
Ok, I think I found the problem. It appears to be at least partially self-inflicted. I was modifying the kernel buffer settings via
/proc/*to improve performance. I noticed that when I used kernel-defaults, the TCP connection hangs stopped (and ran ~20% slower). My personal opinion is that this is still a bug somewhere in TCP, but at least I have a work-around.If you set bogus values, you can end up consuming all the memory on your system.
I still had some sort of memory..ie I could run other applications and could start a second TCP connection on the same interface. It is true that you can truly OOM systems with bad settings here, but my machine has 2GB of RAM, and I think my setting were low enough that it should not have run out. It also had enough memory left to spam the duplicate TCP acks at around 200kpps in both directions! Manually over-riding the write buffer size had little or no effect: It would still hang.
Quoted 8 lines
Quoted 4 lines
The new settings that appear to work around the TCP connection hang, and still give decent performance are below. Documentation being thin, I used the suggested values in
Documentation/networking/ixgb.txtas hints:Settings don't belong in this Documentation file because it a general thing not specific to that hardware.
The file was describing how to tune for good performance using their NICs.
At least most of the TCP settings are described in the networking/ip-sysctl.txt file, but I could find nothing on the core.rmem_max setting, except in examples in the 10Gbe NIC help texts.
It would be nice to either have the core.* values described in the ip-sysctl.txt file, or perhaps add a new file.
There are tuning tips all over the web too...often quite badly outdated it seems. Might be a good thing to add to the linux-net wiki.
I'd give it a try, but obviously my info is suspect! :P
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 9 lines · Ben Greear
May 31 08:07:00 xeon-dt kernel: irqbalance: page allocation failure. order:1, mode:0x20 May 31 08:07:00 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8 May 31 08:07:00 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c May 31 08:07:00 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e May 31 08:07:00 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206 May 31 08:07:00 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80 May 31 08:07:00 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6 May 31 08:07:00 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e
We specifically calculate the packet size here to avoid a 2-page allocation, yet we still do. This is the bug.
[NET]: Fix SKB_MAX_ORDER calculation
The calculation in SKB_MAX_ORDER is incorrect in that it can cause an overflow across a page boundary which is what it's meant to prevent. In particular, the header length (X) should not be lumped together with skb_shared_info. The latter needs to be aligned properly while the header has no choice but to sit in front of wherever the payload is.
Therefore the correct calculation is to take away the aligned size of skb_shared_info, and then subtract the header length. The resulting quantity L satisfies the following inequality:
SKB_DATA_ALIGN(L + X) + sizeof(struct skb_shared_info) <= PAGE_SIZE
This is the quantity used by alloc_skb to do the actual allocation.
While we're at it, there is no need to round up the header size in sk_stream_alloc_pskb. The only thing this achieves is to make the data payload cache-aligned. However, the data payload is just a byte stream with no alignment requirements. So we should take away the rounding.
Signed-off-by Herbert Xu <[email protected]>
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtindex f8f2347..8761000 100644
include/linux/skbuff.h +2 -3
@@ -39,9 +39,8 @@ #define CHECKSUM_UNNECESSARY 2#define SKB_DATA_ALIGN (X ) (((X) + (SMP_CACHE_BYTES - 1)) & \ ~(SMP_CACHE_BYTES - 1)) -#define SKB_MAX_ORDER (X ,ORDER ) (((PAGE_SIZE << (ORDER)) - (X) - \ - sizeof(struct skb_shared_info)) & \ - ~(SMP_CACHE_BYTES - 1)) +#define SKB_MAX_ORDER (X ,ORDER ) ((PAGE_SIZE << (ORDER)) - (X) - \ + SKB_DATA_ALIGN(sizeof(struct skb_shared_info)))#define SKB_MAX_HEAD (X ) (SKB_MAX_ORDER((X), 0))#define SKB_MAX_ALLOC (SKB_MAX_ORDER(0, 2))
index c9fad6f..9694fdf 100644
include/net/sock.h +1 -1
@@ -1162,7 +1162,7 @@ static inline struct sk_buff *sk_stream_struct sk_buff *skb;int hdr_len; - hdr_len =SKB_DATA_ALIGN (sk->sk_prot->max_header); + hdr_len = sk->sk_prot->max_header; skb =alloc_skb_fclone (size + hdr_len, gfp);if (skb) { skb->truesize += mem;