virtio net regression

Hi,

I've got some hosts that were happily running the 2.6.25.x host kernel, kvm-84, kernel.org kvm modules.
The guests were running 2.6.25 to 2.6.29.x quite happily. Network was using virtio.
Since I upgraded one of the hosts (Intel dual core) to 2.6.29.x yesterday, the virtio network performance of the guests on it dropped dramatically. (for some reason another AMD host did not seem to be affected...)
Here are the tests I performed using wget and scp:
* guest to guest: fast
* guest to host: fast
* host to internet: fast
* guest to internet: slow!!!
I was normally getting ~5MB/s to the host (speed to the internet was limited by the capacity of the DSL line), but since the upgrade the performance had dropped to around 20KB/s!
Strangely enough, I could open many new connections to the guest and get more chunks all at 20KB/s!
I switched the guests to using ne2k_pci and the performance has been restored...

And this is where it gets even weirder...
UDP packets get corrupted using ne2k_pci and rtl8139cp but not with virtio...
So I can get performance or UDP, but not both...

Let me know if there is anything more I can provide to help fix this regression.
I can reproduce the problem quite easily without causing problems on the host.

Cheers
Antoine
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Wireshark was showing a huge amount of invalid packets (wrong checksum)
- - that was the cause of the slowdown.
Simply rebooting the host into 2.6.28.9 fixed *everything*, regardless of whether the guests use virtio or ne2k_pci/etc.
The guests are still running 2.6.29.1, but I am not likely to try that release again on the host anytime soon! Ouch!

Antoine

Quoted 33 lines · Antoine Martin

Hi,

I've got some hosts that were happily running the 2.6.25.x host kernel, kvm-84, kernel.org kvm modules.
The guests were running 2.6.25 to 2.6.29.x quite happily. Network was using virtio.
Since I upgraded one of the hosts (Intel dual core) to 2.6.29.x yesterday, the virtio network performance of the guests on it dropped dramatically. (for some reason another AMD host did not seem to be affected...)
Here are the tests I performed using wget and scp:
* guest to guest: fast
* guest to host: fast
* host to internet: fast
* guest to internet: slow!!!
I was normally getting ~5MB/s to the host (speed to the internet was limited by the capacity of the DSL line), but since the upgrade the performance had dropped to around 20KB/s!
Strangely enough, I could open many new connections to the guest and get more chunks all at 20KB/s!
I switched the guests to using ne2k_pci and the performance has been restored...

And this is where it gets even weirder...
UDP packets get corrupted using ne2k_pci and rtl8139cp but not with virtio...
So I can get performance or UDP, but not both...

Let me know if there is anything more I can provide to help fix this regression.
I can reproduce the problem quite easily without causing problems on the host.

Quoted 7 lines

Cheers
Antoine
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEUEAREKAAYFAknmb/sACgkQGK2zHPGK1ruzgwCWPMvAJzToIMbrE7k2K2FHBQlk dQCcCpDrTufqIN4ZSQs/dMLTQMYtTAU=
=lDW9
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 10 lines · Antoine Martin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Wireshark was showing a huge amount of invalid packets (wrong checksum)
- - that was the cause of the slowdown.
Simply rebooting the host into 2.6.28.9 fixed *everything*, regardless of whether the guests use virtio or ne2k_pci/etc.
The guests are still running 2.6.29.1, but I am not likely to try that release again on the host anytime soon! Ouch!

Strange, no significant tun changes between .28 and .29.

Quoted 12 lines · Avi Kivity

Quoted 10 lines · Antoine Martin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Wireshark was showing a huge amount of invalid packets (wrong checksum)
- - that was the cause of the slowdown.
Simply rebooting the host into 2.6.28.9 fixed *everything*, regardless of whether the guests use virtio or ne2k_pci/etc.
The guests are still running 2.6.29.1, but I am not likely to try that release again on the host anytime soon! Ouch!

Quoted 1 line

Strange, no significant tun changes between .28 and .29.

Sounds to me like it's this:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2f181855a0

davem said he was queueing up for stable, but it's not in yet:

http://kerneltrap.org/mailarchive/linux-netdev/2009/3/30/5337934

I'll check that it's in the queue.

Cheers,
Mark.

Hi,

The bug report below does indeed match everything I have experienced. Upon further inspection, 2.6.28.9 is also affected, just less so.

Unfortunately I have applied the patch:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2f181855a0
And if anything, it made things worse.

Cheers
Antoine

Quoted 26 lines · Mark McLoughlin

Quoted 4 lines · Avi Kivity

Quoted 3 lines · Antoine Martin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 8 lines

Quoted 7 lines

Wireshark was showing a huge amount of invalid packets (wrong checksum)
- - that was the cause of the slowdown.
Simply rebooting the host into 2.6.28.9 fixed *everything*, regardless of whether the guests use virtio or ne2k_pci/etc.
The guests are still running 2.6.29.1, but I am not likely to try that release again on the host anytime soon! Ouch!

Strange, no significant tun changes between .28 and .29.

Sounds to me like it's this:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2f181855a0

davem said he was queueing up for stable, but it's not in yet:

http://kerneltrap.org/mailarchive/linux-netdev/2009/3/30/5337934

I'll check that it's in the queue.

Cheers,
Mark.

Hi

Still getting (some but less) network issues with a 2.6.28.9 host.

Found quite a few of these call traces in the 2.6.29.1 guests:
Guest has 512MB of memory and was not all that busy (just network traffic), so I don't understand why it would fail to allocate a page...

[701453.834571] kjournald: page allocation failure. order:0, mode:0x4020
[701453.834574] Pid: 4806, comm: kjournald Not tainted 2.6.29.1 #4
[701453.834576] Call Trace:
[701453.834578]  <IRQ>  [<ffffffff8027fa48>]
__alloc_pages_internal+0x3e1/0x401
[701453.834586]  [<ffffffff802a1ad4>] __slab_alloc+0x17f/0x4ca
[701453.834590]  [<ffffffff8067e322>] tcp_send_ack+0x23/0x105
[701453.834592]  [<ffffffff8067e322>] tcp_send_ack+0x23/0x105
[701453.834595]  [<ffffffff802a2e66>] __kmalloc_track_caller+0xac/0xe1
[701453.834598]  [<ffffffff8062f97e>] __alloc_skb+0x61/0x11e
[701453.834600]  [<ffffffff8067e322>] tcp_send_ack+0x23/0x105
[701453.834603]  [<ffffffff8067c374>] tcp_rcv_established+0x6c7/0x9e6
[701453.834605]  [<ffffffff80683515>] tcp_v4_do_rcv+0x19e/0x324
[701453.834608]  [<ffffffff80683b23>] tcp_v4_rcv+0x488/0x73b
[701453.834611]  [<ffffffff806499c4>] nf_hook_slow+0x62/0xc3
[701453.834615]  [<ffffffff8066925c>] ip_local_deliver_finish+0x0/0x1ee
[701453.834617]  [<ffffffff80669378>] ip_local_deliver_finish+0x11c/0x1ee
[701453.834620]  [<ffffffff80668fcb>] ip_rcv_finish+0x2cf/0x2e9
[701453.834622]  [<ffffffff80669218>] ip_rcv+0x233/0x277
[701453.834626]  [<ffffffff8055d1e7>] virtnet_poll+0x4ca/0x5ab
[701453.834628]  [<ffffffff80633952>] net_rx_action+0x70/0x143
[701453.834631]  [<ffffffff8024030a>] __do_softirq+0x83/0x145
[701453.834634]  [<ffffffff8020eb7a>] timer_interrupt+0x1a/0x21
[701453.834637]  [<ffffffff8020d35c>] call_softirq+0x1c/0x28
[701453.834639]  [<ffffffff8020e2c0>] do_softirq+0x3c/0x85
[701453.834641]  [<ffffffff80240021>] irq_exit+0x3f/0x7a
[701453.834643]  [<ffffffff8020e59c>] do_IRQ+0x12b/0x14f
[701453.834646]  [<ffffffff8020cad3>] ret_from_intr+0x0/0x29
[701453.834647]  <EOI>  [<ffffffff80621b29>] vp_notify+0x0/0x1c
[701453.834653]  [<ffffffff804b099e>] __make_request+0x3e2/0x425
[701453.834656]  [<ffffffff804af1ff>] generic_make_request+0x338/0x389
[701453.834660]  [<ffffffff802986ce>] end_swap_bio_write+0x0/0x66
[701453.834664]  [<ffffffff802c6643>] bio_alloc_bioset+0x73/0xff
[701453.834666]  [<ffffffff804af30d>] submit_bio+0xbd/0xc4
[701453.834669]  [<ffffffff8072a52a>] _spin_lock+0x5/0x7
[701453.834672]  [<ffffffff802986c4>] swap_writepage+0x9b/0xa5
[701453.834675]  [<ffffffff80283dc1>] shrink_page_list+0x358/0x5ff
[701453.834677]  [<ffffffff80284319>] shrink_list+0x2b1/0x5d8
[701453.834680]  [<ffffffff802806e8>] determine_dirtyable_memory+0xd/0x1d
[701453.834682]  [<ffffffff8028075e>] get_dirty_limits+0x1d/0x24f
[701453.834685]  [<ffffffff802237c4>] pvclock_clocksource_read+0x3a/0x70
[701453.834688]  [<ffffffff802848bd>] shrink_zone+0x27d/0x325
[701453.834692]  [<ffffffff80231733>] resched_task+0x2a/0x75
[701453.834694]  [<ffffffff80280990>] background_writeout+0x0/0xce
[701453.834696]  [<ffffffff802855c7>] try_to_free_pages+0x1fa/0x32d
[701453.834699]  [<ffffffff80282bea>] isolate_pages_global+0x0/0x231
[701453.834701]  [<ffffffff8027f8c0>] __alloc_pages_internal+0x259/0x401
[701453.834705]  [<ffffffff8027aefb>] find_or_create_page+0x48/0x88
[701453.834707]  [<ffffffff802c2c31>] __getblk+0x117/0x29d
[701453.834711]  [<ffffffff80357f00>]
journal_get_descriptor_buffer+0x30/0x76
[701453.834713]  [<ffffffff8035478e>] journal_commit_transaction+0x6da/0xdf0
[701453.834716]  [<ffffffff80244218>] lock_timer_base+0x26/0x4b
[701453.834719]  [<ffffffff8024428f>] try_to_del_timer_sync+0x52/0x5b
[701453.834721]  [<ffffffff8072a484>] _spin_lock_irqsave+0x24/0x2c
[701453.834723]  [<ffffffff803578a0>] kjournald+0xe5/0x214
[701453.834726]  [<ffffffff8024d628>] autoremove_wake_function+0x0/0x2e
[701453.834729]  [<ffffffff803577bb>] kjournald+0x0/0x214
[701453.834731]  [<ffffffff8024d2c7>] kthread+0x47/0x73
[701453.834748]  [<ffffffff8020d25a>] child_rip+0xa/0x20
[701453.834751]  [<ffffffff8024d280>] kthread+0x0/0x73
[701453.834753]  [<ffffffff8020d250>] child_rip+0x0/0x20
[701453.834754] Mem-Info:
[701453.834755] DMA per-cpu:
[701453.834757] CPU    0: hi:    0, btch:   1 usd:   0
[701453.834758] DMA32 per-cpu:
[701453.834760] CPU    0: hi:  186, btch:  31 usd: 165
[701453.834763] Active_anon:674 active_file:43401 inactive_anon:11269
[701453.834764]  inactive_file:53885 unevictable:0 dirty:5182
writeback:70 unstable:0
[701453.834765]  free:749 slab:12132 mapped:9094 pagetables:840 bounce:0
[701453.834768] DMA free:1968kB min:28kB low:32kB high:40kB
active_anon:0kB inactive_anon:0kB active_file:1952kB
inactive_file:2380kB unevictable:0kB present:5440kB pages_scanned:0
all_unreclaimable? no
[701453.834770] lowmem_reserve[]: 0 489 489 489
[701453.834774] DMA32 free:1028kB min:2812kB low:3512kB high:4216kB
active_anon:2696kB inactive_anon:45076kB active_file:171652kB
inactive_file:213160kB unevictable:0kB present:500896kB pages_scanned:0
all_unreclaimable? no
[701453.834777] lowmem_reserve[]: 0 0 0 0
[701453.834779] DMA: 78*4kB 7*8kB 12*16kB 8*32kB 10*64kB 4*128kB 0*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 1968kB
[701453.834785] DMA32: 1*4kB 0*8kB 0*16kB 0*32kB 10*64kB 3*128kB 0*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 1028kB
[701453.834791] 99417 total pagecache pages
[701453.834793] 2101 pages in swap cache
[701453.834794] Swap cache stats: add 8718, delete 6617, find 110037/110217
[701453.834796] Free swap  = 1020652kB
[701453.834797] Total swap = 1048568kB
[701453.836985] 131056 pages RAM
[701453.836987] 4801 pages reserved
[701453.836988] 98664 pages shared
[701453.836990] 34608 pages non-shared

Quoted 41 lines · Antoine Martin

Hi,

The bug report below does indeed match everything I have experienced. Upon further inspection, 2.6.28.9 is also affected, just less so.

Unfortunately I have applied the patch:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2f181855a0
And if anything, it made things worse.

Cheers
Antoine

Quoted 26 lines · Mark McLoughlin

Quoted 4 lines · Avi Kivity

Quoted 3 lines · Antoine Martin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 8 lines

Quoted 7 lines

Wireshark was showing a huge amount of invalid packets (wrong checksum)
- - that was the cause of the slowdown.
Simply rebooting the host into 2.6.28.9 fixed *everything*, regardless of whether the guests use virtio or ne2k_pci/etc.
The guests are still running 2.6.29.1, but I am not likely to try that release again on the host anytime soon! Ouch!

Strange, no significant tun changes between .28 and .29.

Sounds to me like it's this:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2f181855a0

davem said he was queueing up for stable, but it's not in yet:

http://kerneltrap.org/mailarchive/linux-netdev/2009/3/30/5337934

I'll check that it's in the queue.

Cheers,
Mark.

Hi,

Here is another one, any ideas?
These oopses do look quite deep. Is it normal to end up in tcp_send_ack from pdflush??

Cheers
Antoine

[929492.154634] pdflush: page allocation failure. order:0, mode:0x20
[929492.154637] Pid: 291, comm: pdflush Not tainted 2.6.29.2 #5
[929492.154639] Call Trace:
[929492.154641]  <IRQ>  [<ffffffff8027e8bc>]
__alloc_pages_internal+0x3e1/0x401
[929492.154649]  [<ffffffff8055b5ea>] try_fill_recv+0xa1/0x182
[929492.154652]  [<ffffffff8055c1fc>] virtnet_poll+0x533/0x5ab
[929492.154655]  [<ffffffff80632bba>] net_rx_action+0x70/0x143
[929492.154658]  [<ffffffff8023f18c>] __do_softirq+0x83/0x123
[929492.154661]  [<ffffffff8020d35c>] call_softirq+0x1c/0x28
[929492.154664]  [<ffffffff8020e2c0>] do_softirq+0x3c/0x85
[929492.154666]  [<ffffffff8023eea3>] irq_exit+0x3f/0x7a
[929492.154668]  [<ffffffff8020e59c>] do_IRQ+0x12b/0x14f
[929492.154670]  [<ffffffff8020cad3>] ret_from_intr+0x0/0x29
[929492.154672]  <EOI>  [<ffffffff802c22b1>]
__set_page_dirty_buffers+0x0/0x8f
[929492.154677]  [<ffffffff8031702b>] bget_one+0x0/0xb
[929492.154680]  [<ffffffff80316fa2>] walk_page_buffers+0x2/0x8b
[929492.154682]  [<ffffffff803185bc>] ext3_ordered_writepage+0xae/0x134
[929492.154685]  [<ffffffff8027ea46>] __writepage+0xa/0x25
[929492.154687]  [<ffffffff8027f19f>] write_cache_pages+0x206/0x322
[929492.154689]  [<ffffffff8027ea3c>] __writepage+0x0/0x25
[929492.154691]  [<ffffffff8027f2fe>] do_writepages+0x27/0x2d
[929492.154694]  [<ffffffff802bd3f6>] __writeback_single_inode+0x1a7/0x3b5
[929492.154696]  [<ffffffff8020a68c>] __switch_to+0xb4/0x38c
[929492.154698]  [<ffffffff802bda76>] generic_sync_sb_inodes+0x2a7/0x458
[929492.154701]  [<ffffffff802bde00>] writeback_inodes+0x8d/0xe6
[929492.154704]  [<ffffffff807296e2>] _spin_lock+0x5/0x7
[929492.155056]  [<ffffffff8027f432>] wb_kupdate+0x9f/0x116
[929492.155058]  [<ffffffff80280095>] pdflush+0x14b/0x202
[929492.155061]  [<ffffffff8027f393>] wb_kupdate+0x0/0x116
[929492.155063]  [<ffffffff8027ff4a>] pdflush+0x0/0x202
[929492.155065]  [<ffffffff8027ff4a>] pdflush+0x0/0x202
[929492.155068]  [<ffffffff8024c127>] kthread+0x47/0x73
[929492.155070]  [<ffffffff8020d25a>] child_rip+0xa/0x20
[929492.155072]  [<ffffffff8024c0e0>] kthread+0x0/0x73
[929492.183142]  [<ffffffff8020d250>] child_rip+0x0/0x20
[929492.183145] Mem-Info:
[929492.183147] DMA per-cpu:
[929492.183149] CPU    0: hi:    0, btch:   1 usd:   0
[929492.183151] DMA32 per-cpu:
[929492.183154] CPU    0: hi:  186, btch:  31 usd: 184
[929492.183158] Active_anon:2755 active_file:39849 inactive_anon:2972
[929492.183159]  inactive_file:70353 unevictable:0 dirty:4172
writeback:1580 unstable:0
[929492.183161]  free:734 slab:5619 mapped:15047 pagetables:927 bounce:0
[929492.183166] DMA free:1968kB min:28kB low:32kB high:40kB
active_anon:0kB inactive_anon:40kB active_file:2116kB
inactive_file:1880kB unevictable:0kB present:5448kB pages_scanned:0
all_unreclaimable? no
[929492.183169] lowmem_reserve[]: 0 489 489 489
[929492.183176] DMA32 free:968kB min:2812kB low:3512kB high:4216kB
active_anon:11020kB inactive_anon:11848kB active_file:157280kB
inactive_file:279532kB unevictable:0kB present:500896kB pages_scanned:0
all_unreclaimable? no
[929492.183180] lowmem_reserve[]: 0 0 0 0
[929492.183183] DMA: 6*4kB 2*8kB 3*16kB 1*32kB 1*64kB 2*128kB 0*256kB
1*512kB 1*1024kB 0*2048kB 0*4096kB = 1976kB
[929492.183235] DMA32: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 3*128kB 2*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 968kB
[929492.183244] 110992 total pagecache pages
[929492.183246] 739 pages in swap cache
[929492.183248] Swap cache stats: add 8996, delete 8257, find 92604/93191
[929492.183250] Free swap  = 1040016kB
[929492.183252] Total swap = 1048568kB
[929492.186003] 131056 pages RAM
[929492.186006] 4799 pages reserved
[929492.186007] 44697 pages shared
[929492.186008] 90516 pages non-shared
[930274.380075] eth0: no IPv6 routers present

Quoted 149 lines · Antoine Martin

Hi

Still getting (some but less) network issues with a 2.6.28.9 host.

Found quite a few of these call traces in the 2.6.29.1 guests:
Guest has 512MB of memory and was not all that busy (just network traffic), so I don't understand why it would fail to allocate a page...

[701453.834571] kjournald: page allocation failure. order:0, mode:0x4020
[701453.834574] Pid: 4806, comm: kjournald Not tainted 2.6.29.1 #4
[701453.834576] Call Trace:
[701453.834578]  <IRQ>  [<ffffffff8027fa48>]
__alloc_pages_internal+0x3e1/0x401
[701453.834586]  [<ffffffff802a1ad4>] __slab_alloc+0x17f/0x4ca
[701453.834590]  [<ffffffff8067e322>] tcp_send_ack+0x23/0x105
[701453.834592]  [<ffffffff8067e322>] tcp_send_ack+0x23/0x105
[701453.834595]  [<ffffffff802a2e66>] __kmalloc_track_caller+0xac/0xe1
[701453.834598]  [<ffffffff8062f97e>] __alloc_skb+0x61/0x11e
[701453.834600]  [<ffffffff8067e322>] tcp_send_ack+0x23/0x105
[701453.834603]  [<ffffffff8067c374>] tcp_rcv_established+0x6c7/0x9e6
[701453.834605]  [<ffffffff80683515>] tcp_v4_do_rcv+0x19e/0x324
[701453.834608]  [<ffffffff80683b23>] tcp_v4_rcv+0x488/0x73b
[701453.834611]  [<ffffffff806499c4>] nf_hook_slow+0x62/0xc3
[701453.834615]  [<ffffffff8066925c>] ip_local_deliver_finish+0x0/0x1ee
[701453.834617]  [<ffffffff80669378>] ip_local_deliver_finish+0x11c/0x1ee
[701453.834620]  [<ffffffff80668fcb>] ip_rcv_finish+0x2cf/0x2e9
[701453.834622]  [<ffffffff80669218>] ip_rcv+0x233/0x277
[701453.834626]  [<ffffffff8055d1e7>] virtnet_poll+0x4ca/0x5ab
[701453.834628]  [<ffffffff80633952>] net_rx_action+0x70/0x143
[701453.834631]  [<ffffffff8024030a>] __do_softirq+0x83/0x145
[701453.834634]  [<ffffffff8020eb7a>] timer_interrupt+0x1a/0x21
[701453.834637]  [<ffffffff8020d35c>] call_softirq+0x1c/0x28
[701453.834639]  [<ffffffff8020e2c0>] do_softirq+0x3c/0x85
[701453.834641]  [<ffffffff80240021>] irq_exit+0x3f/0x7a
[701453.834643]  [<ffffffff8020e59c>] do_IRQ+0x12b/0x14f
[701453.834646]  [<ffffffff8020cad3>] ret_from_intr+0x0/0x29
[701453.834647]  <EOI>  [<ffffffff80621b29>] vp_notify+0x0/0x1c
[701453.834653]  [<ffffffff804b099e>] __make_request+0x3e2/0x425
[701453.834656]  [<ffffffff804af1ff>] generic_make_request+0x338/0x389
[701453.834660]  [<ffffffff802986ce>] end_swap_bio_write+0x0/0x66
[701453.834664]  [<ffffffff802c6643>] bio_alloc_bioset+0x73/0xff
[701453.834666]  [<ffffffff804af30d>] submit_bio+0xbd/0xc4
[701453.834669]  [<ffffffff8072a52a>] _spin_lock+0x5/0x7
[701453.834672]  [<ffffffff802986c4>] swap_writepage+0x9b/0xa5
[701453.834675]  [<ffffffff80283dc1>] shrink_page_list+0x358/0x5ff
[701453.834677]  [<ffffffff80284319>] shrink_list+0x2b1/0x5d8
[701453.834680]  [<ffffffff802806e8>] determine_dirtyable_memory+0xd/0x1d
[701453.834682]  [<ffffffff8028075e>] get_dirty_limits+0x1d/0x24f
[701453.834685]  [<ffffffff802237c4>] pvclock_clocksource_read+0x3a/0x70
[701453.834688]  [<ffffffff802848bd>] shrink_zone+0x27d/0x325
[701453.834692]  [<ffffffff80231733>] resched_task+0x2a/0x75
[701453.834694]  [<ffffffff80280990>] background_writeout+0x0/0xce
[701453.834696]  [<ffffffff802855c7>] try_to_free_pages+0x1fa/0x32d
[701453.834699]  [<ffffffff80282bea>] isolate_pages_global+0x0/0x231
[701453.834701]  [<ffffffff8027f8c0>] __alloc_pages_internal+0x259/0x401
[701453.834705]  [<ffffffff8027aefb>] find_or_create_page+0x48/0x88
[701453.834707]  [<ffffffff802c2c31>] __getblk+0x117/0x29d
[701453.834711]  [<ffffffff80357f00>]
journal_get_descriptor_buffer+0x30/0x76
[701453.834713]  [<ffffffff8035478e>] journal_commit_transaction+0x6da/0xdf0
[701453.834716]  [<ffffffff80244218>] lock_timer_base+0x26/0x4b
[701453.834719]  [<ffffffff8024428f>] try_to_del_timer_sync+0x52/0x5b
[701453.834721]  [<ffffffff8072a484>] _spin_lock_irqsave+0x24/0x2c
[701453.834723]  [<ffffffff803578a0>] kjournald+0xe5/0x214
[701453.834726]  [<ffffffff8024d628>] autoremove_wake_function+0x0/0x2e
[701453.834729]  [<ffffffff803577bb>] kjournald+0x0/0x214
[701453.834731]  [<ffffffff8024d2c7>] kthread+0x47/0x73
[701453.834748]  [<ffffffff8020d25a>] child_rip+0xa/0x20
[701453.834751]  [<ffffffff8024d280>] kthread+0x0/0x73
[701453.834753]  [<ffffffff8020d250>] child_rip+0x0/0x20
[701453.834754] Mem-Info:
[701453.834755] DMA per-cpu:
[701453.834757] CPU    0: hi:    0, btch:   1 usd:   0
[701453.834758] DMA32 per-cpu:
[701453.834760] CPU    0: hi:  186, btch:  31 usd: 165
[701453.834763] Active_anon:674 active_file:43401 inactive_anon:11269
[701453.834764]  inactive_file:53885 unevictable:0 dirty:5182
writeback:70 unstable:0
[701453.834765]  free:749 slab:12132 mapped:9094 pagetables:840 bounce:0
[701453.834768] DMA free:1968kB min:28kB low:32kB high:40kB
active_anon:0kB inactive_anon:0kB active_file:1952kB
inactive_file:2380kB unevictable:0kB present:5440kB pages_scanned:0
all_unreclaimable? no
[701453.834770] lowmem_reserve[]: 0 489 489 489
[701453.834774] DMA32 free:1028kB min:2812kB low:3512kB high:4216kB
active_anon:2696kB inactive_anon:45076kB active_file:171652kB
inactive_file:213160kB unevictable:0kB present:500896kB pages_scanned:0
all_unreclaimable? no
[701453.834777] lowmem_reserve[]: 0 0 0 0
[701453.834779] DMA: 78*4kB 7*8kB 12*16kB 8*32kB 10*64kB 4*128kB 0*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 1968kB
[701453.834785] DMA32: 1*4kB 0*8kB 0*16kB 0*32kB 10*64kB 3*128kB 0*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 1028kB
[701453.834791] 99417 total pagecache pages
[701453.834793] 2101 pages in swap cache
[701453.834794] Swap cache stats: add 8718, delete 6617, find 110037/110217
[701453.834796] Free swap  = 1020652kB
[701453.834797] Total swap = 1048568kB
[701453.836985] 131056 pages RAM
[701453.836987] 4801 pages reserved
[701453.836988] 98664 pages shared
[701453.836990] 34608 pages non-shared

Quoted 41 lines · Antoine Martin

Hi,

The bug report below does indeed match everything I have experienced. Upon further inspection, 2.6.28.9 is also affected, just less so.

Unfortunately I have applied the patch:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2f181855a0
And if anything, it made things worse.

Cheers
Antoine

Quoted 26 lines · Mark McLoughlin

Quoted 4 lines · Avi Kivity

Quoted 3 lines · Antoine Martin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 8 lines

Quoted 7 lines

Wireshark was showing a huge amount of invalid packets (wrong checksum)
- - that was the cause of the slowdown.
Simply rebooting the host into 2.6.28.9 fixed *everything*, regardless of whether the guests use virtio or ne2k_pci/etc.
The guests are still running 2.6.29.1, but I am not likely to try that release again on the host anytime soon! Ouch!

Strange, no significant tun changes between .28 and .29.

Sounds to me like it's this:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2f181855a0

davem said he was queueing up for stable, but it's not in yet:

http://kerneltrap.org/mailarchive/linux-netdev/2009/3/30/5337934

I'll check that it's in the queue.

Cheers,
Mark.

Re-sending as this does not seem to have made it to the list.

Hi,

Here is another one, any ideas?
These oopses do look quite deep. Is it normal to end up in tcp_send_ack from pdflush??

Cheers
Antoine

[929492.154634] pdflush: page allocation failure. order:0, mode:0x20

[929492.154637] Pid: 291, comm: pdflush Not tainted 2.6.29.2 #5
[929492.154639] Call Trace:
[929492.154641]  <IRQ>  [<ffffffff8027e8bc>]
__alloc_pages_internal+0x3e1/0x401
[929492.154649]  [<ffffffff8055b5ea>] try_fill_recv+0xa1/0x182
[929492.154652]  [<ffffffff8055c1fc>] virtnet_poll+0x533/0x5ab
[929492.154655]  [<ffffffff80632bba>] net_rx_action+0x70/0x143
[929492.154658]  [<ffffffff8023f18c>] __do_softirq+0x83/0x123
[929492.154661]  [<ffffffff8020d35c>] call_softirq+0x1c/0x28
[929492.154664]  [<ffffffff8020e2c0>] do_softirq+0x3c/0x85
[929492.154666]  [<ffffffff8023eea3>] irq_exit+0x3f/0x7a
[929492.154668]  [<ffffffff8020e59c>] do_IRQ+0x12b/0x14f
[929492.154670]  [<ffffffff8020cad3>] ret_from_intr+0x0/0x29
[929492.154672]  <EOI>  [<ffffffff802c22b1>]
__set_page_dirty_buffers+0x0/0x8f
[929492.154677]  [<ffffffff8031702b>] bget_one+0x0/0xb
[929492.154680]  [<ffffffff80316fa2>] walk_page_buffers+0x2/0x8b
[929492.154682]  [<ffffffff803185bc>] ext3_ordered_writepage+0xae/0x134
[929492.154685]  [<ffffffff8027ea46>] __writepage+0xa/0x25
[929492.154687]  [<ffffffff8027f19f>] write_cache_pages+0x206/0x322
[929492.154689]  [<ffffffff8027ea3c>] __writepage+0x0/0x25
[929492.154691]  [<ffffffff8027f2fe>] do_writepages+0x27/0x2d
[929492.154694]  [<ffffffff802bd3f6>] __writeback_single_inode+0x1a7/0x3b5
[929492.154696]  [<ffffffff8020a68c>] __switch_to+0xb4/0x38c
[929492.154698]  [<ffffffff802bda76>] generic_sync_sb_inodes+0x2a7/0x458
[929492.154701]  [<ffffffff802bde00>] writeback_inodes+0x8d/0xe6
[929492.154704]  [<ffffffff807296e2>] _spin_lock+0x5/0x7
[929492.155056]  [<ffffffff8027f432>] wb_kupdate+0x9f/0x116
[929492.155058]  [<ffffffff80280095>] pdflush+0x14b/0x202
[929492.155061]  [<ffffffff8027f393>] wb_kupdate+0x0/0x116
[929492.155063]  [<ffffffff8027ff4a>] pdflush+0x0/0x202
[929492.155065]  [<ffffffff8027ff4a>] pdflush+0x0/0x202
[929492.155068]  [<ffffffff8024c127>] kthread+0x47/0x73
[929492.155070]  [<ffffffff8020d25a>] child_rip+0xa/0x20
[929492.155072]  [<ffffffff8024c0e0>] kthread+0x0/0x73
[929492.183142]  [<ffffffff8020d250>] child_rip+0x0/0x20
[929492.183145] Mem-Info:
[929492.183147] DMA per-cpu:
[929492.183149] CPU    0: hi:    0, btch:   1 usd:   0
[929492.183151] DMA32 per-cpu:
[929492.183154] CPU    0: hi:  186, btch:  31 usd: 184
[929492.183158] Active_anon:2755 active_file:39849 inactive_anon:2972
[929492.183159]  inactive_file:70353 unevictable:0 dirty:4172
writeback:1580 unstable:0
[929492.183161]  free:734 slab:5619 mapped:15047 pagetables:927 bounce:0
[929492.183166] DMA free:1968kB min:28kB low:32kB high:40kB
active_anon:0kB inactive_anon:40kB active_file:2116kB
inactive_file:1880kB unevictable:0kB present:5448kB pages_scanned:0
all_unreclaimable? no
[929492.183169] lowmem_reserve[]: 0 489 489 489
[929492.183176] DMA32 free:968kB min:2812kB low:3512kB high:4216kB
active_anon:11020kB inactive_anon:11848kB active_file:157280kB
inactive_file:279532kB unevictable:0kB present:500896kB pages_scanned:0
all_unreclaimable? no
[929492.183180] lowmem_reserve[]: 0 0 0 0
[929492.183183] DMA: 6*4kB 2*8kB 3*16kB 1*32kB 1*64kB 2*128kB 0*256kB
1*512kB 1*1024kB 0*2048kB 0*4096kB = 1976kB
[929492.183235] DMA32: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 3*128kB 2*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 968kB
[929492.183244] 110992 total pagecache pages
[929492.183246] 739 pages in swap cache

[929492.183248] Swap cache stats: add 8996, delete 8257, find 92604/93191
[929492.183250] Free swap  = 1040016kB
[929492.183252] Total swap = 1048568kB
[929492.186003] 131056 pages RAM
[929492.186006] 4799 pages reserved
[929492.186007] 44697 pages shared
[929492.186008] 90516 pages non-shared
[930274.380075] eth0: no IPv6 routers present

Quoted 7 lines · Antoine Martin

Hi,

Here is another one, any ideas?
These oopses do look quite deep. Is it normal to end up in tcp_send_ack from pdflush??

I think it can happen anywhere, part of the net softirq.

Quoted 6 lines

Cheers
Antoine

[929492.154634] pdflush: page allocation failure. order:0, mode:0x20

You're out of memory.

How much memory did you allocate to the guest? did you balloon it?

Quoted 64 lines

[929492.154637] Pid: 291, comm: pdflush Not tainted 2.6.29.2 #5
[929492.154639] Call Trace:
[929492.154641]  <IRQ>  [<ffffffff8027e8bc>]
__alloc_pages_internal+0x3e1/0x401
[929492.154649]  [<ffffffff8055b5ea>] try_fill_recv+0xa1/0x182
[929492.154652]  [<ffffffff8055c1fc>] virtnet_poll+0x533/0x5ab
[929492.154655]  [<ffffffff80632bba>] net_rx_action+0x70/0x143
[929492.154658]  [<ffffffff8023f18c>] __do_softirq+0x83/0x123
[929492.154661]  [<ffffffff8020d35c>] call_softirq+0x1c/0x28
[929492.154664]  [<ffffffff8020e2c0>] do_softirq+0x3c/0x85
[929492.154666]  [<ffffffff8023eea3>] irq_exit+0x3f/0x7a
[929492.154668]  [<ffffffff8020e59c>] do_IRQ+0x12b/0x14f
[929492.154670]  [<ffffffff8020cad3>] ret_from_intr+0x0/0x29
[929492.154672]  <EOI>  [<ffffffff802c22b1>]
__set_page_dirty_buffers+0x0/0x8f
[929492.154677]  [<ffffffff8031702b>] bget_one+0x0/0xb
[929492.154680]  [<ffffffff80316fa2>] walk_page_buffers+0x2/0x8b
[929492.154682]  [<ffffffff803185bc>] ext3_ordered_writepage+0xae/0x134
[929492.154685]  [<ffffffff8027ea46>] __writepage+0xa/0x25
[929492.154687]  [<ffffffff8027f19f>] write_cache_pages+0x206/0x322
[929492.154689]  [<ffffffff8027ea3c>] __writepage+0x0/0x25
[929492.154691]  [<ffffffff8027f2fe>] do_writepages+0x27/0x2d
[929492.154694]  [<ffffffff802bd3f6>]
__writeback_single_inode+0x1a7/0x3b5
[929492.154696]  [<ffffffff8020a68c>] __switch_to+0xb4/0x38c
[929492.154698]  [<ffffffff802bda76>] generic_sync_sb_inodes+0x2a7/0x458
[929492.154701]  [<ffffffff802bde00>] writeback_inodes+0x8d/0xe6
[929492.154704]  [<ffffffff807296e2>] _spin_lock+0x5/0x7
[929492.155056]  [<ffffffff8027f432>] wb_kupdate+0x9f/0x116
[929492.155058]  [<ffffffff80280095>] pdflush+0x14b/0x202
[929492.155061]  [<ffffffff8027f393>] wb_kupdate+0x0/0x116
[929492.155063]  [<ffffffff8027ff4a>] pdflush+0x0/0x202
[929492.155065]  [<ffffffff8027ff4a>] pdflush+0x0/0x202
[929492.155068]  [<ffffffff8024c127>] kthread+0x47/0x73
[929492.155070]  [<ffffffff8020d25a>] child_rip+0xa/0x20
[929492.155072]  [<ffffffff8024c0e0>] kthread+0x0/0x73
[929492.183142]  [<ffffffff8020d250>] child_rip+0x0/0x20
[929492.183145] Mem-Info:
[929492.183147] DMA per-cpu:
[929492.183149] CPU    0: hi:    0, btch:   1 usd:   0
[929492.183151] DMA32 per-cpu:
[929492.183154] CPU    0: hi:  186, btch:  31 usd: 184
[929492.183158] Active_anon:2755 active_file:39849 inactive_anon:2972
[929492.183159]  inactive_file:70353 unevictable:0 dirty:4172
writeback:1580 unstable:0
[929492.183161]  free:734 slab:5619 mapped:15047 pagetables:927 bounce:0
[929492.183166] DMA free:1968kB min:28kB low:32kB high:40kB
active_anon:0kB inactive_anon:40kB active_file:2116kB
inactive_file:1880kB unevictable:0kB present:5448kB pages_scanned:0
all_unreclaimable? no
[929492.183169] lowmem_reserve[]: 0 489 489 489
[929492.183176] DMA32 free:968kB min:2812kB low:3512kB high:4216kB
active_anon:11020kB inactive_anon:11848kB active_file:157280kB
inactive_file:279532kB unevictable:0kB present:500896kB pages_scanned:0
all_unreclaimable? no
[929492.183180] lowmem_reserve[]: 0 0 0 0
[929492.183183] DMA: 6*4kB 2*8kB 3*16kB 1*32kB 1*64kB 2*128kB 0*256kB
1*512kB 1*1024kB 0*2048kB 0*4096kB = 1976kB
[929492.183235] DMA32: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 3*128kB 2*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 968kB
[929492.183244] 110992 total pagecache pages
[929492.183246] 739 pages in swap cache
[929492.183248] Swap cache stats: add 8996, delete 8257, find

Quoted 10 lines

92604/93191

[929492.183250] Free swap  = 1040016kB
[929492.183252] Total swap = 1048568kB
[929492.186003] 131056 pages RAM
[929492.186006] 4799 pages reserved
[929492.186007] 44697 pages shared
[929492.186008] 90516 pages non-shared
[930274.380075] eth0: no IPv6 routers present

Strange, seems to be a bit of free memory here.

Quoted 2 lines

You're out of memory.

That's quite odd, the guest wasn't even hitting the swap at the tine.

Quoted 2 lines

Strange, seems to be a bit of free memory here.

There should be lots, all this host is doing is apache+sftp...

Assuming I can make it re-occur (stress testing it?), how would I dig further to find the cause of this memory exhaustion? /proc/meminfo and friends?

Quoted 3 lines · Antoine Martin

Quoted 2 lines

You're out of memory.

That's quite odd, the guest wasn't even hitting the swap at the tine.

But you do have swap enabled?

Quoted 8 lines

Quoted 2 lines

Strange, seems to be a bit of free memory here.

There should be lots, all this host is doing is apache+sftp...

Assuming I can make it re-occur (stress testing it?), how would I dig further to find the cause of this memory exhaustion? /proc/meminfo and friends?

Yes please. Maybe virtio is leaking memory.

Quoted 3 lines

But you do have swap enabled?

Yes.

I always do this on the guests as it seems fairer to let the guests use swap when they need the extra memory rather than over-committing too much memory on the host. Although it would probably be more efficient overall to let the host manage all swapping.
It consumes more I/O bandwidth, but most guest's memory stay "warm" no matter what other guests are doing.
Does that sound reasonable?