Skip to contentSkip to conversation

win7 bad i/o performance, high insn_emulation and exists

DISCUSSION Peter Lieven 3 participants 19 messages Feb 20, 2012 – Feb 21, 2012 KVM

lore
Share & tools
lore

Peter Lieven

author
#0 lore raw

Hi,

I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest.

If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat

efer_reload 0 0
exits 2260976 79620
fpu_reload 6197 11

halt_exits                114734      5011
halt_wakeup               111195      4876
host_state_reload        1499659     60962
hypercalls                     0         0
insn_emulation           1577325     58488
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  943949     40249
irq_exits                 108679      5434
irq_injections            236545     10788
irq_window                  7606       246
largepages                   672         5
mmio_exits                460020     16082
mmu_cache_miss               119         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write              13474         9
mmu_recycled                   0         0
mmu_shadow_zapped            141         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   22803        35
pf_guest                       0         0
remote_tlb_flush             239         2
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  20933         0

If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane.

efer_reload 0 0
exits 6132004 17931
fpu_reload 19863 3

halt_exits                264961      3083
halt_wakeup               236468      2959
host_state_reload        1104468      3104
hypercalls                     0         0
insn_emulation           1417443      7518
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  869380      2795
irq_exits                 253501      2362
irq_injections            616967      6804
irq_window                201186      2161
largepages                  1019         0
mmio_exits                205268         0
mmu_cache_miss               192         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write            7440546         0
mmu_recycled                   0         0
mmu_shadow_zapped            259         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   38529        30
pf_guest                       0         0
remote_tlb_flush             761         1
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0

I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8
) - no improvement.

My commandline:
/usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on -device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native -m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet

What further information is needed to debug this further?

Thanks,
Peter

#1 lore raw
Quoted 12 lines · Peter Lieven

Hi,

I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest.

If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat

efer_reload 0 0
exits 2260976 79620
fpu_reload 6197 11

Quoted 8 lines
halt_exits                114734      5011
halt_wakeup               111195      4876
host_state_reload        1499659     60962
hypercalls                     0         0
insn_emulation           1577325     58488
insn_emulation_fail            0         0
invlpg                         0         0

io_exits 943949 40249

Hmm, too many of those.

Quoted 73 lines

irq_exits 108679 5434
irq_injections 236545 10788
irq_window 7606 246
largepages 672 5
mmio_exits 460020 16082

mmu_cache_miss               119         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write              13474         9
mmu_recycled                   0         0
mmu_shadow_zapped            141         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   22803        35
pf_guest                       0         0
remote_tlb_flush             239         2
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  20933         0

If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane.

efer_reload 0 0
exits 6132004 17931
fpu_reload 19863 3

halt_exits                264961      3083
halt_wakeup               236468      2959
host_state_reload        1104468      3104
hypercalls                     0         0
insn_emulation           1417443      7518
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  869380      2795
irq_exits                 253501      2362
irq_injections            616967      6804
irq_window                201186      2161
largepages                  1019         0
mmio_exits                205268         0
mmu_cache_miss               192         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write            7440546         0
mmu_recycled                   0         0
mmu_shadow_zapped            259         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   38529        30
pf_guest                       0         0
remote_tlb_flush             761         1
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0

I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase
performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 ) - no improvement.

My commandline:
/usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on

(quoted diff) +0 -2
-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native
-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name

lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet

Quoted 2 lines

What further information is needed to debug this further?

Which kernel version (looks like something recent)? Which host CPU (looks like something old)? Which Windows' virtio drivers are you using?

Take a trace like described here http://www.linux-kvm.org/page/Tracing
(with -no-hpet please).

Try to use -cpu host,+x2apic. It may help Linux guest performance.

--
Gleb.

#2 lore raw
Quoted 96 lines · Gleb Natapov
Quoted 12 lines · Peter Lieven

Hi,

I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest.

If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat

efer_reload 0 0
exits 2260976 79620
fpu_reload 6197 11

Quoted 8 lines
halt_exits                114734      5011
halt_wakeup               111195      4876
host_state_reload        1499659     60962
hypercalls                     0         0
insn_emulation           1577325     58488
insn_emulation_fail            0         0
invlpg                         0         0

io_exits 943949 40249

Hmm, too many of those.

Quoted 73 lines

irq_exits 108679 5434
irq_injections 236545 10788
irq_window 7606 246
largepages 672 5
mmio_exits 460020 16082

mmu_cache_miss               119         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write              13474         9
mmu_recycled                   0         0
mmu_shadow_zapped            141         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   22803        35
pf_guest                       0         0
remote_tlb_flush             239         2
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  20933         0

If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane.

efer_reload 0 0
exits 6132004 17931
fpu_reload 19863 3

halt_exits                264961      3083
halt_wakeup               236468      2959
host_state_reload        1104468      3104
hypercalls                     0         0
insn_emulation           1417443      7518
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  869380      2795
irq_exits                 253501      2362
irq_injections            616967      6804
irq_window                201186      2161
largepages                  1019         0
mmio_exits                205268         0
mmu_cache_miss               192         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write            7440546         0
mmu_recycled                   0         0
mmu_shadow_zapped            259         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   38529        30
pf_guest                       0         0
remote_tlb_flush             761         1
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0

I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase
performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 ) - no improvement.

My commandline:
/usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on

(quoted diff) +0 -2
-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native
-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name

lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet

Quoted 4 lines
Quoted 2 lines

What further information is needed to debug this further?

Which kernel version (looks like something recent)? Which host CPU (looks like something old)?

Output of cat /proc/cpuinfo

Quoted 5 lines

Which Windows' virtio drivers are you using?

Take a trace like described here http://www.linux-kvm.org/page/Tracing
(with -no-hpet please).

And also "info pci" output from qemu monitor while we are at it.

Quoted 8 lines

Try to use -cpu host,+x2apic. It may help Linux guest performance.

--
Gleb.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Gleb.

author
#3 lore raw
Quoted 101 lines · Gleb Natapov
Quoted 96 lines · Gleb Natapov
Quoted 12 lines · Peter Lieven

Hi,

I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest.

If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat

efer_reload 0 0
exits 2260976 79620
fpu_reload 6197 11

Quoted 8 lines
halt_exits                114734      5011
halt_wakeup               111195      4876
host_state_reload        1499659     60962
hypercalls                     0         0
insn_emulation           1577325     58488
insn_emulation_fail            0         0
invlpg                         0         0

io_exits 943949 40249

Hmm, too many of those.

Quoted 73 lines

irq_exits 108679 5434
irq_injections 236545 10788
irq_window 7606 246
largepages 672 5
mmio_exits 460020 16082

mmu_cache_miss               119         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write              13474         9
mmu_recycled                   0         0
mmu_shadow_zapped            141         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   22803        35
pf_guest                       0         0
remote_tlb_flush             239         2
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  20933         0

If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane.

efer_reload 0 0
exits 6132004 17931
fpu_reload 19863 3

halt_exits                264961      3083
halt_wakeup               236468      2959
host_state_reload        1104468      3104
hypercalls                     0         0
insn_emulation           1417443      7518
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  869380      2795
irq_exits                 253501      2362
irq_injections            616967      6804
irq_window                201186      2161
largepages                  1019         0
mmio_exits                205268         0
mmu_cache_miss               192         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write            7440546         0
mmu_recycled                   0         0
mmu_shadow_zapped            259         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   38529        30
pf_guest                       0         0
remote_tlb_flush             761         1
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0

I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase
performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 ) - no improvement.

My commandline:
/usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on

(quoted diff) +0 -2
-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native
-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name

lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet

Quoted 4 lines
Quoted 2 lines

What further information is needed to debug this further?

Which kernel version (looks like something recent)? Which host CPU (looks like something old)?

Quoted 8 lines

Output of cat /proc/cpuinfo

Quoted 5 lines

Which Windows' virtio drivers are you using?

Take a trace like described here http://www.linux-kvm.org/page/Tracing
(with -no-hpet please).

And also "info pci" output from qemu monitor while we are at it.

here we go. i stumbled across IRQ 0 ?

QEMU 1.0 monitor - type 'help' for more information (qemu) info pci
info pci
Bus 0, device 0, function 0:

    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc080 [0xc08f].
      id ""
  Bus  0, device   1, function 2:
    USB controller: PCI device 8086:7020
      IRQ 5.
      BAR4: I/O at 0xc040 [0xc05f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1234:1111
      BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0xc060 [0xc07f].
      BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   4, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xc000 [0xc03f].
      BAR1: 32 bit memory at 0xfebf1000 [0xfebf1fff].
      id ""
Quoted 10 lines
Quoted 8 lines

Try to use -cpu host,+x2apic. It may help Linux guest performance.

--
Gleb.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Gleb.

author
#4 lore raw
In reply to Gleb Natapov
Quoted 101 lines · Gleb Natapov
Quoted 96 lines · Gleb Natapov
Quoted 12 lines · Peter Lieven

Hi,

I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest.

If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat

efer_reload 0 0
exits 2260976 79620
fpu_reload 6197 11

Quoted 8 lines
halt_exits                114734      5011
halt_wakeup               111195      4876
host_state_reload        1499659     60962
hypercalls                     0         0
insn_emulation           1577325     58488
insn_emulation_fail            0         0
invlpg                         0         0

io_exits 943949 40249

Hmm, too many of those.

Quoted 73 lines

irq_exits 108679 5434
irq_injections 236545 10788
irq_window 7606 246
largepages 672 5
mmio_exits 460020 16082

mmu_cache_miss               119         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write              13474         9
mmu_recycled                   0         0
mmu_shadow_zapped            141         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   22803        35
pf_guest                       0         0
remote_tlb_flush             239         2
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  20933         0

If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane.

efer_reload 0 0
exits 6132004 17931
fpu_reload 19863 3

halt_exits                264961      3083
halt_wakeup               236468      2959
host_state_reload        1104468      3104
hypercalls                     0         0
insn_emulation           1417443      7518
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  869380      2795
irq_exits                 253501      2362
irq_injections            616967      6804
irq_window                201186      2161
largepages                  1019         0
mmio_exits                205268         0
mmu_cache_miss               192         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write            7440546         0
mmu_recycled                   0         0
mmu_shadow_zapped            259         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   38529        30
pf_guest                       0         0
remote_tlb_flush             761         1
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0

I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase
performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 ) - no improvement.

My commandline:
/usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on

(quoted diff) +0 -2
-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native
-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name

lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet

Quoted 4 lines
Quoted 2 lines

What further information is needed to debug this further?

Which kernel version (looks like something recent)? Which host CPU (looks like something old)?

Quoted 8 lines

Output of cat /proc/cpuinfo

Quoted 5 lines

Which Windows' virtio drivers are you using?

Take a trace like described here http://www.linux-kvm.org/page/Tracing
(with -no-hpet please).

And also "info pci" output from qemu monitor while we are at it.

here is the output while i was tracing. you can download the trace i took while i did a ftp transfer from the vm:

-> http://82.141.21.156/report.txt.gz

QEMU 1.0 monitor - type 'help' for more information (qemu) info pci
info pci
Bus 0, device 0, function 0:

    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc080 [0xc08f].
      id ""
  Bus  0, device   1, function 2:
    USB controller: PCI device 8086:7020
      IRQ 5.
      BAR4: I/O at 0xc040 [0xc05f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1234:1111
      BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0xc060 [0xc07f].
      BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   4, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xc000 [0xc03f].
      BAR1: 32 bit memory at 0xfebf1000 [0xfebf1fff].
      id ""

thanks for your help,
peter

#5 lore raw
Quoted 106 lines · Peter Lieven
Quoted 105 lines · Gleb Natapov
Quoted 96 lines · Gleb Natapov
Quoted 12 lines · Peter Lieven

Hi,

I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest.

If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat

efer_reload 0 0
exits 2260976 79620
fpu_reload 6197 11

Quoted 8 lines
halt_exits                114734      5011
halt_wakeup               111195      4876
host_state_reload        1499659     60962
hypercalls                     0         0
insn_emulation           1577325     58488
insn_emulation_fail            0         0
invlpg                         0         0

io_exits 943949 40249

Hmm, too many of those.

Quoted 73 lines

irq_exits 108679 5434
irq_injections 236545 10788
irq_window 7606 246
largepages 672 5
mmio_exits 460020 16082

mmu_cache_miss               119         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write              13474         9
mmu_recycled                   0         0
mmu_shadow_zapped            141         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   22803        35
pf_guest                       0         0
remote_tlb_flush             239         2
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  20933         0

If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane.

efer_reload 0 0
exits 6132004 17931
fpu_reload 19863 3

halt_exits                264961      3083
halt_wakeup               236468      2959
host_state_reload        1104468      3104
hypercalls                     0         0
insn_emulation           1417443      7518
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  869380      2795
irq_exits                 253501      2362
irq_injections            616967      6804
irq_window                201186      2161
largepages                  1019         0
mmio_exits                205268         0
mmu_cache_miss               192         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write            7440546         0
mmu_recycled                   0         0
mmu_shadow_zapped            259         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   38529        30
pf_guest                       0         0
remote_tlb_flush             761         1
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0

I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase
performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 ) - no improvement.

My commandline:
/usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on

(quoted diff) +0 -2
-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native
-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name

lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet

Quoted 4 lines
Quoted 2 lines

What further information is needed to debug this further?

Which kernel version (looks like something recent)? Which host CPU (looks like something old)?

Output of cat /proc/cpuinfo

Quoted 2 lines

Which Windows' virtio drivers are you using?

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Take a trace like described here http://www.linux-kvm.org/page/Tracing
(with -no-hpet please).

And also "info pci" output from qemu monitor while we are at it.

here is the output while i was tracing. you can download the trace i took while i did a ftp transfer from the vm:

-> http://82.141.21.156/report.txt.gz

Windows reads PM timer. A lot. 15152 times per second.

Can you try to run this command in Windows guest:

bcdedit /set {default} useplatformclock false

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Quoted 41 lines

QEMU 1.0 monitor - type 'help' for more information (qemu) info pci
info pci
Bus 0, device 0, function 0:

    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc080 [0xc08f].
      id ""
  Bus  0, device   1, function 2:
    USB controller: PCI device 8086:7020
      IRQ 5.
      BAR4: I/O at 0xc040 [0xc05f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1234:1111
      BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0xc060 [0xc07f].
      BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   4, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xc000 [0xc03f].
      BAR1: 32 bit memory at 0xfebf1000 [0xfebf1fff].
      id ""
Quoted 2 lines

thanks for your help,
peter

--
Gleb.

author
#6 lore raw
Quoted 116 lines · Gleb Natapov
Quoted 106 lines · Peter Lieven
Quoted 105 lines · Gleb Natapov
Quoted 96 lines · Gleb Natapov
Quoted 12 lines · Peter Lieven

Hi,

I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest.

If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat

efer_reload 0 0
exits 2260976 79620
fpu_reload 6197 11

Quoted 8 lines
halt_exits                114734      5011
halt_wakeup               111195      4876
host_state_reload        1499659     60962
hypercalls                     0         0
insn_emulation           1577325     58488
insn_emulation_fail            0         0
invlpg                         0         0

io_exits 943949 40249

Hmm, too many of those.

Quoted 73 lines

irq_exits 108679 5434
irq_injections 236545 10788
irq_window 7606 246
largepages 672 5
mmio_exits 460020 16082

mmu_cache_miss               119         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write              13474         9
mmu_recycled                   0         0
mmu_shadow_zapped            141         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   22803        35
pf_guest                       0         0
remote_tlb_flush             239         2
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  20933         0

If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane.

efer_reload 0 0
exits 6132004 17931
fpu_reload 19863 3

halt_exits                264961      3083
halt_wakeup               236468      2959
host_state_reload        1104468      3104
hypercalls                     0         0
insn_emulation           1417443      7518
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  869380      2795
irq_exits                 253501      2362
irq_injections            616967      6804
irq_window                201186      2161
largepages                  1019         0
mmio_exits                205268         0
mmu_cache_miss               192         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write            7440546         0
mmu_recycled                   0         0
mmu_shadow_zapped            259         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   38529        30
pf_guest                       0         0
remote_tlb_flush             761         1
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0

I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase
performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 ) - no improvement.

My commandline:
/usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on

(quoted diff) +0 -2
-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native
-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name

lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet

Quoted 4 lines
Quoted 2 lines

What further information is needed to debug this further?

Which kernel version (looks like something recent)? Which host CPU (looks like something old)?

Output of cat /proc/cpuinfo

Quoted 2 lines

Which Windows' virtio drivers are you using?

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Take a trace like described here http://www.linux-kvm.org/page/Tracing
(with -no-hpet please).

And also "info pci" output from qemu monitor while we are at it.

here is the output while i was tracing. you can download the trace i took while i did a ftp transfer from the vm:

-> http://82.141.21.156/report.txt.gz

Quoted 8 lines

Windows reads PM timer. A lot. 15152 times per second.

Can you try to run this command in Windows guest:

bcdedit /set {default} useplatformclock false

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

I checked with bcedit /v that useplatformclock is set to "No".

I still wonder why both virtio devices are on IRQ0 ?

New Trace:

http://82.141.21.156/report2.txt.gz

efer_reload 0 0
exits 1510993 59343
fpu_reload 6729 10

halt_exits                 93603      5913
halt_wakeup                95698      5849
host_state_reload         738523     24727
hypercalls                     0         0
insn_emulation            678416     20107
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  703291     28436
irq_exits                 102117      7527
irq_injections            217335     14344
irq_window                  9926       650
largepages                   573         8
mmio_exits                    27         0
mmu_cache_miss               148         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write                  0         0
mmu_recycled                   0         0
mmu_shadow_zapped            190         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   21938        38
pf_guest                       0         0
remote_tlb_flush              20         0
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  11711         0

QEMU 1.0 monitor - type 'help' for more information (qemu) info pci
info pci
Bus 0, device 0, function 0:

    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc080 [0xc08f].
      id ""
  Bus  0, device   1, function 2:
    USB controller: PCI device 8086:7020
      IRQ 5.
      BAR4: I/O at 0xc040 [0xc05f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1234:1111
      BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0xc060 [0xc07f].
      BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   4, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xc000 [0xc03f].
      BAR1: 32 bit memory at 0xfebf1000 [0xfebf1fff].
      id ""
#7 lore raw
Quoted 7 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

I checked with bcedit /v that useplatformclock is set to "No".

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Quoted 3 lines

I still wonder why both virtio devices are on IRQ0 ?

They use MSI like they should.

--
Gleb.

author
#8 lore raw
Quoted 7 lines · Gleb Natapov
Quoted 6 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

Quoted 5 lines
Quoted 1 line

I checked with bcedit /v that useplatformclock is set to "No".

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense?

Peter

#9 lore raw
Quoted 8 lines · Peter Lieven
Quoted 7 lines · Gleb Natapov
Quoted 6 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I checked with bcedit /v that useplatformclock is set to "No".

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense?

It make perfect sense! Please try it.

--
Gleb.

author
#10 lore raw
Quoted 10 lines · Gleb Natapov
Quoted 9 lines · Peter Lieven
Quoted 8 lines · Gleb Natapov
Quoted 7 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

I checked with bcedit /v that useplatformclock is set to "No".

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense?

It make perfect sense! Please try it.

~10MB/s. still a lot of 0xb008 reads.

efer_reload 0 0
exits 4389875 72341
fpu_reload 36729 342

halt_exits                206204      3451
halt_wakeup               212953      3474
host_state_reload        2976799     59043
hypercalls                     0         0
insn_emulation           2936091     54921
insn_emulation_fail            0         0
invlpg                         0         0

io_exits 1821386 14108
irq_exits 81999 2798
irq_injections 343720 8560
irq_window 12712 153

largepages                   754         2
mmio_exits                    37         0
mmu_cache_miss               148         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write                  0         0
mmu_recycled                   0         0
mmu_shadow_zapped            189         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                  139461        21
pf_guest                       0         0
remote_tlb_flush             248         0
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  15366         0

trace at http://82.141.21.156/report3.txt.gz

Peter

Quoted 2 lines

--
Gleb.

#11 lore raw
Quoted 15 lines · Peter Lieven
Quoted 14 lines · Gleb Natapov
Quoted 13 lines · Peter Lieven
Quoted 12 lines · Gleb Natapov
Quoted 7 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

I checked with bcedit /v that useplatformclock is set to "No".

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense?

It make perfect sense! Please try it.

~10MB/s. still a lot of 0xb008 reads.

The same amount of reads essentially. So my theory is incorrect. Virtio driver probably calls Windows function to do IO and the function happens to be near the function that access PM timer.

I wonder why time stamps in your traces are so coarse-grained. What do you see in /sys/bus/clocksource/devices/clocksource0/current_clocksource ?

--
Gleb.

author
#12 lore raw
Quoted 20 lines · Gleb Natapov
Quoted 19 lines · Peter Lieven
Quoted 18 lines · Gleb Natapov
Quoted 8 lines · Peter Lieven
Quoted 7 lines · Gleb Natapov
Quoted 6 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I checked with bcedit /v that useplatformclock is set to "No".

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense?

It make perfect sense! Please try it.

Quoted 8 lines
Quoted 2 lines

~10MB/s. still a lot of 0xb008 reads.

The same amount of reads essentially. So my theory is incorrect. Virtio driver probably calls Windows function to do IO and the function happens to be near the function that access PM timer.

I wonder why time stamps in your traces are so coarse-grained. What do you see in /sys/bus/clocksource/devices/clocksource0/current_clocksource ?

its set to acpi_pm on the host. we changed that from tsc (choosen by kernel) after we encountered a kernel bug which ooops all hosts after approx. 270 days uptime. (https://lkml.org/lkml/2011/7/21/343).

i am not sure if this is fixed in 2.6.38 or later kernels and we could go back to tsc.

for testing i already checked this, but it doesn't give better performance.

peter

Quoted 2 lines

--
Gleb.

#13 lore raw

----- Original Message -----
From: "Peter Lieven" <[email protected]>
To: "Gleb Natapov" <[email protected]>
Cc: [email protected], [email protected], [email protected]
Sent: Tuesday, February 21, 2012 2:05:25 PM
Subject: Re: win7 bad i/o performance, high insn_emulation and exists

Quoted 16 lines · Gleb Natapov
Quoted 15 lines · Peter Lieven
Quoted 14 lines · Gleb Natapov
Quoted 13 lines · Peter Lieven
Quoted 12 lines · Gleb Natapov
Quoted 7 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

I checked with bcedit /v that useplatformclock is set to "No".

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense?

It make perfect sense! Please try it.

~10MB/s. still a lot of 0xb008 reads.

[VR]
Could it be that you have Driver Verifier running in you system?

Quoted 6 lines

The same amount of reads essentially. So my theory is incorrect. Virtio driver probably calls Windows function to do IO and the function happens to be near the function that access PM timer.

I wonder why time stamps in your traces are so coarse-grained. What do you see in /sys/bus/clocksource/devices/clocksource0/current_clocksource ?

its set to acpi_pm on the host. we changed that from tsc (choosen by kernel) after we encountered
a kernel bug which ooops all hosts after approx. 270 days uptime.
(https://lkml.org/lkml/2011/7/21/343).

i am not sure if this is fixed in 2.6.38 or later kernels and we could go back to tsc.

for testing i already checked this, but it doesn't give better performance.

peter

Quoted 2 lines

--
Gleb.

author
#14 lore raw
Quoted 25 lines · Vadim Rozenfeld

----- Original Message -----
From: "Peter Lieven"<[email protected]>
To: "Gleb Natapov"<[email protected]>
Cc: [email protected], [email protected], [email protected]
Sent: Tuesday, February 21, 2012 2:05:25 PM
Subject: Re: win7 bad i/o performance, high insn_emulation and exists

Quoted 16 lines · Gleb Natapov
Quoted 15 lines · Peter Lieven
Quoted 14 lines · Gleb Natapov
Quoted 13 lines · Peter Lieven
Quoted 12 lines · Gleb Natapov
Quoted 7 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

I checked with bcedit /v that useplatformclock is set to "No".

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense?

It make perfect sense! Please try it.

~10MB/s. still a lot of 0xb008 reads.

[VR]
Could it be that you have Driver Verifier running in you system?

unfortunately not.

i found the following in an old knowledge base article (http://support.microsoft.com/kb/938448):

"Only Windows Server 2003 with Service Pack 2 uniprocessor ACPI HALs use *PMTIMER* for QPC by default. Multiprocessor ACPI HALs will use *PMTIMER* only if *USE_PLATFORM_CLOCK *flag is set by the BIOS or if the */usepmtimer *boot.ini option is used. Other HAL types don’t support *PMTIMER* and will use *TSC* by default for QPC

By default, Windows Server 2003 Service Pack 2 (SP2) uses the PM timer for all Advanced Configuration and Power Interface (ACPI) HALs unless one of the following conditions aretrue:

* The check process to determine whether the BIOS supports the APIC or ACPI HALs fails.
* *

Note:* If the BIOS does not support the ACPI HAL, contact the original equipment manufacturer to determine whether a BIOS update is available that will resolve the problem. If a BIOS update is not available, you must use the PM timer by using the */usepmtimer* switch.

If you are not running Windows Server 2003 SP2, you must force the AMD computer to use the PM timer by using the */usepmtimer* switch.

*Note* The decision to use the PM timer or the TSC timer is made during a check that is performed at startup to query the BIOS and to determine whether the BIOS will support the PM timer functions. This check is not completely accurate on AMD chipsets. Therefore, you must use the */usepmtimer* switch.

In Windows Server 2003 SP2, this section of code was rewritten. Therefore, the correct performance monitor data appears on AMD chipsets that have Windows Server 2003 SP2 installed, and you do not have to use the */usepmtimer* switch.

For more information about ACPI and APCI hardware support, click the following article number to view the article in the Microsoft Knowledge Base:
309283 <http://support.microsoft.com/kb/309283> HAL options after Windows XP or Windows Server 2003 Setup
The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products."

-

so it seems windows prefers pmtimer over tsc. has anyone an idea/hack to make the acpi_pm timer fail without disabling acpi completely?

thanks,
peter

#15 lore raw

----- Original Message -----
From: "Peter Lieven" <[email protected]>
To: "Vadim Rozenfeld" <[email protected]>
Cc: [email protected], [email protected], "Gleb Natapov" <[email protected]>
Sent: Tuesday, February 21, 2012 4:10:22 PM
Subject: Re: win7 bad i/o performance, high insn_emulation and exists

Quoted 25 lines · Vadim Rozenfeld

----- Original Message -----
From: "Peter Lieven"<[email protected]>
To: "Gleb Natapov"<[email protected]>
Cc: [email protected], [email protected], [email protected]
Sent: Tuesday, February 21, 2012 2:05:25 PM
Subject: Re: win7 bad i/o performance, high insn_emulation and exists

Quoted 16 lines · Gleb Natapov
Quoted 15 lines · Peter Lieven
Quoted 14 lines · Gleb Natapov
Quoted 13 lines · Peter Lieven
Quoted 12 lines · Gleb Natapov
Quoted 7 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

I checked with bcedit /v that useplatformclock is set to "No".

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense?

It make perfect sense! Please try it.

~10MB/s. still a lot of 0xb008 reads.

[VR]
Could it be that you have Driver Verifier running in you system?

unfortunately not.

[VR]
Then could you try booting into "Safe Mode"?

i found the following in an old knowledge base article
(http://support.microsoft.com/kb/938448):

"Only Windows Server 2003 with Service Pack 2 uniprocessor ACPI HALs use *PMTIMER* for QPC by default. Multiprocessor ACPI HALs will use *PMTIMER* only if *USE_PLATFORM_CLOCK *flag is set by the BIOS or if the */usepmtimer *boot.ini option is used. Other HAL types don’t support *PMTIMER* and will use *TSC* by default for QPC

By default, Windows Server 2003 Service Pack 2 (SP2) uses the PM timer for all Advanced Configuration and Power Interface (ACPI) HALs unless one of the following conditions aretrue:

* The check process to determine whether the BIOS supports the APIC or ACPI HALs fails.
* *

Note:* If the BIOS does not support the ACPI HAL, contact the original equipment manufacturer to determine whether a BIOS update is available that will resolve the problem. If a BIOS update is not available, you must use the PM timer by using the */usepmtimer* switch.

If you are not running Windows Server 2003 SP2, you must force the AMD computer to use the PM timer by using the */usepmtimer* switch.

*Note* The decision to use the PM timer or the TSC timer is made during a check that is performed at startup to query the BIOS and to determine whether the BIOS will support the PM timer functions. This check is not completely accurate on AMD chipsets. Therefore, you must use the */usepmtimer* switch.

In Windows Server 2003 SP2, this section of code was rewritten. Therefore, the correct performance monitor data appears on AMD chipsets that have Windows Server 2003 SP2 installed, and you do not have to use the */usepmtimer* switch.

For more information about ACPI and APCI hardware support, click the following article number to view the article in the Microsoft Knowledge Base:
309283 <http://support.microsoft.com/kb/309283> HAL options after Windows XP or Windows Server 2003 Setup
The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products."

-

so it seems windows prefers pmtimer over tsc. has anyone an idea/hack to make the acpi_pm timer fail without disabling acpi completely?

thanks,
peter
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#16 lore raw
Quoted 39 lines · Vadim Rozenfeld

----- Original Message -----
From: "Peter Lieven"<[email protected]>
To: "Vadim Rozenfeld"<[email protected]>
Cc: [email protected], [email protected], "Gleb Natapov"<[email protected]>
Sent: Tuesday, February 21, 2012 4:10:22 PM
Subject: Re: win7 bad i/o performance, high insn_emulation and exists

Quoted 30 lines · Vadim Rozenfeld

----- Original Message -----
From: "Peter Lieven"<[email protected]>
To: "Gleb Natapov"<[email protected]>
Cc: [email protected], [email protected], [email protected]
Sent: Tuesday, February 21, 2012 2:05:25 PM
Subject: Re: win7 bad i/o performance, high insn_emulation and exists

Quoted 22 lines · Gleb Natapov
Quoted 21 lines · Peter Lieven
Quoted 10 lines · Gleb Natapov
Quoted 9 lines · Peter Lieven
Quoted 8 lines · Gleb Natapov
Quoted 7 lines · Peter Lieven
Quoted 2 lines

I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :(

Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell.

I checked with bcedit /v that useplatformclock is set to "No".

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed.

Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense?

It make perfect sense! Please try it.

~10MB/s. still a lot of 0xb008 reads.

Quoted 7 lines
Quoted 3 lines

[VR]
Could it be that you have Driver Verifier running in you system?

unfortunately not.

[VR]
Then could you try booting into "Safe Mode"?

same picture.

author
#17 lore raw
In reply to Gleb Natapov
Quoted 96 lines · Gleb Natapov
Quoted 12 lines · Peter Lieven

Hi,

I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest.

If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat

efer_reload 0 0
exits 2260976 79620
fpu_reload 6197 11

Quoted 8 lines
halt_exits                114734      5011
halt_wakeup               111195      4876
host_state_reload        1499659     60962
hypercalls                     0         0
insn_emulation           1577325     58488
insn_emulation_fail            0         0
invlpg                         0         0

io_exits 943949 40249

Hmm, too many of those.

Quoted 73 lines

irq_exits 108679 5434
irq_injections 236545 10788
irq_window 7606 246
largepages 672 5
mmio_exits 460020 16082

mmu_cache_miss               119         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write              13474         9
mmu_recycled                   0         0
mmu_shadow_zapped            141         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   22803        35
pf_guest                       0         0
remote_tlb_flush             239         2
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  20933         0

If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane.

efer_reload 0 0
exits 6132004 17931
fpu_reload 19863 3

halt_exits                264961      3083
halt_wakeup               236468      2959
host_state_reload        1104468      3104
hypercalls                     0         0
insn_emulation           1417443      7518
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  869380      2795
irq_exits                 253501      2362
irq_injections            616967      6804
irq_window                201186      2161
largepages                  1019         0
mmio_exits                205268         0
mmu_cache_miss               192         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write            7440546         0
mmu_recycled                   0         0
mmu_shadow_zapped            259         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   38529        30
pf_guest                       0         0
remote_tlb_flush             761         1
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0

I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase
performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 ) - no improvement.

My commandline:
/usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on

(quoted diff) +0 -2
-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native
-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name

lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet

Quoted 3 lines
Quoted 2 lines

What further information is needed to debug this further?

Which kernel version (looks like something recent)?

2.6.38 with kvm-kmod 3.2

Quoted 1 line

Which host CPU (looks like something old)?

why? i guess its (quite) new.

vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU L5640 @ 2.27GHz
stepping : 2
cpu MHz : 1596.000
cache size : 12288 KB
physical id : 1
siblings : 6
core id : 10
cpu cores : 6
apicid : 52
initial apicid : 52
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm arat dts tpr_shadow vnmi flexpriority ept vpid
bogomips : 2254.43
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual

Quoted 1 line

Which Windows' virtio drivers are you using?

i used to use 0.1-16 and today also tried 0.1-22 from
http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/

Quoted 2 lines

Take a trace like described here http://www.linux-kvm.org/page/Tracing
(with -no-hpet please).

will prepare this.

Quoted 1 line

Try to use -cpu host,+x2apic. It may help Linux guest performance.

Thanks, it improved throughput a little while lowering the cpu usage. Windows does not support this?

Thanks
Peter

Quoted 2 lines

--
Gleb.

#18 lore raw
Quoted 97 lines · Peter Lieven
Quoted 96 lines · Gleb Natapov
Quoted 12 lines · Peter Lieven

Hi,

I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest.

If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat

efer_reload 0 0
exits 2260976 79620
fpu_reload 6197 11

Quoted 8 lines
halt_exits                114734      5011
halt_wakeup               111195      4876
host_state_reload        1499659     60962
hypercalls                     0         0
insn_emulation           1577325     58488
insn_emulation_fail            0         0
invlpg                         0         0

io_exits 943949 40249

Hmm, too many of those.

Quoted 73 lines

irq_exits 108679 5434
irq_injections 236545 10788
irq_window 7606 246
largepages 672 5
mmio_exits 460020 16082

mmu_cache_miss               119         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write              13474         9
mmu_recycled                   0         0
mmu_shadow_zapped            141         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   22803        35
pf_guest                       0         0
remote_tlb_flush             239         2
request_irq                    0         0
signal_exits                   0         0
tlb_flush                  20933         0

If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane.

efer_reload 0 0
exits 6132004 17931
fpu_reload 19863 3

halt_exits                264961      3083
halt_wakeup               236468      2959
host_state_reload        1104468      3104
hypercalls                     0         0
insn_emulation           1417443      7518
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                  869380      2795
irq_exits                 253501      2362
irq_injections            616967      6804
irq_window                201186      2161
largepages                  1019         0
mmio_exits                205268         0
mmu_cache_miss               192         0
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write            7440546         0
mmu_recycled                   0         0
mmu_shadow_zapped            259         0
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed                   38529        30
pf_guest                       0         0
remote_tlb_flush             761         1
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0

I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase
performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 ) - no improvement.

My commandline:
/usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on

(quoted diff) +0 -2
-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native
-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name

lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

What further information is needed to debug this further?

Which kernel version (looks like something recent)?

2.6.38 with kvm-kmod 3.2

Quoted 1 line

Which host CPU (looks like something old)?

why? i guess its (quite) new.

Yes, it is. I misread the kvm_stat data and thought that CPU does not have ept, but it does.

Quoted 30 lines

vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU L5640 @ 2.27GHz
stepping : 2
cpu MHz : 1596.000
cache size : 12288 KB
physical id : 1
siblings : 6
core id : 10
cpu cores : 6
apicid : 52
initial apicid : 52
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm arat dts tpr_shadow vnmi flexpriority ept vpid
bogomips : 2254.43
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual

Quoted 1 line

Which Windows' virtio drivers are you using?

Quoted 9 lines

i used to use 0.1-16 and today also tried 0.1-22 from
http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/

Quoted 2 lines

Take a trace like described here http://www.linux-kvm.org/page/Tracing
(with -no-hpet please).

will prepare this.

Quoted 1 line

Try to use -cpu host,+x2apic. It may help Linux guest performance.

Thanks, it improved throughput a little while lowering the cpu usage. Windows does not support this?

Not yet. It will support something similar when qemu will have hyper-v support.

--
Gleb.