KVM performance vs. Xen
I wanted to share some performance data for KVM and Xen. I thought it would be interesting to share some performance results especially compared to Xen, using a more complex situation like heterogeneous server consolidation.
The Workload:
The workload is one that simulates a consolidation of servers on to a single host. There are 3 server types: web, imap, and app (j2ee). In addition, there are other "helper" servers which are also consolidated:
a db server, which helps out with the app server, and an nfs server, which helps out with the web server (a portion of the docroot is nfs mounted). There is also one other server that is simply idle. All 6 servers make up one set. The first 3 server types are sent requests, which in turn may send requests to the db and nfs helper servers. The request rate is throttled to produce a fixed amount of work. In order to increase utilization on the host, more sets of these servers are used. The clients which send requests also have a response time requirement which is monitored. The following results have passed the response time requirements.
The host hardware:
A 2 socket, 8 core Nehalem with SMT, and EPT enabled, lots of disks, 4 x 1 GB Ethenret
The host software:
Both Xen and KVM use the same host Linux OS, SLES11. KVM uses the 2.6.27.19-5-default kernel and Xen uses the 2.6.27.19-5-xen kernel. I have tried 2.6.29 for KVM, but results are actually worse. KVM modules are rebuilt with kvm-85. Qemu is also from kvm-85. Xen version is "3.3.1_18546_12-3.1".
The guest software:
All guests are RedHat 5.3. The same disk images are used but different kernels. Xen uses the RedHat Xen kernel and KVM uses 2.6.29 with all paravirt build options enabled. Both use PV I/O drivers. Software
used: Apache, PHP, Java, Glassfish, Postgresql, and Dovecot.
Hypervisor configurations:
Xen guests use "phy:" for disks
KVM guests use "-drive" for disks with cache=none
KVM guests are backed with large pages
Memory and CPU sizings are different for each guest type, but a particular guest's sizings are the same for Xen and KVM
The test run configuration:
There are 4 sets of servers used, so that's 24 guests total (4 idle ones, 20 active ones).
Test Results:
The throughput is equal in these tests, as the clients throttle the work (this is assuming you don't run out of a resource on the host). What's telling is the CPU used to do the same amount of work:
Xen: 52.85% KVM: 66.93%
So, KVM requires 66.93/52.85 = 26.6% more CPU to do the same amount of work. Here's the breakdown:
total user nice system irq softirq guest 66.90 7.20 0.00 12.94 0.35 3.39 43.02
Comparing guest time to all other busy time, that's a 23.88/43.02 = 55% overhead for virtualization. I certainly don't expect it to be 0, but 55% seems a bit high. So, what's the reason for this overhead? At the bottom is oprofile output of top functions for KVM. Some observations:
1) I'm seeing about 2.3% in scheduler functions [that I recognize]. Does that seems a bit excessive?
2) cpu_physical_memory_rw due to not using preadv/pwritev?
3) vmx_[save|load]_host_state: I take it this is from guest switches? We have 180,000 context switches a second. Is this more than expected? I wonder if schedstats can show why we context switch (need to let someone else run, yielded, waiting on io, etc).
samples % name app 385914891 61.3122 kvm-intel.ko vmx_vcpu_run 11413793 1.8134 libc-2.9.so /lib64/libc-2.9.so 8943054 1.4208 qemu-system-x86_64 cpu_physical_memory_rw 6877593 1.0927 kvm.ko kvm_arch_vcpu_ioctl_run 6469799 1.0279 qemu-system-x86_64 phys_page_find_alloc 5080474 0.8072 vmlinux-2.6.27.19-5-default copy_user_generic_string 4154467 0.6600 kvm-intel.ko __vmx_load_host_state 3991060 0.6341 vmlinux-2.6.27.19-5-default schedule 3455331 0.5490 kvm-intel.ko vmx_save_host_state 2582344 0.4103 vmlinux-2.6.27.19-5-default find_busiest_group 2509543 0.3987 qemu-system-x86_64 main_loop_wait 2457476 0.3904 vmlinux-2.6.27.19-5-default kfree 2395296 0.3806 kvm.ko kvm_set_irq 2385298 0.3790 vmlinux-2.6.27.19-5-default fget_light 2229755 0.3543 vmlinux-2.6.27.19-5-default __switch_to 2178739 0.3461 bnx2.ko bnx2_rx_int 2156418 0.3426 vmlinux-2.6.27.19-5-default complete_signal 1854497 0.2946 qemu-system-x86_64 virtqueue_get_head 1833823 0.2913 vmlinux-2.6.27.19-5-default try_to_wake_up 1816954 0.2887 qemu-system-x86_64 cpu_physical_memory_map 1776548 0.2822 oprofiled find_kernel_image 1737294 0.2760 vmlinux-2.6.27.19-5-default kmem_cache_alloc 1662346 0.2641 qemu-system-x86_64 virtqueue_avail_bytes 1651070 0.2623 vmlinux-2.6.27.19-5-default do_select 1643139 0.2611 vmlinux-2.6.27.19-5-default update_curr 1640495 0.2606 vmlinux-2.6.27.19-5-default kmem_cache_free 1606493 0.2552 libpthread-2.9.so pthread_mutex_lock 1549536 0.2462 qemu-system-x86_64 lduw_phys 1535539 0.2440 vmlinux-2.6.27.19-5-default tg_shares_up 1438468 0.2285 vmlinux-2.6.27.19-5-default mwait_idle 1316461 0.2092 vmlinux-2.6.27.19-5-default __down_read 1282486 0.2038 vmlinux-2.6.27.19-5-default native_read_tsc 1226069 0.1948 oprofiled odb_update_node 1224551 0.1946 vmlinux-2.6.27.19-5-default sched_clock_cpu 1222684 0.1943 tun.ko tun_chr_aio_read 1194034 0.1897 vmlinux-2.6.27.19-5-default task_rq_lock 1186129 0.1884 kvm.ko x86_decode_insn 1131644 0.1798 bnx2.ko bnx2_start_xmit 1115575 0.1772 vmlinux-2.6.27.19-5-default enqueue_hrtimer 1044329 0.1659 vmlinux-2.6.27.19-5-default native_sched_clock 988546 0.1571 vmlinux-2.6.27.19-5-default fput 980615 0.1558 vmlinux-2.6.27.19-5-default __up_read 942270 0.1497 qemu-system-x86_64 kvm_run 925076 0.1470 kvm-intel.ko vmcs_writel 889220 0.1413 vmlinux-2.6.27.19-5-default dev_queue_xmit 884786 0.1406 kvm.ko kvm_apic_has_interrupt 880421 0.1399 librt-2.9.so /lib64/librt-2.9.so 880306 0.1399 vmlinux-2.6.27.19-5-default nf_iterate
-Andrew Theurer
Quoted 32 lines · Andrew Theurer
I wanted to share some performance data for KVM and Xen. I thought it would be interesting to share some performance results especially compared to Xen, using a more complex situation like heterogeneous server consolidation.
The Workload:
The workload is one that simulates a consolidation of servers on to a single host. There are 3 server types: web, imap, and app (j2ee). In addition, there are other "helper" servers which are also
consolidated: a db server, which helps out with the app server, and an
nfs server, which helps out with the web server (a portion of the docroot is nfs mounted). There is also one other server that is simply idle. All 6 servers
make up one set. The first 3 server types are sent requests, which in
turn may send requests to the db and nfs helper servers. The request
rate is throttled to produce a fixed amount of work. In order to
increase utilization on the host, more sets of these servers are used.
The clients which send requests also have a response time requirement
which is monitored. The following results have passed the response
time requirements.The host hardware:
A 2 socket, 8 core Nehalem with SMT, and EPT enabled, lots of disks, 4 x
1 GB EthenretThe host software:
Both Xen and KVM use the same host Linux OS, SLES11. KVM uses the 2.6.27.19-5-default kernel and Xen uses the 2.6.27.19-5-xen kernel. I have tried 2.6.29 for KVM, but results are actually worse. KVM modules are rebuilt with kvm-85. Qemu is also from kvm-85. Xen version is "3.3.1_18546_12-3.1".
Quoted 6 lines
The guest software:
All guests are RedHat 5.3. The same disk images are used but different kernels. Xen uses the RedHat Xen kernel and KVM uses 2.6.29 with all paravirt build options enabled. Both use PV I/O drivers. Software used:
Apache, PHP, Java, Glassfish, Postgresql, and Dovecot.
Just for clarification. So are you using PV (Xen) Linux on Xen, not HVM? Is that 32-bit or 64-bit?
.
Jun Nakajima | Intel Open Source Technology Center
Quoted 42 lines · Jun
Quoted 32 lines · Andrew Theurer
I wanted to share some performance data for KVM and Xen. I thought it would be interesting to share some performance results especially compared to Xen, using a more complex situation like heterogeneous server consolidation.
The Workload:
The workload is one that simulates a consolidation of servers on to a single host. There are 3 server types: web, imap, and app (j2ee). In addition, there are other "helper" servers which are also
consolidated: a db server, which helps out with the app server, and an
nfs server, which helps out with the web server (a portion of the docroot is nfs mounted). There is also one other server that is simply idle. All 6 servers
make up one set. The first 3 server types are sent requests, which in
turn may send requests to the db and nfs helper servers. The request
rate is throttled to produce a fixed amount of work. In order to
increase utilization on the host, more sets of these servers are used.
The clients which send requests also have a response time requirement
which is monitored. The following results have passed the response
time requirements.The host hardware:
A 2 socket, 8 core Nehalem with SMT, and EPT enabled, lots of disks, 4 x
1 GB EthenretThe host software:
Both Xen and KVM use the same host Linux OS, SLES11. KVM uses the 2.6.27.19-5-default kernel and Xen uses the 2.6.27.19-5-xen kernel. I have tried 2.6.29 for KVM, but results are actually worse. KVM modules are rebuilt with kvm-85. Qemu is also from kvm-85. Xen version is "3.3.1_18546_12-3.1".Quoted 7 lines
The guest software:
All guests are RedHat 5.3. The same disk images are used but different kernels. Xen uses the RedHat Xen kernel and KVM uses 2.6.29 with all paravirt build options enabled. Both use PV I/O drivers. Software used:
Apache, PHP, Java, Glassfish, Postgresql, and Dovecot.
Quoted 2 lines
Just for clarification. So are you using PV (Xen) Linux on Xen, not HVM? Is that 32-bit or 64-bit?
PV, 64-bit.
-Andrew
Quoted 5 lines · Andrew Theurer
I wanted to share some performance data for KVM and Xen. I thought it would be interesting to share some performance results especially compared to Xen, using a more complex situation like heterogeneous server consolidation.
Quoted 15 lines
The Workload:
The workload is one that simulates a consolidation of servers on to a single host. There are 3 server types: web, imap, and app (j2ee). In addition, there are other "helper" servers which are also consolidated:
a db server, which helps out with the app server, and an nfs server, which helps out with the web server (a portion of the docroot is nfs mounted). There is also one other server that is simply idle. All 6 servers make up one set. The first 3 server types are sent requests, which in turn may send requests to the db and nfs helper servers. The request rate is throttled to produce a fixed amount of work. In order to increase utilization on the host, more sets of these servers are used. The clients which send requests also have a response time requirement which is monitored. The following results have passed the response time requirements.
What's the typical I/O load (disk and network bandwidth) while the tests are running?
Quoted 3 lines
The host hardware:
A 2 socket, 8 core Nehalem with SMT, and EPT enabled, lots of disks, 4 x 1 GB Ethenret
CPU time measurements with SMT can vary wildly if the system is not fully loaded. If the scheduler happens to schedule two threads on a single core, both of these threads will generate less work compared to if they were scheduled on different cores.
Quoted 14 lines
Test Results:
The throughput is equal in these tests, as the clients throttle the work (this is assuming you don't run out of a resource on the host). What's telling is the CPU used to do the same amount of work:Xen: 52.85% KVM: 66.93%So, KVM requires 66.93/52.85 = 26.6% more CPU to do the same amount of work. Here's the breakdown:
total user nice system irq softirq guest 66.90 7.20 0.00 12.94 0.35 3.39 43.02
Quoted 7 lines
Comparing guest time to all other busy time, that's a 23.88/43.02 = 55% overhead for virtualization. I certainly don't expect it to be 0, but 55% seems a bit high. So, what's the reason for this overhead? At the bottom is oprofile output of top functions for KVM. Some observations:
1) I'm seeing about 2.3% in scheduler functions [that I recognize]. Does that seems a bit excessive?
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
Quoted 1 line
2)
cpu_physical_memory_rwdue to not using preadv/pwritev?
I think both virtio-net and virtio-blk use memcpy().
Quoted 1 line
3) vmx_[save|load]_host_state: I take it this is from guest switches?
These are called when you context-switch from a guest, and, much more frequently, when you enter qemu.
Quoted 1 line
We have 180,000 context switches a second. Is this more than expected?
Way more. Across 16 logical cpus, this is >10,000 cs/sec/cpu.
Quoted 3 lines
I wonder if schedstats can show why we context switch (need to let someone else run, yielded, waiting on io, etc).
Yes, there is a scheduler tracer, though I have no idea how to operate it.
Do you have kvm_stat logs?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 22 lines · Avi Kivity
Quoted 5 lines · Andrew Theurer
I wanted to share some performance data for KVM and Xen. I thought it would be interesting to share some performance results especially compared to Xen, using a more complex situation like heterogeneous server consolidation.
Quoted 15 lines
The Workload:
The workload is one that simulates a consolidation of servers on to a single host. There are 3 server types: web, imap, and app (j2ee). In addition, there are other "helper" servers which are also consolidated:
a db server, which helps out with the app server, and an nfs server, which helps out with the web server (a portion of the docroot is nfs mounted). There is also one other server that is simply idle. All 6 servers make up one set. The first 3 server types are sent requests, which in turn may send requests to the db and nfs helper servers. The request rate is throttled to produce a fixed amount of work. In order to increase utilization on the host, more sets of these servers are used. The clients which send requests also have a response time requirement which is monitored. The following results have passed the response time requirements.
Quoted 1 line
What's the typical I/O load (disk and network bandwidth) while the tests are running?
This is average thrgoughput:
network: Tx: 79 MB/sec Rx: 5 MB/sec disk: read: 17 MB/sec write: 40 MB/sec
Quoted 6 lines
Quoted 4 lines
The host hardware:
A 2 socket, 8 core Nehalem with SMT, and EPT enabled, lots of disks, 4 x 1 GB EthenretCPU time measurements with SMT can vary wildly if the system is not fully loaded. If the scheduler happens to schedule two threads on a single core, both of these threads will generate less work compared to if they were scheduled on different cores.
Understood. Even if at low loads, the scheduler does the right thing and spreads out to all the cores first, once it goes beyond 50% util, the CPU util can climb at a much higher rate (compared to a linear increase in work) because it then starts scheduling 2 threads per core, and each thread can do less work. I have always wanted something which could more accurately show the utilization of a processor core, but I guess we have to use what we have today. I will run again with SMT off to see what we get.
Quoted 21 lines
Quoted 21 lines
Test Results:
The throughput is equal in these tests, as the clients throttle the work (this is assuming you don't run out of a resource on the host). What's telling is the CPU used to do the same amount of work:Xen: 52.85% KVM: 66.93%So, KVM requires 66.93/52.85 = 26.6% more CPU to do the same amount of work. Here's the breakdown:
total user nice system irq softirq guest 66.90 7.20 0.00 12.94 0.35 3.39 43.02Comparing guest time to all other busy time, that's a 23.88/43.02 = 55% overhead for virtualization. I certainly don't expect it to be 0, but 55% seems a bit high. So, what's the reason for this overhead? At the bottom is oprofile output of top functions for KVM. Some observations:
Quoted 4 lines
Quoted 2 lines
1) I'm seeing about 2.3% in scheduler functions [that I recognize]. Does that seems a bit excessive?
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
I have a older patch which makes a small change to posix_aio_thread.c by trying to keep the thread pool size a bit lower than it is today. I will dust that off and see if it helps.
Quoted 14 lines
Quoted 2 lines
2)
cpu_physical_memory_rwdue to not using preadv/pwritev?I think both virtio-net and virtio-blk use
memcpy().Quoted 1 line
3) vmx_[save|load]_host_state: I take it this is from guest switches?
These are called when you context-switch from a guest, and, much more frequently, when you enter qemu.
Quoted 1 line
We have 180,000 context switches a second. Is this more than expected?
Way more. Across 16 logical cpus, this is >10,000 cs/sec/cpu.
Quoted 7 lines
Quoted 3 lines
I wonder if schedstats can show why we context switch (need to let someone else run, yielded, waiting on io, etc).
Yes, there is a scheduler tracer, though I have no idea how to operate it.
Do you have
kvm_statlogs?
Sorry, I don't, but I'll run that next time. BTW, I did not notice a batch/log mode the last time I ram kvm_stat. Or maybe it was not obvious to me. Is there an ideal way to run kvm_stat without a curses like output?
-Andrew
Quoted 6 lines · Andrew Theurer
Quoted 3 lines · Avi Kivity
Quoted 1 line
What's the typical I/O load (disk and network bandwidth) while the tests are running?
This is average thrgoughput:
network: Tx: 79 MB/sec Rx: 5 MB/sec
MB as in Byte or Mb as in bit?
Quoted 1 line
disk: read: 17 MB/sec write: 40 MB/sec
This could definitely cause the extra load, especially if it's many small requests (compared to a few large ones).
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
The host hardware:
A 2 socket, 8 core Nehalem with SMT, and EPT enabled, lots of disks, 4 x 1 GB EthenretCPU time measurements with SMT can vary wildly if the system is not fully loaded. If the scheduler happens to schedule two threads on a single core, both of these threads will generate less work compared to if they were scheduled on different cores.
Understood. Even if at low loads, the scheduler does the right thing and spreads out to all the cores first, once it goes beyond 50% util, the CPU util can climb at a much higher rate (compared to a linear increase in work) because it then starts scheduling 2 threads per core, and each thread can do less work. I have always wanted something which could more accurately show the utilization of a processor core, but I guess we have to use what we have today. I will run again with SMT off to see what we get.
On the other hand, without SMT you will get to overcommit much faster, so you'll have scheduling artifacts. Unfortunately there's no good answer here (except to improve the SMT scheduler).
Quoted 2 lines
Quoted 1 line
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
I have a older patch which makes a small change to
posix_aio_thread.cby trying to keep the thread pool size a bit lower than it is today. I will dust that off and see if it helps.
Really, I think linux-aio support can help here.
Quoted 5 lines
Quoted 4 lines
Yes, there is a scheduler tracer, though I have no idea how to operate it.
Do you have
kvm_statlogs?Sorry, I don't, but I'll run that next time. BTW, I did not notice a batch/log mode the last time I ram
kvm_stat. Or maybe it was not obvious to me. Is there an ideal way to runkvm_statwithout a curses like output?
You're probably using an ancient version:
$ kvm_stat --help Usage: kvm_stat [options]
Options:
-h, --help show this help message and exit -1, --once, --batch run in batch mode for one second -l, --log run in logging mode (like vmstat)
-f FIELDS, --fields=FIELDS
fields to display (regex)
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Andrew Theurer
Quoted 3 lines · Avi Kivity
Quoted 1 line
What's the typical I/O load (disk and network bandwidth) while the tests are running?
This is average thrgoughput:
network: Tx: 79 MB/sec Rx: 5 MB/secMB as in Byte or Mb as in bit?
Byte. There are 4 x 1 Gb adapters, each handling about 20 MB/sec or 160 Mbit/sec.
Quoted 4 lines
Quoted 2 lines
disk: read: 17 MB/sec write: 40 MB/sec
This could definitely cause the extra load, especially if it's many small requests (compared to a few large ones).
I don't have the request sizes at my fingertips, but we have to use a lot of disks to support this I/O, so I think it's safe to assume there are a lot more requests than a simple large sequential read/write.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
The host hardware:
A 2 socket, 8 core Nehalem with SMT, and EPT enabled, lots of disks, 4 x 1 GB EthenretCPU time measurements with SMT can vary wildly if the system is not fully loaded. If the scheduler happens to schedule two threads on a single core, both of these threads will generate less work compared to if they were scheduled on different cores.
Quoted 8 lines
Quoted 1 line
Understood. Even if at low loads, the scheduler does the right thing and spreads out to all the cores first, once it goes beyond 50% util, the CPU util can climb at a much higher rate (compared to a linear increase in work) because it then starts scheduling 2 threads per core, and each thread can do less work. I have always wanted something which could more accurately show the utilization of a processor core, but I guess we have to use what we have today. I will run again with SMT off to see what we get.
On the other hand, without SMT you will get to overcommit much faster, so you'll have scheduling artifacts. Unfortunately there's no good answer here (except to improve the SMT scheduler).
Quoted 2 lines
Quoted 1 line
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
I have a older patch which makes a small change to
posix_aio_thread.cby trying to keep the thread pool size a bit lower than it is today. I will dust that off and see if it helps.Really, I think linux-aio support can help here.
Yes, I think that would work for real block devices, but would that help for files? I am using real block devices right now, but it would be nice to also see a benefit for files in a file-system. Or maybe I am mis-understanding this, and linux-aio can be used on files?
-Andrew
Quoted 20 lines
Quoted 6 lines
Quoted 5 lines
Yes, there is a scheduler tracer, though I have no idea how to operate it.
Do you have
kvm_statlogs?Sorry, I don't, but I'll run that next time. BTW, I did not notice a batch/log mode the last time I ram
kvm_stat. Or maybe it was not obvious to me. Is there an ideal way to runkvm_statwithout a curses like output?You're probably using an ancient version:
$ kvm_stat --help Usage: kvm_stat [options]Options:
-h, --help show this help message and exit -1, --once, --batch run in batch mode for one second -l, --log run in logging mode (like vmstat)-f FIELDS, --fields=FIELDS
fields to display (regex)
Quoted 3 lines · Andrew Theurer
Quoted 2 lines
Really, I think linux-aio support can help here.
Yes, I think that would work for real block devices, but would that help for files? I am using real block devices right now, but it would be nice to also see a benefit for files in a file-system. Or maybe I am mis-understanding this, and linux-aio can be used on files?
For cache=off, with some file systems, yes. But not for cache=writethrough/writeback.
Regards,
Anthony Liguori
Quoted 6 lines · Andrew Theurer
Quoted 5 lines
Quoted 3 lines
disk: read: 17 MB/sec write: 40 MB/sec
This could definitely cause the extra load, especially if it's many small requests (compared to a few large ones).
I don't have the request sizes at my fingertips, but we have to use a lot of disks to support this I/O, so I think it's safe to assume there are a lot more requests than a simple large sequential read/write.
Yes. Well the high context switch rate is the scheduler's way of telling us to use linux-aio. If "lot's of disks" == 100, with a 3ms seek time, that's already 60,000 cs/sec.
Quoted 2 lines
Quoted 1 line
Really, I think linux-aio support can help here.
Yes, I think that would work for real block devices, but would that help for files? I am using real block devices right now, but it would be nice to also see a benefit for files in a file-system. Or maybe I am mis-understanding this, and linux-aio can be used on files?
It could work with files with cache=none (though not qcow2 as now written).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 5 lines · Avi Kivity
Quoted 3 lines
1) I'm seeing about 2.3% in scheduler functions [that I recognize]. Does that seems a bit excessive?
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
This is why I wrote the linux-aio patch. It only reduced CPU consumption by about 2% although I'm not sure if that's absolute or relative. Andrew?
Quoted 3 lines
Quoted 1 line
2)
cpu_physical_memory_rwdue to not using preadv/pwritev?I think both virtio-net and virtio-blk use
memcpy().
With latest linux-2.6, and a development snapshot of glibc, virtio-blk will not use memcpy() anymore but virtio-net still does on the receive path (but not transmit).
Regards,
Anthony Liguori
Quoted 6 lines · Anthony Liguori
Quoted 4 lines
Quoted 2 lines
2)
cpu_physical_memory_rwdue to not using preadv/pwritev?I think both virtio-net and virtio-blk use
memcpy().With latest linux-2.6, and a development snapshot of glibc, virtio-blk will not use
memcpy()anymore but virtio-net still does on the receive path (but not transmit).
There's still the kernel/user copy, so we have two copies on rx, one on tx.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Anthony Liguori
Quoted 4 lines
Quoted 2 lines
2)
cpu_physical_memory_rwdue to not using preadv/pwritev?I think both virtio-net and virtio-blk use
memcpy().With latest linux-2.6, and a development snapshot of glibc, virtio-blk will not use
memcpy()anymore but virtio-net still does on the receive path (but not transmit).There's still the kernel/user copy, so we have two copies on rx, one on tx.
That won't show up as cpu_physical_memory_rw. stl_phys/ldl_phys are suspect though as they degrade to cpu_physical_memory_rw.
Regards,
Anthony Liguori
Quoted 8 lines · Anthony Liguori
Quoted 5 lines · Avi Kivity
Quoted 3 lines
1) I'm seeing about 2.3% in scheduler functions [that I recognize]. Does that seems a bit excessive?
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
This is why I wrote the linux-aio patch. It only reduced CPU consumption by about 2% although I'm not sure if that's absolute or relative. Andrew?
Was that before or after the entire path was made copyless?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 5 lines · Avi Kivity
Quoted 3 lines
1) I'm seeing about 2.3% in scheduler functions [that I recognize]. Does that seems a bit excessive?
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
This is why I wrote the linux-aio patch. It only reduced CPU consumption by about 2% although I'm not sure if that's absolute or relative. Andrew?
If I recall correctly, it was 2.4% and relative. But with 2.3% in scheduler functions, that's what I expected.
Quoted 2 lines
Was that before or after the entire path was made copyless?
If this is referring to the preadv/writev support, no, I have not tested with that.
-Andrew
Quoted 7 lines · Andrew Theurer
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Anthony Liguori
Quoted 4 lines · Avi Kivity
Quoted 3 lines
1) I'm seeing about 2.3% in scheduler functions [that I recognize]. Does that seems a bit excessive?
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
This is why I wrote the linux-aio patch. It only reduced CPU consumption by about 2% although I'm not sure if that's absolute or relative. Andrew?
If I recall correctly, it was 2.4% and relative. But with 2.3% in scheduler functions, that's what I expected.
Quoted 2 lines
Was that before or after the entire path was made copyless?
If this is referring to the preadv/writev support, no, I have not tested with that.
Previously, the block API only exposed non-vector interfaces and bounced vectored operations to a linear buffer. That's been eliminated now though so we need to update the linux-aio patch to implement a vectored backend interface.
However, it is an apples to apples comparison in terms of copying since the same is true with the thread pool. My take away was that the thread pool overhead isn't the major source of issues.
Regards,
Anthony Liguori
Quoted 4 lines · Anthony Liguori
Previously, the block API only exposed non-vector interfaces and bounced vectored operations to a linear buffer. That's been eliminated now though so we need to update the linux-aio patch to implement a vectored backend interface.
However, it is an apples to apples comparison in terms of copying since the same is true with the thread pool. My take away was that the thread pool overhead isn't the major source of issues.
If the overhead is dominated by copying, then you won't see the difference. Once the copying is eliminated, the comparison may yield different results. We should certainly see a difference in context switches.
One cause of context switches won't be eliminated - the non-saturating workload causes us to switch to the idle thread, which incurs a heavyweight exit. This doesn't matter since we're idle anyway, but when we switch back, we incur a heavyweight entry.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 4 lines · Anthony Liguori
Previously, the block API only exposed non-vector interfaces and bounced vectored operations to a linear buffer. That's been eliminated now though so we need to update the linux-aio patch to implement a vectored backend interface.
However, it is an apples to apples comparison in terms of copying since the same is true with the thread pool. My take away was that the thread pool overhead isn't the major source of issues.
If the overhead is dominated by copying, then you won't see the difference. Once the copying is eliminated, the comparison may yield different results. We should certainly see a difference in context switches.
Yes, I agree with this. The absence of copying (in both the thread pool and linux-aio) could yield significantly different results.
Regards,
Anthony Liguori
Quoted 2 lines
One cause of context switches won't be eliminated - the non-saturating workload causes us to switch to the idle thread, which incurs a heavyweight exit. This doesn't matter since we're idle anyway, but when we switch back, we incur a heavyweight entry.
Here are the SMT off results. This workload is designed to not over-saturate the CPU, so you have to pick a number of server sets to ensure that. With SMT on, 4 sets was enough for KVM, but 5 was too much (start seeing response time errors). For SMT off, I tried to size the load as high as we can go without running into these errors. For KVM, thats 3 (18 guests) and for Xen, that's 4 (24 guests). The throughout has a fairly linear relationship to the number of server sets used, but has a bit of wiggle room (mostly affected by response times getting longer and longer, but not exceeding the requirement set forth). Anyway, the relative throughput for these are "1.0" for KVM and "1.34" for Xen. The CPU is 78.71% for KVM the CPU is 87.83%.
If we normalize to CPU utilization, Xen is doing 20% more throughput.
Quoted 7 lines · Avi Kivity
Quoted 4 lines · Anthony Liguori
Previously, the block API only exposed non-vector interfaces and bounced vectored operations to a linear buffer. That's been eliminated now though so we need to update the linux-aio patch to implement a vectored backend interface.
However, it is an apples to apples comparison in terms of copying since the same is true with the thread pool. My take away was that the thread pool overhead isn't the major source of issues.
If the overhead is dominated by copying, then you won't see the difference. Once the copying is eliminated, the comparison may yield different results. We should certainly see a difference in context switches.
I would like to test this the proper way. What do I need to do to ensure these copies are eliminated? I am on a 2.6.27 kernel, am I missing anything there? Anthony, would you be willing to provide a patch to support the changes in the block API?
Quoted 2 lines
One cause of context switches won't be eliminated - the non-saturating workload causes us to switch to the idle thread, which incurs a heavyweight exit. This doesn't matter since we're idle anyway, but when we switch back, we incur a heavyweight entry.
I have not looked at the schedstat or ftrace yet, but will soon. Maybe it will tell us a little more about the context switches.
Here's a sample of the kvm_stat:
efer_relo exits fpu_reloa halt_exit halt_wake host_stat hypercall insn_emul insn_emul invlpg io_exits irq_exits irq_injec irq_windo kvm_reque largepage mmio_exit mmu_cache mmu_flood mmu_pde_z mmu_pte_u mmu_pte_w mmu_recyc mmu_shado mmu_unsyn mmu_unsyn nmi_injec nmi_windo pf_fixed pf_guest remote_tl request_n signal_ex tlb_flush
0 233866 53994 20353 16209 119812 0 48879 0 0 75666 44917 34772 3984 0 187 0 10 0 0 0 0 0 0 0 0 0 0 202 0 0 0 0 17698
0 244556 67321 15570 12364 116226 0 49865 0 0 69357 56131 32860 4449 0 -1895 0 19 0 0 0 0 21 21 0 0 0 0 1117 0 0 0 0 21586
0 230788 71382 10619 7920 109151 0 44354 0 0 62561 60074 28322 4841 0 103 0 13 0 0 0 0 0 0 0 0 0 0 122 0 0 0 0 22702
0 275259 82605 14326 11148 127293 0 53738 0 0 73438 70707 34724 5373 0 859 0 15 0 0 0 0 21 21 0 0 0 0 874 0 0 0 0 26723
0 250576 58760 20368 16476 128296 0 50936 0 0 80439 51219 36329 4621 0 -1170 0 8 0 0 0 0 22 22 0 0 0 0 1333 0 0 0 0 18508
0 244746 59650 19480 15657 122721 0 49882 0 0 76011 50453 35352 4523 0 201 0 11 0 0 0 0 21 21 0 0 0 0 212 0 0 0 0 19163
0 251724 71715 14049 10920 117255 0 49924 0 0 70173 58040 32328 5058 0 94 0 7 0 0 0 0 0 0 0 0 0 0 105 0 0 0 0 25405
0 247873 75212 12397 9465 117299 0 47402 0 0 68435 62901 30999 5289 0 36 0 9 0 0 0 0 0 0 0 0 0 0 47 0 0 0 0 24400
0 259105 79515 14060 10713 121489 0 50106 0 0 71847 62392 33165 4802 0 358 0 17 0 0 0 0 0 0 0 0 0 0 375 0 0 0 0 26420
0 255283 74818 13847 10642 120632 0 48832 0 0 70851 65453 32520 5032 0 752 0 6 0 0 0 0 0 0 0 0 0 0 759 0 0 0 0 23764
0 268411 78048 15231 11707 123642 0 52845 0 0 74031 64919 34404 4765 0 639 0 15 0 0 0 0 0 0 0 0 0 0 653 0 0 0 0 25992
0 247064 73794 12554 9522 115026 0 47878 0 0 66357 64359 30727 4884 0 97 0 8 0 0 0 0 0 0 0 0 0 0 107 0 0 0 0 23545
0 259641 79179 11953 9247 117090 0 49836 0 0 68561 67053 31171 5435 0 -2759 0 11 0 0 0 0 21 21 0 0 0 0 245 0 0 0 0 26858
0 258109 77455 13997 10732 121578 0 50559 0 0 71833 63841 33404 4980 0 484 0 14 0 0 0 0 21 21 0 0 0 0 495 0 0 0 0 24509
0 250245 74357 13611 10459 117791 0 49733 0 0 68471 65089 31943 4797 0 581 0 13 0 0 0 0 0 0 0 0 0 0 596 0 0 0 0 22517
0 262114 77257 13614 10499 121082 0 50683 0 0 71242 67844 33234 5031 0 1125 0 8 0 0 0 0 0 0 0 0 0 0 1133 0 0 0 0 24370
0 254914 75937 12784 9809 116020 0 50562 0 0 67452 62673 31249 4903 0 786 0 19 0 0 0 0 0 0 0 0 0 0 806 0 0 0 0 25931
0 249421 75642 12704 9805 116039 0 48426 0 0 66972 62276 31068 4999 0 -1817 0 6 0 0 0 0 21 21 0 0 0 0 187 0 0 0 0 25169
0 274205 79561 13992 10844 126452 0 53165 0 0 74522 68844 34131 5529 0 123 0 20 0 0 0 0 42 42 0 0 0 0 152 0 0 0 0 26633
0 267310 77262 15092 11705 125139 0 52891 0 0 74651 64647 34938 5018 0 195 0 18 0 0 0 0 0 0 0 0 0 0 213 0 0 0 0 25161-Andrew
Quoted 3 lines · Andrew Theurer
Quoted 2 lines
If the overhead is dominated by copying, then you won't see the difference. Once the copying is eliminated, the comparison may yield different results. We should certainly see a difference in context switches.
I would like to test this the proper way. What do I need to do to ensure these copies are eliminated? I am on a 2.6.27 kernel, am I missing anything there? Anthony, would you be willing to provide a patch to support the changes in the block API?
You need a 2.6.30 host kernel plus a libc patch. Or the linux-aio qemu patch.
Quoted 5 lines
Quoted 2 lines
One cause of context switches won't be eliminated - the non-saturating workload causes us to switch to the idle thread, which incurs a heavyweight exit. This doesn't matter since we're idle anyway, but when we switch back, we incur a heavyweight entry.
I have not looked at the schedstat or ftrace yet, but will soon. Maybe it will tell us a little more about the context switches.
Here's a sample of the
kvm_stat:
We have about 120K host_state_reloads/sec, 70K pio/sec, and 35K interrupts/sec.
That corresponds to 35K virtio notifications/sec (reasonable for 8 cores), and 85K excess context switches/sec. These can probably be eliminated by using linux-aio, except those due to idling.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 5 lines · Avi Kivity
Quoted 3 lines
1) I'm seeing about 2.3% in scheduler functions [that I recognize]. Does that seems a bit excessive?
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
This is why I wrote the linux-aio patch. It only reduced CPU consumption by about 2% although I'm not sure if that's absolute or relative. Andrew?
Was that before or after the entire path was made copyless?
Before so it's worth updating and trying again.
Regards,
Anthony Liguori
Quoted 26 lines · Avi Kivity
Quoted 7 lines · Andrew Theurer
Comparing guest time to all other busy time, that's a 23.88/43.02 = 55% overhead for virtualization. I certainly don't expect it to be 0, but 55% seems a bit high. So, what's the reason for this overhead? At the bottom is oprofile output of top functions for KVM. Some observations:
1) I'm seeing about 2.3% in scheduler functions [that I recognize]. Does that seems a bit excessive?
Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O.
Quoted 1 line
2)
cpu_physical_memory_rwdue to not using preadv/pwritev?I think both virtio-net and virtio-blk use
memcpy().Quoted 1 line
3) vmx_[save|load]_host_state: I take it this is from guest switches?
These are called when you context-switch from a guest, and, much more frequently, when you enter qemu.
Quoted 1 line
We have 180,000 context switches a second. Is this more than expected?
Way more. Across 16 logical cpus, this is >10,000 cs/sec/cpu.
Quoted 7 lines
Quoted 3 lines
I wonder if schedstats can show why we context switch (need to let someone else run, yielded, waiting on io, etc).
Yes, there is a scheduler tracer, though I have no idea how to operate it.
Do you have
kvm_statlogs?
In case the kvm_stat logs don't shed enough light, this should help.
Documentation/trace/ftrace.txt:
sched_switch
------------
This tracer simply records schedule switches. Here is an example of how to use it.
# echo sched_switch > /debug/tracing/current_tracer # echo 1 > /debug/tracing/tracing_enabled # sleep 1 # echo 0 > /debug/tracing/tracing_enabled # cat /debug/tracing/trace