Skip to contentSkip to conversation

don't call adjust_vmx_controls() second time

PATCH Applied Gleb Natapov 3 participants 6 messages Aug 27, 2009 – Aug 31, 2009

Subsystems (3)
x86 KVM KVM x86
lore
Share & tools
lore

Gleb Natapov

author
#0 lore raw

PATCH 1/1 don't call adjust_vmx_controls() second time

Don't call adjust_vmx_controls() two times for the same control. It restores options that was dropped earlier.

Signed-off-by Gleb Natapov <[email protected]>
arch/x86/kvm/vmx.c +3 -6
index 6b57eed..78101dd 100644
@@ -1262,12 +1262,9 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf)
 	if (_cpu_based_2nd_exec_control & SECONDARY_EXEC_ENABLE_EPT) {
 		/* CR3 accesses and invlpg don't need to cause VM Exits when EPT
 		   enabled */
-		min &= ~(CPU_BASED_CR3_LOAD_EXITING |
-			 CPU_BASED_CR3_STORE_EXITING |
-			 CPU_BASED_INVLPG_EXITING);
-		if (adjust_vmx_controls(min, opt, MSR_IA32_VMX_PROCBASED_CTLS,
-					&_cpu_based_exec_control) < 0)
-			return -EIO;
+		_cpu_based_exec_control &= ~(CPU_BASED_CR3_LOAD_EXITING |
+					     CPU_BASED_CR3_STORE_EXITING |
+					     CPU_BASED_INVLPG_EXITING);
 		rdmsr(MSR_IA32_VMX_EPT_VPID_CAP,
 		      vmx_capability.ept, vmx_capability.vpid);
 	}

--
Gleb.

Applied
#1 lore raw
Quoted 3 lines · Gleb Natapov

Don't call adjust_vmx_controls() two times for the same control. It restores options that was dropped earlier.

Applied, thanks. Andrew, if you rerun your benchmark atop kvm.git 'next' branch, I believe you will see dramatically better results.

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#2 lore raw
Quoted 7 lines · Avi Kivity
Quoted 3 lines · Gleb Natapov

Don't call adjust_vmx_controls() two times for the same control. It restores options that was dropped earlier.

Applied, thanks. Andrew, if you rerun your benchmark atop kvm.git 'next' branch, I believe you will see dramatically better results.

Yes!  CPU is much lower:
user  nice  system   irq softirq  guest   idle  iowait
5.81  0.00    9.48  0.08    1.04  21.32  57.86    4.41

previous CPU:
user nice system irq softirq guest idle iowait
5.67 0.00 11.64 0.09 1.05 31.90 46.06 3.59

new oprofile:

Quoted 34 lines
samples  %        app name                 symbol name
885444   53.2905  kvm-intel.ko             vmx_vcpu_run
38090     2.2924  qemu-system-x86_64       cpu_physical_memory_rw
34764     2.0923  qemu-system-x86_64       phys_page_find_alloc
25278     1.5214  vmlinux-2.6.31-rc5-autokern1 native_write_msr_safe
18205     1.0957  libc-2.5.so              memcpy
14730     0.8865  qemu-system-x86_64       qemu_get_ram_ptr
14189     0.8540  kvm.ko                   kvm_arch_vcpu_ioctl_run
12380     0.7451  vmlinux-2.6.31-rc5-autokern1 native_set_debugreg
12278     0.7390  vmlinux-2.6.31-rc5-autokern1 native_read_msr_safe
10871     0.6543  qemu-system-x86_64       virtqueue_get_head
10814     0.6508  vmlinux-2.6.31-rc5-autokern1 copy_user_generic_string
9080      0.5465  vmlinux-2.6.31-rc5-autokern1 fget_light
9015      0.5426  vmlinux-2.6.31-rc5-autokern1 schedule
8557      0.5150  qemu-system-x86_64       virtqueue_avail_bytes
7805      0.4697  vmlinux-2.6.31-rc5-autokern1 do_select
7173      0.4317  qemu-system-x86_64       lduw_phys
7019      0.4224  qemu-system-x86_64       main_loop_wait
6979      0.4200  vmlinux-2.6.31-rc5-autokern1 audit_syscall_exit
5571      0.3353  vmlinux-2.6.31-rc5-autokern1 kfree
5170      0.3112  vmlinux-2.6.31-rc5-autokern1 audit_syscall_entry
5086      0.3061  vmlinux-2.6.31-rc5-autokern1 fput
4631      0.2787  vmlinux-2.6.31-rc5-autokern1 mwait_idle
4584      0.2759  kvm.ko                   kvm_load_guest_fpu
4491      0.2703  vmlinux-2.6.31-rc5-autokern1 system_call
4461      0.2685  vmlinux-2.6.31-rc5-autokern1 __switch_to
4431      0.2667  kvm.ko                   kvm_put_guest_fpu
4371      0.2631  vmlinux-2.6.31-rc5-autokern1 __down_read
4290      0.2582  qemu-system-x86_64       kvm_run
4218      0.2539  vmlinux-2.6.31-rc5-autokern1 getnstimeofday
4129      0.2485  libpthread-2.5.so        pthread_mutex_lock
4122      0.2481  qemu-system-x86_64       ldl_phys
4100      0.2468  vmlinux-2.6.31-rc5-autokern1 do_vfs_ioctl
3811      0.2294  kvm.ko                   find_highest_vector
Quoted 9 lines
3593      0.2162  vmlinux-2.6.31-rc5-autokern1 unroll_tree_refs
3560      0.2143  vmlinux-2.6.31-rc5-autokern1 try_to_wake_up
3550      0.2137  vmlinux-2.6.31-rc5-autokern1 native_get_debugreg
3506      0.2110  kvm-intel.ko             vmcs_writel
3487      0.2099  vmlinux-2.6.31-rc5-autokern1 task_rq_lock
3434      0.2067  vmlinux-2.6.31-rc5-autokern1 __up_read
3368      0.2027  librt-2.5.so             clock_gettime
3339      0.2010  qemu-system-x86_64       virtqueue_num_heads

Thanks very much for the fix!

-Andrew

#3 lore raw
Quoted 11 lines · Andrew Theurer
Quoted 9 lines · Avi Kivity
Quoted 4 lines · Gleb Natapov

Don't call adjust_vmx_controls() two times for the same control. It restores options that was dropped earlier.

Applied, thanks. Andrew, if you rerun your benchmark atop kvm.git 'next' branch, I believe you will see dramatically better results.

Quoted 9 lines
Yes!  CPU is much lower:
user  nice  system   irq softirq  guest   idle  iowait
5.81  0.00    9.48  0.08    1.04  21.32  57.86    4.41

previous CPU:
user nice system irq softirq guest idle iowait
5.67 0.00 11.64 0.09 1.05 31.90 46.06 3.59

How does it compare to the other hypervisor now?

Quoted 6 lines

new oprofile:

Quoted 3 lines
samples  %        app name                 symbol name
885444   53.2905  kvm-intel.ko             vmx_vcpu_run

guest mode = good

Quoted 4 lines
Quoted 4 lines
38090     2.2924  qemu-system-x86_64       cpu_physical_memory_rw
34764     2.0923  qemu-system-x86_64       phys_page_find_alloc
14730     0.8865  qemu-system-x86_64       qemu_get_ram_ptr
10814     0.6508  vmlinux-2.6.31-rc5-autokern1 copy_user_generic_string
Quoted 8 lines
Quoted 8 lines
10871     0.6543  qemu-system-x86_64       virtqueue_get_head
8557      0.5150  qemu-system-x86_64       virtqueue_avail_bytes
7173      0.4317  qemu-system-x86_64       lduw_phys
4122      0.2481  qemu-system-x86_64       ldl_phys
3339      0.2010  qemu-system-x86_64       virtqueue_num_heads
4129      0.2485  libpthread-2.5.so        pthread_mutex_lock

virtio and related qemu overhead: 8.2%.

Quoted 3 lines
Quoted 3 lines
25278     1.5214  vmlinux-2.6.31-rc5-autokern1 native_write_msr_safe
12278     0.7390  vmlinux-2.6.31-rc5-autokern1 native_read_msr_safe

This will be reduced to if we move virtio to kernel context.

Quoted 3 lines
Quoted 3 lines
12380     0.7451  vmlinux-2.6.31-rc5-autokern1 native_set_debugreg
3550      0.2137  vmlinux-2.6.31-rc5-autokern1 native_get_debugreg

A lot less than before, but still annoying.

Quoted 2 lines
Quoted 2 lines

4631 0.2787 vmlinux-2.6.31-rc5-autokern1 mwait_idle

idle=halt may improve this, mwait is slow.

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw
Quoted 16 lines · Avi Kivity
Quoted 15 lines · Andrew Theurer
Quoted 9 lines · Avi Kivity
Quoted 4 lines · Gleb Natapov

Don't call adjust_vmx_controls() two times for the same control. It restores options that was dropped earlier.

Applied, thanks. Andrew, if you rerun your benchmark atop kvm.git 'next' branch, I believe you will see dramatically better results.

Yes!  CPU is much lower:
user  nice  system   irq softirq  guest   idle  iowait
5.81  0.00    9.48  0.08    1.04  21.32  57.86    4.41
Quoted 7 lines
Quoted 5 lines

previous CPU:
user nice system irq softirq guest idle iowait
5.67 0.00 11.64 0.09 1.05 31.90 46.06 3.59

How does it compare to the other hypervisor now?

My original results for other hypervisor were a little inaccurate. They mistakenly used 2 vcpu guests. New runs with 1 vcpu guests (as used in kvm) have slightly lower CPU utilization. Anyway, here's the breakdown:

                               CPU       percent more CPU
kvm-master/qemu-kvm-87:        50.15     78%
kvm-next/qemu-kvm-87:          37.73     34%
Quoted 23 lines
Quoted 7 lines

new oprofile:

Quoted 3 lines
samples  %        app name                 symbol name
885444   53.2905  kvm-intel.ko             vmx_vcpu_run

guest mode = good

Quoted 4 lines
Quoted 4 lines
38090     2.2924  qemu-system-x86_64       cpu_physical_memory_rw
34764     2.0923  qemu-system-x86_64       phys_page_find_alloc
14730     0.8865  qemu-system-x86_64       qemu_get_ram_ptr
10814     0.6508  vmlinux-2.6.31-rc5-autokern1 copy_user_generic_string
Quoted 8 lines
Quoted 8 lines
10871     0.6543  qemu-system-x86_64       virtqueue_get_head
8557      0.5150  qemu-system-x86_64       virtqueue_avail_bytes
7173      0.4317  qemu-system-x86_64       lduw_phys
4122      0.2481  qemu-system-x86_64       ldl_phys
3339      0.2010  qemu-system-x86_64       virtqueue_num_heads
4129      0.2485  libpthread-2.5.so        pthread_mutex_lock
Quoted 7 lines

virtio and related qemu overhead: 8.2%.

Quoted 3 lines
Quoted 3 lines
25278     1.5214  vmlinux-2.6.31-rc5-autokern1 native_write_msr_safe
12278     0.7390  vmlinux-2.6.31-rc5-autokern1 native_read_msr_safe

This will be reduced to if we move virtio to kernel context.

Are there plans to move that to kernel for disk, too?

Quoted 11 lines
Quoted 3 lines
Quoted 3 lines
12380     0.7451  vmlinux-2.6.31-rc5-autokern1 native_set_debugreg
3550      0.2137  vmlinux-2.6.31-rc5-autokern1 native_get_debugreg

A lot less than before, but still annoying.

Quoted 3 lines
Quoted 3 lines

4631 0.2787 vmlinux-2.6.31-rc5-autokern1 mwait_idle

idle=halt may improve this, mwait is slow.

I can try idle-halt on the host. I actually assumed it would be using that, but I'll check.

Thanks,

-Andrew

#5 lore raw
Quoted 9 lines · Andrew Theurer
Quoted 1 line

How does it compare to the other hypervisor now?

My original results for other hypervisor were a little inaccurate. They mistakenly used 2 vcpu guests. New runs with 1 vcpu guests (as used in kvm) have slightly lower CPU utilization. Anyway, here's the breakdown:

                               CPU       percent more CPU
kvm-master/qemu-kvm-87:        50.15     78%
kvm-next/qemu-kvm-87:          37.73     34%

Much better, though still a lot of work to do.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
25278     1.5214  vmlinux-2.6.31-rc5-autokern1 native_write_msr_safe
12278     0.7390  vmlinux-2.6.31-rc5-autokern1 native_read_msr_safe

This will be reduced to if we move virtio to kernel context.

Are there plans to move that to kernel for disk, too?

We don't know if disk or net contributed to this. If it turns out that vhost-blk makes sense, we'll do it.

Signature · 1 line
error compiling committee.c: too many arguments to function