KVM: Use thread debug register storage instead of kvm specific data
PATCH 1/1 KVM: Use thread debug register storage instead of kvm specific data
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
--- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/x86.c | 22 +++++++--------------- 2 files changed, 7 insertions(+), 18 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -3
@@ -362,9 +362,6 @@ struct kvm_vcpu_arch {u32 pat;int switch_db_regs; -unsigned long host_db[KVM_NR_DB_REGS ]; -unsigned long host_dr6; -unsigned long host_dr7;unsigned long db[KVM_NR_DB_REGS ];unsigned long dr6;unsigned long dr7;
arch/x86/kvm/x86.c +7 -15
@@ -3627,14 +3627,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)kvm_guest_enter (); -get_debugreg (vcpu->arch.host_dr6,6 ); -get_debugreg (vcpu->arch.host_dr7,7 );if (unlikely (vcpu->arch.switch_db_regs)) { -get_debugreg (vcpu->arch.host_db[0 ],0 ); -get_debugreg (vcpu->arch.host_db[1 ],1 ); -get_debugreg (vcpu->arch.host_db[2 ],2 ); -get_debugreg (vcpu->arch.host_db[3 ],3 ); -set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );
@@ -3645,15 +3638,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu); -if (unlikely (vcpu->arch.switch_db_regs)) { -set_debugreg (0 ,7 ); -set_debugreg (vcpu->arch.host_db[0 ],0 ); -set_debugreg (vcpu->arch.host_db[1 ],1 ); -set_debugreg (vcpu->arch.host_db[2 ],2 ); -set_debugreg (vcpu->arch.host_db[3 ],3 ); +if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) { +set_debugreg (current->thread.debugreg0,0 ); +set_debugreg (current->thread.debugreg1,1 ); +set_debugreg (current->thread.debugreg2,2 ); +set_debugreg (current->thread.debugreg3,3 ); +set_debugreg (current->thread.debugreg6,6 ); +set_debugreg (current->thread.debugreg7,7 ); } -set_debugreg (vcpu->arch.host_dr6,6 ); -set_debugreg (vcpu->arch.host_dr7,7 );set_bit (KVM_REQ_KICK , &vcpu->requests);local_irq_enable ();
Signature · 1 line
1.6.4.1
Quoted 6 lines · Avi Kivity
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
Andrew, this is now available as the 'debugreg' branch of kvm.git. Given the massive performance improvement, it will be interesting to see how the test results change.
Marcelo, please queue this for 2.6.32, and I think it's even suitable for -stable.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Avi Kivity
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
Quoted 7 lines
Andrew, this is now available as the 'debugreg' branch of kvm.git. Given the massive performance improvement, it will be interesting to see how the test results change.
Marcelo, please queue this for 2.6.32, and I think it's even suitable for -stable.
Here's a run from branch debugreg with thread debugreg storage + conditionally reload dr6:
user nice system irq softirq guest idle iowait
5.79 0.00 9.28 0.08 1.00 20.81 58.78 4.26
total busy: 36.97
Previous run that had avoided calling adjust_vmx_controls twice:
user nice system irq softirq guest idle iowait
5.81 0.00 9.48 0.08 1.04 21.32 57.86 4.41
total busy: 37.73
A relative reduction CPU cycles of 2%
new oprofile:
Quoted 26 lines
samples % app name symbol name 876648 54.1555 kvm-intel.ko vmx_vcpu_run 37595 2.3225 qemu-system-x86_64 cpu_physical_memory_rw 35623 2.2006 qemu-system-x86_64 phys_page_find_alloc 24874 1.5366 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 native_write_msr_safe 17710 1.0940 libc-2.5.so memcpy 14664 0.9059 kvm.ko kvm_arch_vcpu_ioctl_run 14577 0.9005 qemu-system-x86_64 qemu_get_ram_ptr 12528 0.7739 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 native_read_msr_safe 10979 0.6782 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 copy_user_generic_string 9979 0.6165 qemu-system-x86_64 virtqueue_get_head 9371 0.5789 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 schedule 8333 0.5148 qemu-system-x86_64 virtqueue_avail_bytes 7899 0.4880 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 fget_light 7289 0.4503 qemu-system-x86_64 main_loop_wait 7217 0.4458 qemu-system-x86_64 lduw_phys 6821 0.4214 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 audit_syscall_exit 6749 0.4169 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 do_select 5919 0.3657 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 audit_syscall_entry 5466 0.3377 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 kfree 4887 0.3019 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 fput 4689 0.2897 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 __switch_to 4636 0.2864 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 mwait_idle 4505 0.2783 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 getnstimeofday 4453 0.2751 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 system_call 4403 0.2720 kvm.ko kvm_load_guest_fpu
Quoted 9 lines
4285 0.2647 kvm.ko kvm_put_guest_fpu 4241 0.2620 libpthread-2.5.so pthread_mutex_lock 4172 0.2577 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 unroll_tree_refs 4100 0.2533 qemu-system-x86_64 kvm_run 4044 0.2498 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 __down_read 3978 0.2457 qemu-system-x86_64 ldl_phys 3669 0.2267 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 do_vfs_ioctl 3655 0.2258 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 __up_read
A diff of this and previous run's oprofile:
Quoted 40 lines
profile1 is [./oprofile.before]
profile2 is [./oprofile.after]
CountedCPU_CLK_UNHALTEDevents (Clock cycles when not halted) with a unit mask of0x00(No unit mask) count 10000000 total samples (ts1) for profile1 is 1661542
total samples (ts2) for profile2 is 1618760 (includes multiplier of 1.000000)
functions which have a abs(pct2-pct1) < 0.02 are not displayedpct2: pct1: 100* 100* pct2 s1 s2 s2/s1 s2/ts1 s1/ts1 -pct1 symbol bin --------- --------- ------- ------- ------- ------ ------ --- 1559 2747 1.76/1 0.165 0.094 0.071 dput vmlinux 34764 35623 1.02/1 2.144 2.092 0.052 phys_page_find_alloc qemu 5170 5919 1.14/1 0.356 0.311 0.045 audit_syscall_entry vmlinux 3593 4172 1.16/1 0.251 0.216 0.035 unroll_tree_refs vmlinux 896 1414 1.58/1 0.085 0.054 0.031 kstat_irqs_cpu vmlinux 199 676 3.40/1 0.041 0.012 0.029 /usr/lib64/perl5/5.8 libperl.so 14189 14664 1.03/1 0.883 0.854 0.029 kvm_arch_vcpu_ioctl_ kvm.ko 1314 1779 1.35/1 0.107 0.079 0.028 kvm_arch_post_kvm_ru qemu 1390 1785 1.28/1 0.107 0.084 0.024 vfs_read vmlinux 9015 9371 1.04/1 0.564 0.543 0.021 schedule vmlinux 1265 1610 1.27/1 0.097 0.076 0.021 qemu_get_clock qemu 600 941 1.57/1 0.057 0.036 0.021 preempt_notifier_reg vmlinux 3339 2996 1/1.11 0.180 0.201 -0.021 virtqueue_num_heads qemu 822 463 1/1.78 0.028 0.049 -0.022 vmx_vcpu_put kvm 1116 743 1/1.50 0.045 0.067 -0.022 cap_file_ioctl vmlinux 2967 2587 1/1.15 0.156 0.179 -0.023 native_read_tsc vmlinux 3487 3096 1/1.13 0.186 0.210 -0.024 task_rq_lock vmlinux 1780 1381 1/1.29 0.083 0.107 -0.024 math_state_restore vmlinux 25278 24874 1/1.02 1.497 1.521 -0.024 native_write_msr_saf vmlinux 1733 1329 1/1.30 0.080 0.104 -0.024 rw_verify_area vmlinux 2305 1897 1/1.22 0.114 0.139 -0.025 __hrtimer_start_rang vmlinux 1685 1273 1/1.32 0.077 0.101 -0.025 tcp_poll vmlinux 1059 637 1/1.66 0.038 0.064 -0.025 kvm_lapic_get_cr8 kvm.ko 2855 2425 1/1.18 0.146 0.172 -0.026 signalfd_poll vmlinux 4100 3669 1/1.12 0.221 0.247 -0.026 do_vfs_ioctl vmlinux 1941 1498 1/1.30 0.090 0.117 -0.027 complete_pio kvm.ko 3506 3052 1/1.15 0.184 0.211 -0.027 vmcs_writel kvm 18205 17710 1/1.03 1.066 1.096 -0.030 memcpy libc 1717 1114 1/1.54 0.067 0.103 -0.036 rb_insert_color vmlinux
Quoted 9 lines
10871 9979 1/1.09 0.601 0.654 -0.054 virtqueue_get_head qemu
7805 6749 1/1.16 0.406 0.470 -0.064 do_select vmlinux
9080 7899 1/1.15 0.475 0.546 -0.071 fget_light vmlinux
3550 0 0.000 0.214 -0.214 native_get_debugreg vmlinux
885444 876648 1/1.01 52.761 53.290 -0.529 vmx_vcpu_run kvm
12380 0 0.000 0.745 -0.745 native_set_debugreg vmlinux
------- ------- ------
94.916 97.469 -2.553 Notice native_set/get_debugreg is now not an issue :)
All of these still use qemu-kvm-87
-Andrew
Quoted 7 lines · Andrew Theurer
Here's a run from branch debugreg with thread debugreg storage + conditionally reload dr6:
user nice system irq softirq guest idle iowait
5.79 0.00 9.28 0.08 1.00 20.81 58.78 4.26
total busy: 36.97
Quoted 8 lines
Previous run that had avoided calling
adjust_vmx_controlstwice:user nice system irq softirq guest idle iowait
5.81 0.00 9.48 0.08 1.04 21.32 57.86 4.41
total busy: 37.73A relative reduction CPU cycles of 2%
That was an wasy fruit to pick. To bad it was a regression that we introduced.
Quoted 20 lines
new oprofile:
Quoted 17 lines
samples % app name symbol name 876648 54.1555 kvm-intel.ko vmx_vcpu_run 37595 2.3225 qemu-system-x86_64 cpu_physical_memory_rw 35623 2.2006 qemu-system-x86_64 phys_page_find_alloc 24874 1.5366 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 native_write_msr_safe 17710 1.0940 libc-2.5.so memcpy 14664 0.9059 kvm.ko kvm_arch_vcpu_ioctl_run 14577 0.9005 qemu-system-x86_64 qemu_get_ram_ptr 12528 0.7739 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 native_read_msr_safe 10979 0.6782 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 copy_user_generic_string 9979 0.6165 qemu-system-x86_64 virtqueue_get_head 9371 0.5789 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 schedule 8333 0.5148 qemu-system-x86_64 virtqueue_avail_bytes 7899 0.4880 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 fget_light 7289 0.4503 qemu-system-x86_64 main_loop_wait 7217 0.4458 qemu-system-x86_64 lduw_phys
This is almost entirely host virtio. I can reduce native_write_msr_safe by a bit, but not much.
Quoted 8 lines
Quoted 8 lines
6821 0.4214 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 audit_syscall_exit 6749 0.4169 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 do_select 5919 0.3657 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 audit_syscall_entry 5466 0.3377 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 kfree 4887 0.3019 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 fput 4689 0.2897 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 __switch_to 4636 0.2864 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 mwait_idle
Still not idle=poll, it may shave off 0.2%.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 53 lines · Avi Kivity
Quoted 7 lines · Andrew Theurer
Here's a run from branch debugreg with thread debugreg storage + conditionally reload dr6:
user nice system irq softirq guest idle iowait
5.79 0.00 9.28 0.08 1.00 20.81 58.78 4.26
total busy: 36.97Quoted 8 lines
Previous run that had avoided calling
adjust_vmx_controlstwice:user nice system irq softirq guest idle iowait
5.81 0.00 9.48 0.08 1.04 21.32 57.86 4.41
total busy: 37.73A relative reduction CPU cycles of 2%
That was an wasy fruit to pick. To bad it was a regression that we introduced.
Quoted 20 lines
new oprofile:
Quoted 17 lines
samples % app name symbol name 876648 54.1555 kvm-intel.ko vmx_vcpu_run 37595 2.3225 qemu-system-x86_64 cpu_physical_memory_rw 35623 2.2006 qemu-system-x86_64 phys_page_find_alloc 24874 1.5366 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 native_write_msr_safe 17710 1.0940 libc-2.5.so memcpy 14664 0.9059 kvm.ko kvm_arch_vcpu_ioctl_run 14577 0.9005 qemu-system-x86_64 qemu_get_ram_ptr 12528 0.7739 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 native_read_msr_safe 10979 0.6782 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 copy_user_generic_string 9979 0.6165 qemu-system-x86_64 virtqueue_get_head 9371 0.5789 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 schedule 8333 0.5148 qemu-system-x86_64 virtqueue_avail_bytes 7899 0.4880 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 fget_light 7289 0.4503 qemu-system-x86_64 main_loop_wait 7217 0.4458 qemu-system-x86_64 lduw_physThis is almost entirely host virtio. I can reduce
native_write_msr_safeby a bit, but not much.Quoted 8 lines
Quoted 8 lines
6821 0.4214 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 audit_syscall_exit 6749 0.4169 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 do_select 5919 0.3657 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 audit_syscall_entry 5466 0.3377 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 kfree 4887 0.3019 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 fput 4689 0.2897 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 __switch_to 4636 0.2864 vmlinux-2.6.31-rc5_debugreg_v2.6.31-rc3-3441-g479fa73-autokern1 mwait_idle
Quoted 1 line
Still not idle=poll, it may shave off 0.2%.
Won't this affect SMT in a negative way? (OK, I am not running SMT now, but eventually we will be) A long time ago, we tested P4's with HT, and a polling idle in one thread always negatively impacted performance in the sibling thread.
FWIW, I did try idle=halt, and it was slightly worse.
I did get a chance to try the latest qemu (master and next heads). I have been running into a problem with virtIO stor driver for windows on anything much newer than kvm-87. I compiled the driver from the new git tree, installed OK, but still had the same error. Finally, I removed the serial number feature in the virtio-blk in qemu, and I can now get the driver to work in Windows.
So, not really any good news on performance with latest qemu builds. Performance is slightly worse:
qemu-kvm-87
user nice system irq softirq guest idle iowait
5.79 0.00 9.28 0.08 1.00 20.81 58.78 4.26
total busy: 36.97
qemu-kvm-88-905-g6025b2d (master)
user nice system irq softirq guest idle iowait
6.57 0.00 10.86 0.08 1.02 21.35 55.90 4.21
total busy: 39.89
qemu-kvm-88-910-gbf8a05b (next)
user nice system irq softirq guest idle iowait 6.60 0.00 10.91 0.09 1.03 21.35 55.71 4.31
total busy: 39.98
diff of profiles, p1=qemu-kvm-87, p2=qemu-master
Quoted 38 lines
profile1 is qemu-kvm-87
profile2 is qemu-master
CountedCPU_CLK_UNHALTEDevents (Clock cycles when not halted) with a unit mask of0x00(No unit mask) count 10000000 total samples (ts1) for profile1 is 1616921
total samples (ts2) for profile2 is 1752347 (includes multiplier of 0.995420)
functions which have a abs(pct2-pct1) < 0.06 are not displayedpct2: pct1: 100* 100* pct2 s1 s2 s2/s1 s2/ts1 s1/ts1 -pct1 symbol bin --------- --------- ------- ------- ------- ------ ------ --- 879611 907883 1.03/1 56.149 54.400 1.749 vmx_vcpu_run kvm 614 11553 18.82/1 0.715 0.038 0.677 gfn_to_memslot_unali kvm.ko 34511 44922 1.30/1 2.778 2.134 0.644 phys_page_find_alloc qemu 2866 9334 3.26/1 0.577 0.177 0.400 paging64_walk_addr kvm.ko 11139 17200 1.54/1 1.064 0.689 0.375 copy_user_generic_st vmlinux 3100 7108 2.29/1 0.440 0.192 0.248 x86_decode_insn kvm.ko 8169 11873 1.45/1 0.734 0.505 0.229 virtqueue_avail_byte qemu 1103 4540 4.12/1 0.281 0.068 0.213 kvm_read_guest kvm.ko 17427 20401 1.17/1 1.262 1.078 0.184 memcpy libc 0 2905 0.180 0.000 0.180 gfn_to_pfn kvm.ko 1831 4328 2.36/1 0.268 0.113 0.154 x86_emulate_insn kvm.ko 65 2431 37.41/1 0.150 0.004 0.146 emulator_read_emulat kvm.ko 14922 17196 1.15/1 1.064 0.923 0.141 qemu_get_ram_ptr qemu 545 2724 5.00/1 0.168 0.034 0.135 emulate_instruction kvm.ko 599 2464 4.11/1 0.152 0.037 0.115 kvm_read_guest_page kvm.ko 503 2355 4.68/1 0.146 0.031 0.115 gfn_to_hva kvm.ko 1076 2918 2.71/1 0.181 0.067 0.114 memcpy_c vmlinux 594 2241 3.77/1 0.139 0.037 0.102 next_segment kvm.ko 1680 3248 1.93/1 0.201 0.104 0.097 pipe_poll vmlinux 0 1463 0.090 0.000 0.090 subpage_readl qemu 0 1363 0.084 0.000 0.084 msix_enabled qemu 527 1883 3.57/1 0.116 0.033 0.084 paging64_gpte_to_gfn kvm.ko 962 2223 2.31/1 0.138 0.059 0.078 do_insn_fetch kvm.ko 348 1605 4.61/1 0.099 0.022 0.078 is_rsvd_bits_set kvm.ko 520 1763 3.39/1 0.109 0.032 0.077 unalias_gfn kvm.ko 1 1163 1163.65/1 0.072 0.000 0.072 tdp_page_fault kvm.ko 3827 4912 1.28/1 0.304 0.237 0.067 __down_read vmlinux
Quoted 8 lines
0 1014 0.063 0.000 0.063 mapping_level kvm.ko
973 0 0.000 0.060 -0.060 pm_ioport_readl qemu
1635 528 1/3.09 0.033 0.101 -0.068 ioport_read qemu
2179 1017 1/2.14 0.063 0.135 -0.072 kvm_emulate_pio kvm.ko
25141 23722 1/1.06 1.467 1.555 -0.088 native_write_msr_saf vmlinux
1560 0 0.000 0.096 -0.096 eventfd_poll vmlinux
------- ------- ------
105.100 97.450 7.650 18x more samples for gfn_to_memslot_unali*, 37x for emulator_read_emula*, and more CPU time in guest mode.
One other thing I decided to try was some cpu binding. I know this is not practical for production, but I wanted to see if there's any benefit at all. One reason was that a coworker here tried binding the qemu thread for the vcpu and the qemu IO thread to the same cpu. On a networking test, guest->local-host, throughput was up about 2x. Obviously there was a nice effect of being on the same cache. I wondered, even without full bore throughput tests, could we see any benefit here. So, I bound each pair of VMs to a dedicated core. What I saw was about a 6% improvement in performance. For a system which has pretty incredible memory performance and is not that busy, I was surprised that I got 6%. I am not advocating binding, but what I do
wonder: on 1-way VMs, if we keep all the qemu threads together on the same CPU, but still allowing the scheduler to move them (all of them at once) to different cpus over time, would we see the same benefit?
One other thing: So far I have not been using preadv/pwritev. I assume I need a more recent glibc (on 2.5 now) for qemu to take advantage of this?
Thanks!
-Andrew
On Friday 04 September 2009 09:48:17 am Andrew Theurer wrote:
<snip>
Quoted 8 lines
Quoted 2 lines
Still not idle=poll, it may shave off 0.2%.
Won't this affect SMT in a negative way? (OK, I am not running SMT now, but eventually we will be) A long time ago, we tested P4's with HT, and a polling idle in one thread always negatively impacted performance in the sibling thread.
Quoted 8 lines
FWIW, I did try idle=halt, and it was slightly worse.
I did get a chance to try the latest qemu (master and next heads). I have been running into a problem with virtIO stor driver for windows on anything much newer than kvm-87. I compiled the driver from the new git tree, installed OK, but still had the same error. Finally, I removed the serial number feature in the virtio-blk in qemu, and I can now get the driver to work in Windows.
What were the symptoms you were seeing (i.e. define "a problem").
Quoted 21 lines
So, not really any good news on performance with latest qemu builds. Performance is slightly worse:
qemu-kvm-87
user nice system irq softirq guest idle iowait
5.79 0.00 9.28 0.08 1.00 20.81 58.78 4.26
total busy: 36.97qemu-kvm-88-905-g6025b2d (master)
user nice system irq softirq guest idle iowait
6.57 0.00 10.86 0.08 1.02 21.35 55.90 4.21
total busy: 39.89qemu-kvm-88-910-gbf8a05b (next)
user nice system irq softirq guest idle iowait 6.60 0.00 10.91 0.09 1.03 21.35 55.71 4.31total busy: 39.98
diff of profiles, p1=qemu-kvm-87, p2=qemu-master
<snip>
Quoted 19 lines
18x more samples for gfn_to_memslot_unali*, 37x for emulator_read_emula*, and more CPU time in guest mode.
One other thing I decided to try was some cpu binding. I know this is not practical for production, but I wanted to see if there's any benefit at all. One reason was that a coworker here tried binding the qemu thread for the vcpu and the qemu IO thread to the same cpu. On a networking test, guest->local-host, throughput was up about 2x. Obviously there was a nice effect of being on the same cache. I wondered, even without full bore throughput tests, could we see any benefit here. So, I bound each pair of VMs to a dedicated core. What I saw was about a 6% improvement in performance. For a system which has pretty incredible memory performance and is not that busy, I was surprised that I got 6%. I am not advocating binding, but what I do
wonder: on 1-way VMs, if we keep all the qemu threads together on the same CPU, but still allowing the scheduler to move them (all of them at once) to different cpus over time, would we see the same benefit?
Quoted 3 lines
One other thing: So far I have not been using preadv/pwritev. I assume I need a more recent glibc (on 2.5 now) for qemu to take advantage of this?
Getting p(read|write)v working almost doubled my virtio-net throughput in a Linux guest. Not quite as much in Windows guests. Yes you need glibc-2.10. I think some distros might have backported it to 2.9. You will also need some support for it in your system includes.
--Iggy
Quoted 5 lines
Thanks!
-Andrew
Quoted 10 lines · Brian Jackson
On Friday 04 September 2009 09:48:17 am Andrew Theurer wrote:
<snip>Quoted 8 lines
Quoted 1 line
Still not idle=poll, it may shave off 0.2%.
Won't this affect SMT in a negative way? (OK, I am not running SMT now, but eventually we will be) A long time ago, we tested P4's with HT, and a polling idle in one thread always negatively impacted performance in the sibling thread.
FWIW, I did try idle=halt, and it was slightly worse.
Quoted 8 lines
Quoted 6 lines
I did get a chance to try the latest qemu (master and next heads). I have been running into a problem with virtIO stor driver for windows on anything much newer than kvm-87. I compiled the driver from the new git tree, installed OK, but still had the same error. Finally, I removed the serial number feature in the virtio-blk in qemu, and I can now get the driver to work in Windows.
What were the symptoms you were seeing (i.e. define "a problem").
Device manager reports "a problem code 10" occurred, and the driver cannot initialize.
Vadim Rozenfeld informed me:
Quoted 2 lines
There is a sanity check in the code, which checks the I/O range and fails if is not equal to 40h.
Resent virtio-blk devices have I/O range equal to0x400(serial number feature). So, out signed viostor driver will fail on the latest KVMs. This problem was fixed
and committed to SVN some time ago.
I assumed the fix was to the virtio windows driver, but I could not get the driver I compiled from latest git to work either (only on qemu-kvm-87). So, I just backed out the serial number feature in qemu, and it worked. FWIW, the linux virtio-blk driver never had a problem.
Quoted 40 lines
Quoted 21 lines
So, not really any good news on performance with latest qemu builds. Performance is slightly worse:
qemu-kvm-87
user nice system irq softirq guest idle iowait
5.79 0.00 9.28 0.08 1.00 20.81 58.78 4.26
total busy: 36.97qemu-kvm-88-905-g6025b2d (master)
user nice system irq softirq guest idle iowait
6.57 0.00 10.86 0.08 1.02 21.35 55.90 4.21
total busy: 39.89qemu-kvm-88-910-gbf8a05b (next)
user nice system irq softirq guest idle iowait 6.60 0.00 10.91 0.09 1.03 21.35 55.71 4.31total busy: 39.98
diff of profiles, p1=qemu-kvm-87, p2=qemu-master
<snip>
Quoted 18 lines
18x more samples for gfn_to_memslot_unali*, 37x for emulator_read_emula*, and more CPU time in guest mode.
One other thing I decided to try was some cpu binding. I know this is not practical for production, but I wanted to see if there's any benefit at all. One reason was that a coworker here tried binding the qemu thread for the vcpu and the qemu IO thread to the same cpu. On a networking test, guest->local-host, throughput was up about 2x. Obviously there was a nice effect of being on the same cache. I wondered, even without full bore throughput tests, could we see any benefit here. So, I bound each pair of VMs to a dedicated core. What I saw was about a 6% improvement in performance. For a system which has pretty incredible memory performance and is not that busy, I was surprised that I got 6%. I am not advocating binding, but what I do
wonder: on 1-way VMs, if we keep all the qemu threads together on the same CPU, but still allowing the scheduler to move them (all of them at once) to different cpus over time, would we see the same benefit?
Quoted 5 lines
Quoted 3 lines
One other thing: So far I have not been using preadv/pwritev. I assume I need a more recent glibc (on 2.5 now) for qemu to take advantage of this?
Getting p(read|write)v working almost doubled my virtio-net throughput in a Linux guest. Not quite as much in Windows guests. Yes you need glibc-2.10. I think some distros might have backported it to 2.9. You will also need some support for it in your system includes.
Thanks, I will try a newer glibc, or maybe just move to a newer Linux installation which happens to have a newer glic.
-Andrew
Quoted 20 lines · Andrew Theurer
Quoted 19 lines · Brian Jackson
On Friday 04 September 2009 09:48:17 am Andrew Theurer wrote:
<snip>Quoted 15 lines
Quoted 1 line
Still not idle=poll, it may shave off 0.2%.
Won't this affect SMT in a negative way? (OK, I am not running SMT now, but eventually we will be) A long time ago, we tested P4's with HT, and a polling idle in one thread always negatively impacted performance in the sibling thread.
FWIW, I did try idle=halt, and it was slightly worse.
I did get a chance to try the latest qemu (master and next heads). I have been running into a problem with virtIO stor driver for windows on anything much newer than kvm-87. I compiled the driver from the new git tree, installed OK, but still had the same error. Finally, I removed the serial number feature in the virtio-blk in qemu, and I can now get the driver to work in Windows.
Quoted 4 lines
Quoted 1 line
What were the symptoms you were seeing (i.e. define "a problem").
Device manager reports "a problem code 10" occurred, and the driver cannot initialize.
Yes! I was getting this after I moved from 0.10.6 to 0.11.0-rc1. Now I know how to fix it. Thank you. Thank you.
Quoted 7 lines
Vadim Rozenfeld informed me:
Quoted 4 lines
There is a sanity check in the code, which checks the I/O range and fails if is not equal to 40h. Resent virtio-blk devices have I/O range equal to
0x400(serial number feature). So, out signed viostor driver will fail on the latest KVMs. This problem was fixed
Quoted 6 lines
and committed to SVN some time ago.
I assumed the fix was to the virtio windows driver, but I could not get the driver I compiled from latest git to work either (only on qemu-kvm-87). So, I just backed out the serial number feature in qemu, and it worked. FWIW, the linux virtio-blk driver never had a problem.
There have been very few changes to the viostor windows git repo since it was opened. Unless it was done before they were open sourced. In any case, it doesn't seem to be working with what's publicly available, so I think maybe there is something missing internal to external.
Quoted 45 lines
Quoted 45 lines
Quoted 20 lines
So, not really any good news on performance with latest qemu builds. Performance is slightly worse:
qemu-kvm-87
user nice system irq softirq guest idle iowait
5.79 0.00 9.28 0.08 1.00 20.81 58.78 4.26
total busy: 36.97qemu-kvm-88-905-g6025b2d (master)
user nice system irq softirq guest idle iowait
6.57 0.00 10.86 0.08 1.02 21.35 55.90 4.21
total busy: 39.89qemu-kvm-88-910-gbf8a05b (next)
user nice system irq softirq guest idle iowait 6.60 0.00 10.91 0.09 1.03 21.35 55.71 4.31total busy: 39.98
diff of profiles, p1=qemu-kvm-87, p2=qemu-master
<snip>
Quoted 21 lines
18x more samples for gfn_to_memslot_unali*, 37x for emulator_read_emula*, and more CPU time in guest mode.
One other thing I decided to try was some cpu binding. I know this is not practical for production, but I wanted to see if there's any benefit at all. One reason was that a coworker here tried binding the qemu thread for the vcpu and the qemu IO thread to the same cpu. On a networking test, guest->local-host, throughput was up about 2x. Obviously there was a nice effect of being on the same cache. I wondered, even without full bore throughput tests, could we see any benefit here. So, I bound each pair of VMs to a dedicated core. What I saw was about a 6% improvement in performance. For a system which has pretty incredible memory performance and is not that busy, I was surprised that I got 6%. I am not advocating binding, but what I do
wonder: on 1-way VMs, if we keep all the qemu threads together on the same CPU, but still allowing the scheduler to move them (all of them at once) to different cpus over time, would we see the same benefit?One other thing: So far I have not been using preadv/pwritev. I assume I need a more recent glibc (on 2.5 now) for qemu to take advantage of this?
Quoted 7 lines
Quoted 4 lines
Getting p(read|write)v working almost doubled my virtio-net throughput in a Linux guest. Not quite as much in Windows guests. Yes you need glibc-2.10. I think some distros might have backported it to 2.9. You will also need some support for it in your system includes.
Thanks, I will try a newer glibc, or maybe just move to a newer Linux installation which happens to have a newer glic.
Fwiw... In Debian, I had to get glibc from the experimental tree. So some distros might not even have it.
Quoted 3 lines
-Andrew
Quoted 50 lines · Brian Jackson
Quoted 22 lines · Andrew Theurer
Quoted 20 lines · Brian Jackson
On Friday 04 September 2009 09:48:17 am Andrew Theurer wrote:
<snip>Quoted 16 lines
Quoted 2 lines
Still not idle=poll, it may shave off 0.2%.
Won't this affect SMT in a negative way? (OK, I am not running SMT now, but eventually we will be) A long time ago, we tested P4's with HT, and a polling idle in one thread always negatively impacted performance in the sibling thread.
FWIW, I did try idle=halt, and it was slightly worse.
I did get a chance to try the latest qemu (master and next heads). I have been running into a problem with virtIO stor driver for windows on anything much newer than kvm-87. I compiled the driver from the new git tree, installed OK, but still had the same error. Finally, I removed the serial number feature in the virtio-blk in qemu, and I can now get the driver to work in Windows.
Quoted 5 lines
Quoted 2 lines
What were the symptoms you were seeing (i.e. define "a problem").
Device manager reports "a problem code 10" occurred, and the driver cannot initialize.
Yes! I was getting this after I moved from 0.10.6 to 0.11.0-rc1. Now I know how to fix it. Thank you. Thank you.
Quoted 7 lines
Vadim Rozenfeld informed me:
Quoted 5 lines
There is a sanity check in the code, which checks the I/O range and fails if is not equal to 40h. Resent virtio-blk devices have I/O range equal to
0x400(serial number feature). So, out signed viostor driver will fail on the latest KVMs. This problem was fixedQuoted 7 lines
and committed to SVN some time ago.
I assumed the fix was to the virtio windows driver, but I could not get the driver I compiled from latest git to work either (only on qemu-kvm-87). So, I just backed out the serial number feature in qemu, and it worked. FWIW, the linux virtio-blk driver never had a problem.
Quoted 5 lines
There have been very few changes to the viostor windows git repo since it was opened. Unless it was done before they were open sourced. In any case, it doesn't seem to be working with what's publicly available, so I think maybe there is something missing internal to external.
I don't believe it was merged to git yet.
The signed viostor driver works with qemu-kvm-87 only, otherwise you need to remove SN from the virtio-blk code, or clip IO range to the original size.
Cheers,
Vadim.
Quoted 70 lines
Quoted 48 lines
Quoted 48 lines
Quoted 23 lines
So, not really any good news on performance with latest qemu builds. Performance is slightly worse:
qemu-kvm-87
user nice system irq softirq guest idle iowait
5.79 0.00 9.28 0.08 1.00 20.81 58.78 4.26
total busy: 36.97qemu-kvm-88-905-g6025b2d (master)
user nice system irq softirq guest idle iowait
6.57 0.00 10.86 0.08 1.02 21.35 55.90 4.21
total busy: 39.89qemu-kvm-88-910-gbf8a05b (next)
user nice system irq softirq guest idle iowait 6.60 0.00 10.91 0.09 1.03 21.35 55.71 4.31total busy: 39.98
diff of profiles, p1=qemu-kvm-87, p2=qemu-master
<snip>
Quoted 22 lines
18x more samples for gfn_to_memslot_unali*, 37x for emulator_read_emula*, and more CPU time in guest mode.
One other thing I decided to try was some cpu binding. I know this is not practical for production, but I wanted to see if there's any benefit at all. One reason was that a coworker here tried binding the qemu thread for the vcpu and the qemu IO thread to the same cpu. On a networking test, guest->local-host, throughput was up about 2x. Obviously there was a nice effect of being on the same cache. I wondered, even without full bore throughput tests, could we see any benefit here. So, I bound each pair of VMs to a dedicated core. What I saw was about a 6% improvement in performance. For a system which has pretty incredible memory performance and is not that busy, I was surprised that I got 6%. I am not advocating binding, but what I do
wonder: on 1-way VMs, if we keep all the qemu threads together on the same CPU, but still allowing the scheduler to move them (all of them at once) to different cpus over time, would we see the same benefit?One other thing: So far I have not been using preadv/pwritev. I assume I need a more recent glibc (on 2.5 now) for qemu to take advantage of this?
Quoted 8 lines
Quoted 5 lines
Getting p(read|write)v working almost doubled my virtio-net throughput in a Linux guest. Not quite as much in Windows guests. Yes you need glibc-2.10. I think some distros might have backported it to 2.9. You will also need some support for it in your system includes.
Thanks, I will try a newer glibc, or maybe just move to a newer Linux installation which happens to have a newer glic.
Fwiw... In Debian, I had to get glibc from the experimental tree. So some distros might not even have it.
Quoted 3 lines
-Andrew
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 81 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Vadim Rozenfeld
Sent: Saturday, September 05, 2009 2:35 PM
To: Brian Jackson
Cc: Andrew Theurer; [email protected]
Subject: Re: [PATCH] KVM: Use thread debug register storage instead of kvm specific dataQuoted 11 lines · Brian Jackson
Quoted 9 lines · Andrew Theurer
Quoted 7 lines · Brian Jackson
On Friday 04 September 2009 09:48:17 am Andrew Theurer wrote:
<snip>Quoted 3 lines
Quoted 2 lines
Still not idle=poll, it may shave off 0.2%.
Won't this affect SMT in a negative way? (OK, I am not running
SMT now,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
but eventually we will be) A long time ago, we tested P4's with
HT, and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
a polling idle in one thread always negatively impacted
performance in
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
the sibling thread.
FWIW, I did try idle=halt, and it was slightly worse.
I did get a chance to try the latest qemu (master and next heads).
I
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
have been running into a problem with virtIO stor driver for
windows on
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
anything much newer than kvm-87. I compiled the driver from the
new git
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
tree, installed OK, but still had the same error. Finally, I
removed
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the serial number feature in the virtio-blk in qemu, and I can now
get
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
the driver to work in Windows.
What were the symptoms you were seeing (i.e. define "a problem").
Device manager reports "a problem code 10" occurred, and the driver cannot initialize.
Yes! I was getting this after I moved from 0.10.6 to 0.11.0-rc1. Now
I know
Quoted 7 lines
how to fix it. Thank you. Thank you.
Quoted 3 lines
Vadim Rozenfeld informed me:
Quoted 1 line
There is a sanity check in the code, which checks the I/O range and
fails
Quoted 1 line
Quoted 1 line
Quoted 1 line
if is not equal to 40h. Resent virtio-blk devices have I/O range
equal to
Quoted 1 line
Quoted 1 line
Quoted 1 line
0x400(serial number feature). So, out signed viostor driver willfail
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
on the latest KVMs. This problem was fixed
and committed to SVN some time ago.
I assumed the fix was to the virtio windows driver, but I could not
get
Quoted 2 lines
Quoted 2 lines
the driver I compiled from latest git to work either (only on qemu-kvm-87). So, I just backed out the serial number feature in
qemu,
Quoted 1 line
Quoted 1 line
and it worked. FWIW, the linux virtio-blk driver never had a
problem.
Quoted 3 lines
Quoted 1 line
There have been very few changes to the viostor windows git repo
since it was
Quoted 1 line
opened. Unless it was done before they were open sourced. In any
case, it
Quoted 8 lines
Quoted 1 line
doesn't seem to be working with what's publicly available, so I think
maybe
Quoted 2 lines
there is something missing internal to external.
I don't believe it was merged to git yet.
The signed viostor driver works with qemu-kvm-87 only, otherwise you need to remove SN from the virtio-blk code, or clip IO range to the original size.
[YV] Please find attached patch with the fix. Kernel.org experiencing some problems now. The moment they will be fixed, I will push this patch to it.
Quoted 90 lines
Cheers,
Vadim.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
So, not really any good news on performance with latest qemu
builds.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Performance is slightly worse:
qemu-kvm-87
user nice system irq softirq guest idle iowait
5.79 0.00 9.28 0.08 1.00 20.81 58.78 4.26
total busy: 36.97qemu-kvm-88-905-g6025b2d (master)
user nice system irq softirq guest idle iowait
6.57 0.00 10.86 0.08 1.02 21.35 55.90 4.21
total busy: 39.89qemu-kvm-88-910-gbf8a05b (next)
user nice system irq softirq guest idle iowait 6.60 0.00 10.91 0.09 1.03 21.35 55.71 4.31total busy: 39.98
diff of profiles, p1=qemu-kvm-87, p2=qemu-master
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
<snip>
Quoted 4 lines
18x more samples for gfn_to_memslot_unali*, 37x for emulator_read_emula*, and more CPU time in guest mode.
One other thing I decided to try was some cpu binding. I know
this is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
not practical for production, but I wanted to see if there's any
benefit
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
at all. One reason was that a coworker here tried binding the
qemu
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
thread for the vcpu and the qemu IO thread to the same cpu. On a networking test, guest->local-host, throughput was up about 2x. Obviously there was a nice effect of being on the same cache. I wondered, even without full bore throughput tests, could we see
any
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
benefit here. So, I bound each pair of VMs to a dedicated core.
What I
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
saw was about a 6% improvement in performance. For a system which
has
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
pretty incredible memory performance and is not that busy, I was surprised that I got 6%. I am not advocating binding, but what I
do
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wonder: on 1-way VMs, if we keep all the qemu threads together on
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
same CPU, but still allowing the scheduler to move them (all of
them at
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
once) to different cpus over time, would we see the same benefit?
One other thing: So far I have not been using preadv/pwritev. I
assume
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I need a more recent glibc (on 2.5 now) for qemu to take advantage
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
this?
Getting p(read|write)v working almost doubled my virtio-net
throughput in
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
a Linux guest. Not quite as much in Windows guests. Yes you need glibc-2.10. I think some distros might have backported it to 2.9.
You
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
will also need some support for it in your system includes.
Thanks, I will try a newer glibc, or maybe just move to a newer
Linux
Quoted 4 lines
Quoted 2 lines
installation which happens to have a newer glic.
Fwiw... In Debian, I had to get glibc from the experimental tree. So
some
Quoted 4 lines
distros might not even have it.
Quoted 8 lines
Quoted 3 lines
-Andrew
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
From1f7dae772d96ba8aebdebd236edc5f0ec5eb2dfaMon Sep 17 00:00:00 2001
From: Yan Vugenfirer <[email protected]>
Date: Sun, 6 Sep 2009 13:54:34 +0300
Subject: [PATCH] Fix IO range verification that cause blue screen on new version of KVM
--- viostor/virtio_stor.c | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-)
viostor/virtio_stor.c +9 -1
@@ -220,10 +220,18 @@ VirtIoFindAdapter( accessRange->RangeStart.QuadPart + accessRange->RangeLength)); -if (IO_PORT_LENGTH != accessRange->RangeLength) { +if ( accessRange->RangeLength <IO_PORT_LENGTH ) { +ScsiPortLogError (DeviceExtension, +NULL , +0 , +0 , +0 , +SP_INTERNAL_ADAPTER_ERROR , + __LINE__);RhelDbgPrint (TRACE_LEVEL_FATAL , ("Wrong access range %x bytes\n" , accessRange->RangeLength));return SP_RETURN_NOT_FOUND ; } + if (!ScsiPortValidateRange(DeviceExtension, ConfigInfo->AdapterInterfaceType, ConfigInfo->SystemIoBusNumber,
Signature · 1 line
1.5.5.1
Quoted 8 lines · Andrew Theurer
Quoted 3 lines
Still not idle=poll, it may shave off 0.2%.
Won't this affect SMT in a negative way? (OK, I am not running SMT now, but eventually we will be) A long time ago, we tested P4's with HT, and a polling idle in one thread always negatively impacted performance in the sibling thread.
Sorry, I meant idle=halt. idle=poll is too wasteful to be used.
Quoted 2 lines
FWIW, I did try idle=halt, and it was slightly worse.
Interesting, I've heard that mwait latency is bad for spinlocks, guess it's fine for idle.
Quoted 49 lines
Quoted 39 lines
profile1 is qemu-kvm-87
profile2 is qemu-master
CountedCPU_CLK_UNHALTEDevents (Clock cycles when not halted) with a unit mask of0x00(No unit mask) count 10000000 total samples (ts1) for profile1 is 1616921
total samples (ts2) for profile2 is 1752347 (includes multiplier of 0.995420)
functions which have a abs(pct2-pct1)< 0.06 are not displayedpct2: pct1: 100* 100* pct2 s1 s2 s2/s1 s2/ts1 s1/ts1 -pct1 symbol bin --------- --------- ------- ------- ------- ------ ------ --- 879611 907883 1.03/1 56.149 54.400 1.749 vmx_vcpu_run kvm 614 11553 18.82/1 0.715 0.038 0.677 gfn_to_memslot_unali kvm.ko 34511 44922 1.30/1 2.778 2.134 0.644 phys_page_find_alloc qemu 2866 9334 3.26/1 0.577 0.177 0.400 paging64_walk_addr kvm.ko 11139 17200 1.54/1 1.064 0.689 0.375 copy_user_generic_st vmlinux 3100 7108 2.29/1 0.440 0.192 0.248 x86_decode_insn kvm.ko 8169 11873 1.45/1 0.734 0.505 0.229 virtqueue_avail_byte qemu 1103 4540 4.12/1 0.281 0.068 0.213 kvm_read_guest kvm.ko 17427 20401 1.17/1 1.262 1.078 0.184 memcpy libc 0 2905 0.180 0.000 0.180 gfn_to_pfn kvm.ko 1831 4328 2.36/1 0.268 0.113 0.154 x86_emulate_insn kvm.ko 65 2431 37.41/1 0.150 0.004 0.146 emulator_read_emulat kvm.ko 14922 17196 1.15/1 1.064 0.923 0.141 qemu_get_ram_ptr qemu 545 2724 5.00/1 0.168 0.034 0.135 emulate_instruction kvm.ko 599 2464 4.11/1 0.152 0.037 0.115 kvm_read_guest_page kvm.ko 503 2355 4.68/1 0.146 0.031 0.115 gfn_to_hva kvm.ko 1076 2918 2.71/1 0.181 0.067 0.114 memcpy_c vmlinux 594 2241 3.77/1 0.139 0.037 0.102 next_segment kvm.ko 1680 3248 1.93/1 0.201 0.104 0.097 pipe_poll vmlinux 0 1463 0.090 0.000 0.090 subpage_readl qemu 0 1363 0.084 0.000 0.084 msix_enabled qemu 527 1883 3.57/1 0.116 0.033 0.084 paging64_gpte_to_gfn kvm.ko 962 2223 2.31/1 0.138 0.059 0.078 do_insn_fetch kvm.ko 348 1605 4.61/1 0.099 0.022 0.078 is_rsvd_bits_set kvm.ko 520 1763 3.39/1 0.109 0.032 0.077 unalias_gfn kvm.ko 1 1163 1163.65/1 0.072 0.000 0.072 tdp_page_fault kvm.ko 3827 4912 1.28/1 0.304 0.237 0.067 __down_read vmlinuxQuoted 9 lines
0 1014 0.063 0.000 0.063 mapping_level kvm.ko 973 0 0.000 0.060 -0.060 pm_ioport_readl qemu 1635 528 1/3.09 0.033 0.101 -0.068 ioport_read qemu 2179 1017 1/2.14 0.063 0.135 -0.072 kvm_emulate_pio kvm.ko 25141 23722 1/1.06 1.467 1.555 -0.088 native_write_msr_saf vmlinux 1560 0 0.000 0.096 -0.096 eventfd_poll vmlinux ------- ------- ------ 105.100 97.450 7.650
Quoted 3 lines
18x more samples for gfn_to_memslot_unali*, 37x for emulator_read_emula*, and more CPU time in guest mode.
And 5x more instructions emulated. I wonder where that comes from.
Quoted 5 lines
One other thing: So far I have not been using preadv/pwritev. I assume I need a more recent glibc (on 2.5 now) for qemu to take advantage of this?
Yes, but it should be easy to write a LD_PRELOAD hack that will work with your current glibc. It should certainly improve things.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 32 lines · Avi Kivity
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
Signed-off-by Avi Kivity <[email protected]>--- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/x86.c | 22 +++++++--------------- 2 files changed, 7 insertions(+), 18 deletions(-)arch/x86/include/asm/kvm_host.h +0
-3@@ -362,9 +362,6 @@ struct kvm_vcpu_arch {u32 pat;int switch_db_regs; -unsigned long host_db[KVM_NR_DB_REGS ]; -unsigned long host_dr6; -unsigned long host_dr7;unsigned long db[KVM_NR_DB_REGS ];unsigned long dr6;unsigned long dr7;arch/x86/kvm/x86.c +0
-0@@ -3627,14 +3627,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3637
Quoted 10 lines
arch/x86/kvm/x86.c +0
-7kvm_guest_enter (); -get_debugreg (vcpu->arch.host_dr6,6 ); -get_debugreg (vcpu->arch.host_dr7,7 );if (unlikely (vcpu->arch.switch_db_regs)) { -get_debugreg (vcpu->arch.host_db[0 ],0 ); -get_debugreg (vcpu->arch.host_db[1 ],1 ); -get_debugreg (vcpu->arch.host_db[2 ],2 ); -get_debugreg (vcpu->arch.host_db[3 ],3 ); -
I'm fine with this for the common case, but we should switch to the safe pattern on test_thread_flag(TIF_DEBUG), ie. if someone debugs qemu using hardware break/watchpoints.
Quoted 4 lines
arch/x86/kvm/x86.c +0
-0set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3641
Quoted 10 lines
arch/x86/kvm/x86.c +1
-6@@ -3645,15 +3638,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu); -if (unlikely (vcpu->arch.switch_db_regs)) { -set_debugreg (0 ,7 ); -set_debugreg (vcpu->arch.host_db[0 ],0 ); -set_debugreg (vcpu->arch.host_db[1 ],1 ); -set_debugreg (vcpu->arch.host_db[2 ],2 ); -set_debugreg (vcpu->arch.host_db[3 ],3 ); +if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) {
IIRC, you must clear dr7 before loading dr0..3 to avoid spurious effects.
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3656
Quoted 9 lines
arch/x86/kvm/x86.c +6
-2+set_debugreg (current->thread.debugreg0,0 ); +set_debugreg (current->thread.debugreg1,1 ); +set_debugreg (current->thread.debugreg2,2 ); +set_debugreg (current->thread.debugreg3,3 ); +set_debugreg (current->thread.debugreg6,6 ); +set_debugreg (current->thread.debugreg7,7 ); } -set_debugreg (vcpu->arch.host_dr6,6 ); -set_debugreg (vcpu->arch.host_dr7,7 );
Unless I miss something ATM, moving this into the if-clause should cause troubles if the guest leaves dr7 armed behind. But I need to refresh my memory on this.
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3651
Quoted 3 lines
arch/x86/kvm/x86.c +0
-0set_bit (KVM_REQ_KICK , &vcpu->requests);local_irq_enable ();
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 65 lines · Jan Kiszka
Quoted 32 lines · Avi Kivity
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
Signed-off-by Avi Kivity <[email protected]>--- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/x86.c | 22 +++++++--------------- 2 files changed, 7 insertions(+), 18 deletions(-)arch/x86/include/asm/kvm_host.h +0
-3@@ -362,9 +362,6 @@ struct kvm_vcpu_arch {u32 pat;int switch_db_regs; -unsigned long host_db[KVM_NR_DB_REGS ]; -unsigned long host_dr6; -unsigned long host_dr7;unsigned long db[KVM_NR_DB_REGS ];unsigned long dr6;unsigned long dr7;arch/x86/kvm/x86.c +0
-0@@ -3627,14 +3627,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3637
Quoted 10 lines
arch/x86/kvm/x86.c +0
-7kvm_guest_enter (); -get_debugreg (vcpu->arch.host_dr6,6 ); -get_debugreg (vcpu->arch.host_dr7,7 );if (unlikely (vcpu->arch.switch_db_regs)) { -get_debugreg (vcpu->arch.host_db[0 ],0 ); -get_debugreg (vcpu->arch.host_db[1 ],1 ); -get_debugreg (vcpu->arch.host_db[2 ],2 ); -get_debugreg (vcpu->arch.host_db[3 ],3 ); -I'm fine with this for the common case, but we should switch to the safe pattern on
test_thread_flag(TIF_DEBUG), ie. if someone debugs qemu using hardware break/watchpoints.Quoted 4 lines
arch/x86/kvm/x86.c +0
-0set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3641
Quoted 10 lines
arch/x86/kvm/x86.c +1
-6@@ -3645,15 +3638,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu); -if (unlikely (vcpu->arch.switch_db_regs)) { -set_debugreg (0 ,7 ); -set_debugreg (vcpu->arch.host_db[0 ],0 ); -set_debugreg (vcpu->arch.host_db[1 ],1 ); -set_debugreg (vcpu->arch.host_db[2 ],2 ); -set_debugreg (vcpu->arch.host_db[3 ],3 ); +if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) {IIRC, you must clear dr7 before loading dr0..3 to avoid spurious effects.
Quoted 13 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3656
Quoted 9 lines
arch/x86/kvm/x86.c +6
-2+set_debugreg (current->thread.debugreg0,0 ); +set_debugreg (current->thread.debugreg1,1 ); +set_debugreg (current->thread.debugreg2,2 ); +set_debugreg (current->thread.debugreg3,3 ); +set_debugreg (current->thread.debugreg6,6 ); +set_debugreg (current->thread.debugreg7,7 ); } -set_debugreg (vcpu->arch.host_dr6,6 ); -set_debugreg (vcpu->arch.host_dr7,7 );Unless I miss something ATM, moving this into the if-clause should cause troubles if the guest leaves dr7 armed behind. But I need to refresh my memory on this.
Looks like VMX loads 0x400 unconditionally into dr7 on vmexit, thus all hw debugging is initially disabled, I must have missed this while reworking the dr switch code. So we can safely load dr0..3 without clearing dr7 first, and we can skip restoring of dr6 and dr7 unless the current process (the hypervisor) is debugged itself.
Quoted 7 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3651
Quoted 4 lines
arch/x86/kvm/x86.c +0
-0set_bit (KVM_REQ_KICK , &vcpu->requests);local_irq_enable ();Jan
Recent Intel CPUs seem to provide control over dr7/debugctl msr saving/restoring (bit 2 in vmexit and vmentry control). Allready thought about if this may further help to reduce the common case (ie. no host and guest drX usage)?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 9 lines · Jan Kiszka
Quoted 9 lines
arch/x86/kvm/x86.c +0
-0@@ -3627,14 +3627,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)kvm_guest_enter ();
Quoted 13 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
Quoted 9 lines
arch/x86/kvm/x86.c +0
-7-get_debugreg (vcpu->arch.host_dr6,6 ); -get_debugreg (vcpu->arch.host_dr7,7 );if (unlikely (vcpu->arch.switch_db_regs)) { -get_debugreg (vcpu->arch.host_db[0 ],0 ); -get_debugreg (vcpu->arch.host_db[1 ],1 ); -get_debugreg (vcpu->arch.host_db[2 ],2 ); -get_debugreg (vcpu->arch.host_db[3 ],3 ); -I'm fine with this for the common case, but we should switch to the safe pattern on
test_thread_flag(TIF_DEBUG), ie. if someone debugs qemu using hardware break/watchpoints.
IIUC, thread.debugregs should be synced with the hardware registers. The kernel itself only writes to the debug registers, so we're safe doing the same.
Quoted 8 lines
Quoted 8 lines
arch/x86/kvm/x86.c +0
-0set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg(vcpu->arch.eff_db[1], 1);arch/x86/kvm/x86.c +0
-0@@ -3645,15 +3638,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu);
Quoted 10 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3649
Quoted 8 lines
arch/x86/kvm/x86.c +1
-6-if (unlikely (vcpu->arch.switch_db_regs)) { -set_debugreg (0 ,7 ); -set_debugreg (vcpu->arch.host_db[0 ],0 ); -set_debugreg (vcpu->arch.host_db[1 ],1 ); -set_debugreg (vcpu->arch.host_db[2 ],2 ); -set_debugreg (vcpu->arch.host_db[3 ],3 ); +if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) {IIRC, you must clear dr7 before loading dr0..3 to avoid spurious effects.
Yeah, I thought of it but forgot to implement it.
Quoted 5 lines
Quoted 5 lines
arch/x86/kvm/x86.c +5
-0+set_debugreg (current->thread.debugreg0,0 ); +set_debugreg (current->thread.debugreg1,1 ); +set_debugreg (current->thread.debugreg2,2 ); +set_debugreg (current->thread.debugreg3,3 ); +set_debugreg (current->thread.debugreg6,6 );
Quoted 9 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3649
Quoted 5 lines
arch/x86/kvm/x86.c +1
-2+set_debugreg (current->thread.debugreg7,7 ); } -set_debugreg (vcpu->arch.host_dr6,6 ); -set_debugreg (vcpu->arch.host_dr7,7 );Unless I miss something ATM, moving this into the if-clause should cause troubles if the guest leaves dr7 armed behind. But I need to refresh my memory on this.
The hardware will clear dr7 on exit.
Updated patch to follow.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 19 lines · Avi Kivity
Quoted 9 lines
Quoted 9 lines
set_debugreg(0, 7); set_debugreg(vcpu->arch.eff_db[0], 0); set_debugreg(vcpu->arch.eff_db[1], 1);arch/x86/kvm/x86.c +0
-0@@ -3645,15 +3638,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu);Quoted 9 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
Quoted 8 lines
arch/x86/kvm/x86.c +1
-6-if (unlikely (vcpu->arch.switch_db_regs)) { -set_debugreg (0 ,7 ); -set_debugreg (vcpu->arch.host_db[0 ],0 ); -set_debugreg (vcpu->arch.host_db[1 ],1 ); -set_debugreg (vcpu->arch.host_db[2 ],2 ); -set_debugreg (vcpu->arch.host_db[3 ],3 ); +if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) {IIRC, you must clear dr7 before loading dr0..3 to avoid spurious effects.
Quoted 1 line
Yeah, I thought of it but forgot to implement it.
Actually on the entry path it is done correctly, and on the exit path dr7 is cleared, so this patch is correct.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 23 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
Quoted 8 lines
arch/x86/kvm/x86.c +0
-0@@ -3627,14 +3627,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)kvm_guest_enter ();Quoted 13 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
Quoted 9 lines
arch/x86/kvm/x86.c +0
-7-get_debugreg (vcpu->arch.host_dr6,6 ); -get_debugreg (vcpu->arch.host_dr7,7 );if (unlikely (vcpu->arch.switch_db_regs)) { -get_debugreg (vcpu->arch.host_db[0 ],0 ); -get_debugreg (vcpu->arch.host_db[1 ],1 ); -get_debugreg (vcpu->arch.host_db[2 ],2 ); -get_debugreg (vcpu->arch.host_db[3 ],3 ); -I'm fine with this for the common case, but we should switch to the safe pattern on
test_thread_flag(TIF_DEBUG), ie. if someone debugs qemu using hardware break/watchpoints.
Quoted 3 lines
IIUC, thread.debugregs should be synced with the hardware registers. The kernel itself only writes to the debug registers, so we're safe doing the same.
Yep, confirmed. There is only an unsync'ed window between debug trap handling and signal injection, but I cannot imagine then kvm could intercept this path with a guest entry.
Quoted 37 lines
Quoted 9 lines
Quoted 9 lines
arch/x86/kvm/x86.c +0
-0set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg(vcpu->arch.eff_db[1], 1);arch/x86/kvm/x86.c +0
-0@@ -3645,15 +3638,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu);Quoted 10 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3649
Quoted 8 lines
arch/x86/kvm/x86.c +1
-6-if (unlikely (vcpu->arch.switch_db_regs)) { -set_debugreg (0 ,7 ); -set_debugreg (vcpu->arch.host_db[0 ],0 ); -set_debugreg (vcpu->arch.host_db[1 ],1 ); -set_debugreg (vcpu->arch.host_db[2 ],2 ); -set_debugreg (vcpu->arch.host_db[3 ],3 ); +if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) {IIRC, you must clear dr7 before loading dr0..3 to avoid spurious effects.
Yeah, I thought of it but forgot to implement it.
Quoted 5 lines
Quoted 5 lines
arch/x86/kvm/x86.c +5
-0+set_debugreg (current->thread.debugreg0,0 ); +set_debugreg (current->thread.debugreg1,1 ); +set_debugreg (current->thread.debugreg2,2 ); +set_debugreg (current->thread.debugreg3,3 ); +set_debugreg (current->thread.debugreg6,6 );Quoted 9 lines
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 3649
Quoted 5 lines
arch/x86/kvm/x86.c +1
-2+set_debugreg (current->thread.debugreg7,7 ); } -set_debugreg (vcpu->arch.host_dr6,6 ); -set_debugreg (vcpu->arch.host_dr7,7 );Unless I miss something ATM, moving this into the if-clause should cause troubles if the guest leaves dr7 armed behind. But I need to refresh my memory on this.
Quoted 1 line
The hardware will clear dr7 on exit.
Given this fact, I wonder why you want to reload host dr0..3 on vcpu->arch.switch_db_regs. if (unlikely(test_thread_flag(TIF_DEBUG))) should suffice then, right?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 6 lines · Jan Kiszka
Quoted 2 lines
The hardware will clear dr7 on exit.
Given this fact, I wonder why you want to reload host dr0..3 on
vcpu->arch.switch_db_regs. if (unlikely(test_thread_flag(TIF_DEBUG))) should suffice then, right?
I think you're right. It would mean unsynced hardware and software registers, but only if debugging is disabled, which should be fine.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 5 lines · Avi Kivity
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
Is this kgdb safe?
Quoted 64 lines
Signed-off-by Avi Kivity <[email protected]>--- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/x86.c | 22 +++++++--------------- 2 files changed, 7 insertions(+), 18 deletions(-)arch/x86/include/asm/kvm_host.h +0
-3@@ -362,9 +362,6 @@ struct kvm_vcpu_arch {u32 pat;int switch_db_regs; -unsigned long host_db[KVM_NR_DB_REGS ]; -unsigned long host_dr6; -unsigned long host_dr7;unsigned long db[KVM_NR_DB_REGS ];unsigned long dr6;unsigned long dr7;arch/x86/kvm/x86.c +7
-15@@ -3627,14 +3627,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)kvm_guest_enter (); -get_debugreg (vcpu->arch.host_dr6,6 ); -get_debugreg (vcpu->arch.host_dr7,7 );if (unlikely (vcpu->arch.switch_db_regs)) { -get_debugreg (vcpu->arch.host_db[0 ],0 ); -get_debugreg (vcpu->arch.host_db[1 ],1 ); -get_debugreg (vcpu->arch.host_db[2 ],2 ); -get_debugreg (vcpu->arch.host_db[3 ],3 ); -set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );@@ -3645,15 +3638,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu); -if (unlikely (vcpu->arch.switch_db_regs)) { -set_debugreg (0 ,7 ); -set_debugreg (vcpu->arch.host_db[0 ],0 ); -set_debugreg (vcpu->arch.host_db[1 ],1 ); -set_debugreg (vcpu->arch.host_db[2 ],2 ); -set_debugreg (vcpu->arch.host_db[3 ],3 ); +if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) { +set_debugreg (current->thread.debugreg0,0 ); +set_debugreg (current->thread.debugreg1,1 ); +set_debugreg (current->thread.debugreg2,2 ); +set_debugreg (current->thread.debugreg3,3 ); +set_debugreg (current->thread.debugreg6,6 ); +set_debugreg (current->thread.debugreg7,7 ); } -set_debugreg (vcpu->arch.host_dr6,6 ); -set_debugreg (vcpu->arch.host_dr7,7 );set_bit (KVM_REQ_KICK , &vcpu->requests);local_irq_enable ();Signature · 1 line
1.6.4.1
Quoted 8 lines · Marcelo Tosatti
Quoted 5 lines · Avi Kivity
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
Is this kgdb safe?
Nope, kgdb writes directly to the debug registers.
I vaguely recall someone trying to push a debug register management framework. Did it hit mainline in the meantime? I do not find any trace on quick glance, at least not in kgdb.
Jan
Quoted 65 lines
Quoted 65 lines
Signed-off-by Avi Kivity <[email protected]>--- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/x86.c | 22 +++++++--------------- 2 files changed, 7 insertions(+), 18 deletions(-)arch/x86/include/asm/kvm_host.h +0
-3@@ -362,9 +362,6 @@ struct kvm_vcpu_arch {u32 pat;int switch_db_regs; -unsigned long host_db[KVM_NR_DB_REGS ]; -unsigned long host_dr6; -unsigned long host_dr7;unsigned long db[KVM_NR_DB_REGS ];unsigned long dr6;unsigned long dr7;arch/x86/kvm/x86.c +7
-15@@ -3627,14 +3627,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)kvm_guest_enter (); -get_debugreg (vcpu->arch.host_dr6,6 ); -get_debugreg (vcpu->arch.host_dr7,7 );if (unlikely (vcpu->arch.switch_db_regs)) { -get_debugreg (vcpu->arch.host_db[0 ],0 ); -get_debugreg (vcpu->arch.host_db[1 ],1 ); -get_debugreg (vcpu->arch.host_db[2 ],2 ); -get_debugreg (vcpu->arch.host_db[3 ],3 ); -set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );@@ -3645,15 +3638,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu); -if (unlikely (vcpu->arch.switch_db_regs)) { -set_debugreg (0 ,7 ); -set_debugreg (vcpu->arch.host_db[0 ],0 ); -set_debugreg (vcpu->arch.host_db[1 ],1 ); -set_debugreg (vcpu->arch.host_db[2 ],2 ); -set_debugreg (vcpu->arch.host_db[3 ],3 ); +if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) { +set_debugreg (current->thread.debugreg0,0 ); +set_debugreg (current->thread.debugreg1,1 ); +set_debugreg (current->thread.debugreg2,2 ); +set_debugreg (current->thread.debugreg3,3 ); +set_debugreg (current->thread.debugreg6,6 ); +set_debugreg (current->thread.debugreg7,7 ); } -set_debugreg (vcpu->arch.host_dr6,6 ); -set_debugreg (vcpu->arch.host_dr7,7 );set_bit (KVM_REQ_KICK , &vcpu->requests);local_irq_enable ();Signature · 1 line
1.6.4.1
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 8 lines · Jan Kiszka
Quoted 7 lines · Marcelo Tosatti
Quoted 5 lines · Avi Kivity
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
Quoted 7 lines
Quoted 1 line
Is this kgdb safe?
Nope, kgdb writes directly to the debug registers.
I vaguely recall someone trying to push a debug register management framework. Did it hit mainline in the meantime? I do not find any trace on quick glance, at least not in kgdb.
A simple kgdb_enabled sort of flag, in addition to TIF_DEBUG would do it?
Quoted 71 lines
Jan
Quoted 65 lines
Quoted 65 lines
Signed-off-by Avi Kivity <[email protected]>--- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/x86.c | 22 +++++++--------------- 2 files changed, 7 insertions(+), 18 deletions(-)arch/x86/include/asm/kvm_host.h +0
-3@@ -362,9 +362,6 @@ struct kvm_vcpu_arch {u32 pat;int switch_db_regs; -unsigned long host_db[KVM_NR_DB_REGS ]; -unsigned long host_dr6; -unsigned long host_dr7;unsigned long db[KVM_NR_DB_REGS ];unsigned long dr6;unsigned long dr7;arch/x86/kvm/x86.c +7
-15@@ -3627,14 +3627,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)kvm_guest_enter (); -get_debugreg (vcpu->arch.host_dr6,6 ); -get_debugreg (vcpu->arch.host_dr7,7 );if (unlikely (vcpu->arch.switch_db_regs)) { -get_debugreg (vcpu->arch.host_db[0 ],0 ); -get_debugreg (vcpu->arch.host_db[1 ],1 ); -get_debugreg (vcpu->arch.host_db[2 ],2 ); -get_debugreg (vcpu->arch.host_db[3 ],3 ); -set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );@@ -3645,15 +3638,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu); -if (unlikely (vcpu->arch.switch_db_regs)) { -set_debugreg (0 ,7 ); -set_debugreg (vcpu->arch.host_db[0 ],0 ); -set_debugreg (vcpu->arch.host_db[1 ],1 ); -set_debugreg (vcpu->arch.host_db[2 ],2 ); -set_debugreg (vcpu->arch.host_db[3 ],3 ); +if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) { +set_debugreg (current->thread.debugreg0,0 ); +set_debugreg (current->thread.debugreg1,1 ); +set_debugreg (current->thread.debugreg2,2 ); +set_debugreg (current->thread.debugreg3,3 ); +set_debugreg (current->thread.debugreg6,6 ); +set_debugreg (current->thread.debugreg7,7 ); } -set_debugreg (vcpu->arch.host_dr6,6 ); -set_debugreg (vcpu->arch.host_dr7,7 );set_bit (KVM_REQ_KICK , &vcpu->requests);local_irq_enable ();Signature · 1 line
1.6.4.1Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 10 lines · Marcelo Tosatti
Quoted 7 lines
Nope, kgdb writes directly to the debug registers.
I vaguely recall someone trying to push a debug register management framework. Did it hit mainline in the meantime? I do not find any trace on quick glance, at least not in kgdb.
A simple
kgdb_enabledsort of flag, in addition toTIF_DEBUGwould do it?
kgdb is per-cpu whereas user debugging is per-task, so we'd need a per-cpu debug register cache and flag.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Jan Kiszka
Quoted 5 lines · Marcelo Tosatti
Quoted 4 lines · Avi Kivity
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
Is this kgdb safe?
Nope, kgdb writes directly to the debug registers.
I vaguely recall someone trying to push a debug register management framework. Did it hit mainline in the meantime? I do not find any trace on quick glance, at least not in kgdb.
For the time being, falling back to conservative save/restore if kgdb is configured in and kgdb_connected != 0 should allow us to apply this optimization otherwise.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 4 lines · Jan Kiszka
For the time being, falling back to conservative save/restore if kgdb is configured in and
kgdb_connected!= 0 should allow us to apply this optimization otherwise.
I prefer a real fix instead of spaghettiing the code this way (and in the meanwhile I'm fine with kgdb hardware breakpoints not working while kvm is running).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 4 lines · Jan Kiszka
For the time being, falling back to conservative save/restore if kgdb is configured in and
kgdb_connected!= 0 should allow us to apply this optimization otherwise.I prefer a real fix instead of spaghettiing the code this way (and in the meanwhile I'm fine with kgdb hardware breakpoints not working while kvm is running).
Then file a patch that makes CONFIG_KVM depend on !CONFIG_KGDB (or vice versa). Silent breakages are evil.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 9 lines · Jan Kiszka
Quoted 7 lines · Avi Kivity
Quoted 5 lines · Jan Kiszka
For the time being, falling back to conservative save/restore if kgdb is configured in and
kgdb_connected!= 0 should allow us to apply this optimization otherwise.
Quoted 7 lines
Quoted 4 lines
I prefer a real fix instead of spaghettiing the code this way (and in the meanwhile I'm fine with kgdb hardware breakpoints not working while kvm is running).
Then file a patch that makes
CONFIG_KVMdepend on !CONFIG_KGDB (or vice versa). Silent breakages are evil.
Distros typically enable both.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Jan Kiszka
Quoted 11 lines · Avi Kivity
Quoted 5 lines · Jan Kiszka
For the time being, falling back to conservative save/restore if kgdb is configured in and
kgdb_connected!= 0 should allow us to apply this optimization otherwise.I prefer a real fix instead of spaghettiing the code this way (and in the meanwhile I'm fine with kgdb hardware breakpoints not working while kvm is running).
Quoted 5 lines
Quoted 3 lines
Then file a patch that makes
CONFIG_KVMdepend on !CONFIG_KGDB (or vice versa). Silent breakages are evil.Distros typically enable both.
Then we need to resolve it right from the start, not in some distant future. We already have everything we need available, we could just try to refactor it to a nicer interface (something that encapsulates both compile and runtime tests).
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 25 lines · Jan Kiszka
Quoted 23 lines · Avi Kivity
Quoted 21 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Jan Kiszka
For the time being, falling back to conservative save/restore if kgdb is configured in and
kgdb_connected!= 0 should allow us to apply this optimization otherwise.Quoted 5 lines
I prefer a real fix instead of spaghettiing the code this way (and in the meanwhile I'm fine with kgdb hardware breakpoints not working while kvm is running).
Then file a patch that makes
CONFIG_KVMdepend on !CONFIG_KGDB (or vice versa). Silent breakages are evil.
Quoted 7 lines
Quoted 2 lines
Distros typically enable both.
Then we need to resolve it right from the start, not in some distant future. We already have everything we need available, we could just try to refactor it to a nicer interface (something that encapsulates both compile and runtime tests).
I'll look at doing something. It shouldn't be difficult.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
Quoted 4 lines
Then we need to resolve it right from the start, not in some distant future. We already have everything we need available, we could just try to refactor it to a nicer interface (something that encapsulates both compile and runtime tests).
I'll look at doing something. It shouldn't be difficult.
btw, it looks like kgdb is already broken wrt prtace hardware breakpoints.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Marcelo Tosatti
Quoted 6 lines · Avi Kivity
Instead of saving the debug registers from the processor to a kvm data structure, rely in the debug registers stored in the thread structure. This allows us not to save dr6 and dr7.
Reduces lightweight vmexit cost by 350 cycles, or 11 percent.
Is this kgdb safe?
I don't think so - kgdb seems to access the debug registers directly. I think we can live with it at the moment, and later add an API to arbitrate debug register usage among the various users (kgdb seems to be unsafe wrt userspace debug as well).
Signature · 1 line
error compiling committee.c: too many arguments to function