KVM: Implement nested TSC scaling
v4 cover KVM: Implement nested TSC scaling
KVM currently supports hardware-assisted TSC scaling but only for L1; the feature is not exposed to nested guests. This patch series adds support for nested TSC scaling and allows both L1 and L2 to be scaled with different scaling factors. That is achieved by "merging" the 01 and 02 values together.
Most of the logic in this series is implemented in common code (by doing the necessary restructurings), however the patches add support for VMX only. Adding support for SVM should be easy at this point and Maxim Levitsky has volunteered to do this (thanks!).
Changelog:
Only patch 9 needs reviewing at this point, but I am re-sending the whole series as I also applied nitpicks suggested to some of the other pathces.
v4:
- Added vendor callbacks for writing the TSC multiplier
- Moved the VMCS multiplier writes from the VMCS load path to common code that only gets called on TSC ratio changes
- Merged together patches 10 and 11 of v3
- Applied all nitpick-feedback of the previous versions
v3:
- Applied Sean's feedback
- Refactored patches 7 to 10
v2:
- Applied all of Maxim's feedback
- Added a mul_s64_u64_shr function in math64.h
- Added a separate kvm_scale_tsc_l1 function instead of passing an argument to kvm_scale_tsc
- Implemented the 02 fields calculations in common code
- Moved all of write_l1_tsc_offset's logic to common code
- Added a check for whether the TSC is stable in patch 10
- Used a random L1 factor and a negative offset in patch 10
Ilias Stamatis (11):
math64.h: Add mul_s64_u64_shr()
KVM: X86: Store L1's TSC scaling ratio in 'struct kvm_vcpu_arch'
KVM: X86: Rename kvm_compute_tsc_offset() to
kvm_compute_l1_tsc_offset()
KVM: X86: Add a ratio parameter to kvm_scale_tsc()
KVM: nVMX: Add a TSC multiplier field in VMCS12
KVM: X86: Add functions for retrieving L2 TSC fields from common code
KVM: X86: Add functions that calculate the nested TSC fields
KVM: X86: Move write_l1_tsc_offset() logic to common code and rename
it
KVM: X86: Add vendor callbacks for writing the TSC multiplier
KVM: nVMX: Enable nested TSC scaling
KVM: selftests: x86: Add vmx_nested_tsc_scaling_test
arch/x86/include/asm/kvm-x86-ops.h | 5 +- arch/x86/include/asm/kvm_host.h | 15 +- arch/x86/kvm/svm/svm.c | 35 ++- arch/x86/kvm/vmx/nested.c | 33 ++- arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +- arch/x86/kvm/vmx/vmx.c | 55 ++-- arch/x86/kvm/vmx/vmx.h | 11 +- arch/x86/kvm/x86.c | 114 +++++++-- include/linux/math64.h | 19 ++ tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../kvm/x86_64/vmx_nested_tsc_scaling_test.c | 242 ++++++++++++++++++ 13 files changed, 451 insertions(+), 85 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/vmx_nested_tsc_scaling_test.c
Signature · 1 line
2.17.1
PATCH v4 1/11 math64.h: Add mul_s64_u64_shr()
This function is needed for KVM's nested virtualization. The nested TSC scaling implementation requires multiplying the signed TSC offset with the unsigned TSC multiplier.
--- include/linux/math64.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
include/linux/math64.h +19 -0
@@ -3,6 +3,7 @@#define _LINUX_MATH64_H#include <linux/types.h> +#include <linux/math.h> #include <vdso/math64.h> #include <asm/div64.h>
@@ -234,6 +235,24 @@ static inline u64 mul_u64_u64_shr(u64 a, u64 b, unsigned int shift)#endif +#ifndef mul_s64_u64_shr +static inline u64 mul_s64_u64_shr (s64 a,u64 b,unsigned int shift) +{ +u64 ret; + +/* +* Extract the sign before the multiplication and put it back +* afterwards if needed. +*/ + ret =mul_u64_u64_shr (abs (a), b, shift); + +if (a <0 ) + ret = -((s64 ) ret); + +return ret; +} +#endif /* mul_s64_u64_shr */ +#ifndef mul_u64_u32_divstatic inline u64 mul_u64_u32_div (u64 a,u32 mul,u32 divisor) {
Signature · 1 line
2.17.1
PATCH v4 2/11 KVM: X86: Store L1's TSC scaling ratio in 'struct kvm_vcpu_arch'
In reply to Ilias StamatisStore L1's scaling ratio in the kvm_vcpu_arch struct like we already do for L1's TSC offset. This allows for easy save/restore when we enter and then exit the nested guest.
--- arch/x86/include/asm/kvm_host.h | 5 +++-- arch/x86/kvm/vmx/vmx.c | 4 ++-- arch/x86/kvm/x86.c | 6 ++++-- 3 files changed, 9 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -2
@@ -707,7 +707,7 @@ struct kvm_vcpu_arch { } st;u64 l1_tsc_offset; -u64 tsc_offset; +u64 tsc_offset;/* current tsc offset */ u64 last_guest_tsc;u64 last_host_tsc;u64 tsc_offset_adjustment;
@@ -721,7 +721,8 @@ struct kvm_vcpu_arch {u32 virtual_tsc_khz;s64 ia32_tsc_adjust_msr;u64 msr_ia32_power_ctl; -u64 tsc_scaling_ratio; +u64 l1_tsc_scaling_ratio; +u64 tsc_scaling_ratio;/* current scaling ratio */ atomic_t nmi_queued;/* unprocessed asynchronous NMIs */ unsigned nmi_pending;/* NMI queued after currently running handler */
arch/x86/kvm/vmx/vmx.c +2 -2
@@ -7453,10 +7453,10 @@ static int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc, delta_tsc =0 ;/* Convert to host delta tsc if tsc scaling is enabled */ -if (vcpu->arch.tsc_scaling_ratio != kvm_default_tsc_scaling_ratio && +if (vcpu->arch.l1_tsc_scaling_ratio != kvm_default_tsc_scaling_ratio && delta_tsc &&u64_shl_div_u64 (delta_tsc, kvm_tsc_scaling_ratio_frac_bits, - vcpu->arch.tsc_scaling_ratio, &delta_tsc)) + vcpu->arch.l1_tsc_scaling_ratio, &delta_tsc))return -ERANGE ;/*
arch/x86/kvm/x86.c +4 -2
@@ -2185,6 +2185,7 @@ static int set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool scale)/* Guest TSC same frequency as host TSC? */ if (!scale) { + vcpu->arch.l1_tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; vcpu->arch.tsc_scaling_ratio = kvm_default_tsc_scaling_ratio;return 0 ; }
@@ -2211,7 +2212,7 @@ static int set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool scale)return -1 ; } - vcpu->arch.tsc_scaling_ratio = ratio; + vcpu->arch.l1_tsc_scaling_ratio = vcpu->arch.tsc_scaling_ratio = ratio;return 0 ; }
@@ -2223,6 +2224,7 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)/* tsc_khz can be zero if TSC calibration fails */ if (user_tsc_khz ==0 ) {/* set tsc_scaling_ratio to a safe value */ + vcpu->arch.l1_tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; vcpu->arch.tsc_scaling_ratio = kvm_default_tsc_scaling_ratio;return -1 ; }
@@ -2459,7 +2461,7 @@ static inline void adjust_tsc_offset_guest(struct kvm_vcpu *vcpu,static inline void adjust_tsc_offset_host (struct kvm_vcpu *vcpu,s64 adjustment) { -if (vcpu->arch.tsc_scaling_ratio != kvm_default_tsc_scaling_ratio) +if (vcpu->arch.l1_tsc_scaling_ratio != kvm_default_tsc_scaling_ratio)WARN_ON (adjustment <0 ); adjustment =kvm_scale_tsc (vcpu, (u64 ) adjustment);adjust_tsc_offset_guest (vcpu, adjustment);
Signature · 1 line
2.17.1
PATCH v4 3/11 KVM: X86: Rename kvm_compute_tsc_offset() to kvm_compute_l1_tsc_offset()
In reply to Ilias StamatisAll existing code uses kvm_compute_tsc_offset() passing L1 TSC values to it. Let's document this by renaming it to kvm_compute_l1_tsc_offset().
--- arch/x86/kvm/x86.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/x86.c +5 -5
@@ -2319,7 +2319,7 @@ u64 kvm_scale_tsc(struct kvm_vcpu *vcpu, u64 tsc) }EXPORT_SYMBOL_GPL (kvm_scale_tsc); -static u64 kvm_compute_tsc_offset (struct kvm_vcpu *vcpu,u64 target_tsc) +static u64 kvm_compute_l1_tsc_offset (struct kvm_vcpu *vcpu,u64 target_tsc) {u64 tsc;
@@ -2363,7 +2363,7 @@ static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 data)bool synchronizing = false;raw_spin_lock_irqsave (&kvm->arch.tsc_write_lock, flags); - offset =kvm_compute_tsc_offset (vcpu, data); + offset =kvm_compute_l1_tsc_offset (vcpu, data); ns =get_kvmclock_base_ns (); elapsed = ns - kvm->arch.last_tsc_nsec;
@@ -2402,7 +2402,7 @@ static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 data) }else {u64 delta =nsec_to_cycles (vcpu, elapsed); data += delta; - offset =kvm_compute_tsc_offset (vcpu, data); + offset =kvm_compute_l1_tsc_offset (vcpu, data); } matched = true; already_matched = (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation);
@@ -3235,7 +3235,7 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (msr_info->host_initiated) {kvm_synchronize_tsc (vcpu, data); }else { -u64 adj =kvm_compute_tsc_offset (vcpu, data) - vcpu->arch.l1_tsc_offset; +u64 adj =kvm_compute_l1_tsc_offset (vcpu, data) - vcpu->arch.l1_tsc_offset;adjust_tsc_offset_guest (vcpu, adj); vcpu->arch.ia32_tsc_adjust_msr += adj; }
@@ -4123,7 +4123,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)mark_tsc_unstable ("KVM discovered backwards TSC" );if (kvm_check_tsc_unstable ()) { -u64 offset =kvm_compute_tsc_offset (vcpu, +u64 offset =kvm_compute_l1_tsc_offset (vcpu, vcpu->arch.last_guest_tsc);kvm_vcpu_write_tsc_offset (vcpu, offset); vcpu->arch.tsc_catchup =1 ;
Signature · 1 line
2.17.1
PATCH v4 4/11 KVM: X86: Add a ratio parameter to kvm_scale_tsc()
In reply to Ilias StamatisSometimes kvm_scale_tsc() needs to use the current scaling ratio and other times (like when reading the TSC from user space) it needs to use L1's scaling ratio. Have the caller specify this by passing the ratio as a parameter.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/x86.c | 27 ++++++++++++++++++--------- 2 files changed, 19 insertions(+), 10 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1788,7 +1788,7 @@ static inline bool kvm_is_supported_user_return_msr(u32 msr)return kvm_find_user_return_msr (msr) >=0 ; } -u64 kvm_scale_tsc (struct kvm_vcpu *vcpu,u64 tsc); +u64 kvm_scale_tsc (struct kvm_vcpu *vcpu,u64 tsc,u64 ratio);u64 kvm_read_l1_tsc (struct kvm_vcpu *vcpu,u64 host_tsc);unsigned long kvm_get_linear_rip (struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +18 -9
@@ -2307,10 +2307,9 @@ static inline u64 __scale_tsc(u64 ratio, u64 tsc)return mul_u64_u64_shr (tsc, ratio, kvm_tsc_scaling_ratio_frac_bits); } -u64 kvm_scale_tsc (struct kvm_vcpu *vcpu,u64 tsc) +u64 kvm_scale_tsc (struct kvm_vcpu *vcpu,u64 tsc,u64 ratio) {u64 _tsc = tsc; -u64 ratio = vcpu->arch.tsc_scaling_ratio;if (ratio != kvm_default_tsc_scaling_ratio) _tsc =__scale_tsc (ratio, tsc);
@@ -2323,14 +2322,15 @@ static u64 kvm_compute_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 target_tsc) {u64 tsc; - tsc =kvm_scale_tsc (vcpu,rdtsc ()); + tsc =kvm_scale_tsc (vcpu,rdtsc (), vcpu->arch.l1_tsc_scaling_ratio);return target_tsc - tsc; }u64 kvm_read_l1_tsc (struct kvm_vcpu *vcpu,u64 host_tsc) { -return vcpu->arch.l1_tsc_offset +kvm_scale_tsc (vcpu, host_tsc); +return vcpu->arch.l1_tsc_offset + +kvm_scale_tsc (vcpu, host_tsc, vcpu->arch.l1_tsc_scaling_ratio); }EXPORT_SYMBOL_GPL (kvm_read_l1_tsc);
@@ -2463,7 +2463,8 @@ static inline void adjust_tsc_offset_host(struct kvm_vcpu *vcpu, s64 adjustment) {if (vcpu->arch.l1_tsc_scaling_ratio != kvm_default_tsc_scaling_ratio)WARN_ON (adjustment <0 ); - adjustment =kvm_scale_tsc (vcpu, (u64 ) adjustment); + adjustment =kvm_scale_tsc (vcpu, (u64 ) adjustment, + vcpu->arch.l1_tsc_scaling_ratio);adjust_tsc_offset_guest (vcpu, adjustment); }
@@ -2846,7 +2847,8 @@ static int kvm_guest_time_update(struct kvm_vcpu *v)/* With all the info we got, fill in the values */ if (kvm_has_tsc_control) - tgt_tsc_khz =kvm_scale_tsc (v, tgt_tsc_khz); + tgt_tsc_khz =kvm_scale_tsc (v, tgt_tsc_khz, + v->arch.l1_tsc_scaling_ratio);if (unlikely (vcpu->hw_tsc_khz != tgt_tsc_khz)) { kvm_get_time_scale(NSEC_PER_SEC, tgt_tsc_khz * 1000LL,
@@ -3537,10 +3539,17 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)* return L1's TSC value to ensure backwards-compatible * behavior for migration. */ -u64 tsc_offset = msr_info->host_initiated ? vcpu->arch.l1_tsc_offset : - vcpu->arch.tsc_offset; +u64 offset, ratio; - msr_info->data =kvm_scale_tsc (vcpu,rdtsc ()) + tsc_offset; +if (msr_info->host_initiated) { + offset = vcpu->arch.l1_tsc_offset; + ratio = vcpu->arch.l1_tsc_scaling_ratio; + }else { + offset = vcpu->arch.tsc_offset; + ratio = vcpu->arch.tsc_scaling_ratio; + } + + msr_info->data =kvm_scale_tsc (vcpu,rdtsc (), ratio) + offset;break ; }case MSR_MTRRcap:
Signature · 1 line
2.17.1
PATCH v4 5/11 KVM: nVMX: Add a TSC multiplier field in VMCS12
In reply to Ilias StamatisThis is required for supporting nested TSC scaling.
--- arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmcs12.c +1 -0
@@ -37,6 +37,7 @@ const unsigned short vmcs_field_to_offset_table[] = {FIELD64 (VM_ENTRY_MSR_LOAD_ADDR , vm_entry_msr_load_addr),FIELD64 (PML_ADDRESS , pml_address),FIELD64 (TSC_OFFSET , tsc_offset), +FIELD64 (TSC_MULTIPLIER , tsc_multiplier),FIELD64 (VIRTUAL_APIC_PAGE_ADDR , virtual_apic_page_addr),FIELD64 (APIC_ACCESS_ADDR , apic_access_addr), FIELD64(POSTED_INTR_DESC_ADDR, posted_intr_desc_addr),
arch/x86/kvm/vmx/vmcs12.h +3 -1
@@ -70,7 +70,8 @@ struct __packed vmcs12 {u64 eptp_list_address;u64 pml_address;u64 encls_exiting_bitmap; -u64 padding64[2 ];/* room for future expansion */ +u64 tsc_multiplier; +u64 padding64[1 ];/* room for future expansion */ /* * To allow migration of L1 (complete with its L2 guests) between * machines of different natural widths (32 or 64 bit), we cannot have
@@ -258,6 +259,7 @@ static inline void vmx_check_vmcs12_offsets(void)CHECK_OFFSET (eptp_list_address,304 );CHECK_OFFSET (pml_address,312 );CHECK_OFFSET (encls_exiting_bitmap,320 ); +CHECK_OFFSET (tsc_multiplier,328 );CHECK_OFFSET (cr0_guest_host_mask,344 );CHECK_OFFSET (cr4_guest_host_mask,352 );CHECK_OFFSET (cr0_read_shadow,360 );
Signature · 1 line
2.17.1
PATCH v4 6/11 KVM: X86: Add functions for retrieving L2 TSC fields from common code
In reply to Ilias StamatisIn order to implement as much of the nested TSC scaling logic as possible in common code, we need these vendor callbacks for retrieving the TSC offset and the TSC multiplier that L1 has set for L2.
--- arch/x86/include/asm/kvm-x86-ops.h | 2 ++ arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/svm/svm.c | 14 ++++++++++++++ arch/x86/kvm/vmx/vmx.c | 23 +++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.h | 3 +++ 5 files changed, 44 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +2 -0
@@ -87,6 +87,8 @@ KVM_X86_OP(set_identity_map_addr) KVM_X86_OP(get_mt_mask) KVM_X86_OP(load_mmu_pgd) KVM_X86_OP_NULL(has_wbinvd_exit) +KVM_X86_OP(get_l2_tsc_offset) +KVM_X86_OP(get_l2_tsc_multiplier) KVM_X86_OP(write_l1_tsc_offset) KVM_X86_OP(get_exit_info) KVM_X86_OP(check_intercept)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1305,6 +1305,8 @@ struct kvm_x86_ops {bool (*has_wbinvd_exit)(void ); +u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu); +u64 (*get_l2_tsc_multiplier)(struct kvm_vcpu *vcpu);/* Returns actual tsc_offset set in active VMCS */ u64 (*write_l1_tsc_offset)(struct kvm_vcpu *vcpu,u64 offset);
arch/x86/kvm/svm/svm.c +14 -0
@@ -1082,6 +1082,18 @@ static void init_sys_seg(struct vmcb_seg *seg, uint32_t type) seg->base =0 ; } +static u64 svm_get_l2_tsc_offset (struct kvm_vcpu *vcpu) +{ +struct vcpu_svm *svm =to_svm (vcpu); + +return svm->nested.ctl.tsc_offset; +} + +static u64 svm_get_l2_tsc_multiplier (struct kvm_vcpu *vcpu) +{ +return kvm_default_tsc_scaling_ratio; +} +static u64 svm_write_l1_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) {struct vcpu_svm *svm =to_svm (vcpu);
@@ -4526,6 +4538,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .has_wbinvd_exit = svm_has_wbinvd_exit, + .get_l2_tsc_offset = svm_get_l2_tsc_offset, + .get_l2_tsc_multiplier = svm_get_l2_tsc_multiplier, .write_l1_tsc_offset = svm_write_l1_tsc_offset, .load_mmu_pgd = svm_load_mmu_pgd,
arch/x86/kvm/vmx/vmx.c +23 -0
@@ -1787,6 +1787,27 @@ static void setup_msrs(struct vcpu_vmx *vmx) vmx->guest_uret_msrs_loaded = false; } +u64 vmx_get_l2_tsc_offset (struct kvm_vcpu *vcpu) +{ +struct vmcs12 *vmcs12 =get_vmcs12 (vcpu); + +if (nested_cpu_has (vmcs12,CPU_BASED_USE_TSC_OFFSETTING )) +return vmcs12->tsc_offset; + +return 0 ; +} + +u64 vmx_get_l2_tsc_multiplier (struct kvm_vcpu *vcpu) +{ +struct vmcs12 *vmcs12 =get_vmcs12 (vcpu); + +if (nested_cpu_has (vmcs12,CPU_BASED_USE_TSC_OFFSETTING ) && +nested_cpu_has2 (vmcs12,SECONDARY_EXEC_TSC_SCALING )) +return vmcs12->tsc_multiplier; + +return kvm_default_tsc_scaling_ratio; +} +static u64 vmx_write_l1_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) {struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);
@@ -7700,6 +7721,8 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .has_wbinvd_exit = cpu_has_vmx_wbinvd_exit, + .get_l2_tsc_offset = vmx_get_l2_tsc_offset, + .get_l2_tsc_multiplier = vmx_get_l2_tsc_multiplier, .write_l1_tsc_offset = vmx_write_l1_tsc_offset, .load_mmu_pgd = vmx_load_mmu_pgd,
arch/x86/kvm/vmx/vmx.h +3 -0
@@ -404,6 +404,9 @@ void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu);void vmx_disable_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type);void vmx_enable_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type); +u64 vmx_get_l2_tsc_offset (struct kvm_vcpu *vcpu); +u64 vmx_get_l2_tsc_multiplier (struct kvm_vcpu *vcpu); +static inline void vmx_set_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type,bool value) {
Signature · 1 line
2.17.1
PATCH v4 7/11 KVM: X86: Add functions that calculate the nested TSC fields
In reply to Ilias StamatisWhen L2 is entered we need to "merge" the TSC multiplier and TSC offset values of 01 and 12 together.
The merging is done using the following equations:
offset_02 = ((offset_01 * mult_12) >> shift_bits) + offset_12 mult_02 = (mult_01 * mult_12) >> shift_bits
Where shift_bits is kvm_tsc_scaling_ratio_frac_bits.
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/x86.c | 25 +++++++++++++++++++++++++ 2 files changed, 27 insertions(+)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1792,6 +1792,8 @@ static inline bool kvm_is_supported_user_return_msr(u32 msr)u64 kvm_scale_tsc (struct kvm_vcpu *vcpu,u64 tsc,u64 ratio);u64 kvm_read_l1_tsc (struct kvm_vcpu *vcpu,u64 host_tsc); +u64 kvm_calc_nested_tsc_offset (u64 l1_offset,u64 l2_offset,u64 l2_multiplier); +u64 kvm_calc_nested_tsc_multiplier (u64 l1_multiplier,u64 l2_multiplier);unsigned long kvm_get_linear_rip (struct kvm_vcpu *vcpu);bool kvm_is_linear_rip (struct kvm_vcpu *vcpu,unsigned long linear_rip);
arch/x86/kvm/x86.c +25 -0
@@ -2334,6 +2334,31 @@ u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc) }EXPORT_SYMBOL_GPL (kvm_read_l1_tsc); +u64 kvm_calc_nested_tsc_offset (u64 l1_offset,u64 l2_offset,u64 l2_multiplier) +{ +u64 nested_offset; + +if (l2_multiplier == kvm_default_tsc_scaling_ratio) + nested_offset = l1_offset; +else + nested_offset =mul_s64_u64_shr ((s64 ) l1_offset, l2_multiplier, + kvm_tsc_scaling_ratio_frac_bits); + + nested_offset += l2_offset; +return nested_offset; +} +EXPORT_SYMBOL_GPL (kvm_calc_nested_tsc_offset); + +u64 kvm_calc_nested_tsc_multiplier (u64 l1_multiplier,u64 l2_multiplier) +{ +if (l2_multiplier != kvm_default_tsc_scaling_ratio) +return mul_u64_u64_shr (l1_multiplier, l2_multiplier, + kvm_tsc_scaling_ratio_frac_bits); + +return l1_multiplier; +} +EXPORT_SYMBOL_GPL (kvm_calc_nested_tsc_multiplier); +static void kvm_vcpu_write_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) { vcpu->arch.l1_tsc_offset = offset;
Signature · 1 line
2.17.1
PATCH v4 8/11 KVM: X86: Move write_l1_tsc_offset() logic to common code and rename it
In reply to Ilias StamatisThe write_l1_tsc_offset() callback has a misleading name. It does not set L1's TSC offset, it rather updates the current TSC offset which might be different if a nested guest is executing. Additionally, both the vmx and svm implementations use the same logic for calculating the current TSC before writing it to hardware.
Rename the function and move the common logic to the caller. The vmx/svm specific code now merely sets the given offset to the corresponding hardware structure.
--- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 3 +-- arch/x86/kvm/svm/svm.c | 21 ++++----------------- arch/x86/kvm/vmx/vmx.c | 23 +++-------------------- arch/x86/kvm/x86.c | 24 +++++++++++++++++++++--- 5 files changed, 30 insertions(+), 43 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -1
@@ -89,7 +89,7 @@ KVM_X86_OP(load_mmu_pgd) KVM_X86_OP_NULL(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier) -KVM_X86_OP(write_l1_tsc_offset) +KVM_X86_OP(write_tsc_offset) KVM_X86_OP(get_exit_info) KVM_X86_OP(check_intercept) KVM_X86_OP(handle_exit_irqoff)
arch/x86/include/asm/kvm_host.h +1 -2
@@ -1307,8 +1307,7 @@ struct kvm_x86_ops {u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu);u64 (*get_l2_tsc_multiplier)(struct kvm_vcpu *vcpu); -/* Returns actual tsc_offset set in active VMCS */ -u64 (*write_l1_tsc_offset)(struct kvm_vcpu *vcpu,u64 offset); +void (*write_tsc_offset)(struct kvm_vcpu *vcpu,u64 offset);/* * Retrieve somewhat arbitrary exit information. Intended to be used
arch/x86/kvm/svm/svm.c +4 -17
@@ -1094,26 +1094,13 @@ static u64 svm_get_l2_tsc_multiplier(struct kvm_vcpu *vcpu)return kvm_default_tsc_scaling_ratio; } -static u64 svm_write_l1_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) +static void svm_write_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) {struct vcpu_svm *svm =to_svm (vcpu); -u64 g_tsc_offset =0 ; - -if (is_guest_mode (vcpu)) { -/* Write L1's TSC offset. */ - g_tsc_offset = svm->vmcb->control.tsc_offset - - svm->vmcb01.ptr->control.tsc_offset; - svm->vmcb01.ptr->control.tsc_offset = offset; - } - -trace_kvm_write_tsc_offset (vcpu->vcpu_id, - svm->vmcb->control.tsc_offset - g_tsc_offset, - offset); - - svm->vmcb->control.tsc_offset = offset + g_tsc_offset; + svm->vmcb01.ptr->control.tsc_offset = vcpu->arch.l1_tsc_offset; + svm->vmcb->control.tsc_offset = offset;vmcb_mark_dirty (svm->vmcb,VMCB_INTERCEPTS ); -return svm->vmcb->control.tsc_offset; }/* Evaluate instruction intercepts that depend on guest CPUID features. */
@@ -4540,7 +4527,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .get_l2_tsc_offset = svm_get_l2_tsc_offset, .get_l2_tsc_multiplier = svm_get_l2_tsc_multiplier, - .write_l1_tsc_offset = svm_write_l1_tsc_offset, + .write_tsc_offset = svm_write_tsc_offset, .load_mmu_pgd = svm_load_mmu_pgd,
arch/x86/kvm/vmx/vmx.c +3 -20
@@ -1808,26 +1808,9 @@ u64 vmx_get_l2_tsc_multiplier(struct kvm_vcpu *vcpu)return kvm_default_tsc_scaling_ratio; } -static u64 vmx_write_l1_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) +static void vmx_write_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) { -struct vmcs12 *vmcs12 =get_vmcs12 (vcpu); -u64 g_tsc_offset =0 ; - -/* -* We're here if L1 chose not to trap WRMSR to TSC. According -* to the spec, this should set L1's TSC; The offset that L1 -* set for L2 remains unchanged, and still needs to be added -* to the newly set TSC to get L2's TSC. -*/ -if (is_guest_mode (vcpu) && - (vmcs12->cpu_based_vm_exec_control &CPU_BASED_USE_TSC_OFFSETTING )) - g_tsc_offset = vmcs12->tsc_offset; - -trace_kvm_write_tsc_offset (vcpu->vcpu_id, - vcpu->arch.tsc_offset - g_tsc_offset, - offset); -vmcs_write64 (TSC_OFFSET , offset + g_tsc_offset); -return offset + g_tsc_offset; +vmcs_write64 (TSC_OFFSET , offset); }/*
@@ -7723,7 +7706,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .get_l2_tsc_offset = vmx_get_l2_tsc_offset, .get_l2_tsc_multiplier = vmx_get_l2_tsc_multiplier, - .write_l1_tsc_offset = vmx_write_l1_tsc_offset, + .write_tsc_offset = vmx_write_tsc_offset, .load_mmu_pgd = vmx_load_mmu_pgd,
arch/x86/kvm/x86.c +21 -3
@@ -2359,10 +2359,28 @@ u64 kvm_calc_nested_tsc_multiplier(u64 l1_multiplier, u64 l2_multiplier) }EXPORT_SYMBOL_GPL (kvm_calc_nested_tsc_multiplier); -static void kvm_vcpu_write_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) +static void kvm_vcpu_write_tsc_offset (struct kvm_vcpu *vcpu,u64 l1_offset) { - vcpu->arch.l1_tsc_offset = offset; - vcpu->arch.tsc_offset =static_call (kvm_x86_write_l1_tsc_offset)(vcpu, offset); +trace_kvm_write_tsc_offset (vcpu->vcpu_id, + vcpu->arch.l1_tsc_offset, + l1_offset); + + vcpu->arch.l1_tsc_offset = l1_offset; + +/* +* If we are here because L1 chose not to trap WRMSR to TSC then +* according to the spec this should set L1's TSC (as opposed to +* setting L1's offset for L2). +*/ +if (is_guest_mode (vcpu)) + vcpu->arch.tsc_offset =kvm_calc_nested_tsc_offset ( + l1_offset, +static_call (kvm_x86_get_l2_tsc_offset)(vcpu), +static_call (kvm_x86_get_l2_tsc_multiplier)(vcpu)); +else + vcpu->arch.tsc_offset = l1_offset; + +static_call (kvm_x86_write_tsc_offset)(vcpu, vcpu->arch.tsc_offset); }static inline bool kvm_check_tsc_unstable (void )
Signature · 1 line
2.17.1
PATCH v4 9/11 KVM: X86: Add vendor callbacks for writing the TSC multiplier
In reply to Ilias StamatisCurrently vmx_vcpu_load_vmcs() writes the TSC_MULTIPLIER field of the VMCS every time the VMCS is loaded. Instead of doing this, set this field from common code on initialization and whenever the scaling ratio changes.
Additionally remove vmx->current_tsc_ratio. This field is redundant as vcpu->arch.tsc_scaling_ratio already tracks the current TSC scaling ratio. The vmx->current_tsc_ratio field is only used for avoiding unnecessary writes but it is no longer needed after removing the code from the VMCS load path.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/svm/svm.c | 6 ++++++ arch/x86/kvm/vmx/nested.c | 9 ++++----- arch/x86/kvm/vmx/vmx.c | 11 ++++++----- arch/x86/kvm/vmx/vmx.h | 8 -------- arch/x86/kvm/x86.c | 28 +++++++++++++++++++++++----- 7 files changed, 41 insertions(+), 23 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -90,6 +90,7 @@ KVM_X86_OP_NULL(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier) KVM_X86_OP(write_tsc_offset) +KVM_X86_OP(write_tsc_multiplier) KVM_X86_OP(get_exit_info) KVM_X86_OP(check_intercept) KVM_X86_OP(handle_exit_irqoff)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1308,6 +1308,7 @@ struct kvm_x86_ops {u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu);u64 (*get_l2_tsc_multiplier)(struct kvm_vcpu *vcpu);void (*write_tsc_offset)(struct kvm_vcpu *vcpu,u64 offset); +void (*write_tsc_multiplier)(struct kvm_vcpu *vcpu,u64 multiplier);/* * Retrieve somewhat arbitrary exit information. Intended to be used
arch/x86/kvm/svm/svm.c +6 -0
@@ -1103,6 +1103,11 @@ static void svm_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset)vmcb_mark_dirty (svm->vmcb,VMCB_INTERCEPTS ); } +static void svm_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 multiplier) +{ +wrmsrl (MSR_AMD64_TSC_RATIO , multiplier); +} +/* Evaluate instruction intercepts that depend on guest CPUID features. */ static void svm_recalc_instruction_intercepts (struct kvm_vcpu *vcpu,struct vcpu_svm *svm)
@@ -4528,6 +4533,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .get_l2_tsc_offset = svm_get_l2_tsc_offset, .get_l2_tsc_multiplier = svm_get_l2_tsc_multiplier, .write_tsc_offset = svm_write_tsc_offset, + .write_tsc_multiplier = svm_write_tsc_multiplier, .load_mmu_pgd = svm_load_mmu_pgd,
arch/x86/kvm/vmx/nested.c +4 -5
@@ -2533,9 +2533,8 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, }vmcs_write64 (TSC_OFFSET , vcpu->arch.tsc_offset); -if (kvm_has_tsc_control) -decache_tsc_multiplier (vmx); +vmcs_write64 (TSC_MULTIPLIER , vcpu->arch.tsc_scaling_ratio);nested_vmx_transition_tlb_flush (vcpu, vmcs12, true);
@@ -4501,12 +4500,12 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmcs_write32 (VM_EXIT_MSR_LOAD_COUNT , vmx->msr_autoload.host.nr);vmcs_write32 (VM_ENTRY_MSR_LOAD_COUNT , vmx->msr_autoload.guest.nr);vmcs_write64 (TSC_OFFSET , vcpu->arch.tsc_offset); +if (kvm_has_tsc_control) +vmcs_write64 (TSC_MULTIPLIER , vcpu->arch.tsc_scaling_ratio); +if (vmx->nested.l1_tpr_threshold !=-1 )vmcs_write32 (TPR_THRESHOLD , vmx->nested.l1_tpr_threshold); -if (kvm_has_tsc_control) -decache_tsc_multiplier (vmx); -if (vmx->nested.change_vmcs01_virtual_apic_mode) { vmx->nested.change_vmcs01_virtual_apic_mode = false;vmx_set_virtual_apic_mode (vcpu);
arch/x86/kvm/vmx/vmx.c +6 -5
@@ -1390,11 +1390,6 @@ void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu, vmx->loaded_vmcs->cpu = cpu; } - -/* Setup TSC multiplier */ -if (kvm_has_tsc_control && - vmx->current_tsc_ratio != vcpu->arch.tsc_scaling_ratio) -decache_tsc_multiplier (vmx); }/*
@@ -1813,6 +1808,11 @@ static void vmx_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset)vmcs_write64 (TSC_OFFSET , offset); } +static void vmx_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 multiplier) +{ +vmcs_write64 (TSC_MULTIPLIER , multiplier); +} +/* * nested_vmx_allowed() checks whether a guest should be allowed to use VMX * instructions and MSRs (i.e., nested VMX). Nested VMX is disabled for
@@ -7707,6 +7707,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .get_l2_tsc_offset = vmx_get_l2_tsc_offset, .get_l2_tsc_multiplier = vmx_get_l2_tsc_multiplier, .write_tsc_offset = vmx_write_tsc_offset, + .write_tsc_multiplier = vmx_write_tsc_multiplier, .load_mmu_pgd = vmx_load_mmu_pgd,
arch/x86/kvm/vmx/vmx.h +0 -8
@@ -322,8 +322,6 @@ struct vcpu_vmx {/* apic deadline value in host tsc */ u64 hv_deadline_tsc; -u64 current_tsc_ratio; -unsigned long host_debugctlmsr;/*
@@ -532,12 +530,6 @@ static inline struct vmcs *alloc_vmcs(bool shadow) GFP_KERNEL_ACCOUNT); } -static inline void decache_tsc_multiplier (struct vcpu_vmx *vmx) -{ - vmx->current_tsc_ratio = vmx->vcpu.arch.tsc_scaling_ratio; -vmcs_write64 (TSC_MULTIPLIER , vmx->current_tsc_ratio); -} -static inline bool vmx_has_waitpkg (struct vcpu_vmx *vmx) { return vmx->secondary_exec_control &
arch/x86/kvm/x86.c +23 -5
@@ -2179,14 +2179,15 @@ static u32 adjust_tsc_khz(u32 khz, s32 ppm)return v; } +static void kvm_vcpu_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 l1_multiplier); +static int set_tsc_khz (struct kvm_vcpu *vcpu,u32 user_tsc_khz,bool scale) {u64 ratio;/* Guest TSC same frequency as host TSC? */ if (!scale) { - vcpu->arch.l1_tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; - vcpu->arch.tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; +kvm_vcpu_write_tsc_multiplier (vcpu, kvm_default_tsc_scaling_ratio);return 0 ; }
@@ -2212,7 +2213,7 @@ static int set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool scale)return -1 ; } - vcpu->arch.l1_tsc_scaling_ratio = vcpu->arch.tsc_scaling_ratio = ratio; +kvm_vcpu_write_tsc_multiplier (vcpu, ratio);return 0 ; }
@@ -2224,8 +2225,7 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)/* tsc_khz can be zero if TSC calibration fails */ if (user_tsc_khz ==0 ) {/* set tsc_scaling_ratio to a safe value */ - vcpu->arch.l1_tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; - vcpu->arch.tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; +kvm_vcpu_write_tsc_multiplier (vcpu, kvm_default_tsc_scaling_ratio);return -1 ; }
@@ -2383,6 +2383,23 @@ static void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset)static_call (kvm_x86_write_tsc_offset)(vcpu, vcpu->arch.tsc_offset); } +static void kvm_vcpu_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 l1_multiplier) +{ + vcpu->arch.l1_tsc_scaling_ratio = l1_multiplier; + +/* Userspace is changing the multiplier while L2 is active */ +if (is_guest_mode (vcpu)) + vcpu->arch.tsc_scaling_ratio =kvm_calc_nested_tsc_multiplier ( + l1_multiplier, +static_call (kvm_x86_get_l2_tsc_multiplier)(vcpu)); +else + vcpu->arch.tsc_scaling_ratio = l1_multiplier; + +if (kvm_has_tsc_control) +static_call (kvm_x86_write_tsc_multiplier)( + vcpu, vcpu->arch.tsc_scaling_ratio); +} +static inline bool kvm_check_tsc_unstable (void ) {#ifdef CONFIG_X86_64
@@ -10444,6 +10461,7 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)return ;vcpu_load (vcpu);kvm_synchronize_tsc (vcpu,0 ); +kvm_vcpu_write_tsc_multiplier (vcpu, kvm_default_tsc_scaling_ratio);vcpu_put (vcpu);/* poll control enabled by default */
Signature · 1 line
2.17.1
Re: [PATCH v4 09/11] KVM: X86: Add vendor callbacks for writing the TSC multiplier
Quoted 230 lines · Ilias Stamatis
Currently
vmx_vcpu_load_vmcs()writes theTSC_MULTIPLIERfield of the VMCS every time the VMCS is loaded. Instead of doing this, set this field from common code on initialization and whenever the scaling ratio changes.Additionally remove
vmx->current_tsc_ratio. This field is redundant asvcpu->arch.tsc_scaling_ratioalready tracks the current TSC scaling ratio. Thevmx->current_tsc_ratiofield is only used for avoiding unnecessary writes but it is no longer needed after removing the code from the VMCS load path.Suggested-by Sean Christopherson <[email protected]>Signed-off-by Ilias Stamatis <[email protected]>--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/svm/svm.c | 6 ++++++ arch/x86/kvm/vmx/nested.c | 9 ++++----- arch/x86/kvm/vmx/vmx.c | 11 ++++++----- arch/x86/kvm/vmx/vmx.h | 8 -------- arch/x86/kvm/x86.c | 28 +++++++++++++++++++++++----- 7 files changed, 41 insertions(+), 23 deletions(-)arch/x86/include/asm/kvm-x86-ops.h +1
-0@@ -90,6 +90,7 @@ KVM_X86_OP_NULL(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier) KVM_X86_OP(write_tsc_offset) +KVM_X86_OP(write_tsc_multiplier) KVM_X86_OP(get_exit_info) KVM_X86_OP(check_intercept) KVM_X86_OP(handle_exit_irqoff)arch/x86/include/asm/kvm_host.h +1
-0@@ -1308,6 +1308,7 @@ struct kvm_x86_ops {u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu);u64 (*get_l2_tsc_multiplier)(struct kvm_vcpu *vcpu);void (*write_tsc_offset)(struct kvm_vcpu *vcpu,u64 offset); +void (*write_tsc_multiplier)(struct kvm_vcpu *vcpu,u64 multiplier);/* * Retrieve somewhat arbitrary exit information. Intended to be used arch/x86/kvm/svm/svm.c +6
-0@@ -1103,6 +1103,11 @@ static void svm_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset)vmcb_mark_dirty (svm->vmcb,VMCB_INTERCEPTS ); } +static void svm_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 multiplier) +{ +wrmsrl (MSR_AMD64_TSC_RATIO , multiplier); +} +/* Evaluate instruction intercepts that depend on guest CPUID features. */ static void svm_recalc_instruction_intercepts (struct kvm_vcpu *vcpu,struct vcpu_svm *svm)@@ -4528,6 +4533,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .get_l2_tsc_offset = svm_get_l2_tsc_offset, .get_l2_tsc_multiplier = svm_get_l2_tsc_multiplier, .write_tsc_offset = svm_write_tsc_offset, + .write_tsc_multiplier = svm_write_tsc_multiplier, .load_mmu_pgd = svm_load_mmu_pgd,arch/x86/kvm/vmx/nested.c +4
-5@@ -2533,9 +2533,8 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, }vmcs_write64 (TSC_OFFSET , vcpu->arch.tsc_offset); -if (kvm_has_tsc_control) -decache_tsc_multiplier (vmx); +vmcs_write64 (TSC_MULTIPLIER , vcpu->arch.tsc_scaling_ratio);nested_vmx_transition_tlb_flush (vcpu, vmcs12, true);@@ -4501,12 +4500,12 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmcs_write32 (VM_EXIT_MSR_LOAD_COUNT , vmx->msr_autoload.host.nr);vmcs_write32 (VM_ENTRY_MSR_LOAD_COUNT , vmx->msr_autoload.guest.nr);vmcs_write64 (TSC_OFFSET , vcpu->arch.tsc_offset); +if (kvm_has_tsc_control) +vmcs_write64 (TSC_MULTIPLIER , vcpu->arch.tsc_scaling_ratio); +if (vmx->nested.l1_tpr_threshold !=-1 )vmcs_write32 (TPR_THRESHOLD , vmx->nested.l1_tpr_threshold); -if (kvm_has_tsc_control) -decache_tsc_multiplier (vmx); -if (vmx->nested.change_vmcs01_virtual_apic_mode) { vmx->nested.change_vmcs01_virtual_apic_mode = false;vmx_set_virtual_apic_mode (vcpu);arch/x86/kvm/vmx/vmx.c +6
-5@@ -1390,11 +1390,6 @@ void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu, vmx->loaded_vmcs->cpu = cpu; } - -/* Setup TSC multiplier */ -if (kvm_has_tsc_control && - vmx->current_tsc_ratio != vcpu->arch.tsc_scaling_ratio) -decache_tsc_multiplier (vmx); }/* @@ -1813,6 +1808,11 @@ static void vmx_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset)vmcs_write64 (TSC_OFFSET , offset); } +static void vmx_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 multiplier) +{ +vmcs_write64 (TSC_MULTIPLIER , multiplier); +} +/* * nested_vmx_allowed() checks whether a guest should be allowed to use VMX * instructions and MSRs (i.e., nested VMX). Nested VMX is disabled for @@ -7707,6 +7707,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .get_l2_tsc_offset = vmx_get_l2_tsc_offset, .get_l2_tsc_multiplier = vmx_get_l2_tsc_multiplier, .write_tsc_offset = vmx_write_tsc_offset, + .write_tsc_multiplier = vmx_write_tsc_multiplier, .load_mmu_pgd = vmx_load_mmu_pgd,arch/x86/kvm/vmx/vmx.h +0
-8@@ -322,8 +322,6 @@ struct vcpu_vmx {/* apic deadline value in host tsc */ u64 hv_deadline_tsc; -u64 current_tsc_ratio; -unsigned long host_debugctlmsr;/* @@ -532,12 +530,6 @@ static inline struct vmcs *alloc_vmcs(bool shadow) GFP_KERNEL_ACCOUNT); } -static inline void decache_tsc_multiplier (struct vcpu_vmx *vmx) -{ - vmx->current_tsc_ratio = vmx->vcpu.arch.tsc_scaling_ratio; -vmcs_write64 (TSC_MULTIPLIER , vmx->current_tsc_ratio); -} -static inline bool vmx_has_waitpkg (struct vcpu_vmx *vmx) { return vmx->secondary_exec_control &arch/x86/kvm/x86.c +22
-5@@ -2179,14 +2179,15 @@ static u32 adjust_tsc_khz(u32 khz, s32 ppm)return v; } +static void kvm_vcpu_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 l1_multiplier); +static int set_tsc_khz (struct kvm_vcpu *vcpu,u32 user_tsc_khz,bool scale) {u64 ratio;/* Guest TSC same frequency as host TSC? */ if (!scale) { - vcpu->arch.l1_tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; - vcpu->arch.tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; +kvm_vcpu_write_tsc_multiplier (vcpu, kvm_default_tsc_scaling_ratio);return 0 ; }@@ -2212,7 +2213,7 @@ static int set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool scale)return -1 ; } - vcpu->arch.l1_tsc_scaling_ratio = vcpu->arch.tsc_scaling_ratio = ratio; +kvm_vcpu_write_tsc_multiplier (vcpu, ratio);return 0 ; }@@ -2224,8 +2225,7 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)/* tsc_khz can be zero if TSC calibration fails */ if (user_tsc_khz ==0 ) {/* set tsc_scaling_ratio to a safe value */ - vcpu->arch.l1_tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; - vcpu->arch.tsc_scaling_ratio = kvm_default_tsc_scaling_ratio; +kvm_vcpu_write_tsc_multiplier (vcpu, kvm_default_tsc_scaling_ratio);return -1 ; }@@ -2383,6 +2383,23 @@ static void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset)static_call (kvm_x86_write_tsc_offset)(vcpu, vcpu->arch.tsc_offset); } +static void kvm_vcpu_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 l1_multiplier) +{ + vcpu->arch.l1_tsc_scaling_ratio = l1_multiplier; + +/* Userspace is changing the multiplier while L2 is active */ +if (is_guest_mode (vcpu)) + vcpu->arch.tsc_scaling_ratio =kvm_calc_nested_tsc_multiplier ( + l1_multiplier, +static_call (kvm_x86_get_l2_tsc_multiplier)(vcpu)); +else + vcpu->arch.tsc_scaling_ratio = l1_multiplier; + +if (kvm_has_tsc_control) +static_call (kvm_x86_write_tsc_multiplier)( + vcpu, vcpu->arch.tsc_scaling_ratio); +} +static inline bool kvm_check_tsc_unstable (void ) {#ifdef CONFIG_X86_64
arch/x86/kvm/x86.c void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) · line 10464
Quoted 4 lines
arch/x86/kvm/x86.c +1
-0@@ -10444,6 +10461,7 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)return ;vcpu_load (vcpu);kvm_synchronize_tsc (vcpu,0 ); +kvm_vcpu_write_tsc_multiplier (vcpu, kvm_default_tsc_scaling_ratio);
Hmm, I'm actually thinking now that this might not be correct. For example in case we hotplug a new vCPU but the other vCPUs don't use the default ratio.
How about keeping vmx->current_tsc_ratio after all? Apparently this is another role it plays. Or move it to kvm->arch?
Quoted 3 lines
arch/x86/kvm/x86.c +0
-0vcpu_put (vcpu);/* poll control enabled by default */
Paolo Bonzini
maintainerRe: [PATCH v4 09/11] KVM: X86: Add vendor callbacks for writing the TSC multiplier
Quoted 8 lines · Ilias
arch/x86/kvm/x86.c void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) · line 10464
Quoted 6 lines
#ifdef CONFIG_X86_64arch/x86/kvm/x86.c +1
-0@@ -10444,6 +10461,7 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)return ;vcpu_load (vcpu);kvm_synchronize_tsc (vcpu,0 ); +kvm_vcpu_write_tsc_multiplier (vcpu, kvm_default_tsc_scaling_ratio);Hmm, I'm actually thinking now that this might not be correct. For example in case we hotplug a new vCPU but the other vCPUs don't use the default ratio.
It is correct, the TSC frequency can be set per CPU (which is useless except possibly for debugging OS timekeeping, but still). So, the default kHz after hotplug is the host frequency.
It doesn't really matter because it only affects the fixed delta between the hotplugged CPU and the others as soon as userspace sets the frequency to the correct value.
Paolo
Re: [PATCH v4 09/11] KVM: X86: Add vendor callbacks for writing the TSC multiplier
Quoted 15 lines · Paolo Bonzini
Quoted 9 lines · Ilias
arch/x86/kvm/x86.c void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) · line 10464
Quoted 6 lines
#ifdef CONFIG_X86_64arch/x86/kvm/x86.c +1
-0@@ -10444,6 +10461,7 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)return ;vcpu_load (vcpu);kvm_synchronize_tsc (vcpu,0 ); +kvm_vcpu_write_tsc_multiplier (vcpu, kvm_default_tsc_scaling_ratio);Hmm, I'm actually thinking now that this might not be correct. For example in case we hotplug a new vCPU but the other vCPUs don't use the default ratio.
It is correct, the TSC frequency can be set per CPU (which is useless except possibly for debugging OS timekeeping, but still). So, the default kHz after hotplug is the host frequency.
Quoted 6 lines
It doesn't really matter because it only affects the fixed delta between the hotplugged CPU and the others as soon as userspace sets the frequency to the correct value.
Paolo
So this patch is wrong anyway.
kvm_arch_vcpu_create() does a kvm_set_tsc_khz(vcpu, max_tsc_khz) when initializing the vcpu. This wouldn't normally result in a VMWRITE, but now (after applying patch 9) it does. The problem is that this write now happens too early and it raises an exception. To fix this, that line needs to be moved to kvm_arch_vcpu_postcreate() (like above) but before calling kvm_synchronize_tsc(vcpu, 0).
I will re-submit this patch with the fix.
Best,
Ilias
PATCH v4 10/11 KVM: nVMX: Enable nested TSC scaling
In reply to Ilias StamatisCalculate the TSC offset and multiplier on nested transitions and expose the TSC scaling feature to L1.
--- arch/x86/kvm/vmx/nested.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/nested.c +18 -6
@@ -2277,7 +2277,8 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE |SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |SECONDARY_EXEC_APIC_REGISTER_VIRT | - SECONDARY_EXEC_ENABLE_VMFUNC); +SECONDARY_EXEC_ENABLE_VMFUNC | + SECONDARY_EXEC_TSC_SCALING);if (nested_cpu_has (vmcs12,CPU_BASED_ACTIVATE_SECONDARY_CONTROLS )) exec_control |= vmcs12->secondary_vm_exec_control;
@@ -2532,6 +2533,15 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,vmcs_write64 (GUEST_IA32_PAT , vmx->vcpu.arch.pat); } + vcpu->arch.tsc_offset =kvm_calc_nested_tsc_offset ( + vcpu->arch.l1_tsc_offset, +vmx_get_l2_tsc_offset (vcpu), +vmx_get_l2_tsc_multiplier (vcpu)); + + vcpu->arch.tsc_scaling_ratio =kvm_calc_nested_tsc_multiplier ( + vcpu->arch.l1_tsc_scaling_ratio, +vmx_get_l2_tsc_multiplier (vcpu)); +vmcs_write64 (TSC_OFFSET , vcpu->arch.tsc_offset);if (kvm_has_tsc_control)vmcs_write64 (TSC_MULTIPLIER , vcpu->arch.tsc_scaling_ratio);
@@ -3353,8 +3363,6 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, }enter_guest_mode (vcpu); -if (vmcs12->cpu_based_vm_exec_control &CPU_BASED_USE_TSC_OFFSETTING ) - vcpu->arch.tsc_offset += vmcs12->tsc_offset;if (prepare_vmcs02 (vcpu, vmcs12, &entry_failure_code)) { exit_reason.basic =EXIT_REASON_INVALID_STATE ;
@@ -4462,8 +4470,11 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,if (nested_cpu_has_preemption_timer (vmcs12))hrtimer_cancel (&to_vmx (vcpu)->nested.preemption_timer); -if (vmcs12->cpu_based_vm_exec_control &CPU_BASED_USE_TSC_OFFSETTING ) - vcpu->arch.tsc_offset -= vmcs12->tsc_offset; +if (nested_cpu_has (vmcs12,CPU_BASED_USE_TSC_OFFSETTING )) { + vcpu->arch.tsc_offset = vcpu->arch.l1_tsc_offset; +if (nested_cpu_has2 (vmcs12,SECONDARY_EXEC_TSC_SCALING )) + vcpu->arch.tsc_scaling_ratio = vcpu->arch.l1_tsc_scaling_ratio; + }if (likely (!vmx->fail)) {sync_vmcs02_to_vmcs12 (vcpu, vmcs12);
@@ -6473,7 +6484,8 @@ void nested_vmx_setup_ctls_msrs(struct nested_vmx_msrs *msrs, u32 ept_caps)SECONDARY_EXEC_RDRAND_EXITING |SECONDARY_EXEC_ENABLE_INVPCID |SECONDARY_EXEC_RDSEED_EXITING | -SECONDARY_EXEC_XSAVES ; +SECONDARY_EXEC_XSAVES | +SECONDARY_EXEC_TSC_SCALING ;/* * We can emulate "VMCS shadowing," even if the hardware
Signature · 1 line
2.17.1
PATCH v4 11/11 KVM: selftests: x86: Add vmx_nested_tsc_scaling_test
In reply to Ilias StamatisTest that nested TSC scaling works as expected with both L1 and L2 scaled.
--- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../kvm/x86_64/vmx_nested_tsc_scaling_test.c | 242 ++++++++++++++++++ 3 files changed, 244 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86_64/vmx_nested_tsc_scaling_test.c
tools/testing/selftests/kvm/.gitignore +1 -0
@@ -29,6 +29,7 @@ /x86_64/vmx_preemption_timer_test /x86_64/vmx_set_nested_state_test /x86_64/vmx_tsc_adjust_test +/x86_64/vmx_nested_tsc_scaling_test /x86_64/xapic_ipi_test /x86_64/xen_shinfo_test /x86_64/xen_vmcall_test
tools/testing/selftests/kvm/Makefile +1 -0
@@ -60,6 +60,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/vmx_close_while_nested_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_dirty_log_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_set_nested_state_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_tsc_adjust_test +TEST_GEN_PROGS_x86_64 += x86_64/vmx_nested_tsc_scaling_test TEST_GEN_PROGS_x86_64 += x86_64/xapic_ipi_test TEST_GEN_PROGS_x86_64 += x86_64/xss_msr_test TEST_GEN_PROGS_x86_64 += x86_64/debug_regs
tools/testing/selftests/kvm/x86_64/vmx_nested_tsc_scaling_test.c new file +242 -0
242 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 53 lines · Ilias Stamatis
KVM currently supports hardware-assisted TSC scaling but only for L1; the feature is not exposed to nested guests. This patch series adds support for nested TSC scaling and allows both L1 and L2 to be scaled with different scaling factors. That is achieved by "merging" the 01 and 02 values together.
Most of the logic in this series is implemented in common code (by doing the necessary restructurings), however the patches add support for VMX only. Adding support for SVM should be easy at this point and Maxim Levitsky has volunteered to do this (thanks!).
Changelog:
Only patch 9 needs reviewing at this point, but I am re-sending the whole series as I also applied nitpicks suggested to some of the other pathces.
v4:
- Added vendor callbacks for writing the TSC multiplier
- Moved the VMCS multiplier writes from the VMCS load path to common code that only gets called on TSC ratio changes
- Merged together patches 10 and 11 of v3
- Applied all nitpick-feedback of the previous versionsv3:
- Applied Sean's feedback
- Refactored patches 7 to 10v2:
- Applied all of Maxim's feedback
- Added amul_s64_u64_shrfunction inmath64.h
- Added a separatekvm_scale_tsc_l1function instead of passing an argument tokvm_scale_tsc
- Implemented the 02 fields calculations in common code
- Moved all of write_l1_tsc_offset's logic to common code
- Added a check for whether the TSC is stable in patch 10
- Used a random L1 factor and a negative offset in patch 10Ilias Stamatis (11):
math64.h: Addmul_s64_u64_shr()
KVM: X86: Store L1's TSC scaling ratio in 'structkvm_vcpu_arch'
KVM: X86: Renamekvm_compute_tsc_offset()tokvm_compute_l1_tsc_offset()KVM: X86: Add a ratio parameter to
kvm_scale_tsc()
KVM: nVMX: Add a TSC multiplier field in VMCS12
KVM: X86: Add functions for retrieving L2 TSC fields from common code
KVM: X86: Add functions that calculate the nested TSC fields
KVM: X86: Movewrite_l1_tsc_offset()logic to common code and rename
it
KVM: X86: Add vendor callbacks for writing the TSC multiplier
KVM: nVMX: Enable nested TSC scaling
KVM: selftests: x86: Addvmx_nested_tsc_scaling_test
Quoted 16 lines
arch/x86/include/asm/kvm-x86-ops.h | 5 +- arch/x86/include/asm/kvm_host.h | 15 +- arch/x86/kvm/svm/svm.c | 35 ++- arch/x86/kvm/vmx/nested.c | 33 ++- arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +- arch/x86/kvm/vmx/vmx.c | 55 ++-- arch/x86/kvm/vmx/vmx.h | 11 +- arch/x86/kvm/x86.c | 114 +++++++-- include/linux/math64.h | 19 ++ tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../kvm/x86_64/vmx_nested_tsc_scaling_test.c | 242 ++++++++++++++++++ 13 files changed, 451 insertions(+), 85 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/vmx_nested_tsc_scaling_test.c
Hello,
What is the status of these? I think all patches have been reviewed and approved at this point.
(There's a new revision v6 of patch 9 that has been reviewed too)
Thanks,
Ilias
Quoted 73 lines · Ilias Stamatis
Quoted 53 lines · Ilias Stamatis
KVM currently supports hardware-assisted TSC scaling but only for L1; the feature is not exposed to nested guests. This patch series adds support for nested TSC scaling and allows both L1 and L2 to be scaled with different scaling factors. That is achieved by "merging" the 01 and 02 values together.
Most of the logic in this series is implemented in common code (by doing the necessary restructurings), however the patches add support for VMX only. Adding support for SVM should be easy at this point and Maxim Levitsky has volunteered to do this (thanks!).
Changelog:
Only patch 9 needs reviewing at this point, but I am re-sending the whole series as I also applied nitpicks suggested to some of the other pathces.
v4:
- Added vendor callbacks for writing the TSC multiplier
- Moved the VMCS multiplier writes from the VMCS load path to common code that only gets called on TSC ratio changes
- Merged together patches 10 and 11 of v3
- Applied all nitpick-feedback of the previous versionsv3:
- Applied Sean's feedback
- Refactored patches 7 to 10v2:
- Applied all of Maxim's feedback
- Added amul_s64_u64_shrfunction inmath64.h
- Added a separatekvm_scale_tsc_l1function instead of passing an argument tokvm_scale_tsc
- Implemented the 02 fields calculations in common code
- Moved all of write_l1_tsc_offset's logic to common code
- Added a check for whether the TSC is stable in patch 10
- Used a random L1 factor and a negative offset in patch 10Ilias Stamatis (11):
math64.h: Addmul_s64_u64_shr()
KVM: X86: Store L1's TSC scaling ratio in 'structkvm_vcpu_arch'
KVM: X86: Renamekvm_compute_tsc_offset()tokvm_compute_l1_tsc_offset()KVM: X86: Add a ratio parameter to
kvm_scale_tsc()
KVM: nVMX: Add a TSC multiplier field in VMCS12
KVM: X86: Add functions for retrieving L2 TSC fields from common code
KVM: X86: Add functions that calculate the nested TSC fields
KVM: X86: Movewrite_l1_tsc_offset()logic to common code and rename
it
KVM: X86: Add vendor callbacks for writing the TSC multiplier
KVM: nVMX: Enable nested TSC scaling
KVM: selftests: x86: Addvmx_nested_tsc_scaling_testQuoted 16 lines
arch/x86/include/asm/kvm-x86-ops.h | 5 +- arch/x86/include/asm/kvm_host.h | 15 +- arch/x86/kvm/svm/svm.c | 35 ++- arch/x86/kvm/vmx/nested.c | 33 ++- arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +- arch/x86/kvm/vmx/vmx.c | 55 ++-- arch/x86/kvm/vmx/vmx.h | 11 +- arch/x86/kvm/x86.c | 114 +++++++-- include/linux/math64.h | 19 ++ tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../kvm/x86_64/vmx_nested_tsc_scaling_test.c | 242 ++++++++++++++++++ 13 files changed, 451 insertions(+), 85 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/vmx_nested_tsc_scaling_test.cHello,
Quoted 7 lines
What is the status of these? I think all patches have been reviewed and approved at this point.
(There's a new revision v6 of patch 9 that has been reviewed too)
Thanks,
Ilias
Never mind this, I was probably checking at the wrong place, I just realised that they have been queued since last Friday.
Thanks!