Add support for the Idle HLT intercept feature
v4 cover Add support for the Idle HLT intercept feature
The upcoming new Idle HLT Intercept feature allows for the HLT instruction execution by a vCPU to be intercepted by the hypervisor only if there are no pending V_INTR and V_NMI events for the vCPU. When the vCPU is expected to service the pending V_INTR and V_NMI events, the Idle HLT intercept won’t trigger. The feature allows the hypervisor to determine if the vCPU is actually idle and reduces wasteful VMEXITs.
The idle HLT intercept feature is used for enlightened guests who wish to securely handle the events. When an enlightened guest does a HLT while an interrupt is pending, hypervisor will not have a way to figure out whether the guest needs to be re-entered or not. The Idle HLT intercept feature allows the HLT execution only if there are no pending V_INTR and V_NMI events.
Presence of the Idle HLT Intercept feature is indicated via CPUID function Fn8000_000A_EDX[30].
Document for the Idle HLT intercept feature is available at [1].
This series is based on kvm-next/next (64dbb3a771a1) + [2].
Experiments done:
----------------
kvm_amd.avic is set to '0' for this experiment.
The below numbers represent the average of 10 runs.
Normal guest (L1)
The below netperf command was run on the guest with smp = 1 (pinned).
netperf -H <host ip> -t TCP_RR -l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 25645.7136 | 25773.2796 |
----------------------------------------------------------------
Number of transactions/sec with and without idle HLT intercept feature are almost same.
Nested guest (L2)
The below netperf command was run on L2 guest with smp = 1 (pinned).
netperf -H <host ip> -t TCP_RR -l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 5655.4468 | 5755.2189 |
----------------------------------------------------------------
Number of transactions/sec with and without idle HLT intercept feature are almost same.
Testing Done:
- Tested the functionality for the Idle HLT intercept feature using selftest svm_idle_hlt_test.
- Tested SEV and SEV-ES guest for the Idle HLT intercept functionality.
- Tested the Idle HLT intercept functionality on nested guest.
v3 -> v4
- Drop the patches to add vcpu_get_stat() into a new series [2].
- Added nested Idle HLT intercept support.
v2 -> v3
- Incorporated Andrew's suggestion to structure vcpu_stat_types in a way that each architecture can share the generic types and also provide its own.
v1 -> v2
- Done changes in svm_idle_hlt_test based on the review comments from Sean.
- Added an enum based approach to get binary stats in vcpu_get_stat() which doesn't use string to get stat data based on the comments from Sean.
- Added self_halt() and cli() helpers based on the comments from Sean.
[1]: AMD64 Architecture Programmer's Manual Pub. 24593, April 2024,
Vol 2, 15.9 Instruction Intercepts (Table 15-7: IDLE_HLT).
https://bugzilla.kernel.org/attachment.cgi?id=306250[2]: https://lore.kernel.org/kvm/[email protected]/T/#t
Manali Shukla (4):
x86/cpufeatures: Add CPUID feature bit for Idle HLT intercept
KVM: SVM: Add Idle HLT intercept support
KVM: nSVM: implement the nested idle halt intercept
KVM: selftests: KVM: SVM: Add Idle HLT intercept test
arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/svm.h | 1 + arch/x86/include/uapi/asm/svm.h | 2 + arch/x86/kvm/governed_features.h | 1 + arch/x86/kvm/svm/nested.c | 7 ++ arch/x86/kvm/svm/svm.c | 15 +++- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/processor.h | 1 + .../selftests/kvm/x86_64/svm_idle_hlt_test.c | 89 +++++++++++++++++++ 9 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c
base-commit: c8d430db8eec7d4fd13a6bea27b7086a54eda6da
prerequisite-patch-id: ca912571db5c004f77b70843b8dd35517ff1267f
prerequisite-patch-id: 164ea3b4346f9e04bc69819278d20f5e1b5df5ed
prerequisite-patch-id: 90d870f426ebc2cec43c0dd89b701ee998385455
prerequisite-patch-id: 45812b799c517a4521782a1fdbcda881237e1eda
Signature · 1 line
2.34.1
PATCH v4 1/4 x86/cpufeatures: Add CPUID feature bit for Idle HLT intercept
From: Manali Shukla <[email protected]>
The Idle HLT Intercept feature allows for the HLT instruction execution by a vCPU to be intercepted by the hypervisor only if there are no pending events (V_INTR and V_NMI) for the vCPU. When the vCPU is expected to service the pending events (V_INTR and V_NMI), the Idle HLT intercept won’t trigger. The feature allows the hypervisor to determine if the vCPU is idle and reduces wasteful VMEXITs.
In addition to the aforementioned use case, the Idle HLT intercept feature is also used for enlightened guests who aim to securely manage events without the hypervisor’s awareness. If a HLT occurs while a virtual event is pending and the hypervisor is unaware of this pending event (as could be the case with enlightened guests), the absence of the Idle HLT intercept feature could result in a vCPU being suspended indefinitely.
Presence of Idle HLT intercept feature for guests is indicated via CPUID function 0x8000000A_EDX[30].
--- arch/x86/include/asm/cpufeatures.h | 1 + 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -382,6 +382,7 @@#define X86_FEATURE_V_SPEC_CTRL (15*32+20)/* "v_spec_ctrl" Virtual SPEC_CTRL */ #define X86_FEATURE_VNMI (15*32+25)/* "vnmi" Virtual NMI */ #define X86_FEATURE_SVME_ADDR_CHK (15*32+28)/* SVME addr check */ +#define X86_FEATURE_IDLE_HLT (15*32+30)/* "" IDLE HLT intercept */ /* Intel-defined CPU features, CPUID level 0x00000007:0 (ECX), word 16 */ #define X86_FEATURE_AVX512VBMI (16*32+ 1)/* "avx512vbmi" AVX512 Vector Bit Manipulation instructions*/
base-commit: c8d430db8eec7d4fd13a6bea27b7086a54eda6da
prerequisite-patch-id: ca912571db5c004f77b70843b8dd35517ff1267f
prerequisite-patch-id: 164ea3b4346f9e04bc69819278d20f5e1b5df5ed
prerequisite-patch-id: 90d870f426ebc2cec43c0dd89b701ee998385455
prerequisite-patch-id: 45812b799c517a4521782a1fdbcda881237e1eda
Signature · 1 line
2.34.1
Borislav Petkov
maintainerRe: [PATCH v4 1/4] x86/cpufeatures: Add CPUID feature bit for Idle HLT intercept
Quoted 31 lines · Manali Shukla
From: Manali Shukla <[email protected]>
The Idle HLT Intercept feature allows for the HLT instruction execution by a vCPU to be intercepted by the hypervisor only if there are no pending events (
V_INTRandV_NMI) for the vCPU. When the vCPU is expected to service the pending events (V_INTRandV_NMI), the Idle HLT intercept won’t trigger. The feature allows the hypervisor to determine if the vCPU is idle and reduces wasteful VMEXITs.In addition to the aforementioned use case, the Idle HLT intercept feature is also used for enlightened guests who aim to securely manage events without the hypervisor’s awareness. If a HLT occurs while a virtual event is pending and the hypervisor is unaware of this pending event (as could be the case with enlightened guests), the absence of the Idle HLT intercept feature could result in a vCPU being suspended indefinitely.
Presence of Idle HLT intercept feature for guests is indicated via CPUID function 0x8000000A_EDX[30].
Signed-off-by Manali Shukla <[email protected]>--- arch/x86/include/asm/cpufeatures.h | 1 + 1 file changed, 1 insertion(+)arch/x86/include/asm/cpufeatures.h +0
-0
arch/x86/include/asm/cpufeatures.h line 385
Quoted 4 lines
arch/x86/include/asm/cpufeatures.h +1
-0@@ -382,6 +382,7 @@#define X86_FEATURE_V_SPEC_CTRL (15*32+20)/* "v_spec_ctrl" Virtual SPEC_CTRL */ #define X86_FEATURE_VNMI (15*32+25)/* "vnmi" Virtual NMI */ #define X86_FEATURE_SVME_ADDR_CHK (15*32+28)/* SVME addr check */ +#define X86_FEATURE_IDLE_HLT (15*32+30)/* "" IDLE HLT intercept */
Whoever commits this, you can remove the "" in the comment now as they're not needed anymore.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerRe: [PATCH v4 1/4] x86/cpufeatures: Add CPUID feature bit for Idle HLT intercept
Quoted 27 lines · Borislav Petkov
Quoted 26 lines · Manali Shukla
From: Manali Shukla <[email protected]>
The Idle HLT Intercept feature allows for the HLT instruction execution by a vCPU to be intercepted by the hypervisor only if there are no pending events (
V_INTRandV_NMI) for the vCPU. When the vCPU is expected to service the pending events (V_INTRandV_NMI), the Idle HLT intercept won’t trigger. The feature allows the hypervisor to determine if the vCPU is idle and reduces wasteful VMEXITs.In addition to the aforementioned use case, the Idle HLT intercept feature is also used for enlightened guests who aim to securely manage events without the hypervisor’s awareness. If a HLT occurs while a virtual event is pending and the hypervisor is unaware of this pending event (as could be the case with enlightened guests), the absence of the Idle HLT intercept feature could result in a vCPU being suspended indefinitely.
Presence of Idle HLT intercept feature for guests is indicated via CPUID function 0x8000000A_EDX[30].
Signed-off-by Manali Shukla <[email protected]>--- arch/x86/include/asm/cpufeatures.h | 1 + 1 file changed, 1 insertion(+)
Quoted 12 lines
arch/x86/include/asm/cpufeatures.h line 385
Quoted 9 lines
arch/x86/include/asm/cpufeatures.h +1
-0@@ -382,6 +382,7 @@#define X86_FEATURE_V_SPEC_CTRL (15*32+20)/* "v_spec_ctrl" Virtual SPEC_CTRL */ #define X86_FEATURE_VNMI (15*32+25)/* "vnmi" Virtual NMI */ #define X86_FEATURE_SVME_ADDR_CHK (15*32+28)/* SVME addr check */ +#define X86_FEATURE_IDLE_HLT (15*32+30)/* "" IDLE HLT intercept */ Whoever commits this, you can remove the "" in the comment now as they're not needed anymore.
Ya, will do.
PATCH v4 2/4 KVM: SVM: Add Idle HLT intercept support
In reply to Manali ShuklaFrom: Manali Shukla <[email protected]>
The hypervisor can intercept the HLT instruction by setting the HLT-Intercept Bit in VMCB, causing a VMEXIT. This can be wasteful if there are pending V_INTR and V_NMI events, as the hypervisor must then initiate a VMRUN to handle them.
If the HLT-Intercept Bit is cleared and the vCPU executes HLT while there are pending V_INTR and V_NMI events, the hypervisor won’t detect them, potentially causing indefinite suspension of the vCPU. This poses a problem for enlightened guests who wish to securely handle the events.
For Secure AVIC scenarios, if a guest does a HLT while an interrupt is pending (in IRR), the hypervisor does not have a way to figure out whether the guest needs to be re-entered, as it cannot read the guest backing page. The Idle HLT intercept feature allows the hypervisor to intercept HLT execution only if there are no pending V_INTR and V_NMI events.
There are two use cases for the Idle HLT intercept feature:
- Secure VMs that wish to handle pending events securely without exiting to the hypervisor on HLT (Secure AVIC).
- Optimization for all the VMs to avoid a wasteful VMEXIT during HLT when there are pending events.
On discovering the Idle HLT Intercept, the KVM hypervisor, Sets the Idle HLT Intercept bit (bit (6), offset 0x14h) in the VMCB. When the Idle HLT Intercept bit is set, HLT Intercept bit (bit (0), offset 0xFh) should be cleared.
Before entering the HLT state, the HLT instruction performs checks in following order:
- The HLT intercept check, if set, it unconditionally triggers SVM_EXIT_HLT (0x78).
- The Idle HLT intercept check, if set and there are no pending V_INTR or V_NMI events, triggers SVM_EXIT_IDLE_HLT (0xA6).
Details about the Idle HLT intercept feature can be found in AMD APM [1].
[1]: AMD64 Architecture Programmer's Manual Pub. 24593, April
2024, Vol 2, 15.9 Instruction Intercepts (Table 15-7: IDLE_HLT).
https://bugzilla.kernel.org/attachment.cgi?id=306250--- arch/x86/include/asm/svm.h | 1 + arch/x86/include/uapi/asm/svm.h | 2 ++ arch/x86/kvm/svm/svm.c | 11 ++++++++--- 3 files changed, 11 insertions(+), 3 deletions(-)
arch/x86/include/asm/svm.h +1 -0
@@ -116,6 +116,7 @@ enum {INTERCEPT_INVPCID ,INTERCEPT_MCOMMIT ,INTERCEPT_TLBSYNC , + INTERCEPT_IDLE_HLT = 166, };
arch/x86/include/uapi/asm/svm.h +2 -0
@@ -95,6 +95,7 @@#define SVM_EXIT_CR14_WRITE_TRAP 0x09e#define SVM_EXIT_CR15_WRITE_TRAP 0x09f#define SVM_EXIT_INVPCID 0x0a2 +#define SVM_EXIT_IDLE_HLT 0x0a6#define SVM_EXIT_NPF 0x400#define SVM_EXIT_AVIC_INCOMPLETE_IPI 0x401#define SVM_EXIT_AVIC_UNACCELERATED_ACCESS 0x402
@@ -224,6 +225,7 @@ { SVM_EXIT_CR4_WRITE_TRAP, "write_cr4_trap" }, \ { SVM_EXIT_CR8_WRITE_TRAP, "write_cr8_trap" }, \ { SVM_EXIT_INVPCID, "invpcid" }, \ + { SVM_EXIT_IDLE_HLT, "idle-halt" }, \ { SVM_EXIT_NPF, "npf" }, \ { SVM_EXIT_AVIC_INCOMPLETE_IPI, "avic_incomplete_ipi" }, \ { SVM_EXIT_AVIC_UNACCELERATED_ACCESS, "avic_unaccelerated_access" }, \
arch/x86/kvm/svm/svm.c +8 -3
@@ -1298,8 +1298,12 @@ static void init_vmcb(struct kvm_vcpu *vcpu)svm_set_intercept (svm,INTERCEPT_MWAIT ); } -if (!kvm_hlt_in_guest (vcpu->kvm)) -svm_set_intercept (svm,INTERCEPT_HLT ); +if (!kvm_hlt_in_guest (vcpu->kvm)) { +if (cpu_feature_enabled (X86_FEATURE_IDLE_HLT )) +svm_set_intercept (svm,INTERCEPT_IDLE_HLT ); +else +svm_set_intercept (svm,INTERCEPT_HLT ); + } control->iopm_base_pa = iopm_base; control->msrpm_base_pa =__sme_set (__pa (svm->msrpm));
@@ -3353,6 +3357,7 @@ static int (*const svm_exit_handlers[])(struct kvm_vcpu *vcpu) = { [SVM_EXIT_CR4_WRITE_TRAP] = cr_trap, [SVM_EXIT_CR8_WRITE_TRAP] = cr_trap, [SVM_EXIT_INVPCID] = invpcid_interception, + [SVM_EXIT_IDLE_HLT] = kvm_emulate_halt, [SVM_EXIT_NPF] = npf_interception, [SVM_EXIT_RSM] = rsm_interception, [SVM_EXIT_AVIC_INCOMPLETE_IPI] = avic_incomplete_ipi_interception,
@@ -3515,7 +3520,7 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)return interrupt_window_interception (vcpu); else if (exit_code == SVM_EXIT_INTR) return intr_interception(vcpu); - else if (exit_code == SVM_EXIT_HLT) + else if (exit_code == SVM_EXIT_HLT || exit_code == SVM_EXIT_IDLE_HLT) return kvm_emulate_halt(vcpu); else if (exit_code == SVM_EXIT_NPF) return npf_interception(vcpu);
Signature · 1 line
2.34.1
PATCH v4 3/4 KVM: nSVM: implement the nested idle halt intercept
In reply to Manali ShuklaEnable the idle HLT intercept for the L2 guest when it is available on the platform and the L2 guest can utilize the Idle HLT intercept feature. This is achieved by clearing the HLT intercept for the L1 hypervisor in the recalc_interrupts() function. The Idle HLT intercept requires that the HLT intercept is cleared and the Idle HLT intercept is set to properly enable the feature.
The nested idle halt intercept was tested by booting L1,L2 (all Linux) and checking there are only idle-hlt vmexits happened.
--- arch/x86/kvm/governed_features.h | 1 + arch/x86/kvm/svm/nested.c | 8 ++++++++ arch/x86/kvm/svm/svm.c | 4 ++++ 3 files changed, 13 insertions(+)
arch/x86/kvm/governed_features.h +1 -0
@@ -17,6 +17,7 @@ KVM_GOVERNED_X86_FEATURE(PFTHRESHOLD) KVM_GOVERNED_X86_FEATURE(VGIF) KVM_GOVERNED_X86_FEATURE(VNMI) KVM_GOVERNED_X86_FEATURE(LAM) +KVM_GOVERNED_X86_FEATURE(IDLE_HLT)#undef KVM_GOVERNED_X86_FEATURE#undef KVM_GOVERNED_FEATURE
arch/x86/kvm/svm/nested.c +8 -0
@@ -178,6 +178,14 @@ void recalc_intercepts(struct vcpu_svm *svm) }else {WARN_ON (!(c->virt_ext &VIRTUAL_VMLOAD_VMSAVE_ENABLE_MASK )); } + +/* +* Clear the HLT intercept for L2 guest when the Idle HLT intercept feature +* is enabled on the platform and the guest can use the Idle HLT intercept +* feature. +*/ +if (guest_can_use (&svm->vcpu,X86_FEATURE_IDLE_HLT )) +vmcb_clr_intercept (c,INTERCEPT_HLT ); }/*
arch/x86/kvm/svm/svm.c +4 -0
@@ -4425,6 +4425,7 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_PFTHRESHOLD );kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_VGIF );kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_VNMI ); +kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_IDLE_HLT );svm_recalc_instruction_intercepts (vcpu, svm);
@@ -5228,6 +5229,9 @@ static __init void svm_set_cpu_caps(void)if (vnmi)kvm_cpu_cap_set (X86_FEATURE_VNMI ); +if (cpu_feature_enabled (X86_FEATURE_IDLE_HLT )) +kvm_cpu_cap_set (X86_FEATURE_IDLE_HLT ); +/* Nested VM can receive #VMEXIT instead of triggering #GP */ kvm_cpu_cap_set (X86_FEATURE_SVME_ADDR_CHK ); }
Signature · 1 line
2.34.1
Sean Christopherson
maintainerRe: [PATCH v4 3/4] KVM: nSVM: implement the nested idle halt intercept
Quoted 6 lines · Manali Shukla
arch/x86/kvm/svm/nested.c +0
-0@@ -178,6 +178,14 @@ void recalc_intercepts(struct vcpu_svm *svm) }else {
arch/x86/kvm/svm/nested.c void recalc_intercepts(struct vcpu_svm *svm) · line 188
Quoted 10 lines
arch/x86/kvm/svm/nested.c +8
-0WARN_ON (!(c->virt_ext &VIRTUAL_VMLOAD_VMSAVE_ENABLE_MASK )); } + +/* +* Clear the HLT intercept for L2 guest when the Idle HLT intercept feature +* is enabled on the platform and the guest can use the Idle HLT intercept +* feature. +*/ +if (guest_can_use (&svm->vcpu,X86_FEATURE_IDLE_HLT )) +vmcb_clr_intercept (c,INTERCEPT_HLT );
This is wrong. KVM needs to honor the intercept of vmcb12. If L1 wants to intercept HLT, then KVM needs to configure vmcb02 to intercept HLT, regradless of whether or not L1 is utilizing INTERCEPT_IDLE_HLT.
Given how KVM currently handles intercepts for nested SVM, I'm pretty sure you can simply do nothing. recalc_intercepts() starts with KVM's intercepts (from vmcb01), and adds in L1's intercepts. So unless there is a special case, the default behavior should Just Work.
for (i = 0; i < MAX_INTERCEPT; i++) c->intercepts[i] = h->intercepts[i]; ... for (i = 0; i < MAX_INTERCEPT; i++) c->intercepts[i] |= g->intercepts[i];
KVM's approach creates all kinds of virtualization holes, e.g. L1 can utilize IDLE_HLT even if the feature isn't advertised to L1. But that's true for quite literally all feature-based intercepts, so for better or worse, I don't think it makes sense to try and change that approach for this feature.
Quoted 13 lines
arch/x86/kvm/svm/svm.c +1
-0@@ -4425,6 +4425,7 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_PFTHRESHOLD );kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_VGIF );kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_VNMI ); +kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_IDLE_HLT );svm_recalc_instruction_intercepts (vcpu, svm);
arch/x86/kvm/svm/svm.c static __init void svm_set_cpu_caps(void) · line 5233
Quoted 5 lines
arch/x86/kvm/svm/svm.c +2
-0@@ -5228,6 +5229,9 @@ static __init void svm_set_cpu_caps(void)if (vnmi)kvm_cpu_cap_set (X86_FEATURE_VNMI ); +if (cpu_feature_enabled (X86_FEATURE_IDLE_HLT )) +kvm_cpu_cap_set (X86_FEATURE_IDLE_HLT );
kvm_cpu_cap_check_and_set() does this for you.
Quoted 7 lines
arch/x86/kvm/svm/svm.c +1
-0+/* Nested VM can receive #VMEXIT instead of triggering #GP */ kvm_cpu_cap_set (X86_FEATURE_SVME_ADDR_CHK ); }Signature · 1 line
2.34.1
Re: [PATCH v4 3/4] KVM: nSVM: implement the nested idle halt intercept
Hi Sean,
Thank you for reviewing my patches.
Quoted 35 lines · Sean Christopherson
Quoted 6 lines · Manali Shukla
arch/x86/kvm/svm/nested.c +0
-0@@ -178,6 +178,14 @@ void recalc_intercepts(struct vcpu_svm *svm) }else {arch/x86/kvm/svm/nested.c void recalc_intercepts(struct vcpu_svm *svm) · line 188
Quoted 10 lines
arch/x86/kvm/svm/nested.c +8
-0WARN_ON (!(c->virt_ext &VIRTUAL_VMLOAD_VMSAVE_ENABLE_MASK )); } + +/* +* Clear the HLT intercept for L2 guest when the Idle HLT intercept feature +* is enabled on the platform and the guest can use the Idle HLT intercept +* feature. +*/ +if (guest_can_use (&svm->vcpu,X86_FEATURE_IDLE_HLT )) +vmcb_clr_intercept (c,INTERCEPT_HLT );This is wrong. KVM needs to honor the intercept of vmcb12. If L1 wants to intercept HLT, then KVM needs to configure vmcb02 to intercept HLT, regradless of whether or not L1 is utilizing
INTERCEPT_IDLE_HLT.Given how KVM currently handles intercepts for nested SVM, I'm pretty sure you can simply do nothing.
recalc_intercepts()starts with KVM's intercepts (from vmcb01), and adds in L1's intercepts. So unless there is a special case, the default behavior should Just Work.for (i = 0; i < MAX_INTERCEPT; i++) c->intercepts[i] = h->intercepts[i]; ... for (i = 0; i < MAX_INTERCEPT; i++) c->intercepts[i] |= g->intercepts[i];
Quoted 5 lines
KVM's approach creates all kinds of virtualization holes, e.g. L1 can utilize
IDLE_HLTeven if the feature isn't advertised to L1. But that's true for quite literally all feature-based intercepts, so for better or worse, I don't think it makes sense to try and change that approach for this feature.
Yeah. Makes sense. I will remove the above condition from V5, so that intercept of vmcb12 is honored.
Quoted 12 lines
Quoted 12 lines
arch/x86/kvm/svm/svm.c +1
-0@@ -4425,6 +4425,7 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_PFTHRESHOLD );kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_VGIF );kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_VNMI ); +kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_IDLE_HLT );svm_recalc_instruction_intercepts (vcpu, svm);
Quoted 16 lines
arch/x86/kvm/svm/svm.c static __init void svm_set_cpu_caps(void) · line 5233
Quoted 6 lines
arch/x86/kvm/svm/svm.c +2
-0@@ -5228,6 +5229,9 @@ static __init void svm_set_cpu_caps(void)if (vnmi)kvm_cpu_cap_set (X86_FEATURE_VNMI ); +if (cpu_feature_enabled (X86_FEATURE_IDLE_HLT )) +kvm_cpu_cap_set (X86_FEATURE_IDLE_HLT );
kvm_cpu_cap_check_and_set()does this for you.Quoted 7 lines
arch/x86/kvm/svm/svm.c +1
-0+/* Nested VM can receive #VMEXIT instead of triggering #GP */ kvm_cpu_cap_set (X86_FEATURE_SVME_ADDR_CHK ); }Signature · 1 line
2.34.1
- Manali
Re: [PATCH v4 3/4] KVM: nSVM: implement the nested idle halt intercept
In reply to Sean ChristophersonQuoted 12 lines · Sean Christopherson
Quoted 12 lines
arch/x86/kvm/svm/svm.c +1
-0@@ -4425,6 +4425,7 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_PFTHRESHOLD );kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_VGIF );kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_VNMI ); +kvm_governed_feature_check_and_set (vcpu,X86_FEATURE_IDLE_HLT );svm_recalc_instruction_intercepts (vcpu, svm);
Quoted 9 lines
arch/x86/kvm/svm/svm.c static __init void svm_set_cpu_caps(void) · line 5233
Quoted 6 lines
arch/x86/kvm/svm/svm.c +2
-0@@ -5228,6 +5229,9 @@ static __init void svm_set_cpu_caps(void)if (vnmi)kvm_cpu_cap_set (X86_FEATURE_VNMI ); +if (cpu_feature_enabled (X86_FEATURE_IDLE_HLT )) +kvm_cpu_cap_set (X86_FEATURE_IDLE_HLT );
kvm_cpu_cap_check_and_set()does this for you.
Sure. I will use kvm_cpu_cap_check_and_set() in V5.
Quoted 7 lines
Quoted 7 lines
arch/x86/kvm/svm/svm.c +1
-0+/* Nested VM can receive #VMEXIT instead of triggering #GP */ kvm_cpu_cap_set (X86_FEATURE_SVME_ADDR_CHK ); }Signature · 1 line
2.34.1
- Manali
PATCH v4 4/4 KVM: selftests: KVM: SVM: Add Idle HLT intercept test
In reply to Manali ShuklaFrom: Manali Shukla <[email protected]>
Execution of the HLT instruction results in VMEXIT. Hypervisor observes pending V_INTR and V_NMI events just after VMEXIT generated by HLT for the vCPU and causes VM entry to service the pending events. The Idle HLT intercept feature avoids the wasteful VMEXIT during halt if there are pending V_INTR and V_NMI events for the vCPU.
The selftest for Idle HLT intercept instruments above-mentioned scenario.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/processor.h | 1 + .../selftests/kvm/x86_64/svm_idle_hlt_test.c | 89 +++++++++++++++++++ 3 files changed, 91 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -94,6 +94,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/smaller_maxphyaddr_emulation_test TEST_GEN_PROGS_x86_64 += x86_64/smm_test TEST_GEN_PROGS_x86_64 += x86_64/state_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_preemption_timer_test +TEST_GEN_PROGS_x86_64 += x86_64/svm_idle_hlt_test TEST_GEN_PROGS_x86_64 += x86_64/svm_vmcall_test TEST_GEN_PROGS_x86_64 += x86_64/svm_int_ctl_test TEST_GEN_PROGS_x86_64 += x86_64/svm_nested_shutdown_test
tools/testing/selftests/kvm/include/x86_64/processor.h +1 -0
@@ -197,6 +197,7 @@ struct kvm_x86_cpu_feature {#define X86_FEATURE_PAUSEFILTER KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 10)#define X86_FEATURE_PFTHRESHOLD KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 12)#define X86_FEATURE_VGIF KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 16) +#define X86_FEATURE_IDLE_HLT KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 30)#define X86_FEATURE_SEV KVM_X86_CPU_FEATURE(0x8000001F, 0, EAX, 1)#define X86_FEATURE_SEV_ES KVM_X86_CPU_FEATURE(0x8000001F, 0, EAX, 3)
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +89 -0
89 lines not on this page · raw message
Signature · 1 line
2.34.1
Sean Christopherson
maintainerRe: [PATCH v4 4/4] KVM: selftests: KVM: SVM: Add Idle HLT intercept test
Quoted 8 lines · Manali Shukla
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +2
-0@@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 12
Quoted 10 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +10
-0+* Copyright (C) 2024 Advanced Micro Devices, Inc. +* +*/ +#include <kvm_util.h> +#include <processor.h> +#include <test_util.h> +#include "svm_util.h" +#include "apic.h" + +#define VINTR_VECTOR 0x30
Drop the "V". From the guest's perspective, it's simply the interrupt vector. The "V" suggests there's nested SVM stuff going on, e.g. to virtualize an interrupt for L2 or something.
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 21
Quoted 9 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +9
-0+#define NUM_ITERATIONS 1000 + +static bool irq_received; + +/* +* The guest code instruments the scenario where there is a V_INTR pending +* event available while hlt instruction is executed. The HLT VM Exit doesn't +* occur in above-mentioned scenario if Idle HLT intercept feature is enabled. +*/
So the only thing thing that is idle-HLT specific in this test is that final TEST_ASSERT_EQ(). Rather than make this test depend on idle-HLT, we should tweak it run on all hardware, and then:
if (kvm_cpu_has(X86_FEATURE_IDLE_HLT)) TEST_ASSERT_EQ(halt_exits, 0); else TEST_ASSERT_EQ(halt_exits, NUM_ITERATIONS);
Not sure about the name. Maybe hlt_ipi_test or ipi_hlt_test?
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 28
Quoted 6 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +6
-0+static void guest_code (void ) +{ +uint32_t icr_val; +int i; + +xapic_enable ();
Hmm, I think we should have this test force x2APIC mode. KVM emulates x2APIC in software (if it's not accerlated by APICv), i.e. it's always available. That will allow using this test to do performance testing of KVM's fastpath handling of handle_fastpath_set_x2apic_icr_irqoff().
Of course, KVM only uses the fastpath for non-shorthand IPIs, and any setup that can do self-IPI fully in the fastpath (via virtual interrupt delivery) won't exit in the first place (virtualized by hardware), i.e. there's probably no point in adding self-IPIs to the fastpath.
But maybe in the future I can convince someone to enhance this test to do cross-vCPU IPI testing.
Quoted 17 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +17
-0+ + icr_val = (APIC_DEST_SELF |APIC_INT_ASSERT |VINTR_VECTOR ); + +for (i =0 ; i <NUM_ITERATIONS ; i++) { +cli (); +xapic_write_reg (APIC_ICR , icr_val); +safe_halt (); +GUEST_ASSERT (READ_ONCE (irq_received)); +WRITE_ONCE (irq_received, false); + } +GUEST_DONE (); +} + +static void guest_vintr_handler (struct ex_regs *regs) +{ +WRITE_ONCE (irq_received, true); +xapic_write_reg (APIC_EOI ,0x00 );
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 55
Quoted 10 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +10
-0+} + +int main (int argc,char *argv[]) +{ +struct kvm_vm *vm; +struct kvm_vcpu *vcpu; +struct ucall uc; +uint64_t halt_exits, vintr_exits; + +/* Check the extension for binary stats */
Pointless comment, the code below is self-explanatory.
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 56
Quoted 1 line
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +1
-0+TEST_REQUIRE (this_cpu_has (X86_FEATURE_IDLE_HLT ));
This needs to check *KVM* support. I.e. kvm_cpu_has().
Quoted 19 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +19
-0+TEST_REQUIRE (kvm_has_cap (KVM_CAP_BINARY_STATS_FD )); + + vm =vm_create_with_one_vcpu (&vcpu, guest_code); + +vm_install_exception_handler (vm,VINTR_VECTOR , guest_vintr_handler); +virt_pg_map (vm,APIC_DEFAULT_GPA ,APIC_DEFAULT_GPA ); + +vcpu_run (vcpu); +TEST_ASSERT_KVM_EXIT_REASON (vcpu,KVM_EXIT_IO ); + + halt_exits =vcpu_get_stat (vcpu, halt_exits); + vintr_exits =vcpu_get_stat (vcpu, irq_window_exits); + +switch (get_ucall (vcpu, &uc)) { +case UCALL_ABORT : +REPORT_GUEST_ASSERT (uc); +/* NOT REACHED */ +case UCALL_DONE : +break ;
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 85
Quoted 10 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +10
-0+ +default : +TEST_FAIL ("Unknown ucall 0x%lx." , uc.cmd); + } + +TEST_ASSERT_EQ (halt_exits,0 ); +pr_debug ("Guest executed VINTR followed by halts: %d times.\n" +"The guest exited due to halt: %ld times and number\n" +"of vintr exits: %ld.\n" , +NUM_ITERATIONS , halt_exits, vintr_exits);
halt_exits obviously is '0' at this point, so I don't see any point in printing it out.
As for vintr_exits, I vote to drop it, for now at least. At some point in the future, I would like to expand this test so that it can be used for a rudimentary IPI+HLT perf test. But for now, I think it makes sense to keep it simple, e.g. so that nothing needs to be unwound if improvements are made in the future.
Quoted 7 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +4
-0+ +kvm_vm_free (vm); +return 0 ; +}Signature · 1 line
2.34.1
Re: [PATCH v4 4/4] KVM: selftests: KVM: SVM: Add Idle HLT intercept test
Hi Sean,
Thank you for reviewing my patches.
Quoted 15 lines · Sean Christopherson
Quoted 14 lines · Manali Shukla
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +8
-0@@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2024 Advanced Micro Devices, Inc. +* +*/ +#include <kvm_util.h> +#include <processor.h> +#include <test_util.h>
Quoted 8 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 12
Quoted 4 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +4
-0+#include "svm_util.h" +#include "apic.h" + +#define VINTR_VECTOR 0x30Drop the "V". From the guest's perspective, it's simply the interrupt vector. The "V" suggests there's nested SVM stuff going on, e.g. to virtualize an interrupt for L2 or something.
Sure I will remove "V".
Quoted 15 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 21
Quoted 10 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +9
-0+#define NUM_ITERATIONS 1000 + +static bool irq_received; + +/* +* The guest code instruments the scenario where there is a V_INTR pending +* event available while hlt instruction is executed. The HLT VM Exit doesn't +* occur in above-mentioned scenario if Idle HLT intercept feature is enabled. +*/ So the only thing thing that is idle-HLT specific in this test is that final
TEST_ASSERT_EQ(). Rather than make this test depend on idle-HLT, we should tweak it run on all hardware, and then:
Quoted 6 lines
if (kvm_cpu_has(X86_FEATURE_IDLE_HLT)) TEST_ASSERT_EQ(halt_exits, 0); else TEST_ASSERT_EQ(halt_exits, NUM_ITERATIONS);Not sure about the name. Maybe
hlt_ipi_testoripi_hlt_test?
Yeah makes sense.
I will keep the name of the test as ipi_hlt_test and make the test common to run on all hardware.
Quoted 13 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 28
Quoted 7 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +6
-0+static void guest_code (void ) +{ +uint32_t icr_val; +int i; + +xapic_enable ();Hmm, I think we should have this test force x2APIC mode. KVM emulates x2APIC in software (if it's not accerlated by APICv), i.e. it's always available. That will allow using this test to do performance testing of KVM's fastpath handling of
handle_fastpath_set_x2apic_icr_irqoff().
Quoted 8 lines
Of course, KVM only uses the fastpath for non-shorthand IPIs, and any setup that can do self-IPI fully in the fastpath (via virtual interrupt delivery) won't exit in the first place (virtualized by hardware), i.e. there's probably no point in adding self-IPIs to the fastpath.
But maybe in the future I can convince someone to enhance this test to do cross-vCPU IPI testing.
Sure. I will make this test work with x2APIC mode.
Quoted 21 lines
Quoted 21 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +21
-0+ + icr_val = (APIC_DEST_SELF |APIC_INT_ASSERT |VINTR_VECTOR ); + +for (i =0 ; i <NUM_ITERATIONS ; i++) { +cli (); +xapic_write_reg (APIC_ICR , icr_val); +safe_halt (); +GUEST_ASSERT (READ_ONCE (irq_received)); +WRITE_ONCE (irq_received, false); + } +GUEST_DONE (); +} + +static void guest_vintr_handler (struct ex_regs *regs) +{ +WRITE_ONCE (irq_received, true); +xapic_write_reg (APIC_EOI ,0x00 ); +} + +int main (int argc,char *argv[]) +{
Quoted 9 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 55
Quoted 6 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +6
-0+struct kvm_vm *vm; +struct kvm_vcpu *vcpu; +struct ucall uc; +uint64_t halt_exits, vintr_exits; + +/* Check the extension for binary stats */ Pointless comment, the code below is self-explanatory.
Sure will remove the comment.
Quoted 1 line
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 56
Quoted 1 line
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +1
-0+TEST_REQUIRE (this_cpu_has (X86_FEATURE_IDLE_HLT ));
I think this is not needed, since I am planning to make this test independent of idle-HLT as per above comments.
Quoted 36 lines
This needs to check *KVM* support. I.e.
kvm_cpu_has().Quoted 19 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +19
-0+TEST_REQUIRE (kvm_has_cap (KVM_CAP_BINARY_STATS_FD )); + + vm =vm_create_with_one_vcpu (&vcpu, guest_code); + +vm_install_exception_handler (vm,VINTR_VECTOR , guest_vintr_handler); +virt_pg_map (vm,APIC_DEFAULT_GPA ,APIC_DEFAULT_GPA ); + +vcpu_run (vcpu); +TEST_ASSERT_KVM_EXIT_REASON (vcpu,KVM_EXIT_IO ); + + halt_exits =vcpu_get_stat (vcpu, halt_exits); + vintr_exits =vcpu_get_stat (vcpu, irq_window_exits); + +switch (get_ucall (vcpu, &uc)) { +case UCALL_ABORT : +REPORT_GUEST_ASSERT (uc); +/* NOT REACHED */ +case UCALL_DONE : +break ;tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c line 85
Quoted 10 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +10
-0+ +default : +TEST_FAIL ("Unknown ucall 0x%lx." , uc.cmd); + } + +TEST_ASSERT_EQ (halt_exits,0 ); +pr_debug ("Guest executed VINTR followed by halts: %d times.\n" +"The guest exited due to halt: %ld times and number\n" +"of vintr exits: %ld.\n" , +NUM_ITERATIONS , halt_exits, vintr_exits);
halt_exitsobviously is '0' at this point, so I don't see any point in printing it out.
Quoted 5 lines
As for
vintr_exits, I vote to drop it, for now at least. At some point in the future, I would like to expand this test so that it can be used for a rudimentary IPI+HLT perf test. But for now, I think it makes sense to keep it simple, e.g. so that nothing needs to be unwound if improvements are made in the future.
Sure I will remove this pr_debug for now.
Quoted 7 lines
Quoted 7 lines
tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c new file +4
-0+ +kvm_vm_free (vm); +return 0 ; +}Signature · 1 line
2.34.1
- Manali
Quoted 101 lines · Manali Shukla
The upcoming new Idle HLT Intercept feature allows for the HLT instruction execution by a vCPU to be intercepted by the hypervisor only if there are no pending
V_INTRandV_NMIevents for the vCPU. When the vCPU is expected to service the pendingV_INTRandV_NMIevents, the Idle HLT intercept won’t trigger. The feature allows the hypervisor to determine if the vCPU is actually idle and reduces wasteful VMEXITs.The idle HLT intercept feature is used for enlightened guests who wish to securely handle the events. When an enlightened guest does a HLT while an interrupt is pending, hypervisor will not have a way to figure out whether the guest needs to be re-entered or not. The Idle HLT intercept feature allows the HLT execution only if there are no pending
V_INTRandV_NMIevents.Presence of the Idle HLT Intercept feature is indicated via CPUID function Fn8000_000A_EDX[30].
Document for the Idle HLT intercept feature is available at [1].
This series is based on kvm-next/next (
64dbb3a771a1) + [2].Experiments done:
----------------kvm_amd.avic is set to '0' for this experiment.
The below numbers represent the average of 10 runs.
Normal guest (L1)
The below netperf command was run on the guest with smp = 1 (pinned).netperf -H <host ip> -t
TCP_RR-l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 25645.7136 | 25773.2796 |
----------------------------------------------------------------Number of transactions/sec with and without idle HLT intercept feature are almost same.
Nested guest (L2)
The below netperf command was run on L2 guest with smp = 1 (pinned).netperf -H <host ip> -t
TCP_RR-l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 5655.4468 | 5755.2189 |
----------------------------------------------------------------Number of transactions/sec with and without idle HLT intercept feature are almost same.
Testing Done:
- Tested the functionality for the Idle HLT intercept feature using selftestsvm_idle_hlt_test.
- Tested SEV and SEV-ES guest for the Idle HLT intercept functionality.
- Tested the Idle HLT intercept functionality on nested guest.v3 -> v4
- Drop the patches to addvcpu_get_stat()into a new series [2].
- Added nested Idle HLT intercept support.v2 -> v3
- Incorporated Andrew's suggestion to structurevcpu_stat_typesin a way that each architecture can share the generic types and also provide its own.v1 -> v2
- Done changes insvm_idle_hlt_testbased on the review comments from Sean.
- Added an enum based approach to get binary stats invcpu_get_stat()which doesn't use string to get stat data based on the comments from Sean.
- Addedself_halt()andcli()helpers based on the comments from Sean.[1]: AMD64 Architecture Programmer's Manual Pub. 24593, April 2024,
Vol 2, 15.9 Instruction Intercepts (Table 15-7: IDLE_HLT). https://bugzilla.kernel.org/attachment.cgi?id=306250[2]: https://lore.kernel.org/kvm/[email protected]/T/#t
Manali Shukla (4):
x86/cpufeatures: Add CPUID feature bit for Idle HLT intercept
KVM: SVM: Add Idle HLT intercept support
KVM: nSVM: implement the nested idle halt intercept
KVM: selftests: KVM: SVM: Add Idle HLT intercept testarch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/svm.h | 1 + arch/x86/include/uapi/asm/svm.h | 2 + arch/x86/kvm/governed_features.h | 1 + arch/x86/kvm/svm/nested.c | 7 ++ arch/x86/kvm/svm/svm.c | 15 +++- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/processor.h | 1 + .../selftests/kvm/x86_64/svm_idle_hlt_test.c | 89 +++++++++++++++++++ 9 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.c
Quoted 5 lines
base-commit:
c8d430db8eec7d4fd13a6bea27b7086a54eda6da
prerequisite-patch-id:ca912571db5c004f77b70843b8dd35517ff1267f
prerequisite-patch-id:164ea3b4346f9e04bc69819278d20f5e1b5df5ed
prerequisite-patch-id:90d870f426ebc2cec43c0dd89b701ee998385455
prerequisite-patch-id:45812b799c517a4521782a1fdbcda881237e1eda
A gentle reminder.
-Manali
Quoted 108 lines · Manali Shukla
Quoted 101 lines · Manali Shukla
The upcoming new Idle HLT Intercept feature allows for the HLT instruction execution by a vCPU to be intercepted by the hypervisor only if there are no pending
V_INTRandV_NMIevents for the vCPU. When the vCPU is expected to service the pendingV_INTRandV_NMIevents, the Idle HLT intercept won’t trigger. The feature allows the hypervisor to determine if the vCPU is actually idle and reduces wasteful VMEXITs.The idle HLT intercept feature is used for enlightened guests who wish to securely handle the events. When an enlightened guest does a HLT while an interrupt is pending, hypervisor will not have a way to figure out whether the guest needs to be re-entered or not. The Idle HLT intercept feature allows the HLT execution only if there are no pending
V_INTRandV_NMIevents.Presence of the Idle HLT Intercept feature is indicated via CPUID function Fn8000_000A_EDX[30].
Document for the Idle HLT intercept feature is available at [1].
This series is based on kvm-next/next (
64dbb3a771a1) + [2].Experiments done:
----------------kvm_amd.avic is set to '0' for this experiment.
The below numbers represent the average of 10 runs.
Normal guest (L1)
The below netperf command was run on the guest with smp = 1 (pinned).netperf -H <host ip> -t
TCP_RR-l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 25645.7136 | 25773.2796 |
----------------------------------------------------------------Number of transactions/sec with and without idle HLT intercept feature are almost same.
Nested guest (L2)
The below netperf command was run on L2 guest with smp = 1 (pinned).netperf -H <host ip> -t
TCP_RR-l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 5655.4468 | 5755.2189 |
----------------------------------------------------------------Number of transactions/sec with and without idle HLT intercept feature are almost same.
Testing Done:
- Tested the functionality for the Idle HLT intercept feature using selftestsvm_idle_hlt_test.
- Tested SEV and SEV-ES guest for the Idle HLT intercept functionality.
- Tested the Idle HLT intercept functionality on nested guest.v3 -> v4
- Drop the patches to addvcpu_get_stat()into a new series [2].
- Added nested Idle HLT intercept support.v2 -> v3
- Incorporated Andrew's suggestion to structurevcpu_stat_typesin a way that each architecture can share the generic types and also provide its own.v1 -> v2
- Done changes insvm_idle_hlt_testbased on the review comments from Sean.
- Added an enum based approach to get binary stats invcpu_get_stat()which doesn't use string to get stat data based on the comments from Sean.
- Addedself_halt()andcli()helpers based on the comments from Sean.[1]: AMD64 Architecture Programmer's Manual Pub. 24593, April 2024,
Vol 2, 15.9 Instruction Intercepts (Table 15-7: IDLE_HLT). https://bugzilla.kernel.org/attachment.cgi?id=306250[2]: https://lore.kernel.org/kvm/[email protected]/T/#t
Manali Shukla (4):
x86/cpufeatures: Add CPUID feature bit for Idle HLT intercept
KVM: SVM: Add Idle HLT intercept support
KVM: nSVM: implement the nested idle halt intercept
KVM: selftests: KVM: SVM: Add Idle HLT intercept testarch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/svm.h | 1 + arch/x86/include/uapi/asm/svm.h | 2 + arch/x86/kvm/governed_features.h | 1 + arch/x86/kvm/svm/nested.c | 7 ++ arch/x86/kvm/svm/svm.c | 15 +++- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/processor.h | 1 + .../selftests/kvm/x86_64/svm_idle_hlt_test.c | 89 +++++++++++++++++++ 9 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.cQuoted 5 lines
base-commit:
c8d430db8eec7d4fd13a6bea27b7086a54eda6da
prerequisite-patch-id:ca912571db5c004f77b70843b8dd35517ff1267f
prerequisite-patch-id:164ea3b4346f9e04bc69819278d20f5e1b5df5ed
prerequisite-patch-id:90d870f426ebc2cec43c0dd89b701ee998385455
prerequisite-patch-id:45812b799c517a4521782a1fdbcda881237e1eda
Quoted 3 lines
A gentle reminder.
-Manali
A Gentle reminder.
-Manali
Quoted 109 lines · Manali Shukla
Quoted 108 lines · Manali Shukla
Quoted 106 lines · Manali Shukla
The upcoming new Idle HLT Intercept feature allows for the HLT instruction execution by a vCPU to be intercepted by the hypervisor only if there are no pending
V_INTRandV_NMIevents for the vCPU. When the vCPU is expected to service the pendingV_INTRandV_NMIevents, the Idle HLT intercept won’t trigger. The feature allows the hypervisor to determine if the vCPU is actually idle and reduces wasteful VMEXITs.The idle HLT intercept feature is used for enlightened guests who wish to securely handle the events. When an enlightened guest does a HLT while an interrupt is pending, hypervisor will not have a way to figure out whether the guest needs to be re-entered or not. The Idle HLT intercept feature allows the HLT execution only if there are no pending
V_INTRandV_NMIevents.Presence of the Idle HLT Intercept feature is indicated via CPUID function Fn8000_000A_EDX[30].
Document for the Idle HLT intercept feature is available at [1].
This series is based on kvm-next/next (
64dbb3a771a1) + [2].Experiments done:
----------------kvm_amd.avic is set to '0' for this experiment.
The below numbers represent the average of 10 runs.
Normal guest (L1)
The below netperf command was run on the guest with smp = 1 (pinned).netperf -H <host ip> -t
TCP_RR-l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 25645.7136 | 25773.2796 |
----------------------------------------------------------------Number of transactions/sec with and without idle HLT intercept feature are almost same.
Nested guest (L2)
The below netperf command was run on L2 guest with smp = 1 (pinned).netperf -H <host ip> -t
TCP_RR-l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 5655.4468 | 5755.2189 |
----------------------------------------------------------------Number of transactions/sec with and without idle HLT intercept feature are almost same.
Testing Done:
- Tested the functionality for the Idle HLT intercept feature using selftestsvm_idle_hlt_test.
- Tested SEV and SEV-ES guest for the Idle HLT intercept functionality.
- Tested the Idle HLT intercept functionality on nested guest.v3 -> v4
- Drop the patches to addvcpu_get_stat()into a new series [2].
- Added nested Idle HLT intercept support.v2 -> v3
- Incorporated Andrew's suggestion to structurevcpu_stat_typesin a way that each architecture can share the generic types and also provide its own.v1 -> v2
- Done changes insvm_idle_hlt_testbased on the review comments from Sean.
- Added an enum based approach to get binary stats invcpu_get_stat()which doesn't use string to get stat data based on the comments from Sean.
- Addedself_halt()andcli()helpers based on the comments from Sean.[1]: AMD64 Architecture Programmer's Manual Pub. 24593, April 2024,
Vol 2, 15.9 Instruction Intercepts (Table 15-7: IDLE_HLT). https://bugzilla.kernel.org/attachment.cgi?id=306250[2]: https://lore.kernel.org/kvm/[email protected]/T/#t
Manali Shukla (4):
x86/cpufeatures: Add CPUID feature bit for Idle HLT intercept
KVM: SVM: Add Idle HLT intercept support
KVM: nSVM: implement the nested idle halt intercept
KVM: selftests: KVM: SVM: Add Idle HLT intercept testarch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/svm.h | 1 + arch/x86/include/uapi/asm/svm.h | 2 + arch/x86/kvm/governed_features.h | 1 + arch/x86/kvm/svm/nested.c | 7 ++ arch/x86/kvm/svm/svm.c | 15 +++- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/processor.h | 1 + .../selftests/kvm/x86_64/svm_idle_hlt_test.c | 89 +++++++++++++++++++ 9 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.cbase-commit:
c8d430db8eec7d4fd13a6bea27b7086a54eda6da
prerequisite-patch-id:ca912571db5c004f77b70843b8dd35517ff1267f
prerequisite-patch-id:164ea3b4346f9e04bc69819278d20f5e1b5df5ed
prerequisite-patch-id:90d870f426ebc2cec43c0dd89b701ee998385455
prerequisite-patch-id:45812b799c517a4521782a1fdbcda881237e1eda
Quoted 8 lines
Quoted 3 lines
A gentle reminder.
-Manali
A Gentle reminder.
-Manali
A Gentle reminder.
-Manali
Quoted 114 lines · Manali Shukla
Quoted 113 lines · Manali Shukla
Quoted 111 lines · Manali Shukla
Quoted 101 lines · Manali Shukla
The upcoming new Idle HLT Intercept feature allows for the HLT instruction execution by a vCPU to be intercepted by the hypervisor only if there are no pending
V_INTRandV_NMIevents for the vCPU. When the vCPU is expected to service the pendingV_INTRandV_NMIevents, the Idle HLT intercept won’t trigger. The feature allows the hypervisor to determine if the vCPU is actually idle and reduces wasteful VMEXITs.The idle HLT intercept feature is used for enlightened guests who wish to securely handle the events. When an enlightened guest does a HLT while an interrupt is pending, hypervisor will not have a way to figure out whether the guest needs to be re-entered or not. The Idle HLT intercept feature allows the HLT execution only if there are no pending
V_INTRandV_NMIevents.Presence of the Idle HLT Intercept feature is indicated via CPUID function Fn8000_000A_EDX[30].
Document for the Idle HLT intercept feature is available at [1].
This series is based on kvm-next/next (
64dbb3a771a1) + [2].Experiments done:
----------------kvm_amd.avic is set to '0' for this experiment.
The below numbers represent the average of 10 runs.
Normal guest (L1)
The below netperf command was run on the guest with smp = 1 (pinned).netperf -H <host ip> -t
TCP_RR-l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 25645.7136 | 25773.2796 |
----------------------------------------------------------------Number of transactions/sec with and without idle HLT intercept feature are almost same.
Nested guest (L2)
The below netperf command was run on L2 guest with smp = 1 (pinned).netperf -H <host ip> -t
TCP_RR-l 60
----------------------------------------------------------------
|with Idle HLT(transactions/Sec)|w/o Idle HLT(transactions/Sec)|
----------------------------------------------------------------
| 5655.4468 | 5755.2189 |
----------------------------------------------------------------Number of transactions/sec with and without idle HLT intercept feature are almost same.
Testing Done:
- Tested the functionality for the Idle HLT intercept feature using selftestsvm_idle_hlt_test.
- Tested SEV and SEV-ES guest for the Idle HLT intercept functionality.
- Tested the Idle HLT intercept functionality on nested guest.v3 -> v4
- Drop the patches to addvcpu_get_stat()into a new series [2].
- Added nested Idle HLT intercept support.v2 -> v3
- Incorporated Andrew's suggestion to structurevcpu_stat_typesin a way that each architecture can share the generic types and also provide its own.v1 -> v2
- Done changes insvm_idle_hlt_testbased on the review comments from Sean.
- Added an enum based approach to get binary stats invcpu_get_stat()which doesn't use string to get stat data based on the comments from Sean.
- Addedself_halt()andcli()helpers based on the comments from Sean.[1]: AMD64 Architecture Programmer's Manual Pub. 24593, April 2024,
Vol 2, 15.9 Instruction Intercepts (Table 15-7: IDLE_HLT). https://bugzilla.kernel.org/attachment.cgi?id=306250[2]: https://lore.kernel.org/kvm/[email protected]/T/#t
Manali Shukla (4):
x86/cpufeatures: Add CPUID feature bit for Idle HLT intercept
KVM: SVM: Add Idle HLT intercept support
KVM: nSVM: implement the nested idle halt intercept
KVM: selftests: KVM: SVM: Add Idle HLT intercept testarch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/svm.h | 1 + arch/x86/include/uapi/asm/svm.h | 2 + arch/x86/kvm/governed_features.h | 1 + arch/x86/kvm/svm/nested.c | 7 ++ arch/x86/kvm/svm/svm.c | 15 +++- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/processor.h | 1 + .../selftests/kvm/x86_64/svm_idle_hlt_test.c | 89 +++++++++++++++++++ 9 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/svm_idle_hlt_test.cQuoted 5 lines
base-commit:
c8d430db8eec7d4fd13a6bea27b7086a54eda6da
prerequisite-patch-id:ca912571db5c004f77b70843b8dd35517ff1267f
prerequisite-patch-id:164ea3b4346f9e04bc69819278d20f5e1b5df5ed
prerequisite-patch-id:90d870f426ebc2cec43c0dd89b701ee998385455
prerequisite-patch-id:45812b799c517a4521782a1fdbcda881237e1edaA gentle reminder.
-Manali
Quoted 8 lines
Quoted 4 lines
A Gentle reminder.
-Manali
A Gentle reminder.
-Manali
Sorry. I just realized that you have already reviewed the patches. Please ignore the reminder.
- Manali