KVM: Fix tick-based accounting for x86 guests
Subsystems (4)
Sean Christopherson
maintainerv3 cover KVM: Fix tick-based accounting for x86 guests
This is a continuation of Wanpeng's series[1] to fix tick-based CPU time accounting on x86, with my cleanups[2] bolted on top. The core premise of Wanpeng's patches are preserved, but they are heavily stripped down. Specifically, only the "guest exit" paths are split, and no code is consolidated. The intent is to do as little as possible in the three patches that need to be backported. Keeping those changes as small as possible also meant that my cleanups did not need to unwind much refactoring.
On x86, tested CONFIG_VIRT_CPU_ACCOUNTING_GEN =y and =n, and with CONFIG_DEBUG_ENTRY=y && CONFIG_VALIDATE_STACKS=y. Compile tested arm64, MIPS, PPC, and s390, the latter with CONFIG_DEBUG_ENTRY=y for giggles.
One last note: I elected to use vtime_account_guest_exit() in the x86 code instead of open coding these equivalents:
if (vtime_accounting_enabled_this_cpu()) vtime_guest_exit(current); ... if (!vtime_accounting_enabled_this_cpu()) current->flags &= ~PF_VCPU;
With CONFIG_VIRT_CPU_ACCOUNTING_GEN=n, this is a complete non-issue, but for the =y case it means context_tracking_enabled_this_cpu() is being checked back-to-back. The redundant checks bug me, but open coding the gory details in x86 or providing funky variants in vtime.h felt worse.
Delta from Wanpeng's v2:
- s/context_guest/context_tracking_guest, purely to match the existing functions. I have no strong opinion either way.
- Split only the "exit" functions.
- Partially open code vcpu_account_guest_exit() and __vtime_account_guest_exit() in x86 to avoid churn when segueing into my cleanups (see above).
[1] https://lkml.kernel.org/r/[email protected]
[2] https://lkml.kernel.org/r/[email protected]
Sean Christopherson (6):
sched/vtime: Move vtime accounting external declarations above inlines sched/vtime: Move guest enter/exit vtime accounting to vtime.h
context_tracking: Consolidate guest enter/exit wrappers
context_tracking: KVM: Move guest enter/exit wrappers to KVM's domain
KVM: x86: Consolidate guest enter/exit logic to common helpers
KVM: Move instrumentation-safe annotations for enter/exit to x86 code
Wanpeng Li (3):
context_tracking: Move guest exit context tracking to separate helpers
context_tracking: Move guest exit vtime accounting to separate helpers
KVM: x86: Defer tick-based accounting 'til after IRQ handling
arch/x86/kvm/svm/svm.c | 39 +-------- arch/x86/kvm/vmx/vmx.c | 39 +-------- arch/x86/kvm/x86.c | 8 ++ arch/x86/kvm/x86.h | 52 ++++++++++++ include/linux/context_tracking.h | 92 ++++----------------- include/linux/kvm_host.h | 38 +++++++++ include/linux/vtime.h | 138 +++++++++++++++++++------------ 7 files changed, 204 insertions(+), 202 deletions(-)
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Sean Christopherson
maintainerPATCH v3 1/9 context_tracking: Move guest exit context tracking to separate helpers
From: Wanpeng Li <[email protected]>
Provide separate context tracking helpers for guest exit, the standalone helpers will be called separately by KVM x86 in later patches to fix tick-based accounting.
--- include/linux/context_tracking.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
include/linux/context_tracking.h +8 -1
@@ -131,10 +131,15 @@ static __always_inline void guest_enter_irqoff(void) } } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void context_tracking_guest_exit_irqoff(void) {if (context_tracking_enabled ())__context_tracking_exit (CONTEXT_GUEST ); +} + +static __always_inline void guest_exit_irqoff(void) +{ +context_tracking_guest_exit_irqoff ();instrumentation_begin ();if (vtime_accounting_enabled_this_cpu ())
@@ -159,6 +164,8 @@ static __always_inline void guest_enter_irqoff(void)instrumentation_end (); } +static __always_inline void context_tracking_guest_exit_irqoff(void) { } + static __always_inline void guest_exit_irqoff(void) {instrumentation_begin ();
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Re: [PATCH v3 1/9] context_tracking: Move guest exit context tracking to separate helpers
Quoted 6 lines · Sean Christopherson
From: Wanpeng Li <[email protected]>
Provide separate context tracking helpers for guest exit, the standalone helpers will be called separately by KVM x86 in later patches to fix tick-based accounting.
Quoted 8 lines
Suggested-by Thomas Gleixner <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Sean Christopherson <[email protected]>Cc Michael Tokarev <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by Wanpeng Li <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
Quoted 35 lines
--- include/linux/context_tracking.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)include/linux/context_tracking.h +8
-1@@ -131,10 +131,15 @@ static __always_inline void guest_enter_irqoff(void) } } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void context_tracking_guest_exit_irqoff(void) {if (context_tracking_enabled ())__context_tracking_exit (CONTEXT_GUEST ); +} + +static __always_inline void guest_exit_irqoff(void) +{ +context_tracking_guest_exit_irqoff ();instrumentation_begin ();if (vtime_accounting_enabled_this_cpu ())@@ -159,6 +164,8 @@ static __always_inline void guest_enter_irqoff(void)instrumentation_end (); } +static __always_inline void context_tracking_guest_exit_irqoff(void) { } + static __always_inline void guest_exit_irqoff(void) {instrumentation_begin ();
Frederic Weisbecker
maintainerRe: [PATCH v3 1/9] context_tracking: Move guest exit context tracking to separate helpers
In reply to Sean ChristophersonQuoted 23 lines · Sean Christopherson
From: Wanpeng Li <[email protected]>
Provide separate context tracking helpers for guest exit, the standalone helpers will be called separately by KVM x86 in later patches to fix tick-based accounting.
Suggested-by Thomas Gleixner <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Sean Christopherson <[email protected]>Cc Michael Tokarev <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by Wanpeng Li <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- include/linux/context_tracking.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)include/linux/context_tracking.h +0
-0
include/linux/context_tracking.h static __always_inline void guest_enter_irqoff(void) · line 134
Quoted 5 lines
include/linux/context_tracking.h +1
-1@@ -131,10 +131,15 @@ static __always_inline void guest_enter_irqoff(void) } } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void context_tracking_guest_exit_irqoff(void)
You can make this context_tracking_guest_exit() since there is no irq-on version.
Thanks.
Sean Christopherson
maintainerPATCH v3 2/9 context_tracking: Move guest exit vtime accounting to separate helpers
In reply to Sean ChristophersonFrom: Wanpeng Li <[email protected]>
Provide separate vtime accounting functions for guest exit instead of open coding the logic within the context tracking code. This will allow KVM x86 to handle vtime accounting slightly differently when using tick-based accounting.
--- include/linux/context_tracking.h | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-)
include/linux/context_tracking.h +17 -7
@@ -137,15 +137,20 @@ static __always_inline void context_tracking_guest_exit_irqoff(void)__context_tracking_exit (CONTEXT_GUEST ); } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void vtime_account_guest_exit(void) { -context_tracking_guest_exit_irqoff (); - -instrumentation_begin ();if (vtime_accounting_enabled_this_cpu ())vtime_guest_exit (current);else current->flags &= ~PF_VCPU ; +} + +static __always_inline void guest_exit_irqoff(void) +{ +context_tracking_guest_exit_irqoff (); + +instrumentation_begin (); +vtime_account_guest_exit ();instrumentation_end (); }
@@ -166,12 +171,17 @@ static __always_inline void guest_enter_irqoff(void) static __always_inline void context_tracking_guest_exit_irqoff(void) { } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void vtime_account_guest_exit(void) { -instrumentation_begin (); -/* Flush the guest cputime we spent on the guest */ vtime_account_kernel (current); current->flags &= ~PF_VCPU ; +} + +static __always_inline void guest_exit_irqoff(void) +{ +instrumentation_begin (); +/* Flush the guest cputime we spent on the guest */ +vtime_account_guest_exit ();instrumentation_end (); }#endif /* CONFIG_VIRT_CPU_ACCOUNTING_GEN */
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Re: [PATCH v3 2/9] context_tracking: Move guest exit vtime accounting to separate helpers
Quoted 7 lines · Sean Christopherson
From: Wanpeng Li <[email protected]>
Provide separate vtime accounting functions for guest exit instead of open coding the logic within the context tracking code. This will allow KVM x86 to handle vtime accounting slightly differently when using tick-based accounting.
Quoted 7 lines
Suggested-by Thomas Gleixner <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Michael Tokarev <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by Wanpeng Li <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
Quoted 55 lines
--- include/linux/context_tracking.h | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-)include/linux/context_tracking.h +17
-7@@ -137,15 +137,20 @@ static __always_inline void context_tracking_guest_exit_irqoff(void)__context_tracking_exit (CONTEXT_GUEST ); } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void vtime_account_guest_exit(void) { -context_tracking_guest_exit_irqoff (); - -instrumentation_begin ();if (vtime_accounting_enabled_this_cpu ())vtime_guest_exit (current);else current->flags &= ~PF_VCPU ; +} + +static __always_inline void guest_exit_irqoff(void) +{ +context_tracking_guest_exit_irqoff (); + +instrumentation_begin (); +vtime_account_guest_exit ();instrumentation_end (); }@@ -166,12 +171,17 @@ static __always_inline void guest_enter_irqoff(void) static __always_inline void context_tracking_guest_exit_irqoff(void) { } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void vtime_account_guest_exit(void) { -instrumentation_begin (); -/* Flush the guest cputime we spent on the guest */ vtime_account_kernel (current); current->flags &= ~PF_VCPU ; +} + +static __always_inline void guest_exit_irqoff(void) +{ +instrumentation_begin (); +/* Flush the guest cputime we spent on the guest */ +vtime_account_guest_exit ();instrumentation_end (); }#endif /* CONFIG_VIRT_CPU_ACCOUNTING_GEN */
Sean Christopherson
maintainerPATCH v3 3/9 KVM: x86: Defer tick-based accounting 'til after IRQ handling
In reply to Sean ChristophersonFrom: Wanpeng Li <[email protected]>
When using tick-based accounting, defer the call to account guest time until after servicing any IRQ(s) that happened in the guest or immediately after VM-Exit. Tick-based accounting of vCPU time relies on PF_VCPU being set when the tick IRQ handler runs, and IRQs are blocked throughout {svm,vmx}_vcpu_enter_exit().
This fixes a bug[*] where reported guest time remains '0', even when running an infinite loop in the guest.
[*] https://bugzilla.kernel.org/show_bug.cgi?id=209831
--- arch/x86/kvm/svm/svm.c | 13 ++++++++++--- arch/x86/kvm/vmx/vmx.c | 13 ++++++++++--- arch/x86/kvm/x86.c | 8 ++++++++ 3 files changed, 28 insertions(+), 6 deletions(-)
arch/x86/kvm/svm/svm.c +10 -3
@@ -3750,17 +3750,24 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu)* have them in state 'on' as recorded before entering guest mode. * Same as enter_from_user_mode(). * -* guest_exit_irqoff() restores host context and reinstates RCU if -* enabled and required. +* context_tracking_guest_exit_irqoff() restores host context and +* reinstates RCU if enabled and required. * * This needs to be done before the below as native_read_msr() * contains a tracepoint and x86_spec_ctrl_restore_host() calls * into world and some more. */ lockdep_hardirqs_off (CALLER_ADDR0 ); -guest_exit_irqoff (); +context_tracking_guest_exit_irqoff ();instrumentation_begin (); +/* +* Account guest time when precise accounting based on context tracking +* is enabled. Tick based accounting is deferred until after IRQs that +* occurred within the VM-Enter/VM-Exit "window" are handled. +*/ +if (vtime_accounting_enabled_this_cpu ()) +vtime_account_guest_exit ();trace_hardirqs_off_finish ();instrumentation_end (); }
arch/x86/kvm/vmx/vmx.c +10 -3
@@ -6639,17 +6639,24 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,* have them in state 'on' as recorded before entering guest mode. * Same as enter_from_user_mode(). * -* guest_exit_irqoff() restores host context and reinstates RCU if -* enabled and required. +* context_tracking_guest_exit_irqoff() restores host context and +* reinstates RCU if enabled and required. * * This needs to be done before the below as native_read_msr() * contains a tracepoint and x86_spec_ctrl_restore_host() calls * into world and some more. */ lockdep_hardirqs_off (CALLER_ADDR0 ); -guest_exit_irqoff (); +context_tracking_guest_exit_irqoff ();instrumentation_begin (); +/* +* Account guest time when precise accounting based on context tracking +* is enabled. Tick based accounting is deferred until after IRQs that +* occurred within the VM-Enter/VM-Exit "window" are handled. +*/ +if (vtime_accounting_enabled_this_cpu ()) +vtime_account_guest_exit ();trace_hardirqs_off_finish ();instrumentation_end (); }
arch/x86/kvm/x86.c +8 -0
@@ -9230,6 +9230,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)local_irq_disable ();kvm_after_interrupt (vcpu); +/* +* When using tick-based accounting, wait until after servicing IRQs to +* account guest time so that any ticks that occurred while running the +* guest are properly accounted to the guest. +*/ +if (!vtime_accounting_enabled_this_cpu ()) +vtime_account_guest_exit (); +if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;if (delta !=S64_MIN ) {
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Frederic Weisbecker
maintainerRe: [PATCH v3 3/9] KVM: x86: Defer tick-based accounting 'til after IRQ handling
Quoted 27 lines · Sean Christopherson
From: Wanpeng Li <[email protected]>
When using tick-based accounting, defer the call to account guest time until after servicing any IRQ(s) that happened in the guest or immediately after VM-Exit. Tick-based accounting of vCPU time relies on
PF_VCPUbeing set when the tick IRQ handler runs, and IRQs are blocked throughout {svm,vmx}_vcpu_enter_exit().This fixes a bug[*] where reported guest time remains '0', even when running an infinite loop in the guest.
[*] https://bugzilla.kernel.org/show_bug.cgi?id=209831
Fixes 87fa7f3e98a131 ("x86/kvm: Move context tracking where it belongs")Cc Thomas Gleixner <[email protected]>Cc Sean Christopherson <[email protected]>Cc Michael Tokarev <[email protected]>Cc [email protected]#v5.9-rc1+Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Wanpeng Li <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9239
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0@@ -9230,6 +9230,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)local_irq_disable ();kvm_after_interrupt (vcpu); +/* +* When using tick-based accounting, wait until after servicing IRQs to +* account guest time so that any ticks that occurred while running the +* guest are properly accounted to the guest. +*/ +if (!vtime_accounting_enabled_this_cpu ()) +vtime_account_guest_exit ();
Can we rather have instead:
static inline void tick_account_guest_exit(void)
{
if (!vtime_accounting_enabled_this_cpu())
current->flags &= ~PF_VCPU;
}It duplicates a bit of code but I think this will read less confusing.
Thanks.
Quoted 7 lines
arch/x86/kvm/x86.c +1
-0+if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;if (delta !=S64_MIN ) {Signature · 1 line
2.31.1.368.gbe11c130af-goog
Sean Christopherson
maintainerRe: [PATCH v3 3/9] KVM: x86: Defer tick-based accounting 'til after IRQ handling
Quoted 19 lines · Frederic Weisbecker
Quoted 5 lines · Sean Christopherson
arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9239
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0@@ -9230,6 +9230,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)local_irq_disable ();kvm_after_interrupt (vcpu); +/* +* When using tick-based accounting, wait until after servicing IRQs to +* account guest time so that any ticks that occurred while running the +* guest are properly accounted to the guest. +*/ +if (!vtime_accounting_enabled_this_cpu ()) +vtime_account_guest_exit ();Can we rather have instead:
Quoted 7 lines
static inline void
tick_account_guest_exit(void){ if (!vtime_accounting_enabled_this_cpu()) current->flags &= ~PF_VCPU; }It duplicates a bit of code but I think this will read less confusing.
Either way works for me. I used vtime_account_guest_exit() to try to keep as many details as possible inside vtime, e.g. in case the implemenation is tweaked in the future. But I agree that pretending KVM isn't already deeply intertwined with the details is a lie.
Frederic Weisbecker
maintainerRe: [PATCH v3 3/9] KVM: x86: Defer tick-based accounting 'til after IRQ handling
Quoted 26 lines · Sean Christopherson
Quoted 25 lines · Frederic Weisbecker
Quoted 5 lines · Sean Christopherson
arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9239
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0@@ -9230,6 +9230,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)local_irq_disable ();kvm_after_interrupt (vcpu); +/* +* When using tick-based accounting, wait until after servicing IRQs to +* account guest time so that any ticks that occurred while running the +* guest are properly accounted to the guest. +*/ +if (!vtime_accounting_enabled_this_cpu ()) +vtime_account_guest_exit ();Can we rather have instead:
static inline void
tick_account_guest_exit(void){ if (!vtime_accounting_enabled_this_cpu()) current->flags &= ~PF_VCPU; }
Quoted 6 lines
Quoted 1 line
It duplicates a bit of code but I think this will read less confusing.
Either way works for me. I used
vtime_account_guest_exit()to try to keep as many details as possible inside vtime, e.g. in case the implemenation is tweaked in the future. But I agree that pretending KVM isn't already deeply intertwined with the details is a lie.
Ok let's keep it as is then. It reads funny but can we perhaps hope that other archs have the same issue and in the future we'll need to have this split generalized outside x86 with:
* guest_exit_vtime()
* guest_exit_tick()
Frederic Weisbecker
maintainerRe: [PATCH v3 3/9] KVM: x86: Defer tick-based accounting 'til after IRQ handling
In reply to Sean ChristophersonQuoted 26 lines · Sean Christopherson
Quoted 25 lines · Frederic Weisbecker
Quoted 5 lines · Sean Christopherson
arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9239
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0@@ -9230,6 +9230,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)local_irq_disable ();kvm_after_interrupt (vcpu); +/* +* When using tick-based accounting, wait until after servicing IRQs to +* account guest time so that any ticks that occurred while running the +* guest are properly accounted to the guest. +*/ +if (!vtime_accounting_enabled_this_cpu ()) +vtime_account_guest_exit ();Can we rather have instead:
static inline void
tick_account_guest_exit(void){ if (!vtime_accounting_enabled_this_cpu()) current->flags &= ~PF_VCPU; }
Quoted 6 lines
Quoted 1 line
It duplicates a bit of code but I think this will read less confusing.
Either way works for me. I used
vtime_account_guest_exit()to try to keep as many details as possible inside vtime, e.g. in case the implemenation is tweaked in the future. But I agree that pretending KVM isn't already deeply intertwined with the details is a lie.
Ah I see, before 87fa7f3e98a131 the vtime was accounted after interrupts get processed. So it used to work until then. I see that ARM64 waits for IRQs to be enabled too.
PPC/book3s_hv, MIPS, s390 do it before IRQs get re-enabled (weird, how does that work?)
And PPC/book3s_pr calls guest_exit() so I guess it has interrupts enabled.
The point is: does it matter to call vtime_account_guest_exit() before or after interrupts? If it doesn't matter, we can simply call vtime_account_guest_exit() once and for all once IRQs are re-enabled.
If it does matter because we don't want to account the host IRQs firing at the end of vcpu exit, then probably we should standardize that behaviour and have guest_exit_vtime() called before interrupts get enabled and guest_exit_tick() called after interrupts get enabled. It's probably then beyond the scope of this patchset but I would like to poke your opinion on that.
Thanks.
Sean Christopherson
maintainerRe: [PATCH v3 3/9] KVM: x86: Defer tick-based accounting 'til after IRQ handling
Apologies for the slow response.
Quoted 34 lines · Frederic Weisbecker
Quoted 26 lines · Sean Christopherson
Quoted 25 lines · Frederic Weisbecker
Quoted 5 lines · Sean Christopherson
arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9239
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0@@ -9230,6 +9230,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)local_irq_disable ();kvm_after_interrupt (vcpu); +/* +* When using tick-based accounting, wait until after servicing IRQs to +* account guest time so that any ticks that occurred while running the +* guest are properly accounted to the guest. +*/ +if (!vtime_accounting_enabled_this_cpu ()) +vtime_account_guest_exit ();Can we rather have instead:
static inline void
tick_account_guest_exit(void){ if (!vtime_accounting_enabled_this_cpu()) current->flags &= ~PF_VCPU; }Quoted 6 lines
Quoted 1 line
It duplicates a bit of code but I think this will read less confusing.
Either way works for me. I used
vtime_account_guest_exit()to try to keep as many details as possible inside vtime, e.g. in case the implemenation is tweaked in the future. But I agree that pretending KVM isn't already deeply intertwined with the details is a lie.
Quoted 6 lines
Ah I see, before
87fa7f3e98a131the vtime was accounted after interrupts get processed. So it used to work until then. I see that ARM64 waits for IRQs to be enabled too.PPC/book3s_hv, MIPS, s390 do it before IRQs get re-enabled (weird, how does that work?)
No idea. It's entirely possible it doesn't work on one or more of those architectures.
Based on init/Kconfig, s390 doesn't support tick-based accounting, so I assume s390 is ok.
config TICK_CPU_ACCOUNTING
bool "Simple tick based cputime accounting" depends on !S390 && !NO_HZ_FULL
Quoted 6 lines
And PPC/book3s_pr calls
guest_exit()so I guess it has interrupts enabled.The point is: does it matter to call
vtime_account_guest_exit()before or after interrupts? If it doesn't matter, we can simply callvtime_account_guest_exit()once and for all once IRQs are re-enabled.
Quoted 7 lines
If it does matter because we don't want to account the host IRQs firing at the end of vcpu exit, then probably we should standardize that behaviour and have
guest_exit_vtime()called before interrupts get enabled andguest_exit_tick()called after interrupts get enabled. It's probably then beyond the scope of this patchset but I would like to poke your opinion on that.Thanks.
I don't know. For x86, I would be ok with simply moving the call to vtime_account_guest_exit() to after IRQs are enabled. It would bug me a little bit that KVM _could_ be more precise when running with CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, and KVM would still be poking into the details of vtime_account_guest_exit() to some extent, but overall it would be an improvement from a code cleanliness perspective.
The problem is I have no clue who, if anyone, deploys KVM on x86 with CONFIG_VIRT_CPU_ACCOUNTING_GEN=y. On the other hand, AMD/SVM has always had the "inaccurate" accounting, and Intel/VMX has been inaccurate since commit d7a08882a0a4 ("KVM: x86: Unconditionally enable irqs in guest context"), which amusingly was a fix for an edge case in tick-based accounting.
Anyone have an opinion either way? I'm very tempted to go with Frederic's suggestion of moving the time accounting back to where it was, it makes KVM just a little less ugly.
Frederic Weisbecker
maintainerRe: [PATCH v3 3/9] KVM: x86: Defer tick-based accounting 'til after IRQ handling
In reply to Sean ChristophersonQuoted 7 lines · Sean Christopherson
From: Wanpeng Li <[email protected]>
When using tick-based accounting, defer the call to account guest time until after servicing any IRQ(s) that happened in the guest or immediately after VM-Exit. Tick-based accounting of vCPU time relies on
PF_VCPUbeing set when the tick IRQ handler runs, and IRQs are blocked throughout {svm,vmx}_vcpu_enter_exit().
Out of curiosity, how has guest accounting ever managed to work if we have always cleared PF_VCPU before re-enabling interrupt?
Thanks.
Sean Christopherson
maintainerPATCH v3 4/9 sched/vtime: Move vtime accounting external declarations above inlines
In reply to Sean ChristophersonMove the blob of external declarations (and their stubs) above the set of inline definitions (and their stubs) for vtime accounting. This will allow a future patch to bring in more inline definitions without also having to shuffle large chunks of code.
No functional change intended.
--- include/linux/vtime.h | 94 +++++++++++++++++++++---------------------- 1 file changed, 47 insertions(+), 47 deletions(-)
include/linux/vtime.h +47 -47
@@ -10,53 +10,6 @@struct task_struct ; -/* -* vtime_accounting_enabled_this_cpu() definitions/declarations -*/ -#if defined(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE ) - -static inline bool vtime_accounting_enabled_this_cpu (void ) {return true; } -extern void vtime_task_switch (struct task_struct *prev); - -#elif defined(CONFIG_VIRT_CPU_ACCOUNTING_GEN ) - -/* -* Checks if vtime is enabled on some CPU. Cputime readers want to be careful -* in that case and compute the tickless cputime. -* For now vtime state is tied to context tracking. We might want to decouple -* those later if necessary. -*/ -static inline bool vtime_accounting_enabled (void ) -{ -return context_tracking_enabled (); -} - -static inline bool vtime_accounting_enabled_cpu (int cpu) -{ -return context_tracking_enabled_cpu (cpu); -} - -static inline bool vtime_accounting_enabled_this_cpu (void ) -{ -return context_tracking_enabled_this_cpu (); -} - -extern void vtime_task_switch_generic (struct task_struct *prev); - -static inline void vtime_task_switch (struct task_struct *prev) -{ -if (vtime_accounting_enabled_this_cpu ()) -vtime_task_switch_generic (prev); -} - -#else /* !CONFIG_VIRT_CPU_ACCOUNTING */ - -static inline bool vtime_accounting_enabled_cpu (int cpu) {return false; } -static inline bool vtime_accounting_enabled_this_cpu (void ) {return false; } -static inline void vtime_task_switch (struct task_struct *prev) { } - -#endif -/* * Common vtime APIs */
@@ -94,6 +47,53 @@ static inline void vtime_account_hardirq(struct task_struct *tsk) { }static inline void vtime_flush (struct task_struct *tsk) { }#endif +/* +* vtime_accounting_enabled_this_cpu() definitions/declarations +*/ +#if defined(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE ) + +static inline bool vtime_accounting_enabled_this_cpu (void ) {return true; } +extern void vtime_task_switch (struct task_struct *prev); + +#elif defined(CONFIG_VIRT_CPU_ACCOUNTING_GEN ) + +/* +* Checks if vtime is enabled on some CPU. Cputime readers want to be careful +* in that case and compute the tickless cputime. +* For now vtime state is tied to context tracking. We might want to decouple +* those later if necessary. +*/ +static inline bool vtime_accounting_enabled (void ) +{ +return context_tracking_enabled (); +} + +static inline bool vtime_accounting_enabled_cpu (int cpu) +{ +return context_tracking_enabled_cpu (cpu); +} + +static inline bool vtime_accounting_enabled_this_cpu (void ) +{ +return context_tracking_enabled_this_cpu (); +} + +extern void vtime_task_switch_generic (struct task_struct *prev); + +static inline void vtime_task_switch (struct task_struct *prev) +{ +if (vtime_accounting_enabled_this_cpu ()) +vtime_task_switch_generic (prev); +} + +#else /* !CONFIG_VIRT_CPU_ACCOUNTING */ + +static inline bool vtime_accounting_enabled_cpu (int cpu) {return false; } +static inline bool vtime_accounting_enabled_this_cpu (void ) {return false; } +static inline void vtime_task_switch (struct task_struct *prev) { } + +#endif +#ifdef CONFIG_IRQ_TIME_ACCOUNTING extern void irqtime_account_irq (struct task_struct *tsk,unsigned int offset);
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Re: [PATCH v3 4/9] sched/vtime: Move vtime accounting external declarations above inlines
Quoted 8 lines · Sean Christopherson
Move the blob of external declarations (and their stubs) above the set of inline definitions (and their stubs) for vtime accounting. This will allow a future patch to bring in more inline definitions without also having to shuffle large chunks of code.
No functional change intended.
Signed-off-by Sean Christopherson<[email protected]>
Quoted 118 lines
--- include/linux/vtime.h | 94 +++++++++++++++++++++---------------------- 1 file changed, 47 insertions(+), 47 deletions(-)include/linux/vtime.h +47
-48@@ -10,53 +10,6 @@struct task_struct ; -/* -* vtime_accounting_enabled_this_cpu() definitions/declarations -*/ -#if defined(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE ) - -static inline bool vtime_accounting_enabled_this_cpu (void ) {return true; } -extern void vtime_task_switch (struct task_struct *prev); - -#elif defined(CONFIG_VIRT_CPU_ACCOUNTING_GEN ) - -/* -* Checks if vtime is enabled on some CPU. Cputime readers want to be careful -* in that case and compute the tickless cputime. -* For now vtime state is tied to context tracking. We might want to decouple -* those later if necessary. -*/ -static inline bool vtime_accounting_enabled (void ) -{ -return context_tracking_enabled (); -} - -static inline bool vtime_accounting_enabled_cpu (int cpu) -{ -return context_tracking_enabled_cpu (cpu); -} - -static inline bool vtime_accounting_enabled_this_cpu (void ) -{ -return context_tracking_enabled_this_cpu (); -} - -extern void vtime_task_switch_generic (struct task_struct *prev); - -static inline void vtime_task_switch (struct task_struct *prev) -{ -if (vtime_accounting_enabled_this_cpu ()) -vtime_task_switch_generic (prev); -} - -#else /* !CONFIG_VIRT_CPU_ACCOUNTING */ - -static inline bool vtime_accounting_enabled_cpu (int cpu) {return false; } -static inline bool vtime_accounting_enabled_this_cpu (void ) {return false; } -static inline void vtime_task_switch (struct task_struct *prev) { } - -#endif -/* * Common vtime APIs */ @@ -94,6 +47,53 @@ static inline void vtime_account_hardirq(struct task_struct *tsk) { }static inline void vtime_flush (struct task_struct *tsk) { }#endif +/* +* vtime_accounting_enabled_this_cpu() definitions/declarations +*/ +#if defined(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE ) + +static inline bool vtime_accounting_enabled_this_cpu (void ) {return true; } +extern void vtime_task_switch (struct task_struct *prev); + +#elif defined(CONFIG_VIRT_CPU_ACCOUNTING_GEN ) + +/* +* Checks if vtime is enabled on some CPU. Cputime readers want to be careful +* in that case and compute the tickless cputime. +* For now vtime state is tied to context tracking. We might want to decouple +* those later if necessary. +*/ +static inline bool vtime_accounting_enabled (void ) +{ +return context_tracking_enabled (); +} + +static inline bool vtime_accounting_enabled_cpu (int cpu) +{ +return context_tracking_enabled_cpu (cpu); +} + +static inline bool vtime_accounting_enabled_this_cpu (void ) +{ +return context_tracking_enabled_this_cpu (); +} + +extern void vtime_task_switch_generic (struct task_struct *prev); + +static inline void vtime_task_switch (struct task_struct *prev) +{ +if (vtime_accounting_enabled_this_cpu ()) +vtime_task_switch_generic (prev); +} + +#else /* !CONFIG_VIRT_CPU_ACCOUNTING */ + +static inline bool vtime_accounting_enabled_cpu (int cpu) {return false; } +static inline bool vtime_accounting_enabled_this_cpu (void ) {return false; } +static inline void vtime_task_switch (struct task_struct *prev) { } + +#endif +#ifdef CONFIG_IRQ_TIME_ACCOUNTING extern void irqtime_account_irq (struct task_struct *tsk,unsigned int offset); -- 2.31.1.368.gbe11c130af-goog
Sean Christopherson
maintainerPATCH v3 5/9 sched/vtime: Move guest enter/exit vtime accounting to vtime.h
In reply to Sean ChristophersonProvide separate helpers for guest enter vtime accounting (in addition to the existing guest exit helpers), and move all vtime accounting helpers to vtime.h where the existing #ifdef infrastructure can be leveraged to better delineate the different types of accounting. This will also allow future cleanups via deduplication of context tracking code.
Opportunstically delete the vtime_account_kernel() stub now that all callers are wrapped with CONFIG_VIRT_CPU_ACCOUNTING_NATIVE=y.
No functional change intended.
--- include/linux/context_tracking.h | 17 +----------- include/linux/vtime.h | 46 +++++++++++++++++++++++++++----- 2 files changed, 41 insertions(+), 22 deletions(-)
include/linux/context_tracking.h +1 -16
@@ -137,14 +137,6 @@ static __always_inline void context_tracking_guest_exit_irqoff(void)__context_tracking_exit (CONTEXT_GUEST ); } -static __always_inline void vtime_account_guest_exit(void) -{ -if (vtime_accounting_enabled_this_cpu ()) -vtime_guest_exit (current); -else - current->flags &= ~PF_VCPU ; -} - static __always_inline void guest_exit_irqoff(void) {context_tracking_guest_exit_irqoff ();
@@ -163,20 +155,13 @@ static __always_inline void guest_enter_irqoff(void)* to flush. */ instrumentation_begin (); -vtime_account_kernel (current); - current->flags |=PF_VCPU ; +vtime_account_guest_enter ();rcu_virt_note_context_switch (smp_processor_id ());instrumentation_end (); } static __always_inline void context_tracking_guest_exit_irqoff(void) { } -static __always_inline void vtime_account_guest_exit(void) -{ -vtime_account_kernel (current); - current->flags &= ~PF_VCPU ; -} - static __always_inline void guest_exit_irqoff(void) {instrumentation_begin ();
include/linux/vtime.h +40 -6
79 lines not on this page · raw message
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Sean Christopherson
maintainerPATCH v3 6/9 context_tracking: Consolidate guest enter/exit wrappers
In reply to Sean ChristophersonConsolidate the guest enter/exit wrappers, providing and tweaking stubs as needed. This will allow moving the wrappers under KVM without having to bleed #ifdefs into the soon-to-be KVM code.
No functional change intended.
--- include/linux/context_tracking.h | 65 ++++++++++++-------------------- 1 file changed, 24 insertions(+), 41 deletions(-)
include/linux/context_tracking.h +24 -41
@@ -71,6 +71,19 @@ static inline void exception_exit(enum ctx_state prev_ctx) } } +static __always_inline bool context_tracking_guest_enter_irqoff(void) +{ +if (context_tracking_enabled ()) +__context_tracking_enter (CONTEXT_GUEST ); + +return context_tracking_enabled_this_cpu (); +} + +static __always_inline void context_tracking_guest_exit_irqoff(void) +{ +if (context_tracking_enabled ()) +__context_tracking_exit (CONTEXT_GUEST ); +}/** * ct_state() - return the current context tracking state if known
@@ -92,6 +105,9 @@ static inline void user_exit_irqoff(void) { }static inline enum ctx_state exception_enter (void ) {return 0 ; }static inline void exception_exit (enum ctx_state prev_ctx) { }static inline enum ctx_state ct_state (void ) {return CONTEXT_DISABLED ; } +static inline bool context_tracking_guest_enter_irqoff (void ) {return false; } +static inline void context_tracking_guest_exit_irqoff (void ) { } +#endif /* !CONFIG_CONTEXT_TRACKING */ #define CT_WARN_ON (cond) WARN_ON(context_tracking_enabled() && (cond))
@@ -102,74 +118,41 @@ extern void context_tracking_init(void);static inline void context_tracking_init (void ) { }#endif /* CONFIG_CONTEXT_TRACKING_FORCE */ - -#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN /* must be called with irqs disabled */ static __always_inline void guest_enter_irqoff(void) { +/* +* This is running in ioctl context so its safe to assume that it's the +* stime pending cputime to flush. +*/ instrumentation_begin (); -if (vtime_accounting_enabled_this_cpu ()) -vtime_guest_enter (current); -else - current->flags |=PF_VCPU ; +vtime_account_guest_enter ();instrumentation_end (); -if (context_tracking_enabled ()) -__context_tracking_enter (CONTEXT_GUEST ); - -/* KVM does not hold any references to rcu protected data when it +/* +* KVM does not hold any references to rcu protected data when it * switches CPU into a guest mode. In fact switching to a guest mode * is very similar to exiting to userspace from rcu point of view. In * addition CPU may stay in a guest mode for quite a long time (up to * one time slice). Lets treat guest mode as quiescent state, just like * we do with user-mode execution. */ -if (!context_tracking_enabled_this_cpu ()) { +if (!context_tracking_guest_enter_irqoff ()) {instrumentation_begin ();rcu_virt_note_context_switch (smp_processor_id ());instrumentation_end (); } } -static __always_inline void context_tracking_guest_exit_irqoff(void) -{ -if (context_tracking_enabled ()) -__context_tracking_exit (CONTEXT_GUEST ); -} - static __always_inline void guest_exit_irqoff(void) {context_tracking_guest_exit_irqoff (); -instrumentation_begin (); -vtime_account_guest_exit (); -instrumentation_end (); -} - -#else -static __always_inline void guest_enter_irqoff(void) -{ -/* -* This is running in ioctl context so its safe -* to assume that it's the stime pending cputime -* to flush. -*/ -instrumentation_begin (); -vtime_account_guest_enter (); -rcu_virt_note_context_switch (smp_processor_id ()); -instrumentation_end (); -} - -static __always_inline void context_tracking_guest_exit_irqoff(void) { } - -static __always_inline void guest_exit_irqoff(void) -{instrumentation_begin ();/* Flush the guest cputime we spent on the guest */ vtime_account_guest_exit ();instrumentation_end (); } -#endif /* CONFIG_VIRT_CPU_ACCOUNTING_GEN */ static inline void guest_exit (void ) {
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Sean Christopherson
maintainerPATCH v3 7/9 context_tracking: KVM: Move guest enter/exit wrappers to KVM's domain
In reply to Sean ChristophersonMove the guest enter/exit wrappers to kvm_host.h so that KVM can manage its context tracking vs. vtime accounting without bleeding too many KVM details into the context tracking code.
No functional change intended.
--- include/linux/context_tracking.h | 45 -------------------------------- include/linux/kvm_host.h | 45 ++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 45 deletions(-)
include/linux/context_tracking.h +0 -45
@@ -118,49 +118,4 @@ extern void context_tracking_init(void);static inline void context_tracking_init (void ) { }#endif /* CONFIG_CONTEXT_TRACKING_FORCE */ -/* must be called with irqs disabled */ -static __always_inline void guest_enter_irqoff(void) -{ -/* -* This is running in ioctl context so its safe to assume that it's the -* stime pending cputime to flush. -*/ -instrumentation_begin (); -vtime_account_guest_enter (); -instrumentation_end (); - -/* -* KVM does not hold any references to rcu protected data when it -* switches CPU into a guest mode. In fact switching to a guest mode -* is very similar to exiting to userspace from rcu point of view. In -* addition CPU may stay in a guest mode for quite a long time (up to -* one time slice). Lets treat guest mode as quiescent state, just like -* we do with user-mode execution. -*/ -if (!context_tracking_guest_enter_irqoff ()) { -instrumentation_begin (); -rcu_virt_note_context_switch (smp_processor_id ()); -instrumentation_end (); - } -} - -static __always_inline void guest_exit_irqoff(void) -{ -context_tracking_guest_exit_irqoff (); - -instrumentation_begin (); -/* Flush the guest cputime we spent on the guest */ -vtime_account_guest_exit (); -instrumentation_end (); -} - -static inline void guest_exit (void ) -{ -unsigned long flags; - -local_irq_save (flags); -guest_exit_irqoff (); -local_irq_restore (flags); -} -#endif
include/linux/kvm_host.h +45 -0
51 lines not on this page · raw message
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Re: [PATCH v3 7/9] context_tracking: KVM: Move guest enter/exit wrappers to KVM's domain
Quoted 5 lines · Sean Christopherson
Move the guest enter/exit wrappers to
kvm_host.hso that KVM can manage its context tracking vs. vtime accounting without bleeding too many KVM details into the context tracking code.No functional change intended.
Funny story. This used to be in kvm code and it was moved to context_tracking by commit 521921bad1192fb1b8f9b6a5aa673635848b8b5f
kvm: Move guest entry/exit APIs to context_tracking
I think with all your cleanup it can now move back as it no longer deals with context tracking.
Quoted 118 lines
Signed-off-by Sean Christopherson <[email protected]>--- include/linux/context_tracking.h | 45 -------------------------------- include/linux/kvm_host.h | 45 ++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 45 deletions(-)include/linux/context_tracking.h +0
-45@@ -118,49 +118,4 @@ extern void context_tracking_init(void);static inline void context_tracking_init (void ) { }#endif /* CONFIG_CONTEXT_TRACKING_FORCE */ -/* must be called with irqs disabled */ -static __always_inline void guest_enter_irqoff(void) -{ -/* -* This is running in ioctl context so its safe to assume that it's the -* stime pending cputime to flush. -*/ -instrumentation_begin (); -vtime_account_guest_enter (); -instrumentation_end (); - -/* -* KVM does not hold any references to rcu protected data when it -* switches CPU into a guest mode. In fact switching to a guest mode -* is very similar to exiting to userspace from rcu point of view. In -* addition CPU may stay in a guest mode for quite a long time (up to -* one time slice). Lets treat guest mode as quiescent state, just like -* we do with user-mode execution. -*/ -if (!context_tracking_guest_enter_irqoff ()) { -instrumentation_begin (); -rcu_virt_note_context_switch (smp_processor_id ()); -instrumentation_end (); - } -} - -static __always_inline void guest_exit_irqoff(void) -{ -context_tracking_guest_exit_irqoff (); - -instrumentation_begin (); -/* Flush the guest cputime we spent on the guest */ -vtime_account_guest_exit (); -instrumentation_end (); -} - -static inline void guest_exit (void ) -{ -unsigned long flags; - -local_irq_save (flags); -guest_exit_irqoff (); -local_irq_restore (flags); -} -#endif include/linux/kvm_host.h +45
-0@@ -332,6 +332,51 @@ struct kvm_vcpu {struct kvm_dirty_ring dirty_ring; }; +/* must be called with irqs disabled */ +static __always_inline void guest_enter_irqoff(void) +{ +/* +* This is running in ioctl context so its safe to assume that it's the +* stime pending cputime to flush. +*/ +instrumentation_begin (); +vtime_account_guest_enter (); +instrumentation_end (); + +/* +* KVM does not hold any references to rcu protected data when it +* switches CPU into a guest mode. In fact switching to a guest mode +* is very similar to exiting to userspace from rcu point of view. In +* addition CPU may stay in a guest mode for quite a long time (up to +* one time slice). Lets treat guest mode as quiescent state, just like +* we do with user-mode execution. +*/ +if (!context_tracking_guest_enter_irqoff ()) { +instrumentation_begin (); +rcu_virt_note_context_switch (smp_processor_id ()); +instrumentation_end (); + } +} + +static __always_inline void guest_exit_irqoff(void) +{ +context_tracking_guest_exit_irqoff (); + +instrumentation_begin (); +/* Flush the guest cputime we spent on the guest */ +vtime_account_guest_exit (); +instrumentation_end (); +} + +static inline void guest_exit (void ) +{ +unsigned long flags; + +local_irq_save (flags); +guest_exit_irqoff (); +local_irq_restore (flags); +} +static inline int kvm_vcpu_exiting_guest_mode (struct kvm_vcpu *vcpu) {/*
Sean Christopherson
maintainerPATCH v3 8/9 KVM: x86: Consolidate guest enter/exit logic to common helpers
In reply to Sean ChristophersonMove the enter/exit logic in {svm,vmx}_vcpu_enter_exit() to common helpers. Opportunistically update the somewhat stale comment about the updates needing to occur immediately after VM-Exit.
No functional change intended.
--- arch/x86/kvm/svm/svm.c | 46 ++----------------------------------- arch/x86/kvm/vmx/vmx.c | 46 ++----------------------------------- arch/x86/kvm/x86.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 56 insertions(+), 88 deletions(-)
arch/x86/kvm/svm/svm.c +2 -44
@@ -3713,25 +3713,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); -/* -* VMENTER enables interrupts (host state), but the kernel state is -* interrupts disabled when this is invoked. Also tell RCU about -* it. This is the same logic as for exit_to_user_mode(). -* -* This ensures that e.g. latency analysis on the host observes -* guest mode as interrupt enabled. -* -* guest_enter_irqoff() informs context tracking about the -* transition to guest mode and if enabled adjusts RCU state -* accordingly. -*/ -instrumentation_begin (); -trace_hardirqs_on_prepare (); -lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); -instrumentation_end (); - -guest_enter_irqoff (); -lockdep_hardirqs_on (CALLER_ADDR0 ); +kvm_guest_enter_irqoff ();if (sev_es_guest (vcpu->kvm)) {__svm_sev_es_vcpu_run (svm->vmcb_pa);
@@ -3745,31 +3727,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu)vmload (__sme_page_pa (sd->save_area)); } -/* -* VMEXIT disables interrupts (host state), but tracing and lockdep -* have them in state 'on' as recorded before entering guest mode. -* Same as enter_from_user_mode(). -* -* context_tracking_guest_exit_irqoff() restores host context and -* reinstates RCU if enabled and required. -* -* This needs to be done before the below as native_read_msr() -* contains a tracepoint and x86_spec_ctrl_restore_host() calls -* into world and some more. -*/ -lockdep_hardirqs_off (CALLER_ADDR0 ); -context_tracking_guest_exit_irqoff (); - -instrumentation_begin (); -/* -* Account guest time when precise accounting based on context tracking -* is enabled. Tick based accounting is deferred until after IRQs that -* occurred within the VM-Enter/VM-Exit "window" are handled. -*/ -if (vtime_accounting_enabled_this_cpu ()) -vtime_account_guest_exit (); -trace_hardirqs_off_finish (); -instrumentation_end (); +kvm_guest_exit_irqoff (); } static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
arch/x86/kvm/vmx/vmx.c +2 -44
58 lines not on this page · raw message
arch/x86/kvm/x86.h +52 -0
58 lines not on this page · raw message
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Sean Christopherson
maintainerPATCH v3 9/9 KVM: Move instrumentation-safe annotations for enter/exit to x86 code
In reply to Sean ChristophersonDrop the instrumentation_{begin,end}() annonations from the common KVM guest enter/exit helpers, and massage the x86 code as needed to preserve the necessary annotations. x86 is the only architecture whose transition flow is tagged as noinstr, and more specifically, it is the only architecture for which instrumentation_{begin,end}() can be non-empty.
No other architecture supports CONFIG_STACK_VALIDATION=y, and s390 is the only other architecture that support CONFIG_DEBUG_ENTRY=y. For instrumentation annontations to be meaningful, both aformentioned configs must be enabled.
Letting x86 deal with the annotations avoids unnecessary nops by squashing back-to-back instrumention-safe sequences.
--- arch/x86/kvm/x86.h | 4 ++-- include/linux/kvm_host.h | 9 +-------- 2 files changed, 3 insertions(+), 10 deletions(-)
arch/x86/kvm/x86.h +2 -2
@@ -25,9 +25,9 @@ static __always_inline void kvm_guest_enter_irqoff(void)instrumentation_begin ();trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); -instrumentation_end (); -guest_enter_irqoff (); +instrumentation_end (); +lockdep_hardirqs_on (CALLER_ADDR0 ); }
include/linux/kvm_host.h +1 -8
@@ -339,9 +339,7 @@ static __always_inline void guest_enter_irqoff(void)* This is running in ioctl context so its safe to assume that it's the * stime pending cputime to flush. */ -instrumentation_begin ();vtime_account_guest_enter (); -instrumentation_end ();/* * KVM does not hold any references to rcu protected data when it
@@ -351,21 +349,16 @@ static __always_inline void guest_enter_irqoff(void)* one time slice). Lets treat guest mode as quiescent state, just like * we do with user-mode execution. */ -if (!context_tracking_guest_enter_irqoff ()) { -instrumentation_begin (); +if (!context_tracking_guest_enter_irqoff ())rcu_virt_note_context_switch (smp_processor_id ()); -instrumentation_end (); - } } static __always_inline void guest_exit_irqoff(void) {context_tracking_guest_exit_irqoff (); -instrumentation_begin ();/* Flush the guest cputime we spent on the guest */ vtime_account_guest_exit (); -instrumentation_end (); }static inline void guest_exit (void )
Signature · 1 line
2.31.1.368.gbe11c130af-goog
Re: [PATCH v3 9/9] KVM: Move instrumentation-safe annotations for enter/exit to x86 code
Quoted 6 lines · Sean Christopherson
Drop the instrumentation_{begin,end}() annonations from the common KVM guest enter/exit helpers, and massage the x86 code as needed to preserve the necessary annotations. x86 is the only architecture whose transition flow is tagged as noinstr, and more specifically, it is the only architecture for which instrumentation_{begin,end}() can be non-empty.
Quoted 7 lines
No other architecture supports
CONFIG_STACK_VALIDATION=y, and s390 is the only other architecture that supportCONFIG_DEBUG_ENTRY=y. For instrumentation annontations to be meaningful, both aformentioned configs must be enabled.Letting x86 deal with the annotations avoids unnecessary nops by squashing back-to-back instrumention-safe sequences.
We have considered implementing objtool for s390. Not sure where we stand and if we will do this or not. Sven/Heiko?
So maybe drop this patch until every other arch agrees that there are no plans to implement this.
Re: [PATCH v3 9/9] KVM: Move instrumentation-safe annotations for enter/exit to x86 code
Quoted 14 lines · Christian Borntraeger
Quoted 4 lines · Sean Christopherson
Drop the instrumentation_{begin,end}() annonations from the common KVM guest enter/exit helpers, and massage the x86 code as needed to preserve the necessary annotations. x86 is the only architecture whose transition flow is tagged as noinstr, and more specifically, it is the only
Quoted 8 lines
architecture for which instrumentation_{begin,end}() can be non-empty. No other architecture supports
CONFIG_STACK_VALIDATION=y, and s390 is the
only other architecture that supportCONFIG_DEBUG_ENTRY=y. For instrumentation annontations to be meaningful, both aformentioned configs must be enabled.
Letting x86 deal with the annotations avoids unnecessary nops by squashing back-to-back instrumention-safe sequences.
Quoted 2 lines
We have considered implementing objtool for s390. Not sure where we stand and if we will do this or not. Sven/Heiko?
We are planning to support objtool on s390. Vasily is working on it - maybe he has some thoughts about this.
Re: [PATCH v3 9/9] KVM: Move instrumentation-safe annotations for enter/exit to x86 code
Quoted 16 lines · Sven Schnelle
Quoted 14 lines · Christian Borntraeger
Quoted 12 lines · Sean Christopherson
Drop the instrumentation_{begin,end}() annonations from the common KVM guest enter/exit helpers, and massage the x86 code as needed to preserve the necessary annotations. x86 is the only architecture whose transition flow is tagged as noinstr, and more specifically, it is the only architecture for which instrumentation_{begin,end}() can be non-empty. No other architecture supports
CONFIG_STACK_VALIDATION=y, and s390 is the
only other architecture that supportCONFIG_DEBUG_ENTRY=y. For instrumentation annontations to be meaningful, both aformentioned configs must be enabled.
Letting x86 deal with the annotations avoids unnecessary nops by squashing back-to-back instrumention-safe sequences.
Quoted 5 lines
Quoted 2 lines
We have considered implementing objtool for s390. Not sure where we stand and if we will do this or not. Sven/Heiko?
We are planning to support objtool on s390. Vasily is working on it - maybe he has some thoughts about this.
We got CONFIG_DEBUG_ENTRY=y since 5.12, objtool runs on vmlinux.o but I have not yet enabled --noinstr option in s390 objtool. So, it's hard to say in advance if this particular change would make things better or worse. In general, common code annotations are problematic, because arch specific code is still not identical and this leads sometimes to different needs for common code annotations.
I'll try to experiment with --noinstr on s390 shortly.
Frederic Weisbecker
maintainerQuoted 52 lines · Sean Christopherson
This is a continuation of Wanpeng's series[1] to fix tick-based CPU time accounting on x86, with my cleanups[2] bolted on top. The core premise of Wanpeng's patches are preserved, but they are heavily stripped down. Specifically, only the "guest exit" paths are split, and no code is consolidated. The intent is to do as little as possible in the three patches that need to be backported. Keeping those changes as small as possible also meant that my cleanups did not need to unwind much refactoring.
On x86, tested
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y and =n, and withCONFIG_DEBUG_ENTRY=y&&CONFIG_VALIDATE_STACKS=y. Compile tested arm64, MIPS, PPC, and s390, the latter withCONFIG_DEBUG_ENTRY=yfor giggles.One last note: I elected to use
vtime_account_guest_exit()in the x86 code instead of open coding these equivalents:if (vtime_accounting_enabled_this_cpu()) vtime_guest_exit(current); ... if (!vtime_accounting_enabled_this_cpu()) current->flags &= ~PF_VCPU;With
CONFIG_VIRT_CPU_ACCOUNTING_GEN=n, this is a complete non-issue, but for the =y case it meanscontext_tracking_enabled_this_cpu()is being checked back-to-back. The redundant checks bug me, but open coding the gory details in x86 or providing funky variants invtime.hfelt worse.Delta from Wanpeng's v2:
- s/context_guest/context_tracking_guest, purely to match the existing functions. I have no strong opinion either way.
- Split only the "exit" functions.
- Partially open codevcpu_account_guest_exit()and__vtime_account_guest_exit()in x86 to avoid churn when segueing into my cleanups (see above).[1] https://lkml.kernel.org/r/[email protected]
[2] https://lkml.kernel.org/r/[email protected]Sean Christopherson (6):
sched/vtime: Move vtime accounting external declarations above inlines sched/vtime: Move guest enter/exit vtime accounting tovtime.h
context_tracking: Consolidate guest enter/exit wrappers
context_tracking: KVM: Move guest enter/exit wrappers to KVM's domain
KVM: x86: Consolidate guest enter/exit logic to common helpers
KVM: Move instrumentation-safe annotations for enter/exit to x86 codeWanpeng Li (3):
context_tracking: Move guest exit context tracking to separate helpers
context_tracking: Move guest exit vtime accounting to separate helpers
KVM: x86: Defer tick-based accounting 'til after IRQ handling
Quoted 8 lines
arch/x86/kvm/svm/svm.c | 39 +-------- arch/x86/kvm/vmx/vmx.c | 39 +-------- arch/x86/kvm/x86.c | 8 ++ arch/x86/kvm/x86.h | 52 ++++++++++++ include/linux/context_tracking.h | 92 ++++----------------- include/linux/kvm_host.h | 38 +++++++++ include/linux/vtime.h | 138 +++++++++++++++++++------------ 7 files changed, 204 insertions(+), 202 deletions(-)
Please Cc me on any follow-up of this patchset. I have set up a lot of booby traps on purpose in this cave and I might be able to remember a few on the way. Should you meet one of the poisoned arrows, rest assured that you were not the aimed target though.
Thanks.