Revert "KVM: x86: Unconditionally enable irqs in guest context"
PATCH 1/1 Revert "KVM: x86: Unconditionally enable irqs in guest context"
This reverts commit d7a08882a0a4b4e176691331ee3f492996579534.
After the introduction of the patch:
87fa7f3e9: x86/kvm: Move context tracking where it belongs
since we have moved guest_exit_irqoff closer to the VM-Exit, explicit enabling of irqs to process pending interrupts should not be required within vcpu_enter_guest anymore.
Conflicts:
arch/x86/kvm/svm.c
--- arch/x86/kvm/svm/svm.c | 9 +++++++++ arch/x86/kvm/x86.c | 11 ----------- 2 files changed, 9 insertions(+), 11 deletions(-)
arch/x86/kvm/svm/svm.c +9 -0
@@ -4187,6 +4187,15 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,static void svm_handle_exit_irqoff (struct kvm_vcpu *vcpu) { +kvm_before_interrupt (vcpu); +local_irq_enable (); +/* +* We must have an instruction with interrupts enabled, so +* the timer interrupt isn't delayed by the interrupt shadow. +*/ + asm("nop" ); +local_irq_disable (); +kvm_after_interrupt (vcpu); }static void svm_sched_in (struct kvm_vcpu *vcpu,int cpu)
arch/x86/kvm/x86.c +0 -11
@@ -9023,18 +9023,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_x86_ops.handle_exit_irqoff (vcpu); -/* -* Consume any pending interrupts, including the possible source of -* VM-Exit on SVM and any ticks that occur between VM-Exit and now. -* An instruction is required after local_irq_enable() to fully unblock -* interrupts on processors that implement an interrupt shadow, the -* stat.exits increment will do nicely. -*/ -kvm_before_interrupt (vcpu); -local_irq_enable (); ++vcpu->stat.exits; -local_irq_disable (); -kvm_after_interrupt (vcpu);if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;
Signature · 1 line
2.27.0
Quoted 28 lines · Nitesh Narayan Lal
This reverts commit
d7a08882a0a4b4e176691331ee3f492996579534.After the introduction of the patch:
87fa7f3e9: x86/kvm: Move context tracking where it belongssince we have moved
guest_exit_irqoffcloser to the VM-Exit, explicit enabling of irqs to process pending interrupts should not be required withinvcpu_enter_guestanymore.Conflicts:
arch/x86/kvm/svm.cSigned-off-by Nitesh Narayan Lal <[email protected]>--- arch/x86/kvm/svm/svm.c | 9 +++++++++ arch/x86/kvm/x86.c | 11 ----------- 2 files changed, 9 insertions(+), 11 deletions(-)arch/x86/kvm/svm/svm.c +0
-0@@ -4187,6 +4187,15 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,static void svm_handle_exit_irqoff (struct kvm_vcpu *vcpu) {
arch/x86/kvm/svm/svm.c static int svm_check_intercept(struct kvm_vcpu *vcpu, · line 4199
Quoted 10 lines
arch/x86/kvm/svm/svm.c +9
-0+kvm_before_interrupt (vcpu); +local_irq_enable (); +/* +* We must have an instruction with interrupts enabled, so +* the timer interrupt isn't delayed by the interrupt shadow. +*/ + asm("nop" ); +local_irq_disable (); +kvm_after_interrupt (vcpu); }
Why do we need to reintroduce this part?
Wanpeng
Sean Christopherson
maintainer+tglx
Quoted 9 lines · Nitesh Narayan Lal
This reverts commit
d7a08882a0a4b4e176691331ee3f492996579534.After the introduction of the patch:
87fa7f3e9: x86/kvm: Move context tracking where it belongssince we have moved
guest_exit_irqoffcloser to the VM-Exit, explicit enabling of irqs to process pending interrupts should not be required withinvcpu_enter_guestanymore.
Ugh, except that commit completely broke tick-based accounting, on both Intel and AMD. With guest_exit_irqoff() being called immediately after VM-Exit, any tick that happens after IRQs are disabled will be accounted to the host. E.g. on Intel, even an IRQ VM-Exit that has already been acked by the CPU isn't processed until kvm_x86_ops.handle_exit_irqoff(), well after PF_VCPU has been cleared.
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y should still work (I didn't bother to verify).
Thomas, any clever ideas? Handling IRQs in {vmx,svm}_vcpu_enter_exit() isn't an option as KVM hasn't restored enough state to handle an IRQ, e.g. PKRU and XCR0 are still guest values. Is it too heinous to fudge PF_VCPU across KVM's "pending" IRQ handling? E.g. this god-awful hack fixes the accounting:
arch/x86/kvm/x86.c +2 -0
@@ -9028,6 +9028,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) vcpu->mode =OUTSIDE_GUEST_MODE ;smp_wmb (); + current->flags |=PF_VCPU ; kvm_x86_ops.handle_exit_irqoff (vcpu);/*
@@ -9042,6 +9043,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) ++vcpu->stat.exits;local_irq_disable ();kvm_after_interrupt (vcpu); + current->flags &= ~PF_VCPU ;if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;
Quoted 55 lines
Conflicts:
arch/x86/kvm/svm.cSigned-off-by Nitesh Narayan Lal <[email protected]>--- arch/x86/kvm/svm/svm.c | 9 +++++++++ arch/x86/kvm/x86.c | 11 ----------- 2 files changed, 9 insertions(+), 11 deletions(-)arch/x86/kvm/svm/svm.c +9
-0@@ -4187,6 +4187,15 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,static void svm_handle_exit_irqoff (struct kvm_vcpu *vcpu) { +kvm_before_interrupt (vcpu); +local_irq_enable (); +/* +* We must have an instruction with interrupts enabled, so +* the timer interrupt isn't delayed by the interrupt shadow. +*/ + asm("nop" ); +local_irq_disable (); +kvm_after_interrupt (vcpu); }static void svm_sched_in (struct kvm_vcpu *vcpu,int cpu)arch/x86/kvm/x86.c +0
-11@@ -9023,18 +9023,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_x86_ops.handle_exit_irqoff (vcpu); -/* -* Consume any pending interrupts, including the possible source of -* VM-Exit on SVM and any ticks that occur between VM-Exit and now. -* An instruction is required after local_irq_enable() to fully unblock -* interrupts on processors that implement an interrupt shadow, the -* stat.exits increment will do nicely. -*/ -kvm_before_interrupt (vcpu); -local_irq_enable (); ++vcpu->stat.exits; -local_irq_disable (); -kvm_after_interrupt (vcpu);if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;Signature · 1 line
2.27.0
Quoted 7 lines · Sean Christopherson
+tglx
Quoted 4 lines · Nitesh Narayan Lal
This reverts commit
d7a08882a0a4b4e176691331ee3f492996579534.After the introduction of the patch:
Quoted 7 lines
Quoted 5 lines
87fa7f3e9: x86/kvm: Move context tracking where it belongssince we have moved
guest_exit_irqoffcloser to the VM-Exit, explicit enabling of irqs to process pending interrupts should not be required withinvcpu_enter_guestanymore.Ugh, except that commit completely broke tick-based accounting, on both Intel and AMD.
I did notice some discrepancies in the system time reported after the introduction of this patch but I wrongly concluded that the behavior is correct.
I reported this yesterday [1] but I think I added your old email ID in that thread (sorry about that).
Quoted 5 lines
With
guest_exit_irqoff()being called immediately after VM-Exit, any
tick that happens after IRQs are disabled will be accounted to the host. E.g. on Intel, even an IRQ VM-Exit that has already been acked by the CPU isn't processed untilkvm_x86_ops.handle_exit_irqoff(), well afterPF_VCPUhas been cleared.
Right that also explains the higher system time reported by the cpuacct.stats.
Quoted 2 lines
CONFIG_VIRT_CPU_ACCOUNTING_GEN=yshould still work (I didn't bother to verify).
For the cpuacct stats that I have shared in the other thread, this config was enabled.
Quoted 20 lines
Thomas, any clever ideas? Handling IRQs in {vmx,svm}_vcpu_enter_exit() isn't an option as KVM hasn't restored enough state to handle an IRQ, e.g. PKRU and XCR0 are still guest values. Is it too heinous to fudge
PF_VCPUacross KVM's "pending" IRQ handling? E.g. this god-awful hack fixes the accounting:arch/x86/kvm/x86.c +1
-0@@ -9028,6 +9028,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) vcpu->mode =OUTSIDE_GUEST_MODE ;smp_wmb (); + current->flags |=PF_VCPU ; kvm_x86_ops.handle_exit_irqoff (vcpu);/*
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9049
Quoted 7 lines
arch/x86/kvm/x86.c +1
-0@@ -9042,6 +9043,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) ++vcpu->stat.exits;local_irq_disable ();kvm_after_interrupt (vcpu); + current->flags &= ~PF_VCPU ;if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;
I can give this a try.
What is the right way to test this (via cpuacct stats maybe)?
[1] https://lore.kernel.org/lkml/[email protected]/
Signature · 2 lines
Thanks Nitesh
Quoted 14 lines · Sean Christopherson
+tglx
Quoted 9 lines · Nitesh Narayan Lal
This reverts commit
d7a08882a0a4b4e176691331ee3f492996579534.After the introduction of the patch:
87fa7f3e9: x86/kvm: Move context tracking where it belongssince we have moved
guest_exit_irqoffcloser to the VM-Exit, explicit enabling of irqs to process pending interrupts should not be required withinvcpu_enter_guestanymore.
Quoted 7 lines
Ugh, except that commit completely broke tick-based accounting, on both Intel and AMD. With
guest_exit_irqoff()being called immediately after VM-Exit, any tick that happens after IRQs are disabled will be accounted to the host. E.g. on Intel, even an IRQ VM-Exit that has already been acked by the CPU isn't processed untilkvm_x86_ops.handle_exit_irqoff(), well afterPF_VCPUhas been cleared.
This issue can be 100% reproduced.
https://bugzilla.kernel.org/show_bug.cgi?id=204177
Quoted 83 lines
CONFIG_VIRT_CPU_ACCOUNTING_GEN=yshould still work (I didn't bother to verify).Thomas, any clever ideas? Handling IRQs in {vmx,svm}_vcpu_enter_exit() isn't an option as KVM hasn't restored enough state to handle an IRQ, e.g. PKRU and XCR0 are still guest values. Is it too heinous to fudge
PF_VCPUacross KVM's "pending" IRQ handling? E.g. this god-awful hack fixes the accounting:arch/x86/kvm/x86.c +2
-0@@ -9028,6 +9028,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) vcpu->mode =OUTSIDE_GUEST_MODE ;smp_wmb (); + current->flags |=PF_VCPU ; kvm_x86_ops.handle_exit_irqoff (vcpu);/* @@ -9042,6 +9043,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) ++vcpu->stat.exits;local_irq_disable ();kvm_after_interrupt (vcpu); + current->flags &= ~PF_VCPU ;if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;Quoted 56 lines
Conflicts:
arch/x86/kvm/svm.cSigned-off-by Nitesh Narayan Lal <[email protected]>--- arch/x86/kvm/svm/svm.c | 9 +++++++++ arch/x86/kvm/x86.c | 11 ----------- 2 files changed, 9 insertions(+), 11 deletions(-)arch/x86/kvm/svm/svm.c +9
-0@@ -4187,6 +4187,15 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,static void svm_handle_exit_irqoff (struct kvm_vcpu *vcpu) { +kvm_before_interrupt (vcpu); +local_irq_enable (); +/* +* We must have an instruction with interrupts enabled, so +* the timer interrupt isn't delayed by the interrupt shadow. +*/ + asm("nop" ); +local_irq_disable (); +kvm_after_interrupt (vcpu); }static void svm_sched_in (struct kvm_vcpu *vcpu,int cpu)arch/x86/kvm/x86.c +0
-12@@ -9023,18 +9023,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_x86_ops.handle_exit_irqoff (vcpu); -/* -* Consume any pending interrupts, including the possible source of -* VM-Exit on SVM and any ticks that occur between VM-Exit and now. -* An instruction is required after local_irq_enable() to fully unblock -* interrupts on processors that implement an interrupt shadow, the -* stat.exits increment will do nicely. -*/ -kvm_before_interrupt (vcpu); -local_irq_enable (); ++vcpu->stat.exits; -local_irq_disable (); -kvm_after_interrupt (vcpu);if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta; --2.27.0
Quoted 24 lines · Wanpeng Li
Quoted 14 lines · Sean Christopherson
+tglx
Quoted 9 lines · Nitesh Narayan Lal
This reverts commit
d7a08882a0a4b4e176691331ee3f492996579534.After the introduction of the patch:
87fa7f3e9: x86/kvm: Move context tracking where it belongssince we have moved
guest_exit_irqoffcloser to the VM-Exit, explicit enabling of irqs to process pending interrupts should not be required withinvcpu_enter_guestanymore.Quoted 7 lines
Ugh, except that commit completely broke tick-based accounting, on both Intel and AMD. With
guest_exit_irqoff()being called immediately after VM-Exit, any tick that happens after IRQs are disabled will be accounted to the host. E.g. on Intel, even an IRQ VM-Exit that has already been acked by the CPU isn't processed untilkvm_x86_ops.handle_exit_irqoff(), well afterPF_VCPUhas been cleared.
Quoted 2 lines
This issue can be 100% reproduced.
https://bugzilla.kernel.org/show_bug.cgi?id=204177
Sorry, the posted link should be
https://bugzilla.kernel.org/show_bug.cgi?id=209831
Wanpeng
Quoted 44 lines · Nitesh Narayan Lal
This reverts commit
d7a08882a0a4b4e176691331ee3f492996579534.After the introduction of the patch:
87fa7f3e9: x86/kvm: Move context tracking where it belongssince we have moved
guest_exit_irqoffcloser to the VM-Exit, explicit enabling of irqs to process pending interrupts should not be required withinvcpu_enter_guestanymore.Conflicts:
arch/x86/kvm/svm.cSigned-off-by Nitesh Narayan Lal <[email protected]>--- arch/x86/kvm/svm/svm.c | 9 +++++++++ arch/x86/kvm/x86.c | 11 ----------- 2 files changed, 9 insertions(+), 11 deletions(-)arch/x86/kvm/svm/svm.c +9
-0@@ -4187,6 +4187,15 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,static void svm_handle_exit_irqoff (struct kvm_vcpu *vcpu) { +kvm_before_interrupt (vcpu); +local_irq_enable (); +/* +* We must have an instruction with interrupts enabled, so +* the timer interrupt isn't delayed by the interrupt shadow. +*/ + asm("nop" ); +local_irq_disable (); +kvm_after_interrupt (vcpu); }static void svm_sched_in (struct kvm_vcpu *vcpu,int cpu)arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9028
Quoted 6 lines
arch/x86/kvm/x86.c +0
-3@@ -9023,18 +9023,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_x86_ops.handle_exit_irqoff (vcpu); -/* -* Consume any pending interrupts, including the possible source of -* VM-Exit on SVM
I kind of liked this part of the comment, the new (old) one in svm_handle_exit_irqoff() doesn't actually explain what's going on.
Quoted 1 line
and any ticks that occur between VM-Exit and now.
Looking back, I don't quite understand why we wanted to account ticks between vmexit and exiting guest context as 'guest' in the first place; to my understanging 'guest time' is time spent within VMX non-root operation, the rest is KVM overhead (system). It seems to match how the accounting is done nowadays after Tglx's 87fa7f3e98a1 ("x86/kvm: Move context tracking where it belongs").
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9029
Quoted 12 lines
arch/x86/kvm/x86.c +0
-8-* An instruction is required after local_irq_enable() to fully unblock -* interrupts on processors that implement an interrupt shadow, the -* stat.exits increment will do nicely. -*/ -kvm_before_interrupt (vcpu); -local_irq_enable (); ++vcpu->stat.exits; -local_irq_disable (); -kvm_after_interrupt (vcpu);if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;
FWIW,
Signature · 1 line
Vitaly
Sean Christopherson
maintainerQuoted 16 lines · Vitaly Kuznetsov
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9028
Quoted 11 lines · Nitesh Narayan Lal
arch/x86/kvm/x86.c +0
-3@@ -9023,18 +9023,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_x86_ops.handle_exit_irqoff (vcpu); -/* -* Consume any pending interrupts, including the possible source of -* VM-Exit on SVM I kind of liked this part of the comment, the new (old) one in
svm_handle_exit_irqoff()doesn't actually explain what's going on.
Quoted 6 lines
Quoted 1 line
and any ticks that occur between VM-Exit and now.
Looking back, I don't quite understand why we wanted to account ticks between vmexit and exiting guest context as 'guest' in the first place; to my understanging 'guest time' is time spent within VMX non-root operation, the rest is KVM overhead (system).
With tick-based accounting, if the tick IRQ is received after PF_VCPU is cleared then that tick will be accounted to the host/system. The motivation for opening an IRQ window after VM-Exit is to handle the case where the guest is constantly exiting for a different reason _just_ before the tick arrives, e.g. if the guest has its tick configured such that the guest and host ticks get synchronized in a bad way.
This is a non-issue when using CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, at least with a stable TSC, as the accounting happens during guest_exit_irqoff() itself. Accounting might be less-than-stellar if TSC is unstable, but I don't think it would be as binary of a failure as tick-based accounting.
Quoted 23 lines
It seems to match how the accounting is done nowadays after Tglx's
87fa7f3e98a1("x86/kvm: Move context tracking where it belongs").arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9029
Quoted 12 lines
arch/x86/kvm/x86.c +0
-8-* An instruction is required after local_irq_enable() to fully unblock -* interrupts on processors that implement an interrupt shadow, the -* stat.exits increment will do nicely. -*/ -kvm_before_interrupt (vcpu); -local_irq_enable (); ++vcpu->stat.exits; -local_irq_disable (); -kvm_after_interrupt (vcpu);if (lapic_in_kernel (vcpu)) {s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;FWIW,
Reviewed-by Vitaly Kuznetsov <[email protected]>Signature · 1 line
Vitaly
Quoted 14 lines · Sean Christopherson
Quoted 5 lines · Vitaly Kuznetsov
Looking back, I don't quite understand why we wanted to account ticks between vmexit and exiting guest context as 'guest' in the first place; to my understanging 'guest time' is time spent within VMX non-root operation, the rest is KVM overhead (system).
With tick-based accounting, if the tick IRQ is received after
PF_VCPUis cleared then that tick will be accounted to the host/system. The motivation for opening an IRQ window after VM-Exit is to handle the case where the guest is constantly exiting for a different reason _just_ before the tick arrives, e.g. if the guest has its tick configured such that the guest and host ticks get synchronized in a bad way.
Quoted 5 lines
This is a non-issue when using
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, at least with a stable TSC, as the accounting happens duringguest_exit_irqoff()itself. Accounting might be less-than-stellar if TSC is unstable, but I don't think it would be as binary of a failure as tick-based accounting.
Oh, yea, I vaguely remember we had to deal with a very similar problem but for userspace/kernel accounting. It was possible to observe e.g. a userspace task going 100% kernel while in reality it was just perfectly synchronized with the tick and doing a syscall just before it arrives (or something like that, I may be misremembering the details).
So depending on the frequency, it is probably possible to e.g observe '100% host' with tick based accounting, the guest just has to synchronize exiting to KVM in a way that the tick will always arrive past guest_exit_irqoff().
It seems to me this is a fundamental problem in case the frequency of guest exits can match the frequency of the time accounting tick.
Signature · 1 line
Vitaly
Quoted 23 lines · Vitaly Kuznetsov
Quoted 14 lines · Sean Christopherson
Quoted 5 lines · Vitaly Kuznetsov
Looking back, I don't quite understand why we wanted to account ticks between vmexit and exiting guest context as 'guest' in the first place; to my understanging 'guest time' is time spent within VMX non-root operation, the rest is KVM overhead (system).
With tick-based accounting, if the tick IRQ is received after
PF_VCPUis cleared then that tick will be accounted to the host/system. The motivation for opening an IRQ window after VM-Exit is to handle the case where the guest is constantly exiting for a different reason _just_ before the tick arrives, e.g. if the guest has its tick configured such that the guest and host ticks get synchronized in a bad way.Quoted 5 lines
This is a non-issue when using
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, at least with a stable TSC, as the accounting happens duringguest_exit_irqoff()itself. Accounting might be less-than-stellar if TSC is unstable, but I don't think it would be as binary of a failure as tick-based accounting.
Quoted 5 lines
Oh, yea, I vaguely remember we had to deal with a very similar problem but for userspace/kernel accounting. It was possible to observe e.g. a userspace task going 100% kernel while in reality it was just perfectly synchronized with the tick and doing a syscall just before it arrives (or something like that, I may be misremembering the details).
Yes. :) commit 2a42eb9594a1 ("sched/cputime: Accumulate vtime on top of nsec clocksource")
Quoted 11 lines
So depending on the frequency, it is probably possible to e.g observe '100% host' with tick based accounting, the guest just has to synchronize exiting to KVM in a way that the tick will always arrive past
guest_exit_irqoff().It seems to me this is a fundamental problem in case the frequency of guest exits can match the frequency of the time accounting tick.
--
Vitaly
Quoted 25 lines · Vitaly Kuznetsov
Quoted 12 lines · Sean Christopherson
Quoted 4 lines · Vitaly Kuznetsov
Looking back, I don't quite understand why we wanted to account ticks between vmexit and exiting guest context as 'guest' in the first place; to my understanging 'guest time' is time spent within VMX non-root operation, the rest is KVM overhead (system).
With tick-based accounting, if the tick IRQ is received after
PF_VCPUis cleared then that tick will be accounted to the host/system. The motivation for opening an IRQ window after VM-Exit is to handle the case where the guest is constantly exiting for a different reason _just_ before the tick arrives, e.g. if the guest has its tick configured such that the guest and host ticks get synchronized in a bad way.Quoted 5 lines
This is a non-issue when using
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, at least with a stable TSC, as the accounting happens duringguest_exit_irqoff()itself. Accounting might be less-than-stellar if TSC is unstable, but I don't think it would be as binary of a failure as tick-based accounting.Oh, yea, I vaguely remember we had to deal with a very similar problem but for userspace/kernel accounting. It was possible to observe e.g. a userspace task going 100% kernel while in reality it was just perfectly synchronized with the tick and doing a syscall just before it arrives (or something like that, I may be misremembering the details).
Quoted 8 lines
So depending on the frequency, it is probably possible to e.g observe '100% host' with tick based accounting, the guest just has to synchronize exiting to KVM in a way that the tick will always arrive past
guest_exit_irqoff().It seems to me this is a fundamental problem in case the frequency of guest exits can match the frequency of the time accounting tick.
Just to make sure that I am understanding things correctly. There are two issues:
1. The first issue is with the tick IRQs that arrive after PF_VCPU is cleared as they are then accounted into the system context atleast on the setup where CONFIG_VIRT_CPU_ACCOUNTING_GEN is not enabled. With the patch "KVM: x86: Unconditionally enable irqs in guest context", we are atleast taking care of the scenario where the guest context is exiting constantly just before the arrival of the tick.
2. The second issue that Sean mentioned was introduced because of moving guest_exit_irqoff() closer to VM-exit. Due to this change, any ticks that happen after IRQs are disabled are incorrectly accounted into the system context. This is because we exit the guest context early without ensuring if the required guest states to handle IRQs are restored.
So, the increase in the system time (reported by cpuacct.stats) that I was observing is not entirely correct after all.
Am I missing anything here?
--
Thanks
Nitesh
Sean Christopherson
maintainerQuoted 36 lines · Nitesh Narayan Lal
Quoted 25 lines · Vitaly Kuznetsov
Quoted 12 lines · Sean Christopherson
Quoted 4 lines · Vitaly Kuznetsov
Looking back, I don't quite understand why we wanted to account ticks between vmexit and exiting guest context as 'guest' in the first place; to my understanging 'guest time' is time spent within VMX non-root operation, the rest is KVM overhead (system).
With tick-based accounting, if the tick IRQ is received after
PF_VCPUis cleared then that tick will be accounted to the host/system. The motivation for opening an IRQ window after VM-Exit is to handle the case where the guest is constantly exiting for a different reason _just_ before the tick arrives, e.g. if the guest has its tick configured such that the guest and host ticks get synchronized in a bad way.Quoted 5 lines
This is a non-issue when using
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, at least with a stable TSC, as the accounting happens duringguest_exit_irqoff()itself. Accounting might be less-than-stellar if TSC is unstable, but I don't think it would be as binary of a failure as tick-based accounting.Oh, yea, I vaguely remember we had to deal with a very similar problem but for userspace/kernel accounting. It was possible to observe e.g. a userspace task going 100% kernel while in reality it was just perfectly synchronized with the tick and doing a syscall just before it arrives (or something like that, I may be misremembering the details).
Quoted 8 lines
So depending on the frequency, it is probably possible to e.g observe '100% host' with tick based accounting, the guest just has to synchronize exiting to KVM in a way that the tick will always arrive past
guest_exit_irqoff().It seems to me this is a fundamental problem in case the frequency of guest exits can match the frequency of the time accounting tick.
Quoted 8 lines
Just to make sure that I am understanding things correctly. There are two issues:
1. The first issue is with the tick IRQs that arrive afterPF_VCPUis cleared as they are then accounted into the system context atleast on the setup whereCONFIG_VIRT_CPU_ACCOUNTING_GENis not enabled. With the patch "KVM: x86: Unconditionally enable irqs in guest context", we are atleast taking care of the scenario where the guest context is exiting constantly just before the arrival of the tick.
Yep.
Quoted 5 lines
2. The second issue that Sean mentioned was introduced because of moving
guest_exit_irqoff()closer to VM-exit. Due to this change, any ticks that happen after IRQs are disabled are incorrectly accounted into the system context. This is because we exit the guest context early without ensuring if the required guest states to handle IRQs are restored.
Yep.
Quoted 2 lines
So, the increase in the system time (reported by cpuacct.stats) that I was observing is not entirely correct after all.
It's correct, but iff CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, as that doesn't rely on ticks and so closer to VM-Enter is better. The problem is that it completely breaks CONFIG_VIRT_CPU_ACCOUNTING_GEN=n (#2 above) because KVM will never service an IRQ, ticks included, with PF_VCPU set.
Quoted 1 line
Am I missing anything here?
Quoted 31 lines · Sean Christopherson
Quoted 16 lines · Vitaly Kuznetsov
arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
Quoted 11 lines · Nitesh Narayan Lal
arch/x86/kvm/x86.c +0
-0@@ -9023,18 +9023,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
kvm_x86_ops.handle_exit_irqoff(vcpu);arch/x86/kvm/x86.c +0
-3-/* -* Consume any pending interrupts, including the possible source of -* VM-Exit on SVM I kind of liked this part of the comment, the new (old) one in
svm_handle_exit_irqoff()doesn't actually explain what's going on.Quoted 6 lines
Quoted 1 line
and any ticks that occur between VM-Exit and now.
Looking back, I don't quite understand why we wanted to account ticks between vmexit and exiting guest context as 'guest' in the first place; to my understanging 'guest time' is time spent within VMX non-root operation, the rest is KVM overhead (system).
With tick-based accounting, if the tick IRQ is received after
PF_VCPUis cleared then that tick will be accounted to the host/system. The motivation for opening an IRQ window after VM-Exit is to handle the case where the guest is constantly exiting for a different reason _just_ before the tick arrives, e.g. if the guest has its tick configured such that the guest and host ticks get synchronized in a bad way.
Quoted 4 lines
This is a non-issue when using
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, at least with a stable TSC, as the accounting happens duringguest_exit_irqoff()itself. Accounting might be less-than-stellar if TSC is unstable, but I don't think it would be as binary of a failure as tick-based accounting.
If I don't specify "nohz_full" in boot command line when using CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, Will the problem still exist?
Quoted 24 lines
Quoted 24 lines
It seems to match how the accounting is done nowadays after Tglx's
87fa7f3e98a1("x86/kvm: Move context tracking where it belongs").Quoted 9 lines
arch/x86/kvm/x86.c +0
-6-* An instruction is required after local_irq_enable() to fully unblock -* interrupts on processors that implement an interrupt shadow, the -* stat.exits increment will do nicely. -*/ -kvm_before_interrupt (vcpu); -local_irq_enable ();++vcpu->stat.exits;
arch/x86/kvm/x86.c +0
-2-local_irq_disable (); -kvm_after_interrupt (vcpu);Quoted 3 lines
if (lapic_in_kernel(vcpu)) {
s64 delta =vcpu->arch.apic->lapic_timer.advance_expire_delta;FWIW,
Reviewed-by Vitaly Kuznetsov <[email protected]>--
Vitaly