Skip to contentSkip to conversation

x86/kvm: fix interrupts based APF mechanism

PATCH Applied Vitaly Kuznetsov 3 participants 6 messages Sep 8, 2020 – Sep 12, 2020

Subsystems (3)
lore
Share & tools
lore

Vitaly Kuznetsov

author reviewer
#0 lore raw

cover x86/kvm: fix interrupts based APF mechanism

Linux-5.9 switched KVM guests to interrupt based APF mechanism for 'page ready' events (instead of the previously used '#PF' exception) but a collision with the newly introduced IDTENTRY magic happened and it wasn't properly resolved. QEMU doesn't currently enable KVM_FEATURE_ASYNC_PF_INT bit so the breakage is invisible but all KVM guests will hang as soon as the bit will get exposed.

Vitaly Kuznetsov (2):
x86/kvm: properly use DEFINE_IDTENTRY_SYSVEC() macro x86/kvm: don't forget to ACK async PF IRQ

 arch/x86/kernel/kvm.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
Signature · 1 line
2.25.4
author reviewer
#1 lore raw

PATCH 1/2 x86/kvm: properly use DEFINE_IDTENTRY_SYSVEC() macro

DEFINE_IDTENTRY_SYSVEC() already contains irqentry_enter()/ irqentry_exit().

---
 arch/x86/kernel/kvm.c | 4 ----
 1 file changed, 4 deletions(-)
arch/x86/kernel/kvm.c +0 -4
index 08320b0b2b27..d45f34cbe1ef 100644
@@ -270,9 +270,6 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_kvm_asyncpf_interrupt)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 	u32 token;
-	irqentry_state_t state;
-
-	state = irqentry_enter(regs);
 
 	inc_irq_stat(irq_hv_callback_count);
 
@@ -283,7 +280,6 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_kvm_asyncpf_interrupt)
 		wrmsrl(MSR_KVM_ASYNC_PF_ACK, 1);
 	}
 
-	irqentry_exit(regs, state);
 	set_irq_regs(old_regs);
 }
 
Signature · 1 line
2.25.4
author reviewer
#2 lore raw

PATCH 2/2 x86/kvm: don't forget to ACK async PF IRQ

In reply to Vitaly Kuznetsov

Merge commit 26d05b368a5c0 ("Merge branch 'kvm-async-pf-int' into HEAD") tried to adapt the new interrupt based async PF mechanism to the newly introduced IDTENTRY magic but unfortunately it missed the fact that DEFINE_IDTENTRY_SYSVEC() doesn't call ack_APIC_irq() on its own and all DEFINE_IDTENTRY_SYSVEC() users have to call it manually.

As the result all multi-CPU KVM guest hang on boot when KVM_FEATURE_ASYNC_PF_INT is present. The breakage went unnoticed because no KVM userspace (e.g. QEMU) currently set it (and thus async PF mechanism is currently disabled) but we're about to change that.

Fixes 26d05b368a5c0 ("Merge branch 'kvm-async-pf-int' into HEAD")
---
 arch/x86/kernel/kvm.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kernel/kvm.c +2 -0
index d45f34cbe1ef..9663ba31347c 100644
@@ -271,6 +271,8 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_kvm_asyncpf_interrupt)
 	struct pt_regs *old_regs = set_irq_regs(regs);
 	u32 token;
 
+	ack_APIC_irq();
+
 	inc_irq_stat(irq_hv_callback_count);
 
 	if (__this_cpu_read(apf_reason.enabled)) {
Signature · 1 line
2.25.4
Tested-by on v1 2/2
#3 lore raw

Re: [PATCH 2/2] x86/kvm: don't forget to ACK async PF IRQ

Quoted 6 lines · Vitaly Kuznetsov

Merge commit 26d05b368a5c0 ("Merge branch 'kvm-async-pf-int' into HEAD") tried to adapt the new interrupt based async PF mechanism to the newly introduced IDTENTRY magic but unfortunately it missed the fact that DEFINE_IDTENTRY_SYSVEC() doesn't call ack_APIC_irq() on its own and all DEFINE_IDTENTRY_SYSVEC() users have to call it manually.

Quoted 7 lines

As the result all multi-CPU KVM guest hang on boot when KVM_FEATURE_ASYNC_PF_INT is present. The breakage went unnoticed because no KVM userspace (e.g. QEMU) currently set it (and thus async PF mechanism is currently disabled) but we're about to change that.

Fixes 26d05b368a5c0 ("Merge branch 'kvm-async-pf-int' into HEAD")

This also fixes a kvmtool regression, but interestingly it does not set KVM_FEATURE_ASYNC_PF_INT either AFAICS:

kepler:~/kvmtool.git> git grep KVM_FEATURE_ASYNC_PF_INT kepler:~/kvmtool.git>

kepler:~/kvmtool.git> grep url .git/config
url = https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git

So either I missed the flag-setting in the kvmtools.git source, or maybe there's some other way to trigger this bug?

Anyway, please handle this as a v5.9 regression:

Thanks,

Ingo

author reviewer
#4 lore raw

Re: [PATCH 2/2] x86/kvm: don't forget to ACK async PF IRQ

Quoted 13 lines · Ingo Molnar
Quoted 11 lines · Vitaly Kuznetsov

Merge commit 26d05b368a5c0 ("Merge branch 'kvm-async-pf-int' into HEAD") tried to adapt the new interrupt based async PF mechanism to the newly introduced IDTENTRY magic but unfortunately it missed the fact that DEFINE_IDTENTRY_SYSVEC() doesn't call ack_APIC_irq() on its own and all DEFINE_IDTENTRY_SYSVEC() users have to call it manually.

As the result all multi-CPU KVM guest hang on boot when KVM_FEATURE_ASYNC_PF_INT is present. The breakage went unnoticed because no KVM userspace (e.g. QEMU) currently set it (and thus async PF mechanism is currently disabled) but we're about to change that.

Quoted 8 lines
Quoted 2 lines
Fixes 26d05b368a5c0 ("Merge branch 'kvm-async-pf-int' into HEAD")

This also fixes a kvmtool regression, but interestingly it does not set KVM_FEATURE_ASYNC_PF_INT either AFAICS:

kepler:~/kvmtool.git> git grep KVM_FEATURE_ASYNC_PF_INT kepler:~/kvmtool.git>

My wild guess would be that kvmtool doesn't manually set any of the KVM PV features:

[vitty@vitty kvmtool]$ git grep KVM_FEATURE_
[vitty@vitty kvmtool]$

it just blindly passes whatever it gets from KVM via KVM_GET_SUPPORTED_CPUID to KVM_SET_CPUID2 and KVM_FEATURE_ASYNC_PF_INT among other PV features is set there by default.

Quoted 10 lines

kepler:~/kvmtool.git> grep url .git/config
url = https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git

So either I missed the flag-setting in the kvmtools.git source, or maybe there's some other way to trigger this bug?

Anyway, please handle this as a v5.9 regression:

Thanks!

Signature · 1 line
Vitaly
maintainer Applied
#5 lore raw

Re: [PATCH 2/2] x86/kvm: don't forget to ACK async PF IRQ

Quoted 38 lines · Vitaly Kuznetsov
Quoted 13 lines · Ingo Molnar
Quoted 11 lines · Vitaly Kuznetsov

Merge commit 26d05b368a5c0 ("Merge branch 'kvm-async-pf-int' into HEAD") tried to adapt the new interrupt based async PF mechanism to the newly introduced IDTENTRY magic but unfortunately it missed the fact that DEFINE_IDTENTRY_SYSVEC() doesn't call ack_APIC_irq() on its own and all DEFINE_IDTENTRY_SYSVEC() users have to call it manually.

As the result all multi-CPU KVM guest hang on boot when KVM_FEATURE_ASYNC_PF_INT is present. The breakage went unnoticed because no KVM userspace (e.g. QEMU) currently set it (and thus async PF mechanism is currently disabled) but we're about to change that.

Quoted 8 lines
Quoted 2 lines
Fixes 26d05b368a5c0 ("Merge branch 'kvm-async-pf-int' into HEAD")

This also fixes a kvmtool regression, but interestingly it does not set KVM_FEATURE_ASYNC_PF_INT either AFAICS:

kepler:~/kvmtool.git> git grep KVM_FEATURE_ASYNC_PF_INT kepler:~/kvmtool.git>

My wild guess would be that kvmtool doesn't manually set any of the KVM PV features:

[vitty@vitty kvmtool]$ git grep KVM_FEATURE_
[vitty@vitty kvmtool]$

it just blindly passes whatever it gets from KVM via KVM_GET_SUPPORTED_CPUID to KVM_SET_CPUID2 and KVM_FEATURE_ASYNC_PF_INT among other PV features is set there by default.

Quoted 4 lines

kepler:~/kvmtool.git> grep url .git/config
url = https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git

Quoted 9 lines
Quoted 6 lines

So either I missed the flag-setting in the kvmtools.git source, or maybe there's some other way to trigger this bug?

Anyway, please handle this as a v5.9 regression:

Thanks!

Queued, thanks.

Paolo