Skip to contentSkip to conversation

i386: Don't try to set MSR_KVM_ASYNC_PF_EN if kernel-irqchip=off

PATCH In review Eduardo Habkost 3 participants 8 messages Sep 22, 2020 KVM

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 i386: Don't try to set MSR_KVM_ASYNC_PF_EN if kernel-irqchip=off

This addresses the following crash when running Linux v5.8 with kernel-irqchip=off:

qemu-system-x86_64: error: failed to set MSR 0x4b564d02 to 0x0
qemu-system-x86_64: ../target/i386/kvm.c:2714: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

There is a kernel-side fix for the issue too (kernel commit d831de177217 "KVM: x86: always allow writing '0' to MSR_KVM_ASYNC_PF_EN"), but it's nice to simply not trigger the bug if running an older kernel.

---
 target/i386/kvm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
target/i386/kvm.c +6 -1
index 9efb07e7c83..1492f41349f 100644
@@ -2818,7 +2818,12 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
         kvm_msr_entry_add(cpu, MSR_IA32_TSC, env->tsc);
         kvm_msr_entry_add(cpu, MSR_KVM_SYSTEM_TIME, env->system_time_msr);
         kvm_msr_entry_add(cpu, MSR_KVM_WALL_CLOCK, env->wall_clock_msr);
-        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF)) {
+        /*
+         * Some kernel versions (v5.8) won't let MSR_KVM_ASYNC_PF_EN to be set
+         * at all if kernel-irqchip=off, so don't try to set it in that case.
+         */
+        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF) &&
+            kvm_irqchip_in_kernel()) {
             kvm_msr_entry_add(cpu, MSR_KVM_ASYNC_PF_EN, env->async_pf_en_msr);
         }
         if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_PV_EOI)) {
Signature · 1 line
2.26.2
#1 lore raw
Quoted 25 lines · Eduardo Habkost

This addresses the following crash when running Linux v5.8 with kernel-irqchip=off:

qemu-system-x86_64: error: failed to set MSR 0x4b564d02 to 0x0
qemu-system-x86_64: ../target/i386/kvm.c:2714: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

There is a kernel-side fix for the issue too (kernel commit d831de177217 "KVM: x86: always allow writing '0' to MSR_KVM_ASYNC_PF_EN"), but it's nice to simply not trigger the bug if running an older kernel.

---
 target/i386/kvm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
target/i386/kvm.c +0 -0
index 9efb07e7c83..1492f41349f 100644
@@ -2818,7 +2818,12 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
         kvm_msr_entry_add(cpu, MSR_IA32_TSC, env->tsc);
         kvm_msr_entry_add(cpu, MSR_KVM_SYSTEM_TIME, env->system_time_msr);
         kvm_msr_entry_add(cpu, MSR_KVM_WALL_CLOCK, env->wall_clock_msr);

target/i386/kvm.c static int kvm_put_msrs(X86CPU *cpu, int level) · line 2829

Quoted 10 lines
target/i386/kvm.c +6 -1
-        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF)) {
+        /*
+         * Some kernel versions (v5.8) won't let MSR_KVM_ASYNC_PF_EN to be set
+         * at all if kernel-irqchip=off, so don't try to set it in that case.
+         */
+        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF) &&
+            kvm_irqchip_in_kernel()) {
             kvm_msr_entry_add(cpu, MSR_KVM_ASYNC_PF_EN, env->async_pf_en_msr);
         }
         if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_PV_EOI)) {

I'm not sure kvm_irqchip_in_kernel() was required before we switched to interrupt-based APF (as we were always injecting #PF) but with kernel-5.8+ this should work. We'll need to merge this with

https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg02963.html
(queued by Paolo) and
https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg06196.html
which fixes a bug in it.

as kvm_irqchip_in_kernel() should go around both KVM_FEATURE_ASYNC_PF and KVM_FEATURE_ASYNC_PF_INT I believe.

Signature · 1 line
Vitaly
#2 lore raw
Quoted 33 lines · Vitaly Kuznetsov
Quoted 31 lines · Eduardo Habkost

This addresses the following crash when running Linux v5.8 with kernel-irqchip=off:

qemu-system-x86_64: error: failed to set MSR 0x4b564d02 to 0x0
qemu-system-x86_64: ../target/i386/kvm.c:2714: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

There is a kernel-side fix for the issue too (kernel commit d831de177217 "KVM: x86: always allow writing '0' to MSR_KVM_ASYNC_PF_EN"), but it's nice to simply not trigger the bug if running an older kernel.

---
 target/i386/kvm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
target/i386/kvm.c +5 -1
index 9efb07e7c83..1492f41349f 100644
@@ -2818,7 +2818,12 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
         kvm_msr_entry_add(cpu, MSR_IA32_TSC, env->tsc);
         kvm_msr_entry_add(cpu, MSR_KVM_SYSTEM_TIME, env->system_time_msr);
         kvm_msr_entry_add(cpu, MSR_KVM_WALL_CLOCK, env->wall_clock_msr);
-        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF)) {
+        /*
+         * Some kernel versions (v5.8) won't let MSR_KVM_ASYNC_PF_EN to be set
+         * at all if kernel-irqchip=off, so don't try to set it in that case.
+         */
+        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF) &&
Quoted 8 lines

target/i386/kvm.c static int kvm_put_msrs(X86CPU *cpu, int level) · line 2829

Quoted 4 lines
target/i386/kvm.c +1 -0
+            kvm_irqchip_in_kernel()) {
             kvm_msr_entry_add(cpu, MSR_KVM_ASYNC_PF_EN, env->async_pf_en_msr);
         }
         if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_PV_EOI)) {

I'm not sure kvm_irqchip_in_kernel() was required before we switched to interrupt-based APF (as we were always injecting #PF) but with kernel-5.8+ this should work. [...]

Were guests able to set MSR_KVM_ASYNC_PF_EN to non-zero with kernel-irqchip=off on hosts running Linux <= 5.7? I am assuming kvm-asyncpf never worked with kernel-irqchip=off (and enabling it by default with kernel-irqchip=off was a mistake).

Quoted 9 lines

[...] We'll need to merge this with

https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg02963.html
(queued by Paolo) and
https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg06196.html
which fixes a bug in it.

as kvm_irqchip_in_kernel() should go around both KVM_FEATURE_ASYNC_PF and KVM_FEATURE_ASYNC_PF_INT I believe.

Shouldn't we just disallow kvm-asyncpf-int=on if kernel-irqchip=off?

Signature · 1 line
Eduardo
#3 lore raw
Quoted 43 lines · Eduardo Habkost
Quoted 33 lines · Vitaly Kuznetsov
Quoted 31 lines · Eduardo Habkost

This addresses the following crash when running Linux v5.8 with kernel-irqchip=off:

qemu-system-x86_64: error: failed to set MSR 0x4b564d02 to 0x0
qemu-system-x86_64: ../target/i386/kvm.c:2714: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

There is a kernel-side fix for the issue too (kernel commit d831de177217 "KVM: x86: always allow writing '0' to MSR_KVM_ASYNC_PF_EN"), but it's nice to simply not trigger the bug if running an older kernel.

---
 target/i386/kvm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
target/i386/kvm.c +5 -1
index 9efb07e7c83..1492f41349f 100644
@@ -2818,7 +2818,12 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
         kvm_msr_entry_add(cpu, MSR_IA32_TSC, env->tsc);
         kvm_msr_entry_add(cpu, MSR_KVM_SYSTEM_TIME, env->system_time_msr);
         kvm_msr_entry_add(cpu, MSR_KVM_WALL_CLOCK, env->wall_clock_msr);
-        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF)) {
+        /*
+         * Some kernel versions (v5.8) won't let MSR_KVM_ASYNC_PF_EN to be set
+         * at all if kernel-irqchip=off, so don't try to set it in that case.
+         */
+        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF) &&
Quoted 8 lines

target/i386/kvm.c static int kvm_put_msrs(X86CPU *cpu, int level) · line 2829

Quoted 4 lines
target/i386/kvm.c +1 -0
+            kvm_irqchip_in_kernel()) {
             kvm_msr_entry_add(cpu, MSR_KVM_ASYNC_PF_EN, env->async_pf_en_msr);
         }
         if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_PV_EOI)) {

I'm not sure kvm_irqchip_in_kernel() was required before we switched to interrupt-based APF (as we were always injecting #PF) but with kernel-5.8+ this should work. [...]

Quoted 2 lines

Were guests able to set MSR_KVM_ASYNC_PF_EN to non-zero with kernel-irqchip=off on hosts running Linux <= 5.7?

lapic_in_kernel() check appeared in kernel with the following commit:

commit 9d3c447c72fb2337ca39f245c6ae89f2369de216

Author: Wanpeng Li <[email protected]>
Date:   Mon Jun 29 18:26:31 2020 +0800

KVM: X86: Fix async pf caused null-ptr-deref

which was post-interrupt-based-APF. I *think* it was OK to enable APF with !lapic_in_kernel() before (at least I don't see what would not allow that).

Quoted 12 lines

I am assuming
kvm-asyncpf never worked with kernel-irqchip=off (and enabling it by default with kernel-irqchip=off was a mistake).

Quoted 7 lines

[...] We'll need to merge this with

https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg02963.html
(queued by Paolo) and
https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg06196.html
which fixes a bug in it.

Quoted 4 lines
Quoted 2 lines

as kvm_irqchip_in_kernel() should go around both KVM_FEATURE_ASYNC_PF and KVM_FEATURE_ASYNC_PF_INT I believe.

Shouldn't we just disallow kvm-asyncpf-int=on if kernel-irqchip=off?

(Sarcasm: if disallowing 'kernel-irqchip=off' is not an option, then) yes, we probably can, but kvm-asyncpf-int=on is the default we have so we can't just implicitly change it underneath or migration will break...

Signature · 1 line
Vitaly
#4 lore raw
Quoted 51 lines · Vitaly Kuznetsov
Quoted 43 lines · Eduardo Habkost
Quoted 33 lines · Vitaly Kuznetsov
Quoted 31 lines · Eduardo Habkost

This addresses the following crash when running Linux v5.8 with kernel-irqchip=off:

qemu-system-x86_64: error: failed to set MSR 0x4b564d02 to 0x0
qemu-system-x86_64: ../target/i386/kvm.c:2714: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

There is a kernel-side fix for the issue too (kernel commit d831de177217 "KVM: x86: always allow writing '0' to MSR_KVM_ASYNC_PF_EN"), but it's nice to simply not trigger the bug if running an older kernel.

---
 target/i386/kvm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
target/i386/kvm.c +5 -1
index 9efb07e7c83..1492f41349f 100644
@@ -2818,7 +2818,12 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
         kvm_msr_entry_add(cpu, MSR_IA32_TSC, env->tsc);
         kvm_msr_entry_add(cpu, MSR_KVM_SYSTEM_TIME, env->system_time_msr);
         kvm_msr_entry_add(cpu, MSR_KVM_WALL_CLOCK, env->wall_clock_msr);
-        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF)) {
+        /*
+         * Some kernel versions (v5.8) won't let MSR_KVM_ASYNC_PF_EN to be set
+         * at all if kernel-irqchip=off, so don't try to set it in that case.
+         */
+        if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_ASYNC_PF) &&
Quoted 8 lines

target/i386/kvm.c static int kvm_put_msrs(X86CPU *cpu, int level) · line 2829

Quoted 4 lines
target/i386/kvm.c +1 -0
+            kvm_irqchip_in_kernel()) {
             kvm_msr_entry_add(cpu, MSR_KVM_ASYNC_PF_EN, env->async_pf_en_msr);
         }
         if (env->features[FEAT_KVM] & (1 << KVM_FEATURE_PV_EOI)) {

I'm not sure kvm_irqchip_in_kernel() was required before we switched to interrupt-based APF (as we were always injecting #PF) but with kernel-5.8+ this should work. [...]

Quoted 2 lines

Were guests able to set MSR_KVM_ASYNC_PF_EN to non-zero with kernel-irqchip=off on hosts running Linux <= 5.7?

lapic_in_kernel() check appeared in kernel with the following commit:

commit 9d3c447c72fb2337ca39f245c6ae89f2369de216

Quoted 8 lines
Author: Wanpeng Li <[email protected]>
Date:   Mon Jun 29 18:26:31 2020 +0800

KVM: X86: Fix async pf caused null-ptr-deref

which was post-interrupt-based-APF. I *think* it was OK to enable APF with !lapic_in_kernel() before (at least I don't see what would not allow that).

If it was possible, did KVM break live migration of kernel-irqchip=off guests that enabled APF? This would mean my patch is replacing a crash with a silent migration bug. Not nice either way.

Quoted 13 lines
Quoted 13 lines

I am assuming
kvm-asyncpf never worked with kernel-irqchip=off (and enabling it by default with kernel-irqchip=off was a mistake).

Quoted 7 lines

[...] We'll need to merge this with

https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg02963.html
(queued by Paolo) and
https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg06196.html
which fixes a bug in it.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

as kvm_irqchip_in_kernel() should go around both KVM_FEATURE_ASYNC_PF and KVM_FEATURE_ASYNC_PF_INT I believe.

Shouldn't we just disallow kvm-asyncpf-int=on if kernel-irqchip=off?

(Sarcasm: if disallowing 'kernel-irqchip=off' is not an option, then)

I'm working on it. :-)

Quoted 2 lines

yes, we probably can, but kvm-asyncpf-int=on is the default we have so we can't just implicitly change it underneath or migration will break...

kvm-asyncpf-int wasn't merged yet, was it? This means we don't have compatibility issues to care about.

Signature · 1 line
Eduardo
maintainer
#5 lore raw
Quoted 4 lines · Eduardo Habkost

If it was possible, did KVM break live migration of kernel-irqchip=off guests that enabled APF? This would mean my patch is replacing a crash with a silent migration bug. Not nice either way.

Let's drop kernel-irqchip=off completely so migration is not broken. :)

I'm actually serious, I don't think we need a deprecation period even.

Paolo

#6 lore raw
Quoted 9 lines · Paolo Bonzini
Quoted 4 lines · Eduardo Habkost

If it was possible, did KVM break live migration of kernel-irqchip=off guests that enabled APF? This would mean my patch is replacing a crash with a silent migration bug. Not nice either way.

Let's drop kernel-irqchip=off completely so migration is not broken. :)

I'm actually serious, I don't think we need a deprecation period even.

I wasn't sure about this, but then I've noticed the man page says "disabling the in-kernel irqchip completely is not recommended except for debugging purposes."

Does this note apply to all architectures?

Signature · 1 line
Eduardo
maintainer
#7 lore raw

Il mar 22 set 2020, 21:39 Eduardo Habkost <[email protected]> ha scritto:

Quoted 9 lines · Paolo Bonzini
Quoted 8 lines · Eduardo Habkost
Quoted 4 lines

If it was possible, did KVM break live migration of kernel-irqchip=off guests that enabled APF? This would mean my patch is replacing a crash with a silent migration bug. Not nice either way.

Let's drop kernel-irqchip=off completely so migration is not broken. :)

Quoted 8 lines
Quoted 1 line

I'm actually serious, I don't think we need a deprecation period even.

I wasn't sure about this, but then I've noticed the man page says "disabling the in-kernel irqchip completely is not recommended except for debugging purposes."

Does this note apply to all architectures?

At least on ARM, Raspberry Pi 4 cannot use kernel-irqchip=on at all. But remember I am only talking about x86.

Paolo

Quoted 4 lines

--
Eduardo