cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly
PATCH v2 1/2 cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly
The return value for kernel VM exit handlers is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.
This worked in kvmtool because the exit_reason defaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a VMM exit with error on KVM_EXIT_UNKNOWN exit_reson would help catch these bugs early.
--- kvm-cpu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kvm-cpu.c +1 -1
@@ -170,7 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)switch (cpu->kvm_run->exit_reason) {case KVM_EXIT_UNKNOWN : -break ; + goto panic_kvm;case KVM_EXIT_DEBUG :kvm_cpu__show_registers (cpu);kvm_cpu__show_code (cpu);
Signature · 1 line
2.43.0
PATCH v2 2/2 cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT in KVM_RUN ioctl return
Linux kernel documentation states:
"Note! KVM_EXIT_MEMORY_FAULT is unique among all KVM exit reasons in that it accompanies a return code of '-1', not '0'! errno will always be set to EFAULT or EHWPOISON when KVM exits with KVM_EXIT_MEMORY_FAULT, userspace should assume kvm_run.exit_reason is stale/undefined for all other error numbers." "
Update KVM_RUN ioctl error handling to correctly handle KVM_EXIT_MEMORY_FAULT. This enables the memory fault exit handlers in the kernel to return -EFAULT as the return value. VMM support is still required to handle these memory fault exits, but that is not included in this change
--- kvm-cpu.c | 15 +++++++++++++-- kvm.c | 1 + 2 files changed, 14 insertions(+), 2 deletions(-)
kvm-cpu.c +13 -2
@@ -41,8 +41,19 @@ void kvm_cpu__run(struct kvm_cpu *vcpu)return ; err =ioctl (vcpu->vcpu_fd,KVM_RUN ,0 ); -if (err <0 && (errno !=EINTR && errno !=EAGAIN )) -die_perror ("KVM_RUN failed" ); +if (err <0 ) { +switch (errno) { +case EINTR : +case EAGAIN : +return ; +case EFAULT : +if (vcpu->kvm_run->exit_reason ==KVM_EXIT_MEMORY_FAULT ) +return ; +/* faullthrough */ +default : +die_perror ("KVM_RUN failed" ); + } + } }static void kvm_cpu_signal_handler (int signum)
kvm.c +1 -0
@@ -55,6 +55,7 @@ const char *kvm_exit_reasons[] = {#ifdef CONFIG_PPC64 DEFINE_KVM_EXIT_REASON (KVM_EXIT_PAPR_HCALL ), #endif + DEFINE_KVM_EXIT_REASON(KVM_EXIT_MEMORY_FAULT), };static int pause_event;
Signature · 1 line
2.43.0
Re: [PATCH kvmtool v2 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT in KVM_RUN ioctl return
Hi Aneesh,
Quoted 11 lines · Aneesh Kumar K.V (Arm)
Linux kernel documentation states:
"Note!
KVM_EXIT_MEMORY_FAULTis unique among all KVM exit reasons in that it accompanies a return code of '-1', not '0'! errno will always be set to EFAULT or EHWPOISON when KVM exits withKVM_EXIT_MEMORY_FAULT, userspace should assume kvm_run.exit_reason is stale/undefined for all other error numbers." "Update
KVM_RUNioctl error handling to correctly handleKVM_EXIT_MEMORY_FAULT. This enables the memory fault exit handlers in the kernel to return-EFAULTas the return value. VMM support is
I think this is too much information. Are all architectures that kvmtool supports going to/have implemented forwarding memory faults to userspace this same way? I would think not.
The first paragraph should be enough - kvmtool implements the KVM ABI, it shouldn't care about how that is implemented by KVM.
Other than that, looks correct to me, and more readable (but that might just be my personal bias):
Thanks,
Alex
Quoted 50 lines
still required to handle these memory fault exits, but that is not included in this change
Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 15 +++++++++++++-- kvm.c | 1 + 2 files changed, 14 insertions(+), 2 deletions(-)kvm-cpu.c +13
-2@@ -41,8 +41,19 @@ void kvm_cpu__run(struct kvm_cpu *vcpu)return ; err =ioctl (vcpu->vcpu_fd,KVM_RUN ,0 ); -if (err <0 && (errno !=EINTR && errno !=EAGAIN )) -die_perror ("KVM_RUN failed" ); +if (err <0 ) { +switch (errno) { +case EINTR : +case EAGAIN : +return ; +case EFAULT : +if (vcpu->kvm_run->exit_reason ==KVM_EXIT_MEMORY_FAULT ) +return ; +/* faullthrough */ +default : +die_perror ("KVM_RUN failed" ); + } + } }static void kvm_cpu_signal_handler (int signum)kvm.c +1
-0@@ -55,6 +55,7 @@ const char *kvm_exit_reasons[] = {#ifdef CONFIG_PPC64 DEFINE_KVM_EXIT_REASON (KVM_EXIT_PAPR_HCALL ), #endif + DEFINE_KVM_EXIT_REASON(KVM_EXIT_MEMORY_FAULT), };static int pause_event;Signature · 1 line
2.43.0
Hi Aneesh,
Quoted 10 lines · Aneesh Kumar K.V (Arm)
The return value for kernel VM exit handlers is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.
This worked in kvmtool because the
exit_reasondefaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a VMM exit with error onKVM_EXIT_UNKNOWNexit_resonwould help catch these bugs early.
I think I understand what you're saying - if there's a bug in handle_exit() in KVM that triggers an erroneous exit to userspace, exit_reason and the exit information struct could still be at their default values, which are 0 from when kvm_run was allocated (in kvm_vm_ioctl_create_vcpu()).
An exit_reason of 0 is interpreted by userspace as KVM_EXIT_UNKNOWN, but kvmtool on KVM_EXIT_UNKNOWN resumes the guest instead of signalling the error, thus masking the buggy KVM behaviour.
The patch looks good to me:
Thanks,
Alex
Quoted 22 lines
Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kvm-cpu.c +1
-1@@ -170,7 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)switch (cpu->kvm_run->exit_reason) {case KVM_EXIT_UNKNOWN : -break ; + goto panic_kvm;case KVM_EXIT_DEBUG :kvm_cpu__show_registers (cpu);kvm_cpu__show_code (cpu);Signature · 1 line
2.43.0
Quoted 31 lines · Aneesh Kumar K.V (Arm)
The return value for kernel VM exit handlers is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.
This worked in kvmtool because the
exit_reasondefaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a VMM exit with error onKVM_EXIT_UNKNOWNexit_resonwould help catch these bugs early.Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kvm-cpu.c +1
-1@@ -170,7 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)switch (cpu->kvm_run->exit_reason) {case KVM_EXIT_UNKNOWN : -break ; + goto panic_kvm;case KVM_EXIT_DEBUG :kvm_cpu__show_registers (cpu);kvm_cpu__show_code (cpu);Signature · 1 line
2.43.0
This breaks SMP boot on my x86 machine:
# ./lkvm run
...
[ 0.628472] smp: Bringing up secondary CPUs ...
[ 0.630401] smpboot: x86: Booting SMP configuration:
Error: KVM exit reason: 0 ("KVM_EXIT_UNKNOWN")
Error: KVM exit code: 0 Registers:
----------
rip: 0000000000000000 rsp: 0000000000000000 flags: 0000000000000002 rax: 0000000000000000 rbx: 0000000000000000 rcx: 0000000000000000 rdx: 0000000000050654 rsi: 0000000000000000 rdi: 0000000000000000 rbp: 0000000000000000 r8: 0000000000000000 r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 r15: 0000000000000000 cr0: 0000000060000010 cr2: 0000000000000000 cr3: 0000000000000000 cr4: 0000000000000000 cr8: 0000000000000000
Segment registers:
------------------
register selector base limit type p dpl db s l g avl cs 9900 0000000000099000 0000ffff 0b 1 0 0 1 0 0 0 ss 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0 ds 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0 es 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0 fs 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0 gs 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0 tr 0000 0000000000000000 0000ffff 0b 1 0 0 0 0 0 0 ldt 0000 0000000000000000 0000ffff 02 1 0 0 0 0 0 0 gdt 0000000000000000 0000ffff idt 0000000000000000 0000ffff
APIC:
-----
efer: 0000000000000000 apic base: 00000000fee00800 nmi: enabled
Interrupt bitmap:
-----------------
0000000000000000 0000000000000000 0000000000000000 0000000000000000
Code:
-----
Warning: symbol_lookup() failed to find symbol with error: -2
rip: [<0000000000000000>] <unknown>
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <fa> 0f 09 ea 08 10 00 98 8c c8 8e d8 8e c0 8e d0 66 f0 0f ba 2e
Stack: ------ rsp: [<0000000000000000>] 0x00000000: 30 00 00 f0 30 00 00 f0 0x00000008: 30 00 00 f0 30 00 00 f0 0x00000010: 30 00 00 f0 30 00 00 f0 0x00000018: 30 00 00 f0 30 00 00 f0
Page Tables:
------
Not in protected mode
Quoted 35 lines · Will Deacon
Quoted 31 lines · Aneesh Kumar K.V (Arm)
The return value for kernel VM exit handlers is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.
This worked in kvmtool because the
exit_reasondefaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a VMM exit with error onKVM_EXIT_UNKNOWNexit_resonwould help catch these bugs early.Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kvm-cpu.c +1
-1@@ -170,7 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)switch (cpu->kvm_run->exit_reason) {case KVM_EXIT_UNKNOWN : -break ; + goto panic_kvm;case KVM_EXIT_DEBUG :kvm_cpu__show_registers (cpu);kvm_cpu__show_code (cpu);Signature · 1 line
2.43.0This breaks SMP boot on my x86 machine:
Quoted 7 lines
# ./lkvm run
...
[ 0.628472] smp: Bringing up secondary CPUs ...
[ 0.630401] smpboot: x86: Booting SMP configuration:
Error: KVM exit reason: 0 ("KVM_EXIT_UNKNOWN")
Error: KVM exit code: 0I can recreate this with distro kernel as the guest kernel but not with CONFIG_SMP enabled with kvm_guest.config. I will have to find out what is causing that KVM_EXIT_UNKNOWN on the secondary cpu boot.
-aneesh
Quoted 35 lines · Will Deacon
Quoted 31 lines · Aneesh Kumar K.V (Arm)
The return value for kernel VM exit handlers is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.
This worked in kvmtool because the
exit_reasondefaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a VMM exit with error onKVM_EXIT_UNKNOWNexit_resonwould help catch these bugs early.Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kvm-cpu.c +1
-1@@ -170,7 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)switch (cpu->kvm_run->exit_reason) {case KVM_EXIT_UNKNOWN : -break ; + goto panic_kvm;case KVM_EXIT_DEBUG :kvm_cpu__show_registers (cpu);kvm_cpu__show_code (cpu);Signature · 1 line
2.43.0This breaks SMP boot on my x86 machine:
Quoted 7 lines
# ./lkvm run
...
[ 0.628472] smp: Bringing up secondary CPUs ...
[ 0.630401] smpboot: x86: Booting SMP configuration:
Error: KVM exit reason: 0 ("KVM_EXIT_UNKNOWN")
Error: KVM exit code: 0Turns out we should handle EINTR and EAGAIN as special such that we do an retry of KVM_RUN ioctl without checking the exit_reason. I can send a v3 if you are ok with the change below.
@@ -16,7 +16,7 @@ void kvm_cpu__delete(struct kvm_cpu *vcpu);
void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu);
void kvm_cpu__setup_cpuid(struct kvm_cpu *vcpu);
void kvm_cpu__enable_singlestep(struct kvm_cpu *vcpu);
-void kvm_cpu__run(struct kvm_cpu *vcpu);
+int kvm_cpu__run(struct kvm_cpu *vcpu);
int kvm_cpu__start(struct kvm_cpu *cpu);
bool kvm_cpu__handle_exit(struct kvm_cpu *vcpu);
int kvm_cpu__get_endianness(struct kvm_cpu *vcpu);
modified kvm-cpu.c
@@ -35,27 +35,32 @@ void kvm_cpu__enable_singlestep(struct kvm_cpu *vcpu)
pr_warning("KVM_SET_GUEST_DEBUG failed");
}
-void kvm_cpu__run(struct kvm_cpu *vcpu)
+/*
+ * return value -1 if we need to call the kvm_cpu__run again without checking
+ * exit_reason. return value 0 results in taking action based on exit_reason.
+ */
+int kvm_cpu__run(struct kvm_cpu *vcpu)
{
int err;
if (!vcpu->is_running)
- return;
+ return -1;
err = ioctl(vcpu->vcpu_fd, KVM_RUN, 0);
if (err < 0) {
switch (errno) {
case EINTR:
case EAGAIN:
- return;
+ return -1;
case EFAULT:
if (vcpu->kvm_run->exit_reason == KVM_EXIT_MEMORY_FAULT)
- return;
+ return 0;
/* faullthrough */
default:
die_perror("KVM_RUN failed");
}
}
+ return 0;
}
static void kvm_cpu_signal_handler(int signum)
@@ -179,11 +184,13 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
if (cpu->task)
kvm_cpu__run_task(cpu);
- kvm_cpu__run(cpu);
+ if (kvm_cpu__run(cpu) == -1)
+ /* retry without an exit_reason check */
+ continue;
-aneesh