cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly
PATCH 1/2 cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly
The return value for the KVM_RUN ioctl is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.
This worked in kvmtool because the exit_reason defaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a KVM panic on an unknown exit reason would help catch these bugs early.
--- kvm-cpu.c | 1 + 1 file changed, 1 insertion(+)
kvm-cpu.c +1 -0
@@ -170,6 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)switch (cpu->kvm_run->exit_reason) {case KVM_EXIT_UNKNOWN : + goto panic_kvm;break ;case KVM_EXIT_DEBUG :kvm_cpu__show_registers (cpu);
base-commit: 6d754d01fe2cb366f3b636d8a530f46ccf3b10d8
Signature · 1 line
2.43.0
PATCH 2/2 cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT correctly
Linux kernel documentation states:
"Note! KVM_EXIT_MEMORY_FAULT is unique among all KVM exit reasons in that it accompanies a return code of '-1', not '0'! errno will always be set to EFAULT or EHWPOISON when KVM exits with KVM_EXIT_MEMORY_FAULT, userspace should assume kvm_run.exit_reason is stale/undefined for all other error numbers." "
Update the KVM_RUN ioctl error handling to correctly handle KVM_EXIT_MEMORY_FAULT.
--- kvm-cpu.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kvm-cpu.c +9 -2
@@ -41,8 +41,15 @@ void kvm_cpu__run(struct kvm_cpu *vcpu)return ; err =ioctl (vcpu->vcpu_fd,KVM_RUN ,0 ); -if (err <0 && (errno !=EINTR && errno !=EAGAIN )) -die_perror ("KVM_RUN failed" ); +if (err <0 ) { +if (errno ==EINTR || errno ==EAGAIN ) +return ; +else if (errno ==EFAULT && + vcpu->kvm_run->exit_reason ==KVM_EXIT_MEMORY_FAULT ) +return ; +else +die_perror ("KVM_RUN failed" ); + } }static void kvm_cpu_signal_handler (int signum)
Signature · 1 line
2.43.0
Re: [PATCH kvmtool 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT correctly
Quoted 25 lines · Aneesh Kumar K.V (Arm)
Linux kernel documentation states:
"Note!
KVM_EXIT_MEMORY_FAULTis unique among all KVM exit reasons in that it accompanies a return code of '-1', not '0'! errno will always be set to EFAULT or EHWPOISON when KVM exits withKVM_EXIT_MEMORY_FAULT, userspace should assume kvm_run.exit_reason is stale/undefined for all other error numbers." "Update the
KVM_RUNioctl error handling to correctly handleKVM_EXIT_MEMORY_FAULT.Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)kvm-cpu.c +0
-1@@ -41,8 +41,15 @@ void kvm_cpu__run(struct kvm_cpu *vcpu)return ; err =ioctl (vcpu->vcpu_fd,KVM_RUN ,0 ); -if (err <0 && (errno !=EINTR && errno !=EAGAIN ))
kvm-cpu.c void kvm_cpu__run(struct kvm_cpu *vcpu) · line 52
Quoted 10 lines
kvm-cpu.c +9
-1-die_perror ("KVM_RUN failed" ); +if (err <0 ) { +if (errno ==EINTR || errno ==EAGAIN ) +return ; +else if (errno ==EFAULT && + vcpu->kvm_run->exit_reason ==KVM_EXIT_MEMORY_FAULT ) +return ; +else +die_perror ("KVM_RUN failed" ); + }
Probably cleaner to switch on errno?
Will
Re: [PATCH kvmtool 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT correctly
Quoted 24 lines · Will Deacon
Quoted 23 lines · Aneesh Kumar K.V (Arm)
Linux kernel documentation states:
"Note!
KVM_EXIT_MEMORY_FAULTis unique among all KVM exit reasons in that it accompanies a return code of '-1', not '0'! errno will always be set to EFAULT or EHWPOISON when KVM exits withKVM_EXIT_MEMORY_FAULT, userspace should assume kvm_run.exit_reason is stale/undefined for all other error numbers." "Update the
KVM_RUNioctl error handling to correctly handleKVM_EXIT_MEMORY_FAULT.Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)kvm-cpu.c +0
-0@@ -41,8 +41,15 @@ void kvm_cpu__run(struct kvm_cpu *vcpu)return ;
Quoted 14 lines
kvm-cpu.c void kvm_cpu__run(struct kvm_cpu *vcpu) · line 52
Quoted 12 lines
kvm-cpu.c +9
-2err =ioctl (vcpu->vcpu_fd,KVM_RUN ,0 ); -if (err <0 && (errno !=EINTR && errno !=EAGAIN )) -die_perror ("KVM_RUN failed" ); +if (err <0 ) { +if (errno ==EINTR || errno ==EAGAIN ) +return ; +else if (errno ==EFAULT && + vcpu->kvm_run->exit_reason ==KVM_EXIT_MEMORY_FAULT ) +return ; +else +die_perror ("KVM_RUN failed" ); + }Probably cleaner to switch on errno?
This? . I will update.
if (err < 0) {
switch (errno) {
case EINTR:
case EAGAIN:
return;
case EFAULT:
if (vcpu->kvm_run->exit_reason == KVM_EXIT_MEMORY_FAULT)
return;
/* fallthrough */
default:
die_perror("KVM_RUN failed");
}
}-aneesh
Re: [PATCH kvmtool 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT correctly
In reply to Aneesh Kumar K.V (Arm)Hi,
Quoted 10 lines · Aneesh Kumar K.V (Arm)
Linux kernel documentation states:
"Note!
KVM_EXIT_MEMORY_FAULTis unique among all KVM exit reasons in that it accompanies a return code of '-1', not '0'! errno will always be set to EFAULT or EHWPOISON when KVM exits withKVM_EXIT_MEMORY_FAULT, userspace should assume kvm_run.exit_reason is stale/undefined for all other error numbers." "Update the
KVM_RUNioctl error handling to correctly handleKVM_EXIT_MEMORY_FAULT.
I've tried to follow how kvmtool handles KVM_EXIT_MEMORY_FAULT before and after this patch.
Before: calls die_perror().
After: prints more information about the error, in kvm_cpu_thread().
Is that what you want? Because "correctly handle KVM_EXIT_MEMORY_FAULT" can be interpreted as kvmtool resolving the memory fault, which is something that kvmtool does not do.
Also, can you update kvm_exit_reasons with KVM_EXIT_MEMORY_FAULT, because otherwise kvm_cpu_thread() will segfault when it tries to access kvm_exit_reasons[KVM_EXIT_MEMORY_FAULT].
Thanks,
Alex
Quoted 32 lines
Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)kvm-cpu.c +9
-2@@ -41,8 +41,15 @@ void kvm_cpu__run(struct kvm_cpu *vcpu)return ; err =ioctl (vcpu->vcpu_fd,KVM_RUN ,0 ); -if (err <0 && (errno !=EINTR && errno !=EAGAIN )) -die_perror ("KVM_RUN failed" ); +if (err <0 ) { +if (errno ==EINTR || errno ==EAGAIN ) +return ; +else if (errno ==EFAULT && + vcpu->kvm_run->exit_reason ==KVM_EXIT_MEMORY_FAULT ) +return ; +else +die_perror ("KVM_RUN failed" ); + } }static void kvm_cpu_signal_handler (int signum)Signature · 1 line
2.43.0
Re: [PATCH kvmtool 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT correctly
Quoted 17 lines · Alexandru Elisei
Hi,
Quoted 10 lines · Aneesh Kumar K.V (Arm)
Linux kernel documentation states:
"Note!
KVM_EXIT_MEMORY_FAULTis unique among all KVM exit reasons in that it accompanies a return code of '-1', not '0'! errno will always be set to EFAULT or EHWPOISON when KVM exits withKVM_EXIT_MEMORY_FAULT, userspace should assume kvm_run.exit_reason is stale/undefined for all other error numbers." "Update the
KVM_RUNioctl error handling to correctly handleKVM_EXIT_MEMORY_FAULT.I've tried to follow how kvmtool handles
KVM_EXIT_MEMORY_FAULTbefore and after this patch.
Quoted 7 lines
Before: calls
die_perror().
After: prints more information about the error, inkvm_cpu_thread().Is that what you want? Because "correctly handle
KVM_EXIT_MEMORY_FAULT" can be interpreted as kvmtool resolving the memory fault, which is something that kvmtool does not do.
That is correct. The changes to enable the handling of KVM_EXIT_MEMORY_FAULT is not yet part of upstream [1]. But then the return value for KVM_RUN ioctl() is defined as part of Documentation/virt/kvm/api.rst in the Linux kernel.
[1] https://gitlab.arm.com/linux-arm/kvmtool-cca/-/blob/cca/v4/arm/kvm-cpu.c?ref_type=heads#L247
Quoted 5 lines
Also, can you update
kvm_exit_reasonswithKVM_EXIT_MEMORY_FAULT, because otherwisekvm_cpu_thread()will segfault when it tries to access kvm_exit_reasons[KVM_EXIT_MEMORY_FAULT].
Will update.
-aneesh
Quoted 5 lines · Aneesh Kumar K.V (Arm)
The return value for the
KVM_RUNioctl is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.
Oops. Did any of those broken handlers reach mainline?
Quoted 15 lines
This worked in kvmtool because the
exit_reasondefaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a KVM panic on an unknown exit reason would help catch these bugs early.Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 1 + 1 file changed, 1 insertion(+)kvm-cpu.c +0
-0
kvm-cpu.c int kvm_cpu__start(struct kvm_cpu *cpu) · line 174
Quoted 5 lines
kvm-cpu.c +1
-0@@ -170,6 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)switch (cpu->kvm_run->exit_reason) {case KVM_EXIT_UNKNOWN : + goto panic_kvm;break ;
The break is no longer needed.
Will
Quoted 9 lines · Will Deacon
Quoted 5 lines · Aneesh Kumar K.V (Arm)
The return value for the
KVM_RUNioctl is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.Oops. Did any of those broken handlers reach mainline?
Not that I noticed. We do have patches in review.
https://lore.kernel.org/all/[email protected]
Quoted 16 lines
Quoted 16 lines
This worked in kvmtool because the
exit_reasondefaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a KVM panic on an unknown exit reason would help catch these bugs early.Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 1 + 1 file changed, 1 insertion(+)kvm-cpu.c +0
-0@@ -170,6 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
Quoted 7 lines
kvm-cpu.c int kvm_cpu__start(struct kvm_cpu *cpu) · line 174
Quoted 4 lines
kvm-cpu.c +1
-0switch (cpu->kvm_run->exit_reason) {case KVM_EXIT_UNKNOWN : + goto panic_kvm;break ;The break is no longer needed.
ok.
-aneesh
Hi,
Quoted 5 lines · Aneesh Kumar K.V (Arm)
The return value for the
KVM_RUNioctl is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.
I find this paragraph confusing. KVM_RUN, as per the documentation, returns 0 or -1 (on error). As far as I can tell, at least on arm64, KVM_RUN can never return 1.
Are you referring to the loop in kvm_arch_vcpu_ioctl_run() by any chance? That's the only place I found where a value of 1 from the handlers signifies return to the guest.
Quoted 5 lines
This worked in kvmtool because the
exit_reasondefaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a KVM panic on an unknown exit reason would help catch these bugs early.
I would hope that a VMM cannot force KVM to panic at will, which will bring down the host. Are you referring to kvmtool exiting with an error? That's what the unfortunately named 'panic_kvm' label seems to be doing.
Thanks,
Alex
Quoted 24 lines
Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>--- kvm-cpu.c | 1 + 1 file changed, 1 insertion(+)kvm-cpu.c +1
-0@@ -170,6 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)switch (cpu->kvm_run->exit_reason) {case KVM_EXIT_UNKNOWN : + goto panic_kvm;break ;case KVM_EXIT_DEBUG :kvm_cpu__show_registers (cpu);base-commit:
6d754d01fe2cb366f3b636d8a530f46ccf3b10d8Signature · 1 line
2.43.0
Quoted 9 lines · Alexandru Elisei
Hi,
Quoted 5 lines · Aneesh Kumar K.V (Arm)
The return value for the
KVM_RUNioctl is confusing and has led to errors in different kernel exit handlers. A return value of 0 indicates a return to the VMM, whereas a return value of 1 indicates resuming execution in the guest. Some handlers mistakenly return 0 to force a return to the guest.
Quoted 8 lines
I find this paragraph confusing.
KVM_RUN, as per the documentation, returns 0 or -1 (on error). As far as I can tell, at least on arm64,KVM_RUNcan never return 1.Are you referring to the loop in
kvm_arch_vcpu_ioctl_run()by any chance? That's the only place I found where a value of 1 from the handlers signifies return to the guest.
Yes. I will update the commit message to reflect that. It is the exit handler return value rather than KVM_RUN ioctl return value.
Quoted 11 lines
Quoted 6 lines
This worked in kvmtool because the
exit_reasondefaulted to 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit reason. However, forcing a KVM panic on an unknown exit reason would help catch these bugs early.I would hope that a VMM cannot force KVM to panic at will, which will bring down the host. Are you referring to kvmtool exiting with an error? That's what the unfortunately named '
panic_kvm' label seems to be doing.
yes. I will update the commit message to indicate that for KVM_EXIT_UNKNOWN exit reason, kvmtool will now exit with an error instead of returning to the guest."
-aneesh