Hypercall UCALL for guest/userspace communication
Subsystems (6)
cover Hypercall UCALL for guest/userspace communication
This RFC is to add a hypercall UCALL so that guest can communicate with the userspace hypervisor. A guest may want to do this in different scenarios. For example, a guest can ask the userspace hypervisor to harden security by setting restricted access permissions on guest SLAT entries.
Please note this hypercall provides the infrastructure for this type of communication but does not enforce the protocol. A proposal for the guest/userspace communication protocol will follow if this RFC is accepted.
Forrest Yuan Yu (1):
KVM: x86: add KVM_HC_UCALL hypercall
Documentation/virt/kvm/api.rst | 15 +- Documentation/virt/kvm/cpuid.rst | 3 + Documentation/virt/kvm/hypercalls.rst | 14 ++ arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/uapi/asm/kvm_para.h | 1 + arch/x86/kvm/x86.c | 39 +++- include/uapi/linux/kvm.h | 1 + include/uapi/linux/kvm_para.h | 1 + tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/x86_64/hypercall_ucall.c | 195 ++++++++++++++++++ 11 files changed, 264 insertions(+), 8 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/hypercall_ucall.c
Signature · 1 line
2.26.2.526.g744177e7f7-goog
RFC PATCH 1/1 KVM: x86: add KVM_HC_UCALL hypercall
The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send a message to the hypervisor which will do its job and send back anything it wants the guest to know.
--- Documentation/virt/kvm/api.rst | 15 +- Documentation/virt/kvm/cpuid.rst | 3 + Documentation/virt/kvm/hypercalls.rst | 14 ++ arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/uapi/asm/kvm_para.h | 1 + arch/x86/kvm/x86.c | 39 +++- include/uapi/linux/kvm.h | 1 + include/uapi/linux/kvm_para.h | 1 + tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/x86_64/hypercall_ucall.c | 195 ++++++++++++++++++ 11 files changed, 264 insertions(+), 8 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/hypercall_ucall.c
Documentation/virt/kvm/api.rst +13 -2
@@ -4854,8 +4854,8 @@ to the byte array. .. note:: - For KVM_EXIT_IO, KVM_EXIT_MMIO, KVM_EXIT_OSI, KVM_EXIT_PAPR and - KVM_EXIT_EPR the corresponding + For KVM_EXIT_IO, KVM_EXIT_MMIO, KVM_EXIT_HYPERCALL, KVM_EXIT_OSI, + KVM_EXIT_PAPR and KVM_EXIT_EPR the corresponding operations are complete (and guest state is consistent) only after userspace has re-entered the kernel with KVM_RUN. The kernel side will first finish
@@ -5802,6 +5802,17 @@ If present, this capability can be enabled for a VM, meaning that KVM will allow the transition to secure guest mode. Otherwise KVM will veto the transition. +7.20 KVM_CAP_UCALL +------------------------------ + +:Architectures: x86 + +This capability indicates that KVM supports hypercall ucall. It being enabled +means userspace is ready to receive a message sent by a guest using hypercall +ucall. When it is not enabled, a hypercall ucall made by a guest will not cause +control to be handed to userspace. Instead, kvm will return -KVM_ENOSYS without +userspace participation. + 8. Other capabilities. ======================
Documentation/virt/kvm/cpuid.rst +3 -0
@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit before using paravirtualized sched yield. +KVM_FEATURE_UCALL 14 guest checks this feature bit + before calling hypercall ucall. + KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24 host will warn if no guest-side per-cpu warps are expeced in kvmclock
Documentation/virt/kvm/hypercalls.rst +14 -0
@@ -169,3 +169,17 @@ a0: destination APIC ID :Usage example: When sending a call-function IPI-many to vCPUs, yield if any of the IPI target vCPUs was preempted. + +8. KVM_HC_UCALL +--------------------- + +:Architecture: x86 +:Status: active +:Purpose: Hypercall used to exchange message between VM and hypervisor. + +a0: message type + +a1, a2, a3: dependent on message type + +:Usage example: A guest asks hypervisor to harden security by setting +restricted access permission on guest SLAT entry.
arch/x86/include/asm/kvm_host.h +1 -0
@@ -979,6 +979,7 @@ struct kvm_arch {bool guest_can_read_msr_platform_info;bool exception_payload_enabled; +bool hypercall_ucall_enabled;struct kvm_pmu_event_filter *pmu_event_filter;struct task_struct *nx_lpage_recovery_thread;
arch/x86/include/uapi/asm/kvm_para.h +1 -0
@@ -31,6 +31,7 @@#define KVM_FEATURE_PV_SEND_IPI 11#define KVM_FEATURE_POLL_CONTROL 12#define KVM_FEATURE_PV_SCHED_YIELD 13 +#define KVM_FEATURE_UCALL 14#define KVM_HINTS_REALTIME 0
arch/x86/kvm/x86.c +33 -6
@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_GET_MSR_FEATURES :case KVM_CAP_MSR_PLATFORM_INFO :case KVM_CAP_EXCEPTION_PAYLOAD : +case KVM_CAP_UCALL : r =1 ;break ;case KVM_CAP_SYNC_REGS :
@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, kvm->arch.exception_payload_enabled = cap->args[0 ]; r =0 ;break ; +case KVM_CAP_UCALL : + kvm->arch.hypercall_ucall_enabled = cap->args[0 ]; + r =0 ; +break ;default : r = -EINVAL ;break ;
@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)kvm_vcpu_yield_to (target); } +static int complete_hypercall (struct kvm_vcpu *vcpu) +{ +u64 ret = vcpu->run->hypercall.ret; + +if (!is_64_bit_mode (vcpu)) + ret = (u32 )ret; +kvm_rax_write (vcpu, ret); + + ++vcpu->stat.hypercalls; + +return kvm_skip_emulated_instruction (vcpu); +} +int kvm_emulate_hypercall (struct kvm_vcpu *vcpu) {unsigned long nr, a0, a1, a2, a3, ret;
@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)kvm_sched_yield (vcpu->kvm, a0); ret =0 ;break ; +case KVM_HC_UCALL : +if (vcpu->kvm->arch.hypercall_ucall_enabled) { + vcpu->run->hypercall.nr = nr; + vcpu->run->hypercall.args[0 ] = a0; + vcpu->run->hypercall.args[1 ] = a1; + vcpu->run->hypercall.args[2 ] = a2; + vcpu->run->hypercall.args[3 ] = a3; + vcpu->run->exit_reason =KVM_EXIT_HYPERCALL ; + vcpu->arch.complete_userspace_io = complete_hypercall; +return 0 ;// message is going to userspace + } + ret = -KVM_ENOSYS ; +break ;default : ret = -KVM_ENOSYS ;break ; } out: -if (!op_64_bit) - ret = (u32 )ret; -kvm_rax_write (vcpu, ret); - - ++vcpu->stat.hypercalls; -return kvm_skip_emulated_instruction (vcpu); + vcpu->run->hypercall.ret = ret; +return complete_hypercall (vcpu); }EXPORT_SYMBOL_GPL (kvm_emulate_hypercall);
include/uapi/linux/kvm.h +1 -0
@@ -1017,6 +1017,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_S390_VCPU_RESETS 179#define KVM_CAP_S390_PROTECTED 180#define KVM_CAP_PPC_SECURE_GUEST 181 +#define KVM_CAP_UCALL 182#ifdef KVM_CAP_IRQ_ROUTING
include/uapi/linux/kvm_para.h +1 -0
@@ -29,6 +29,7 @@#define KVM_HC_CLOCK_PAIRING 9#define KVM_HC_SEND_IPI 10#define KVM_HC_SCHED_YIELD 11 +#define KVM_HC_UCALL 12/* * hypercalls use architecture specific
tools/testing/selftests/kvm/.gitignore +1 -0
@@ -18,6 +18,7 @@ /x86_64/vmx_set_nested_state_test /x86_64/vmx_tsc_adjust_test /x86_64/xss_msr_test +/x86_64/hypercall_ucall /clear_dirty_log_test /demand_paging_test /dirty_log_test
tools/testing/selftests/kvm/Makefile +1 -0
@@ -33,6 +33,7 @@ TEST_GEN_PROGS_x86_64 += demand_paging_test TEST_GEN_PROGS_x86_64 += dirty_log_test TEST_GEN_PROGS_x86_64 += kvm_create_max_vcpus TEST_GEN_PROGS_x86_64 += steal_time +TEST_GEN_PROGS_x86_64 += x86_64/hypercall_ucall TEST_GEN_PROGS_aarch64 += clear_dirty_log_test TEST_GEN_PROGS_aarch64 += demand_paging_test
tools/testing/selftests/kvm/x86_64/hypercall_ucall.c new file +195 -0
195 lines not on this page · raw message
Signature · 1 line
2.26.2.526.g744177e7f7-goog
Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall
Quoted 4 lines · Forrest Yuan Yu
The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send
Please do s/SLAT/TDP everywhere. IMO, TDP is a less than stellar acronym, e.g. what will we do if we go to three dimensions!?!? But, we're stuck with it :-)
Quoted 2 lines
a message to the hypervisor which will do its job and send back anything it wants the guest to know.
Hrm, so this reintroduces KVM_EXIT_HYPERCALL without justifying _why_ it needs to be reintroduced. I'm not familiar with the history, but the comments in the documentation advise "use KVM_EXIT_IO or KVM_EXIT_MMIO".
Off the top of my head, IO and/or MMIO has a few advantages:
- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded to
L1.
- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.
Documentation/virt/kvm/cpuid.rst KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit · line 90
Quoted 12 lines
Signed-off-by Forrest Yuan Yu <[email protected]>---
Documentation/virt/kvm/cpuid.rst +2
-0@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit before using paravirtualized sched yield. +KVM_FEATURE_UCALL 14 guest checks this feature bit + before calling hypercall ucall.
Why make the UCALL a KVM CPUID feature? I can understand wanting to query KVM support from host userspace, but presumably the guest will care about capabiliteis built on top of the UCALL, not the UCALL itself.
arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 3388
Quoted 13 lines
Documentation/virt/kvm/cpuid.rst +1
-0+ KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24 host will warn if no guest-side per-cpu warps are expeced in kvmclockarch/x86/kvm/x86.c +1
-0@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_GET_MSR_FEATURES :case KVM_CAP_MSR_PLATFORM_INFO :case KVM_CAP_EXCEPTION_PAYLOAD : +case KVM_CAP_UCALL :
For whatever reason I have a metnal block with UCALL, can we call this KVM_CAP_USERSPACE_HYPERCALL
Quoted 15 lines
arch/x86/kvm/x86.c +4
-0r =1 ;break ;case KVM_CAP_SYNC_REGS :@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, kvm->arch.exception_payload_enabled = cap->args[0 ]; r =0 ;break ; +case KVM_CAP_UCALL : + kvm->arch.hypercall_ucall_enabled = cap->args[0 ]; + r =0 ; +break ;default : r = -EINVAL ;break ;
arch/x86/kvm/x86.c static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id) · line 7566
Quoted 8 lines
arch/x86/kvm/x86.c +5
-0@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)kvm_vcpu_yield_to (target); } +static int complete_hypercall (struct kvm_vcpu *vcpu) +{ +u64 ret = vcpu->run->hypercall.ret; + +if (!is_64_bit_mode (vcpu))
Do we really anticipate adding support in 32-bit guests? Honest question.
Quoted 12 lines
arch/x86/kvm/x86.c +8
-0+ ret = (u32 )ret; +kvm_rax_write (vcpu, ret); + + ++vcpu->stat.hypercalls; + +return kvm_skip_emulated_instruction (vcpu); +} +int kvm_emulate_hypercall (struct kvm_vcpu *vcpu) {unsigned long nr, a0, a1, a2, a3, ret;
arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) · line 7632
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)kvm_sched_yield (vcpu->kvm, a0); ret =0 ;break ; +case KVM_HC_UCALL : +if (vcpu->kvm->arch.hypercall_ucall_enabled) { + vcpu->run->hypercall.nr = nr; + vcpu->run->hypercall.args[0 ] = a0; + vcpu->run->hypercall.args[1 ] = a1; + vcpu->run->hypercall.args[2 ] = a2; + vcpu->run->hypercall.args[3 ] = a3;
If performance is a justification for a more direct userspace exit, why limit it to just four parameters? E.g. why not copy all registers to kvm_sync_regs and reverse the process on the way back in?
Quoted 21 lines
arch/x86/kvm/x86.c +8
-6+ vcpu->run->exit_reason =KVM_EXIT_HYPERCALL ; + vcpu->arch.complete_userspace_io = complete_hypercall; +return 0 ;// message is going to userspace + } + ret = -KVM_ENOSYS ; +break ;default : ret = -KVM_ENOSYS ;break ; } out: -if (!op_64_bit) - ret = (u32 )ret; -kvm_rax_write (vcpu, ret); - - ++vcpu->stat.hypercalls; -return kvm_skip_emulated_instruction (vcpu); + vcpu->run->hypercall.ret = ret; +return complete_hypercall (vcpu); }EXPORT_SYMBOL_GPL (kvm_emulate_hypercall);
Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall
Quoted 11 lines · Sean Christopherson
Quoted 7 lines · Forrest Yuan Yu
The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send
a message to the hypervisor which will do its job and send back anything it wants the guest to know.
Hrm, so this reintroduces
KVM_EXIT_HYPERCALLwithout justifying _why_ it needs to be reintroduced. I'm not familiar with the history, but the comments in the documentation advise "useKVM_EXIT_IOorKVM_EXIT_MMIO".
Both of these options have the disadvantage of requiring instruction emulation (Although a trivial one for PIO). Which enlarge host attack surface. I think this is one of the reasons why Hyper-V defined their PV devices (E.g. NetVSC/StorVSC) doorbell kick with hypercall instead of PIO/MMIO. (This is currently not much relevant as KVM's instruction emulator is not optional and is not even offloaded to host userspace. But relevant for the future...)
Quoted 9 lines
Off the top of my head, IO and/or MMIO has a few advantages:
- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded toL1.- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.
I agree with all the above (I believe similar rational had led VMware to design their Backdoor PIO interface).
Also note that recently AWS introduced Nitro Enclave PV device which is also de-facto a PV control-plane interface between guest and host userspace. Why similar approach couldn't have been used here?
(Capability is exposed on a per-VM basis by attaching PV device to VM, communication interface is device specific and no KVM changes, only host userspace changes).
Quoted 11 lines
Quoted 11 lines
Signed-off-by Forrest Yuan Yu <[email protected]>---
Documentation/virt/kvm/cpuid.rst +0
-0@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit before using paravirtualized sched yield.
Quoted 5 lines
Documentation/virt/kvm/cpuid.rst KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit · line 90
Quoted 2 lines
Documentation/virt/kvm/cpuid.rst +2
-0+KVM_FEATURE_UCALL 14 guest checks this feature bit + before calling hypercall ucall.Why make the UCALL a KVM CPUID feature? I can understand wanting to query KVM support from host userspace, but presumably the guest will care about capabiliteis built on top of the UCALL, not the UCALL itself.
I agree with this.
In case of PV device approach, device detection by guest will be the capability discovery.
Quoted 16 lines
Quoted 10 lines
Documentation/virt/kvm/cpuid.rst +1
-0+ KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24 host will warn if no guest-side per-cpu warps are expeced in kvmclockarch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 3388
Quoted 4 lines
arch/x86/kvm/x86.c +1
-0@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_GET_MSR_FEATURES :case KVM_CAP_MSR_PLATFORM_INFO :case KVM_CAP_EXCEPTION_PAYLOAD : +case KVM_CAP_UCALL :For whatever reason I have a metnal block with UCALL, can we call this
KVM_CAP_USERSPACE_HYPERCALL
+1
Quoted 43 lines
Quoted 16 lines
arch/x86/kvm/x86.c +4
-0r =1 ;break ;case KVM_CAP_SYNC_REGS :@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, kvm->arch.exception_payload_enabled = cap->args[0 ]; r =0 ;break ; +case KVM_CAP_UCALL : + kvm->arch.hypercall_ucall_enabled = cap->args[0 ]; + r =0 ; +break ;default : r = -EINVAL ;break ;arch/x86/kvm/x86.c static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id) · line 7566
Quoted 8 lines
arch/x86/kvm/x86.c +5
-0@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)kvm_vcpu_yield_to (target); } +static int complete_hypercall (struct kvm_vcpu *vcpu) +{ +u64 ret = vcpu->run->hypercall.ret; + +if (!is_64_bit_mode (vcpu))Do we really anticipate adding support in 32-bit guests? Honest question.
Quoted 17 lines
arch/x86/kvm/x86.c +10
-0+ ret = (u32 )ret; +kvm_rax_write (vcpu, ret); + + ++vcpu->stat.hypercalls; + +return kvm_skip_emulated_instruction (vcpu); +} +int kvm_emulate_hypercall (struct kvm_vcpu *vcpu) {unsigned long nr, a0, a1, a2, a3, ret;@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)kvm_sched_yield (vcpu->kvm, a0); ret =0 ;break ; +case KVM_HC_UCALL : +if (vcpu->kvm->arch.hypercall_ucall_enabled) {
Quoted 8 lines
arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) · line 7632
Quoted 5 lines
arch/x86/kvm/x86.c +5
-0+ vcpu->run->hypercall.nr = nr; + vcpu->run->hypercall.args[0 ] = a0; + vcpu->run->hypercall.args[1 ] = a1; + vcpu->run->hypercall.args[2 ] = a2; + vcpu->run->hypercall.args[3 ] = a3;If performance is a justification for a more direct userspace exit, why limit it to just four parameters? E.g. why not copy all registers to
kvm_sync_regsand reverse the process on the way back in?
I don't think performance should be relevant for a hypercall interface. It's control-plane path.
If a fast-path is required, guest should use this interface to coordinate a separate fast-path (e.g. via ring-buffer on some guest memory page).
Anyway, these kind of questions is another reason why I agree with Sean it seems using a PV device is preferred.
Instead of forcing a general userspace hypercall interface standard, one could just implement whatever PV device it wants in host userspace which is device specific.
In QEMU's VMPort implementation BTW, userspace calls cpu_synchronize_state() which de-facto syncs tons of vCPU state from KVM to userspace. Not just the GP registers. Because it's a slow-path, it's considered fine. :P
-Liran
Quoted 22 lines
Quoted 22 lines
arch/x86/kvm/x86.c +8
-6+ vcpu->run->exit_reason =KVM_EXIT_HYPERCALL ; + vcpu->arch.complete_userspace_io = complete_hypercall; +return 0 ;// message is going to userspace + } + ret = -KVM_ENOSYS ; +break ;default : ret = -KVM_ENOSYS ;break ; } out: -if (!op_64_bit) - ret = (u32 )ret; -kvm_rax_write (vcpu, ret); - - ++vcpu->stat.hypercalls; -return kvm_skip_emulated_instruction (vcpu); + vcpu->run->hypercall.ret = ret; +return complete_hypercall (vcpu); }EXPORT_SYMBOL_GPL (kvm_emulate_hypercall);
Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall
Quoted 5 lines · Liran Alon
Quoted 2 lines · Sean Christopherson
Off the top of my head, IO and/or MMIO has a few advantages:
Quoted 8 lines
Quoted 6 lines
- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded toL1.- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.
I agree with all the above (I believe similar rational had led VMware to design their Backdoor PIO interface).
Just to set the record straight...
VMware's backdoor PIO interface predates both VMX and SVM, so VMCALL and VMMCALL played no role whatsoever in its design. Moreover, VMware's backdoor PIO interface actually does not allow the guest kernel to delegate permissions to guest userspace. VMware ignores the I/O permission bitmap in the TSS for the backdoor ports, so userspace always has access to them. It's the VMware hypervisor that decides whether or not to accept certain hypercalls at CPL>0.
Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall
In reply to Liran AlonQuoted 13 lines · Liran Alon
Quoted 11 lines · Sean Christopherson
Quoted 7 lines · Forrest Yuan Yu
The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send
a message to the hypervisor which will do its job and send back anything it wants the guest to know.
Hrm, so this reintroduces
KVM_EXIT_HYPERCALLwithout justifying _why_ it needs to be reintroduced. I'm not familiar with the history, but the comments in the documentation advise "useKVM_EXIT_IOorKVM_EXIT_MMIO".
Quoted 8 lines
Both of these options have the disadvantage of requiring instruction emulation (Although a trivial one for PIO). Which enlarge host attack surface.
I think this is one of the reasons why Hyper-V defined their PV devices (E.g. NetVSC/StorVSC) doorbell kick with hypercall instead of PIO/MMIO. (This is currently not much relevant as KVM's instruction emulator is not optional and is not even offloaded to host userspace. But relevant for the future...)
Good point. Again to me the simplicity of the hypercall interface really makes it an ideal mechanism for message passing.
Quoted 102 lines
Quoted 9 lines
Off the top of my head, IO and/or MMIO has a few advantages:
- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded toL1.- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.
I agree with all the above (I believe similar rational had led VMware to design their Backdoor PIO interface).
Also note that recently AWS introduced Nitro Enclave PV device which is also de-facto a PV control-plane interface between guest and host userspace. Why similar approach couldn't have been used here?
(Capability is exposed on a per-VM basis by attaching PV device to VM, communication interface is device specific and no KVM changes, only host userspace changes).Quoted 6 lines
Quoted 6 lines
Signed-off-by Forrest Yuan Yu <[email protected]>---
Documentation/virt/kvm/cpuid.rst +0
-0Quoted 8 lines
Documentation/virt/kvm/cpuid.rst KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit · line 89
Quoted 5 lines
Documentation/virt/kvm/cpuid.rst +2
-0@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit before using paravirtualized sched yield. +KVM_FEATURE_UCALL 14 guest checks this feature bit + before calling hypercall ucall.Why make the UCALL a KVM CPUID feature? I can understand wanting to query KVM support from host userspace, but presumably the guest will care about capabiliteis built on top of the UCALL, not the UCALL itself.
I agree with this.
In case of PV device approach, device detection by guest will be the capability discovery.Quoted 16 lines
Quoted 10 lines
Documentation/virt/kvm/cpuid.rst +1
-0+ KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24 host will warn if no guest-side per-cpu warps are expeced in kvmclockarch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 3388
Quoted 4 lines
arch/x86/kvm/x86.c +1
-0@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_GET_MSR_FEATURES :case KVM_CAP_MSR_PLATFORM_INFO :case KVM_CAP_EXCEPTION_PAYLOAD : +case KVM_CAP_UCALL :For whatever reason I have a metnal block with UCALL, can we call this
KVM_CAP_USERSPACE_HYPERCALL+1
Quoted 42 lines
Quoted 16 lines
arch/x86/kvm/x86.c +4
-0r =1 ;break ;case KVM_CAP_SYNC_REGS :@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, kvm->arch.exception_payload_enabled = cap->args[0 ]; r =0 ;break ; +case KVM_CAP_UCALL : + kvm->arch.hypercall_ucall_enabled = cap->args[0 ]; + r =0 ; +break ;default : r = -EINVAL ;break ;arch/x86/kvm/x86.c static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id) · line 7565
Quoted 7 lines
arch/x86/kvm/x86.c +5
-0@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)kvm_vcpu_yield_to (target); } +static int complete_hypercall (struct kvm_vcpu *vcpu) +{ +u64 ret = vcpu->run->hypercall.ret; + +if (!is_64_bit_mode (vcpu))Do we really anticipate adding support in 32-bit guests? Honest question.
Quoted 17 lines
arch/x86/kvm/x86.c +10
-0+ ret = (u32 )ret; +kvm_rax_write (vcpu, ret); + + ++vcpu->stat.hypercalls; + +return kvm_skip_emulated_instruction (vcpu); +} +int kvm_emulate_hypercall (struct kvm_vcpu *vcpu) {unsigned long nr, a0, a1, a2, a3, ret;@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)kvm_sched_yield (vcpu->kvm, a0); ret =0 ;break ; +case KVM_HC_UCALL : +if (vcpu->kvm->arch.hypercall_ucall_enabled) {Quoted 8 lines
arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) · line 7632
Quoted 5 lines
arch/x86/kvm/x86.c +5
-0+ vcpu->run->hypercall.nr = nr; + vcpu->run->hypercall.args[0 ] = a0; + vcpu->run->hypercall.args[1 ] = a1; + vcpu->run->hypercall.args[2 ] = a2; + vcpu->run->hypercall.args[3 ] = a3;If performance is a justification for a more direct userspace exit, why limit it to just four parameters? E.g. why not copy all registers to
kvm_sync_regsand reverse the process on the way back in?
Quoted 7 lines
I don't think performance should be relevant for a hypercall interface. It's control-plane path.
If a fast-path is required, guest should use this interface to coordinate a separate fast-path (e.g. via ring-buffer on some guest memory page).Anyway, these kind of questions is another reason why I agree with Sean it seems using a PV device is preferred.
I agree. Performance is not an issue here. However, I also don't think this would make a PV device very suitable for a task like message passing. When I try to imagine using a device to pass several bytes of messages, it doesn't feel right ...
Quoted 33 lines
Instead of forcing a general userspace hypercall interface standard, one could just implement whatever PV device it wants in host userspace which is device specific.
In QEMU's VMPort implementation BTW, userspace calls
cpu_synchronize_state()which de-facto syncs tons of vCPU state from KVM to userspace. Not just the GP registers.
Because it's a slow-path, it's considered fine. :P-Liran
Quoted 22 lines
Quoted 22 lines
arch/x86/kvm/x86.c +8
-6+ vcpu->run->exit_reason =KVM_EXIT_HYPERCALL ; + vcpu->arch.complete_userspace_io = complete_hypercall; +return 0 ;// message is going to userspace + } + ret = -KVM_ENOSYS ; +break ;default : ret = -KVM_ENOSYS ;break ; } out: -if (!op_64_bit) - ret = (u32 )ret; -kvm_rax_write (vcpu, ret); - - ++vcpu->stat.hypercalls; -return kvm_skip_emulated_instruction (vcpu); + vcpu->run->hypercall.ret = ret; +return complete_hypercall (vcpu); }EXPORT_SYMBOL_GPL (kvm_emulate_hypercall);
Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall
In reply to Sean ChristophersonQuoted 9 lines · Sean Christopherson
Quoted 4 lines · Forrest Yuan Yu
The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send
Please do s/SLAT/TDP everywhere. IMO, TDP is a less than stellar acronym, e.g. what will we do if we go to three dimensions!?!? But, we're stuck with it :-)
Will do.
Quoted 8 lines
Quoted 3 lines
a message to the hypervisor which will do its job and send back anything it wants the guest to know.
Hrm, so this reintroduces
KVM_EXIT_HYPERCALLwithout justifying _why_ it needs to be reintroduced. I'm not familiar with the history, but the comments in the documentation advise "useKVM_EXIT_IOorKVM_EXIT_MMIO".
Quoted 8 lines
Off the top of my head, IO and/or MMIO has a few advantages:
- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded toL1.- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.
Yes I'm aware of the document advice but still feel hypercall is the right thing to do. First of all, hypercalls, by definition, exist for exactly these kind of jobs. When a guest wants to talk to the hypervisor, it makes a hypercall. It just feels natural.
Of course IO/MMIO can do the same thing, but for simple jobs like asking hypervisor to set restricted access permission, IO/MMIO feels heavy (and IMHO feels a little hacky).
Also I'm not reintroducing KVM_EXIT_HYPERCALL the same way as before, in that we don't pass all unkown hypercalls to userspace. Instead, we just pass one specific call to userspace, one that userspace has opted into.
When userspace does nothing -- not opting in -- the behavior is the same as the current code base.
There are many ways to allow certain hypercalls to be delegated from the guest kernel to guest userspace, and we would probably want a mechanism that allows finer control than I/O permission bitmaps (for PIO) or mapping MMIO regions as writable in a userspace context. Something like this is not tied to a particular communication channel.
Regarding the nested scenario, I would argue there should not be a hypercall interface from a guest to any hypervisor other than its immediate parent.
VMCALL and VMMCALL having different opcodes shouldn't be a problem since kvm already supports hypercalls using these opcodes.
Quoted 11 lines
Quoted 11 lines
Signed-off-by Forrest Yuan Yu <[email protected]>---
Documentation/virt/kvm/cpuid.rst +0
-0@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit before using paravirtualized sched yield.
Quoted 6 lines
Documentation/virt/kvm/cpuid.rst KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit · line 90
Quoted 2 lines
Documentation/virt/kvm/cpuid.rst +2
-0+KVM_FEATURE_UCALL 14 guest checks this feature bit + before calling hypercall ucall.Why make the UCALL a KVM CPUID feature? I can understand wanting to query KVM support from host userspace, but presumably the guest will care about capabiliteis built on top of the UCALL, not the UCALL itself.
This is to keep the behavior the same when UCALL is not recognized by the current userspaces which are not aware of this hypercall. Without this, userspace may break if a guest invokes UCALL.
Quoted 17 lines
Quoted 10 lines
Documentation/virt/kvm/cpuid.rst +1
-0+ KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24 host will warn if no guest-side per-cpu warps are expeced in kvmclockarch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 3388
Quoted 4 lines
arch/x86/kvm/x86.c +1
-0@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_GET_MSR_FEATURES :case KVM_CAP_MSR_PLATFORM_INFO :case KVM_CAP_EXCEPTION_PAYLOAD : +case KVM_CAP_UCALL :For whatever reason I have a metnal block with UCALL, can we call this
KVM_CAP_USERSPACE_HYPERCALL
I was trying to follow the convention. The name UCALL was used in places like tools/testing/selftests/kvm/include/kvm_util.h for similar purposes. I can rename it for better readability.
Quoted 19 lines
Quoted 19 lines
arch/x86/kvm/x86.c +4
-0r =1 ;break ;case KVM_CAP_SYNC_REGS :@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, kvm->arch.exception_payload_enabled = cap->args[0 ]; r =0 ;break ; +case KVM_CAP_UCALL : + kvm->arch.hypercall_ucall_enabled = cap->args[0 ]; + r =0 ; +break ;default : r = -EINVAL ;break ;@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)kvm_vcpu_yield_to (target); }
Quoted 7 lines
arch/x86/kvm/x86.c static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id) · line 7566
Quoted 5 lines
arch/x86/kvm/x86.c +5
-0+static int complete_hypercall (struct kvm_vcpu *vcpu) +{ +u64 ret = vcpu->run->hypercall.ret; + +if (!is_64_bit_mode (vcpu))Do we really anticipate adding support in 32-bit guests? Honest question.
Oh I was trying to keep the behavior exactly the same. It was there under kvm_emulate_hypercall() so I pasted this block here.
Quoted 49 lines
Quoted 13 lines
arch/x86/kvm/x86.c +8
-0+ ret = (u32 )ret; +kvm_rax_write (vcpu, ret); + + ++vcpu->stat.hypercalls; + +return kvm_skip_emulated_instruction (vcpu); +} +int kvm_emulate_hypercall (struct kvm_vcpu *vcpu) {unsigned long nr, a0, a1, a2, a3, ret;arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) · line 7632
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)kvm_sched_yield (vcpu->kvm, a0); ret =0 ;break ; +case KVM_HC_UCALL : +if (vcpu->kvm->arch.hypercall_ucall_enabled) { + vcpu->run->hypercall.nr = nr; + vcpu->run->hypercall.args[0 ] = a0; + vcpu->run->hypercall.args[1 ] = a1; + vcpu->run->hypercall.args[2 ] = a2; + vcpu->run->hypercall.args[3 ] = a3;If performance is a justification for a more direct userspace exit, why limit it to just four parameters? E.g. why not copy all registers to
kvm_sync_regsand reverse the process on the way back in?Quoted 21 lines
arch/x86/kvm/x86.c +8
-6+ vcpu->run->exit_reason =KVM_EXIT_HYPERCALL ; + vcpu->arch.complete_userspace_io = complete_hypercall; +return 0 ;// message is going to userspace + } + ret = -KVM_ENOSYS ; +break ;default : ret = -KVM_ENOSYS ;break ; } out: -if (!op_64_bit) - ret = (u32 )ret; -kvm_rax_write (vcpu, ret); - - ++vcpu->stat.hypercalls; -return kvm_skip_emulated_instruction (vcpu); + vcpu->run->hypercall.ret = ret; +return complete_hypercall (vcpu); }EXPORT_SYMBOL_GPL (kvm_emulate_hypercall);
Quoted 10 lines · Forrest Yuan Yu
This RFC is to add a hypercall UCALL so that guest can communicate with the userspace hypervisor. A guest may want to do this in different scenarios. For example, a guest can ask the userspace hypervisor to harden security by setting restricted access permissions on guest SLAT entries.
Please note this hypercall provides the infrastructure for this type of communication but does not enforce the protocol. A proposal for the guest/userspace communication protocol will follow if this RFC is accepted.
It's hard to review certain aspects of the proposal without seeing the usage of it. Can you post the whole thing as an RFC?