Skip to contentSkip to conversation

Hypercall UCALL for guest/userspace communication

RFC PATCH In review Forrest Yuan Yu 4 participants 8 messages May 1, 2020 – May 5, 2020

Subsystems (6)
Documentation x86 KVM Kselftest +2 more subsystems
lore
Share & tools
lore
#0 lore raw

cover Hypercall UCALL for guest/userspace communication

This RFC is to add a hypercall UCALL so that guest can communicate with the userspace hypervisor. A guest may want to do this in different scenarios. For example, a guest can ask the userspace hypervisor to harden security by setting restricted access permissions on guest SLAT entries.

Please note this hypercall provides the infrastructure for this type of communication but does not enforce the protocol. A proposal for the guest/userspace communication protocol will follow if this RFC is accepted.

Forrest Yuan Yu (1):
KVM: x86: add KVM_HC_UCALL hypercall

 Documentation/virt/kvm/api.rst                |  15 +-
 Documentation/virt/kvm/cpuid.rst              |   3 +
 Documentation/virt/kvm/hypercalls.rst         |  14 ++
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/include/uapi/asm/kvm_para.h          |   1 +
 arch/x86/kvm/x86.c                            |  39 +++-
 include/uapi/linux/kvm.h                      |   1 +
 include/uapi/linux/kvm_para.h                 |   1 +
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/x86_64/hypercall_ucall.c    | 195 ++++++++++++++++++
 11 files changed, 264 insertions(+), 8 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/hypercall_ucall.c
Signature · 1 line
2.26.2.526.g744177e7f7-goog
#1 lore raw

RFC PATCH 1/1 KVM: x86: add KVM_HC_UCALL hypercall

The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send a message to the hypervisor which will do its job and send back anything it wants the guest to know.

---
 Documentation/virt/kvm/api.rst                |  15 +-
 Documentation/virt/kvm/cpuid.rst              |   3 +
 Documentation/virt/kvm/hypercalls.rst         |  14 ++
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/include/uapi/asm/kvm_para.h          |   1 +
 arch/x86/kvm/x86.c                            |  39 +++-
 include/uapi/linux/kvm.h                      |   1 +
 include/uapi/linux/kvm_para.h                 |   1 +
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/x86_64/hypercall_ucall.c    | 195 ++++++++++++++++++
 11 files changed, 264 insertions(+), 8 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/hypercall_ucall.c
Documentation/virt/kvm/api.rst +13 -2
index efbbe570aa9b..ae8958a7ad15 100644
@@ -4854,8 +4854,8 @@ to the byte array.
 
 .. note::
 
-      For KVM_EXIT_IO, KVM_EXIT_MMIO, KVM_EXIT_OSI, KVM_EXIT_PAPR and
-      KVM_EXIT_EPR the corresponding
+      For KVM_EXIT_IO, KVM_EXIT_MMIO, KVM_EXIT_HYPERCALL, KVM_EXIT_OSI,
+      KVM_EXIT_PAPR and KVM_EXIT_EPR the corresponding
 
 operations are complete (and guest state is consistent) only after userspace
 has re-entered the kernel with KVM_RUN.  The kernel side will first finish
@@ -5802,6 +5802,17 @@ If present, this capability can be enabled for a VM, meaning that KVM
 will allow the transition to secure guest mode.  Otherwise KVM will
 veto the transition.
 
+7.20 KVM_CAP_UCALL
+------------------------------
+
+:Architectures: x86
+
+This capability indicates that KVM supports hypercall ucall. It being enabled
+means userspace is ready to receive a message sent by a guest using hypercall
+ucall. When it is not enabled, a hypercall ucall made by a guest will not cause
+control to be handed to userspace. Instead, kvm will return -KVM_ENOSYS without
+userspace participation.
+
 8. Other capabilities.
 ======================
 
Documentation/virt/kvm/cpuid.rst +3 -0
index 01b081f6e7ea..ff313f6827bf 100644
@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD        13          guest checks this feature bit
                                               before using paravirtualized
                                               sched yield.
 
+KVM_FEATURE_UCALL                 14          guest checks this feature bit
+                                              before calling hypercall ucall.
+
 KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24          host will warn if no guest-side
                                               per-cpu warps are expeced in
                                               kvmclock
Documentation/virt/kvm/hypercalls.rst +14 -0
index dbaf207e560d..ce3f30d5b2ee 100644
@@ -169,3 +169,17 @@ a0: destination APIC ID
 
 :Usage example: When sending a call-function IPI-many to vCPUs, yield if
 	        any of the IPI target vCPUs was preempted.
+
+8. KVM_HC_UCALL
+---------------------
+
+:Architecture: x86
+:Status: active
+:Purpose: Hypercall used to exchange message between VM and hypervisor.
+
+a0: message type
+
+a1, a2, a3: dependent on message type
+
+:Usage example: A guest asks hypervisor to harden security by setting
+restricted access permission on guest SLAT entry.
arch/x86/include/asm/kvm_host.h +1 -0
index 42a2d0d3984a..433e96c126a5 100644
@@ -979,6 +979,7 @@ struct kvm_arch {
 
 	bool guest_can_read_msr_platform_info;
 	bool exception_payload_enabled;
+	bool hypercall_ucall_enabled;
 
 	struct kvm_pmu_event_filter *pmu_event_filter;
 	struct task_struct *nx_lpage_recovery_thread;
arch/x86/include/uapi/asm/kvm_para.h +1 -0
index 2a8e0b6b9805..9524434463f2 100644
@@ -31,6 +31,7 @@
 #define KVM_FEATURE_PV_SEND_IPI	11
 #define KVM_FEATURE_POLL_CONTROL	12
 #define KVM_FEATURE_PV_SCHED_YIELD	13
+#define KVM_FEATURE_UCALL		14
 
 #define KVM_HINTS_REALTIME      0
 
arch/x86/kvm/x86.c +33 -6
index c5835f9cb9ad..388a4f89464d 100644
@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_GET_MSR_FEATURES:
 	case KVM_CAP_MSR_PLATFORM_INFO:
 	case KVM_CAP_EXCEPTION_PAYLOAD:
+	case KVM_CAP_UCALL:
 		r = 1;
 		break;
 	case KVM_CAP_SYNC_REGS:
@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		kvm->arch.exception_payload_enabled = cap->args[0];
 		r = 0;
 		break;
+	case KVM_CAP_UCALL:
+		kvm->arch.hypercall_ucall_enabled = cap->args[0];
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)
 		kvm_vcpu_yield_to(target);
 }
 
+static int complete_hypercall(struct kvm_vcpu *vcpu)
+{
+	u64 ret = vcpu->run->hypercall.ret;
+
+	if (!is_64_bit_mode(vcpu))
+		ret = (u32)ret;
+	kvm_rax_write(vcpu, ret);
+
+	++vcpu->stat.hypercalls;
+
+	return kvm_skip_emulated_instruction(vcpu);
+}
+
 int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 {
 	unsigned long nr, a0, a1, a2, a3, ret;
@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 		kvm_sched_yield(vcpu->kvm, a0);
 		ret = 0;
 		break;
+	case KVM_HC_UCALL:
+		if (vcpu->kvm->arch.hypercall_ucall_enabled) {
+			vcpu->run->hypercall.nr = nr;
+			vcpu->run->hypercall.args[0] = a0;
+			vcpu->run->hypercall.args[1] = a1;
+			vcpu->run->hypercall.args[2] = a2;
+			vcpu->run->hypercall.args[3] = a3;
+			vcpu->run->exit_reason = KVM_EXIT_HYPERCALL;
+			vcpu->arch.complete_userspace_io = complete_hypercall;
+			return 0; // message is going to userspace
+		}
+		ret = -KVM_ENOSYS;
+		break;
 	default:
 		ret = -KVM_ENOSYS;
 		break;
 	}
 out:
-	if (!op_64_bit)
-		ret = (u32)ret;
-	kvm_rax_write(vcpu, ret);
-
-	++vcpu->stat.hypercalls;
-	return kvm_skip_emulated_instruction(vcpu);
+	vcpu->run->hypercall.ret = ret;
+	return complete_hypercall(vcpu);
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_hypercall);
 
include/uapi/linux/kvm.h +1 -0
index 428c7dde6b4b..c1fcac311c76 100644
@@ -1017,6 +1017,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_VCPU_RESETS 179
 #define KVM_CAP_S390_PROTECTED 180
 #define KVM_CAP_PPC_SECURE_GUEST 181
+#define KVM_CAP_UCALL 182
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
include/uapi/linux/kvm_para.h +1 -0
index 8b86609849b9..4e5ad8dec801 100644
@@ -29,6 +29,7 @@
 #define KVM_HC_CLOCK_PAIRING		9
 #define KVM_HC_SEND_IPI		10
 #define KVM_HC_SCHED_YIELD		11
+#define KVM_HC_UCALL			12
 
 /*
  * hypercalls use architecture specific
tools/testing/selftests/kvm/.gitignore +1 -0
index a9b2b48947ff..c796c8efaa23 100644
@@ -18,6 +18,7 @@
 /x86_64/vmx_set_nested_state_test
 /x86_64/vmx_tsc_adjust_test
 /x86_64/xss_msr_test
+/x86_64/hypercall_ucall
 /clear_dirty_log_test
 /demand_paging_test
 /dirty_log_test
tools/testing/selftests/kvm/Makefile +1 -0
index 712a2ddd2a27..b3aeec375644 100644
@@ -33,6 +33,7 @@ TEST_GEN_PROGS_x86_64 += demand_paging_test
 TEST_GEN_PROGS_x86_64 += dirty_log_test
 TEST_GEN_PROGS_x86_64 += kvm_create_max_vcpus
 TEST_GEN_PROGS_x86_64 += steal_time
+TEST_GEN_PROGS_x86_64 += x86_64/hypercall_ucall
 
 TEST_GEN_PROGS_aarch64 += clear_dirty_log_test
 TEST_GEN_PROGS_aarch64 += demand_paging_test
tools/testing/selftests/kvm/x86_64/hypercall_ucall.c new file +195 -0

195 lines not on this page · raw message

Signature · 1 line
2.26.2.526.g744177e7f7-goog
#2 lore raw

Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall

Quoted 4 lines · Forrest Yuan Yu

The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send

Please do s/SLAT/TDP everywhere. IMO, TDP is a less than stellar acronym, e.g. what will we do if we go to three dimensions!?!? But, we're stuck with it :-)

Quoted 2 lines

a message to the hypervisor which will do its job and send back anything it wants the guest to know.

Hrm, so this reintroduces KVM_EXIT_HYPERCALL without justifying _why_ it needs to be reintroduced. I'm not familiar with the history, but the comments in the documentation advise "use KVM_EXIT_IO or KVM_EXIT_MMIO".

Off the top of my head, IO and/or MMIO has a few advantages:

- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded to

    L1.

- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.

Documentation/virt/kvm/cpuid.rst KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit · line 90

Quoted 12 lines

---

Documentation/virt/kvm/cpuid.rst +2 -0
index 01b081f6e7ea..ff313f6827bf 100644
@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD        13          guest checks this feature bit
                                               before using paravirtualized
                                               sched yield.
 
+KVM_FEATURE_UCALL                 14          guest checks this feature bit
+                                              before calling hypercall ucall.

Why make the UCALL a KVM CPUID feature? I can understand wanting to query KVM support from host userspace, but presumably the guest will care about capabiliteis built on top of the UCALL, not the UCALL itself.

arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 3388

Quoted 13 lines
Documentation/virt/kvm/cpuid.rst +1 -0
+
 KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24          host will warn if no guest-side
                                               per-cpu warps are expeced in
                                               kvmclock
arch/x86/kvm/x86.c +1 -0
index c5835f9cb9ad..388a4f89464d 100644
@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_GET_MSR_FEATURES:
 	case KVM_CAP_MSR_PLATFORM_INFO:
 	case KVM_CAP_EXCEPTION_PAYLOAD:
+	case KVM_CAP_UCALL:

For whatever reason I have a metnal block with UCALL, can we call this KVM_CAP_USERSPACE_HYPERCALL

Quoted 15 lines
arch/x86/kvm/x86.c +4 -0
 		r = 1;
 		break;
 	case KVM_CAP_SYNC_REGS:
@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		kvm->arch.exception_payload_enabled = cap->args[0];
 		r = 0;
 		break;
+	case KVM_CAP_UCALL:
+		kvm->arch.hypercall_ucall_enabled = cap->args[0];
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;

arch/x86/kvm/x86.c static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id) · line 7566

Quoted 8 lines
arch/x86/kvm/x86.c +5 -0
@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)
 		kvm_vcpu_yield_to(target);
 }
 
+static int complete_hypercall(struct kvm_vcpu *vcpu)
+{
+	u64 ret = vcpu->run->hypercall.ret;
+
+	if (!is_64_bit_mode(vcpu))

Do we really anticipate adding support in 32-bit guests? Honest question.

Quoted 12 lines
arch/x86/kvm/x86.c +8 -0
+		ret = (u32)ret;
+	kvm_rax_write(vcpu, ret);
+
+	++vcpu->stat.hypercalls;
+
+	return kvm_skip_emulated_instruction(vcpu);
+}
+
 int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 {
 	unsigned long nr, a0, a1, a2, a3, ret;

arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) · line 7632

Quoted 10 lines
arch/x86/kvm/x86.c +7 -0
@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 		kvm_sched_yield(vcpu->kvm, a0);
 		ret = 0;
 		break;
+	case KVM_HC_UCALL:
+		if (vcpu->kvm->arch.hypercall_ucall_enabled) {
+			vcpu->run->hypercall.nr = nr;
+			vcpu->run->hypercall.args[0] = a0;
+			vcpu->run->hypercall.args[1] = a1;
+			vcpu->run->hypercall.args[2] = a2;
+			vcpu->run->hypercall.args[3] = a3;

If performance is a justification for a more direct userspace exit, why limit it to just four parameters? E.g. why not copy all registers to kvm_sync_regs and reverse the process on the way back in?

Quoted 21 lines
arch/x86/kvm/x86.c +8 -6
+			vcpu->run->exit_reason = KVM_EXIT_HYPERCALL;
+			vcpu->arch.complete_userspace_io = complete_hypercall;
+			return 0; // message is going to userspace
+		}
+		ret = -KVM_ENOSYS;
+		break;
 	default:
 		ret = -KVM_ENOSYS;
 		break;
 	}
 out:
-	if (!op_64_bit)
-		ret = (u32)ret;
-	kvm_rax_write(vcpu, ret);
-
-	++vcpu->stat.hypercalls;
-	return kvm_skip_emulated_instruction(vcpu);
+	vcpu->run->hypercall.ret = ret;
+	return complete_hypercall(vcpu);
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_hypercall);
#3 lore raw

Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall

Quoted 11 lines · Sean Christopherson
Quoted 7 lines · Forrest Yuan Yu

The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send

a message to the hypervisor which will do its job and send back anything it wants the guest to know.

Hrm, so this reintroduces KVM_EXIT_HYPERCALL without justifying _why_ it needs to be reintroduced. I'm not familiar with the history, but the comments in the documentation advise "use KVM_EXIT_IO or KVM_EXIT_MMIO".

Both of these options have the disadvantage of requiring instruction emulation (Although a trivial one for PIO). Which enlarge host attack surface. I think this is one of the reasons why Hyper-V defined their PV devices (E.g. NetVSC/StorVSC) doorbell kick with hypercall instead of PIO/MMIO. (This is currently not much relevant as KVM's instruction emulator is not optional and is not even offloaded to host userspace. But relevant for the future...)

Quoted 9 lines

Off the top of my head, IO and/or MMIO has a few advantages:

- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded to

     L1.

- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.

I agree with all the above (I believe similar rational had led VMware to design their Backdoor PIO interface).

Also note that recently AWS introduced Nitro Enclave PV device which is also de-facto a PV control-plane interface between guest and host userspace. Why similar approach couldn't have been used here?
(Capability is exposed on a per-VM basis by attaching PV device to VM, communication interface is device specific and no KVM changes, only host userspace changes).

Quoted 11 lines
Quoted 11 lines

---

Documentation/virt/kvm/cpuid.rst +0 -0
index 01b081f6e7ea..ff313f6827bf 100644
@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD        13          guest checks this feature bit
                                                before using paravirtualized
                                                sched yield.
 
Quoted 5 lines

Documentation/virt/kvm/cpuid.rst KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit · line 90

Quoted 2 lines
Documentation/virt/kvm/cpuid.rst +2 -0
+KVM_FEATURE_UCALL                 14          guest checks this feature bit
+                                              before calling hypercall ucall.

Why make the UCALL a KVM CPUID feature? I can understand wanting to query KVM support from host userspace, but presumably the guest will care about capabiliteis built on top of the UCALL, not the UCALL itself.

I agree with this.
In case of PV device approach, device detection by guest will be the capability discovery.

Quoted 16 lines
Quoted 10 lines
Documentation/virt/kvm/cpuid.rst +1 -0
+
  KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24          host will warn if no guest-side
                                                per-cpu warps are expeced in
                                                kvmclock
arch/x86/kvm/x86.c +0 -0
index c5835f9cb9ad..388a4f89464d 100644

arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 3388

Quoted 4 lines
arch/x86/kvm/x86.c +1 -0
@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
  	case KVM_CAP_GET_MSR_FEATURES:
  	case KVM_CAP_MSR_PLATFORM_INFO:
  	case KVM_CAP_EXCEPTION_PAYLOAD:
+	case KVM_CAP_UCALL:

For whatever reason I have a metnal block with UCALL, can we call this KVM_CAP_USERSPACE_HYPERCALL

+1

Quoted 43 lines
Quoted 16 lines
arch/x86/kvm/x86.c +4 -0
  		r = 1;
  		break;
  	case KVM_CAP_SYNC_REGS:
@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
  		kvm->arch.exception_payload_enabled = cap->args[0];
  		r = 0;
  		break;
+	case KVM_CAP_UCALL:
+		kvm->arch.hypercall_ucall_enabled = cap->args[0];
+		r = 0;
+		break;
  	default:
  		r = -EINVAL;
  		break;

arch/x86/kvm/x86.c static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id) · line 7566

Quoted 8 lines
arch/x86/kvm/x86.c +5 -0
@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)
  		kvm_vcpu_yield_to(target);
  }
 
+static int complete_hypercall(struct kvm_vcpu *vcpu)
+{
+	u64 ret = vcpu->run->hypercall.ret;
+
+	if (!is_64_bit_mode(vcpu))

Do we really anticipate adding support in 32-bit guests? Honest question.

Quoted 17 lines
arch/x86/kvm/x86.c +10 -0
+		ret = (u32)ret;
+	kvm_rax_write(vcpu, ret);
+
+	++vcpu->stat.hypercalls;
+
+	return kvm_skip_emulated_instruction(vcpu);
+}
+
  int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
  {
  	unsigned long nr, a0, a1, a2, a3, ret;
@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
  		kvm_sched_yield(vcpu->kvm, a0);
  		ret = 0;
  		break;
+	case KVM_HC_UCALL:
+		if (vcpu->kvm->arch.hypercall_ucall_enabled) {
Quoted 8 lines

arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) · line 7632

Quoted 5 lines
arch/x86/kvm/x86.c +5 -0
+			vcpu->run->hypercall.nr = nr;
+			vcpu->run->hypercall.args[0] = a0;
+			vcpu->run->hypercall.args[1] = a1;
+			vcpu->run->hypercall.args[2] = a2;
+			vcpu->run->hypercall.args[3] = a3;

If performance is a justification for a more direct userspace exit, why limit it to just four parameters? E.g. why not copy all registers to kvm_sync_regs and reverse the process on the way back in?

I don't think performance should be relevant for a hypercall interface. It's control-plane path.
If a fast-path is required, guest should use this interface to coordinate a separate fast-path (e.g. via ring-buffer on some guest memory page).

Anyway, these kind of questions is another reason why I agree with Sean it seems using a PV device is preferred.
Instead of forcing a general userspace hypercall interface standard, one could just implement whatever PV device it wants in host userspace which is device specific.

In QEMU's VMPort implementation BTW, userspace calls cpu_synchronize_state() which de-facto syncs tons of vCPU state from KVM to userspace. Not just the GP registers. Because it's a slow-path, it's considered fine. :P

-Liran

Quoted 22 lines
Quoted 22 lines
arch/x86/kvm/x86.c +8 -6
+			vcpu->run->exit_reason = KVM_EXIT_HYPERCALL;
+			vcpu->arch.complete_userspace_io = complete_hypercall;
+			return 0; // message is going to userspace
+		}
+		ret = -KVM_ENOSYS;
+		break;
  	default:
  		ret = -KVM_ENOSYS;
  		break;
  	}
  out:
-	if (!op_64_bit)
-		ret = (u32)ret;
-	kvm_rax_write(vcpu, ret);
-
-	++vcpu->stat.hypercalls;
-	return kvm_skip_emulated_instruction(vcpu);
+	vcpu->run->hypercall.ret = ret;
+	return complete_hypercall(vcpu);
  }
  EXPORT_SYMBOL_GPL(kvm_emulate_hypercall);
#4 lore raw

Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall

Quoted 5 lines · Liran Alon
Quoted 2 lines · Sean Christopherson

Off the top of my head, IO and/or MMIO has a few advantages:

Quoted 8 lines
Quoted 6 lines

- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded to

     L1.

- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.

I agree with all the above (I believe similar rational had led VMware to design their Backdoor PIO interface).

Just to set the record straight...

VMware's backdoor PIO interface predates both VMX and SVM, so VMCALL and VMMCALL played no role whatsoever in its design. Moreover, VMware's backdoor PIO interface actually does not allow the guest kernel to delegate permissions to guest userspace. VMware ignores the I/O permission bitmap in the TSS for the backdoor ports, so userspace always has access to them. It's the VMware hypervisor that decides whether or not to accept certain hypercalls at CPL>0.

#5 lore raw

Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall

In reply to Liran Alon
Quoted 13 lines · Liran Alon
Quoted 11 lines · Sean Christopherson
Quoted 7 lines · Forrest Yuan Yu

The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send

a message to the hypervisor which will do its job and send back anything it wants the guest to know.

Hrm, so this reintroduces KVM_EXIT_HYPERCALL without justifying _why_ it needs to be reintroduced. I'm not familiar with the history, but the comments in the documentation advise "use KVM_EXIT_IO or KVM_EXIT_MMIO".

Quoted 8 lines

Both of these options have the disadvantage of requiring instruction emulation (Although a trivial one for PIO). Which enlarge host attack surface.
I think this is one of the reasons why Hyper-V defined their PV devices (E.g. NetVSC/StorVSC) doorbell kick with hypercall instead of PIO/MMIO. (This is currently not much relevant as KVM's instruction emulator is not optional and is not even offloaded to host userspace. But relevant for the future...)

Good point. Again to me the simplicity of the hypercall interface really makes it an ideal mechanism for message passing.

Quoted 102 lines
Quoted 9 lines

Off the top of my head, IO and/or MMIO has a few advantages:

- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded to

     L1.

- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.

I agree with all the above (I believe similar rational had led VMware to design their Backdoor PIO interface).

Also note that recently AWS introduced Nitro Enclave PV device which is also de-facto a PV control-plane interface between guest and host userspace. Why similar approach couldn't have been used here?
(Capability is exposed on a per-VM basis by attaching PV device to VM, communication interface is device specific and no KVM changes, only host userspace changes).

Quoted 6 lines
Quoted 6 lines

---

Documentation/virt/kvm/cpuid.rst +0 -0
index 01b081f6e7ea..ff313f6827bf 100644
Quoted 8 lines

Documentation/virt/kvm/cpuid.rst KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit · line 89

Quoted 5 lines
Documentation/virt/kvm/cpuid.rst +2 -0
@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD        13          guest checks this feature bit
                                                before using paravirtualized
                                                sched yield.
+KVM_FEATURE_UCALL                 14          guest checks this feature bit
+                                              before calling hypercall ucall.

Why make the UCALL a KVM CPUID feature? I can understand wanting to query KVM support from host userspace, but presumably the guest will care about capabiliteis built on top of the UCALL, not the UCALL itself.

I agree with this.
In case of PV device approach, device detection by guest will be the capability discovery.

Quoted 16 lines
Quoted 10 lines
Documentation/virt/kvm/cpuid.rst +1 -0
+
  KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24          host will warn if no guest-side
                                                per-cpu warps are expeced in
                                                kvmclock
arch/x86/kvm/x86.c +0 -0
index c5835f9cb9ad..388a4f89464d 100644

arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 3388

Quoted 4 lines
arch/x86/kvm/x86.c +1 -0
@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
  	case KVM_CAP_GET_MSR_FEATURES:
  	case KVM_CAP_MSR_PLATFORM_INFO:
  	case KVM_CAP_EXCEPTION_PAYLOAD:
+	case KVM_CAP_UCALL:

For whatever reason I have a metnal block with UCALL, can we call this KVM_CAP_USERSPACE_HYPERCALL

+1

Quoted 42 lines
Quoted 16 lines
arch/x86/kvm/x86.c +4 -0
  		r = 1;
  		break;
  	case KVM_CAP_SYNC_REGS:
@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
  		kvm->arch.exception_payload_enabled = cap->args[0];
  		r = 0;
  		break;
+	case KVM_CAP_UCALL:
+		kvm->arch.hypercall_ucall_enabled = cap->args[0];
+		r = 0;
+		break;
  	default:
  		r = -EINVAL;
  		break;

arch/x86/kvm/x86.c static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id) · line 7565

Quoted 7 lines
arch/x86/kvm/x86.c +5 -0
@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)
  		kvm_vcpu_yield_to(target);
  }
+static int complete_hypercall(struct kvm_vcpu *vcpu)
+{
+	u64 ret = vcpu->run->hypercall.ret;
+
+	if (!is_64_bit_mode(vcpu))

Do we really anticipate adding support in 32-bit guests? Honest question.

Quoted 17 lines
arch/x86/kvm/x86.c +10 -0
+		ret = (u32)ret;
+	kvm_rax_write(vcpu, ret);
+
+	++vcpu->stat.hypercalls;
+
+	return kvm_skip_emulated_instruction(vcpu);
+}
+
  int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
  {
  	unsigned long nr, a0, a1, a2, a3, ret;
@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
  		kvm_sched_yield(vcpu->kvm, a0);
  		ret = 0;
  		break;
+	case KVM_HC_UCALL:
+		if (vcpu->kvm->arch.hypercall_ucall_enabled) {
Quoted 8 lines

arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) · line 7632

Quoted 5 lines
arch/x86/kvm/x86.c +5 -0
+			vcpu->run->hypercall.nr = nr;
+			vcpu->run->hypercall.args[0] = a0;
+			vcpu->run->hypercall.args[1] = a1;
+			vcpu->run->hypercall.args[2] = a2;
+			vcpu->run->hypercall.args[3] = a3;

If performance is a justification for a more direct userspace exit, why limit it to just four parameters? E.g. why not copy all registers to kvm_sync_regs and reverse the process on the way back in?

Quoted 7 lines

I don't think performance should be relevant for a hypercall interface. It's control-plane path.
If a fast-path is required, guest should use this interface to coordinate a separate fast-path (e.g. via ring-buffer on some guest memory page).

Anyway, these kind of questions is another reason why I agree with Sean it seems using a PV device is preferred.

I agree. Performance is not an issue here. However, I also don't think this would make a PV device very suitable for a task like message passing. When I try to imagine using a device to pass several bytes of messages, it doesn't feel right ...

Quoted 33 lines

Instead of forcing a general userspace hypercall interface standard, one could just implement whatever PV device it wants in host userspace which is device specific.

In QEMU's VMPort implementation BTW, userspace calls cpu_synchronize_state() which de-facto syncs tons of vCPU state from KVM to userspace. Not just the GP registers.
Because it's a slow-path, it's considered fine. :P

-Liran

Quoted 22 lines
Quoted 22 lines
arch/x86/kvm/x86.c +8 -6
+			vcpu->run->exit_reason = KVM_EXIT_HYPERCALL;
+			vcpu->arch.complete_userspace_io = complete_hypercall;
+			return 0; // message is going to userspace
+		}
+		ret = -KVM_ENOSYS;
+		break;
  	default:
  		ret = -KVM_ENOSYS;
  		break;
  	}
  out:
-	if (!op_64_bit)
-		ret = (u32)ret;
-	kvm_rax_write(vcpu, ret);
-
-	++vcpu->stat.hypercalls;
-	return kvm_skip_emulated_instruction(vcpu);
+	vcpu->run->hypercall.ret = ret;
+	return complete_hypercall(vcpu);
  }
  EXPORT_SYMBOL_GPL(kvm_emulate_hypercall);
#6 lore raw

Re: [PATCH RFC 1/1] KVM: x86: add KVM_HC_UCALL hypercall

In reply to Sean Christopherson
Quoted 9 lines · Sean Christopherson
Quoted 4 lines · Forrest Yuan Yu

The purpose of this new hypercall is to exchange message between guest and hypervisor. For example, a guest may want to ask hypervisor to harden security by setting restricted access permission on guest SLAT entry. In this case, the guest can use this hypercall to send

Please do s/SLAT/TDP everywhere. IMO, TDP is a less than stellar acronym, e.g. what will we do if we go to three dimensions!?!? But, we're stuck with it :-)

Will do.

Quoted 8 lines
Quoted 3 lines

a message to the hypervisor which will do its job and send back anything it wants the guest to know.

Hrm, so this reintroduces KVM_EXIT_HYPERCALL without justifying _why_ it needs to be reintroduced. I'm not familiar with the history, but the comments in the documentation advise "use KVM_EXIT_IO or KVM_EXIT_MMIO".

Quoted 8 lines

Off the top of my head, IO and/or MMIO has a few advantages:

- Allows the guest kernel to delegate permissions to guest userspace, whereas KVM restrict hypercalls to CPL0.
- Allows "pass-through", whereas VMCALL is unconditionally forwarded to

    L1.

- Is vendor agnostic, e.g. VMX and SVM recognized different opcodes for VMCALL vs VMMCALL.

Yes I'm aware of the document advice but still feel hypercall is the right thing to do. First of all, hypercalls, by definition, exist for exactly these kind of jobs. When a guest wants to talk to the hypervisor, it makes a hypercall. It just feels natural.

Of course IO/MMIO can do the same thing, but for simple jobs like asking hypervisor to set restricted access permission, IO/MMIO feels heavy (and IMHO feels a little hacky).

Also I'm not reintroducing KVM_EXIT_HYPERCALL the same way as before, in that we don't pass all unkown hypercalls to userspace. Instead, we just pass one specific call to userspace, one that userspace has opted into.

When userspace does nothing -- not opting in -- the behavior is the same as the current code base.

There are many ways to allow certain hypercalls to be delegated from the guest kernel to guest userspace, and we would probably want a mechanism that allows finer control than I/O permission bitmaps (for PIO) or mapping MMIO regions as writable in a userspace context. Something like this is not tied to a particular communication channel.

Regarding the nested scenario, I would argue there should not be a hypercall interface from a guest to any hypervisor other than its immediate parent.

VMCALL and VMMCALL having different opcodes shouldn't be a problem since kvm already supports hypercalls using these opcodes.

Quoted 11 lines
Quoted 11 lines

---

Documentation/virt/kvm/cpuid.rst +0 -0
index 01b081f6e7ea..ff313f6827bf 100644
@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD        13          guest checks this feature bit
                                               before using paravirtualized
                                               sched yield.
 
Quoted 6 lines

Documentation/virt/kvm/cpuid.rst KVM_FEATURE_PV_SCHED_YIELD 13 guest checks this feature bit · line 90

Quoted 2 lines
Documentation/virt/kvm/cpuid.rst +2 -0
+KVM_FEATURE_UCALL                 14          guest checks this feature bit
+                                              before calling hypercall ucall.

Why make the UCALL a KVM CPUID feature? I can understand wanting to query KVM support from host userspace, but presumably the guest will care about capabiliteis built on top of the UCALL, not the UCALL itself.

This is to keep the behavior the same when UCALL is not recognized by the current userspaces which are not aware of this hypercall. Without this, userspace may break if a guest invokes UCALL.

Quoted 17 lines
Quoted 10 lines
Documentation/virt/kvm/cpuid.rst +1 -0
+
 KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24          host will warn if no guest-side
                                               per-cpu warps are expeced in
                                               kvmclock
arch/x86/kvm/x86.c +0 -0
index c5835f9cb9ad..388a4f89464d 100644

arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 3388

Quoted 4 lines
arch/x86/kvm/x86.c +1 -0
@@ -3385,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_GET_MSR_FEATURES:
 	case KVM_CAP_MSR_PLATFORM_INFO:
 	case KVM_CAP_EXCEPTION_PAYLOAD:
+	case KVM_CAP_UCALL:

For whatever reason I have a metnal block with UCALL, can we call this KVM_CAP_USERSPACE_HYPERCALL

I was trying to follow the convention. The name UCALL was used in places like tools/testing/selftests/kvm/include/kvm_util.h for similar purposes. I can rename it for better readability.

Quoted 19 lines
Quoted 19 lines
arch/x86/kvm/x86.c +4 -0
 		r = 1;
 		break;
 	case KVM_CAP_SYNC_REGS:
@@ -4895,6 +4896,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		kvm->arch.exception_payload_enabled = cap->args[0];
 		r = 0;
 		break;
+	case KVM_CAP_UCALL:
+		kvm->arch.hypercall_ucall_enabled = cap->args[0];
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -7554,6 +7559,19 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)
 		kvm_vcpu_yield_to(target);
 }
 
Quoted 7 lines

arch/x86/kvm/x86.c static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id) · line 7566

Quoted 5 lines
arch/x86/kvm/x86.c +5 -0
+static int complete_hypercall(struct kvm_vcpu *vcpu)
+{
+	u64 ret = vcpu->run->hypercall.ret;
+
+	if (!is_64_bit_mode(vcpu))

Do we really anticipate adding support in 32-bit guests? Honest question.

Oh I was trying to keep the behavior exactly the same. It was there under kvm_emulate_hypercall() so I pasted this block here.

Quoted 49 lines
Quoted 13 lines
arch/x86/kvm/x86.c +8 -0
+		ret = (u32)ret;
+	kvm_rax_write(vcpu, ret);
+
+	++vcpu->stat.hypercalls;
+
+	return kvm_skip_emulated_instruction(vcpu);
+}
+
 int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 {
 	unsigned long nr, a0, a1, a2, a3, ret;

arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) · line 7632

Quoted 10 lines
arch/x86/kvm/x86.c +7 -0
@@ -7605,17 +7623,26 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 		kvm_sched_yield(vcpu->kvm, a0);
 		ret = 0;
 		break;
+	case KVM_HC_UCALL:
+		if (vcpu->kvm->arch.hypercall_ucall_enabled) {
+			vcpu->run->hypercall.nr = nr;
+			vcpu->run->hypercall.args[0] = a0;
+			vcpu->run->hypercall.args[1] = a1;
+			vcpu->run->hypercall.args[2] = a2;
+			vcpu->run->hypercall.args[3] = a3;

If performance is a justification for a more direct userspace exit, why limit it to just four parameters? E.g. why not copy all registers to kvm_sync_regs and reverse the process on the way back in?

Quoted 21 lines
arch/x86/kvm/x86.c +8 -6
+			vcpu->run->exit_reason = KVM_EXIT_HYPERCALL;
+			vcpu->arch.complete_userspace_io = complete_hypercall;
+			return 0; // message is going to userspace
+		}
+		ret = -KVM_ENOSYS;
+		break;
 	default:
 		ret = -KVM_ENOSYS;
 		break;
 	}
 out:
-	if (!op_64_bit)
-		ret = (u32)ret;
-	kvm_rax_write(vcpu, ret);
-
-	++vcpu->stat.hypercalls;
-	return kvm_skip_emulated_instruction(vcpu);
+	vcpu->run->hypercall.ret = ret;
+	return complete_hypercall(vcpu);
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_hypercall);
#7 lore raw
In reply to Forrest Yuan Yu
Quoted 10 lines · Forrest Yuan Yu

This RFC is to add a hypercall UCALL so that guest can communicate with the userspace hypervisor. A guest may want to do this in different scenarios. For example, a guest can ask the userspace hypervisor to harden security by setting restricted access permissions on guest SLAT entries.

Please note this hypercall provides the infrastructure for this type of communication but does not enforce the protocol. A proposal for the guest/userspace communication protocol will follow if this RFC is accepted.

It's hard to review certain aspects of the proposal without seeing the usage of it. Can you post the whole thing as an RFC?