TDX attestation support and GHCI fixup
Subsystems (5)
cover TDX attestation support and GHCI fixup
Hi,
This patch set includes TDX attestation support patch from [0] and some fixups according to the proposed GHCI spec changes below. Tag the patch set as RFC since it's based on the proposed GHCI spec changes. However, the proposal and the patch set are expected to be discussed/finalized. The attestation support and the fixups are hoped to be merged during the 6.16 merge window as part of the initial TDX support.
Paolo once suggested to implement all the TDVMCALLs defined in the current GHCI 1.5 spec [1]. However, since there is a proposal to fix the GHCI spec issues, this patch set doesn't implement the support for Service SetupEventNotifyInterrupt, Instruction.WBINVD and Instruction.PCONFIG since these TDVMCALLs have no real users for now.
Notable changes since attestation v2 [0]
========================================
- Use TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED instead of TDVMCALL_STATUS_INVALID_OPERAND for unsupported TDVMCALLs.
- Document the definition of "The base GHCI TDVMCALLs".
- Forward GetTdVmCallInfo to userspace with the new exit reason KVM_EXIT_TDX_GET_TDVMCALL_INFO when leaf (r12) input is 1 to allow userspace to provide the information of TDVMCALLs supported in userspace.
- Move the check of userspace's opt-in of KVM exit on KVM_HC_MAP_GPA_RANGE to KVM_TDX_FINALIZE_VM, since MapGPA is one of the GHCI base TDVMCALLs according to the proposal below.
This requires some changes to the TDX KVM selftests cases posted [2] to always opt-in KVM exit on KVM_HC_MAP_GPA_RANGE before KVM_TDX_FINALIZE_VM.
- Since there is no opt-in from userspace for GetTdVmCallInfo and GetQuote, userspace is required to handle the exit reasons KVM_EXIT_TDX_GET_TDVMCALL_INFO and KVM_EXIT_TDX_GET_QUOTE as the initial support for TDX. To simplify the implementation in userspace, userspace could return TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED for GetQuote.
GHCI Change Proposal
====================
Current TDX Guest Host communication Interface (GHCI) spec[3] implies that VMM should only return success for TDG.VP.VMCALL<GetTdVmCallInfo> if *all* TDVMCALLs defined in the GHCI spec are supported. The spec is ambiguous on the following perspectives:
- The description "all TDG.VP.VMCALLs defined in this specification" is not forward-compatible since more and more TDVMCALLs will be added when the GHCI spec evolves.
- It actually doesn't cover how to handle if the guest calls an unsupported TDVMCALL.
Historically, KVM has returned TDVMCALL_STATUS_INVALID_OPERAND for any unknown TDVMCALL, as a reasonable interpretation of the ambiguous spec. However, TDX guests can't distinguish the error is due to the TDVMCALL is not supported or an invalid input of the TDVMCALL.
Also, enforce VMMs to implement the TDVMCALLs without real users is an unnecessary burden.
To address the issues, the following are the proposed GHCI spec changes:
- Define "the GHCI base TDVMCALLs", which are: <GetTdVmCallInfo>, <MapGPA>, <ReportFatalError>, <Instruction.CPUID>, <#VE.RequestMMIO>, <Instruction.HLT>, <Instruction.IO>, <Instruction.RDMSR> and <Instruction.WRMSR>.
- Limit the scope of the TDG.VP.VMCALL<GetTdVmCallInfo> with leaf (R12) set to 0 to the GHCI base VMCALLs, so that the meaning is clear and unambiguous.
- Extend the TDG.VP.VMCALL<GetTdVmCallInfo> with leaf (R12) set to 1 to allow TDX guests to query the supported TDVMCALLs beyond the GHCI base TDVMCALLs.
Use R11 - R14 to return the supported TDVMCALLs, which are defined as
* R11
bit 0: <GetQuote>
bit 1: <SetupEventNotifyInterrupt>
bit 2: <Service>
bit 3: <MigTD>
Other bits of R11 are reserved and must be 0.
* R12
bit 0: <Instruction.WBINVD>
bit 1: <Instruction.PCONFIG>
Other bits of R12 are reserved and must be 0.
* R13 and R14 are reserved and must be 0.
- Add TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED to the TDVMCALL status codes for the ones that beyond GHCI base TDVMCALLs to indicate the subfunction is not supported.
For the back-compatibility analysis, please refer to the change log of "KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs".
Testing
=======
This series is based on kvm/next with the commit:
- '61374cc145f4' ("Merge tag 'kvmarm-fixes-6.16-1' of
https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD").
A matching QEMU is here:
https://github.com/intel-staging/qemu-tdx/commits/binbinwu/GetTdVmCallInfo_fixup
It requires TDX module 1.5.06.00.0744 [4], or later.
A working edk2 commit is 95d8a1c ("UnitTestFrameworkPkg: Use TianoCore mirror of subhook submodule").
This patch series passed the TDX kvm-unit-tests, booting a Linux TD, and TDX enhanced KVM selftests. It also passed the TDX related test cases defined in the LKVS test suite as described in:
https://github.com/intel/lkvs/blob/main/KVM/docs/lkvs_on_avocado.md
KVM selftests patches based on the latest TDX KVM selftests patch series [2], were used to test the flows of GetQuote, GetTdVmCallInfo with leaf 1, unsupported TDVMCALL, and some modifications were made because the opt-in of KVM exit on KVM_HC_MAP_GPA_RANGE should be done before KVM_TDX_FINALIZE_VM.
[0] https://lore.kernel.org/kvm/[email protected]
[1] https://lore.kernel.org/kvm/[email protected]
[2] https://lore.kernel.org/kvm/[email protected]
[3] https://cdrdv2.intel.com/v1/dl/getContent/726792
[4] https://github.com/intel/tdx-module/releases/tag/TDX_1.5.06
Binbin Wu (4):
KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs
KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>
KVM: TDX: Exit to userspace for GetTdVmCallInfo
KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
Documentation/virt/kvm/api.rst | 44 ++++++++++++ Documentation/virt/kvm/x86/intel-tdx.rst | 12 ++++ arch/x86/include/asm/shared/tdx.h | 1 + arch/x86/kvm/vmx/tdx.c | 85 +++++++++++++++++++----- include/uapi/linux/kvm.h | 14 ++++ 5 files changed, 141 insertions(+), 15 deletions(-)
base-commit: 61374cc145f4a56377eaf87c7409a97ec7a34041
Signature · 1 line
2.46.0
RFC PATCH 1/4 KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs
Add the new TDVMCALL status code TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED and return it for unimplemented TDVMCALL subfunctions.
Returning TDVMCALL_STATUS_INVALID_OPERAND when a subfunction is not implemented is vague because TDX guests can't tell the error is due to the subfunction is not supported or an invalid input of the subfunction. New GHCI spec adds TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED to avoid the ambiguity. Use it instead of TDVMCALL_STATUS_INVALID_OPERAND.
Before the change, for common guest implementations, when a TDX guest receives TDVMCALL_STATUS_INVALID_OPERAND, it has two cases:
1. Some operand is invalid. It could change the operand to another value retry.
2. The subfunction is not supported.
For case 1, an invalid operand usually means the guest implementation bug. Since the TDX guest can't tell which case is, the best practice for handling TDVMCALL_STATUS_INVALID_OPERAND is stopping calling such leaf. Treat it as fatal if the TDVMCALL is essential or ignore it if the TDVMCALL is optional.
After the change, TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED is unknown to the old TDX guest, the guest will make the same action as TDVMCALL_STATUS_INVALID_OPERAND, unless the guest check the TDVMCALL_STATUS_INVALID_OPERAND specifically. Currently, no known TDX guests do it, e.g., Linux TDX guests just check for success.
--- arch/x86/include/asm/shared/tdx.h | 1 + arch/x86/kvm/vmx/tdx.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/asm/shared/tdx.h +1 -0
@@ -79,6 +79,7 @@#define TDVMCALL_STATUS_RETRY 0x0000000000000001ULL#define TDVMCALL_STATUS_INVALID_OPERAND 0x8000000000000000ULL#define TDVMCALL_STATUS_ALIGN_ERROR 0x8000000000000002ULL +#define TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED 0x8000000000000003ULL/* * Bitmasks of exposed registers (with VMM).
arch/x86/kvm/vmx/tdx.c +1 -1
@@ -1476,7 +1476,7 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)break ; } -tdvmcall_set_return_code (vcpu,TDVMCALL_STATUS_INVALID_OPERAND ); +tdvmcall_set_return_code (vcpu,TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED );return 1 ; }
Signature · 1 line
2.46.0
RFC PATCH 2/4 KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>
In reply to Binbin WuHandle TDVMCALL for GetQuote to generate a TD-Quote.
GetQuote is a doorbell-like interface used by TDX guests to request VMM to generate a TD-Quote signed by a service hosting TD-Quoting Enclave operating on the host. A TDX guest passes a TD Report (TDREPORT_STRUCT) in a shared-memory area as parameter. Host VMM can access it and queue the operation for a service hosting TD-Quoting enclave. When completed, the Quote is returned via the same shared-memory area.
KVM only checks the GPA from the TDX guest has the shared-bit set and drops the shared-bit before exiting to userspace to avoid bleeding the shared-bit into KVM's exit ABI. KVM forwards the request to userspace VMM (e.g. QEMU) and userspace VMM queues the operation asynchronously. KVM sets the return code according to the 'ret' field set by userspace to notify the TDX guest whether the request has been queued successfully or not. When the request has been queued successfully, the TDX guest can poll the status field in the shared-memory area to check whether the Quote generation is completed or not. When completed, the generated Quote is returned via the same buffer.
Add KVM_EXIT_TDX_GET_QUOTE as a new exit reason to userspace. Userspace is required to handle the KVM exit reason as the initial support for TDX, however, userspace is allowed to set 'ret' filed to TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED.
---
v2:
- Skip sanity checks except for the check of the shared-bit. (Rick, Sean)
- Drop the shared-bit of the GPA before exiting to userspace. (Sean)
- Improve the description of 'ret' field in the document. (Kai)
- Use number 40 for KVM_EXIT_TDX_GET_QUOTE. (Xiaoyao, Rick)
- Update the changelog and the description of KVM_EXIT_TDX_GET_QUOTE in the document according to the code changes.
RFC v1:
- Add "Acked-by" from Kai.
- State that KVM_EXIT_TDX_GET_QUOTE should be handled by userspace as the initial support for TDX.
- Remove a blank line.
--- Documentation/virt/kvm/api.rst | 27 +++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.c | 29 +++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 7 +++++++ 3 files changed, 63 insertions(+)
Documentation/virt/kvm/api.rst +27 -0
@@ -7174,6 +7174,33 @@ The valid value for 'flags' is: - KVM_NOTIFY_CONTEXT_INVALID -- the VM context is corrupted and not valid in VMCS. It would run into unknown result if resume the target VM. +:: + + /* KVM_EXIT_TDX_GET_QUOTE */ + struct tdx_get_quote { + __u64 ret; + __u64 gpa; + __u64 size; + }; + +If the exit reason is KVM_EXIT_TDX_GET_QUOTE, then it indicates that a TDX +guest has requested to generate a TD-Quote signed by a service hosting +TD-Quoting Enclave operating on the host. The 'gpa' field and 'size' specify +the guest physical address and size of a shared-memory buffer, in which the +TDX guest passes a TD Report. KVM checks the GPA from the TDX guest has the +shared-bit set and drops the shared-bit in 'gpa' field before exiting to +userspace. KVM doesn't do other sanity checks. The 'ret' field represents the +return value of the GetQuote request. KVM only bridges the request to the +userspace VMM, and the userspace VMM is responsible for setting up the return +value since only userspace knows whether the request has been queued +successfully or not. KVM sets the return code according to the 'ret' field +before returning back to the TDX guest. When the request has been queued +successfully, the TDX guest can poll the status field in the shared-memory area +to check whether the Quote generation is completed or not. When completed, the +generated Quote is returned via the same buffer. Userspace is required to handle +the KVM exit reason as the initial support for TDX, however, userspace is +allowed to set 'ret' filed to TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED. + :: /* Fix the size of the union. */
arch/x86/kvm/vmx/tdx.c +29 -0
@@ -1463,6 +1463,33 @@ static int tdx_get_td_vm_call_info(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_complete_get_quote (struct kvm_vcpu *vcpu) +{ +tdvmcall_set_return_code (vcpu, vcpu->run->tdx_get_quote.ret); +return 1 ; +} + +static int tdx_get_quote (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); +u64 gpa = tdx->vp_enter_args.r12; +u64 size = tdx->vp_enter_args.r13; + +/* The gpa of buffer must have shared bit set. */ +if (vt_is_tdx_private_gpa (vcpu->kvm, gpa)) { +tdvmcall_set_return_code (vcpu,TDVMCALL_STATUS_INVALID_OPERAND ); +return 1 ; + } + + vcpu->run->exit_reason =KVM_EXIT_TDX_GET_QUOTE ; + vcpu->run->tdx_get_quote.gpa = gpa & ~gfn_to_gpa (kvm_gfn_direct_bits (tdx->vcpu.kvm)); + vcpu->run->tdx_get_quote.size = size; + + vcpu->arch.complete_userspace_io = tdx_complete_get_quote; + +return 0 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {switch (tdvmcall_leaf (vcpu)) {
@@ -1472,6 +1499,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_report_fatal_error (vcpu);case TDVMCALL_GET_TD_VM_CALL_INFO :return tdx_get_td_vm_call_info (vcpu); +case TDVMCALL_GET_QUOTE : +return tdx_get_quote (vcpu);default :break ; }
include/uapi/linux/kvm.h +7 -0
@@ -178,6 +178,7 @@ struct kvm_xen_exit {#define KVM_EXIT_NOTIFY 37#define KVM_EXIT_LOONGARCH_IOCSR 38#define KVM_EXIT_MEMORY_FAULT 39 +#define KVM_EXIT_TDX_GET_QUOTE 40/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -447,6 +448,12 @@ struct kvm_run {__u64 gpa;__u64 size; } memory_fault; +/* KVM_EXIT_TDX_GET_QUOTE */ +struct { +__u64 ret; +__u64 gpa; +__u64 size; + } tdx_get_quote;/* Fix the size of the union. */ char padding[256 ]; };
Signature · 1 line
2.46.0
RFC PATCH 3/4 KVM: TDX: Exit to userspace for GetTdVmCallInfo
In reply to Binbin WuExit to userspace for TDG.VP.VMCALL<GetTdVmCallInfo> via a new KVM exit reason to allow userspace to provide information about the support of TDVMCALLs when r12 is 1 for the TDVMCALLs beyond the GHCI base API.
GHCI spec defines the GHCI base TDVMCALLs: <GetTdVmCallInfo>, <MapGPA>, <ReportFatalError>, <Instruction.CPUID>, <#VE.RequestMMIO>, <Instruction.HLT>, <Instruction.IO>, <Instruction.RDMSR> and <Instruction.WRMSR>. They must be supported by VMM to support TDX guests.
For GetTdVmCallInfo
- When leaf (r12) to enumerate TDVMCALL functionality is set to 0, successful execution indicates all GHCI base TDVMCALLs listed above are supported.
Update the KVM TDX document with the set of the GHCI base APIs.
- When leaf (r12) to enumerate TDVMCALL functionality is set to 1, it indicates the TDX guest is querying the supported TDVMCALLs beyond the GHCI base TDVMCALLs.
Exit to userspace to let userspace set the TDVMCALL sub-function bit(s) accordingly to the leaf outputs. KVM could set the TDVMCALL bit(s) supported by itself when the TDVMCALLs don't need support from userspace after returning from userspace and before entering guest. Currently, no such TDVMCALLs implemented, KVM just sets the values returned from userspace.
A new KVM exit reason KVM_EXIT_TDX_GET_TDVMCALL_INFO and its structure are added. Userspace is required to handle the exit reason as the initial support for TDX.
--- Documentation/virt/kvm/api.rst | 17 +++++++++++ Documentation/virt/kvm/x86/intel-tdx.rst | 9 ++++++ arch/x86/kvm/vmx/tdx.c | 36 +++++++++++++++++++++--- include/uapi/linux/kvm.h | 7 +++++ 4 files changed, 65 insertions(+), 4 deletions(-)
Documentation/virt/kvm/api.rst +17 -0
@@ -7201,6 +7201,23 @@ generated Quote is returned via the same buffer. Userspace is required to handle the KVM exit reason as the initial support for TDX, however, userspace is allowed to set 'ret' filed to TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED. +:: + + /* KVM_EXIT_TDX_GET_TDVMCALL_INFO */ + struct tdx_get_tdvmcall_info { + __u64 ret; + __u64 leaf; + __u64 leaf_output[4]; + }; + +If the exit reason is KVM_EXIT_TDX_GET_TDVMCALL_INFO, then it indicates that a +TDX guest has requested to get the supporting status of TDVMCALLs. Currently, +KVM only exits to userspace when the leaf value is 1, i.e, the TDX guest is +querying the supporting status of TDVMCALLs beyond the GHCI base TDVMCALLs. +Userspace is expected to set leaf outputs according to the layout defined in +the GHCI spec if they are supported by userspace. Userspace is required to +handle the exit reason as the initial support for TDX. + :: /* Fix the size of the union. */
Documentation/virt/kvm/x86/intel-tdx.rst +9 -0
@@ -249,6 +249,15 @@ control flow is as follows: #. Run VCPU +The GHCI base TDVMCALLs +======================= +The GHCI base TDVMCALLs are: <GetTdVmCallInfo>, <MapGPA>, <ReportFatalError>, +<Instruction.CPUID>, <#VE.RequestMMIO>, <Instruction.HLT>, <Instruction.IO>, +<Instruction.RDMSR> and <Instruction.WRMSR>. These base TDVMCALLs are mandatory +for VMMs to support TDX guests. +For the TDVMCALLs beyond the GHCI base TDVMCALLs, TDX guests can query the +support status via GetTdVmCallInfo with leaf set to 1. + References ==========
arch/x86/kvm/vmx/tdx.c +32 -4
@@ -1449,18 +1449,46 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_complete_get_tdcall_info (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + +tdvmcall_set_return_code (vcpu, vcpu->run->tdx_get_tdvmcall_info.ret); + +/* +* For now, there is no TDVMCALL beyond GHCI base API supported by KVM +* directly without the support from userspace, just set the value +* returned from userspace. +*/ + tdx->vp_enter_args.r11 = vcpu->run->tdx_get_tdvmcall_info.leaf_output[0 ]; + tdx->vp_enter_args.r12 = vcpu->run->tdx_get_tdvmcall_info.leaf_output[1 ]; + tdx->vp_enter_args.r13 = vcpu->run->tdx_get_tdvmcall_info.leaf_output[2 ]; + tdx->vp_enter_args.r14 = vcpu->run->tdx_get_tdvmcall_info.leaf_output[3 ]; + +return 1 ; +} +static int tdx_get_td_vm_call_info (struct kvm_vcpu *vcpu) {struct vcpu_tdx *tdx =to_tdx (vcpu); -if (tdx->vp_enter_args.r12) -tdvmcall_set_return_code (vcpu,TDVMCALL_STATUS_INVALID_OPERAND ); -else { +switch (tdx->vp_enter_args.r12) { +case 0 : tdx->vp_enter_args.r11 =0 ; tdx->vp_enter_args.r13 =0 ; tdx->vp_enter_args.r14 =0 ; +return 1 ; +case 1 : + vcpu->run->tdx_get_tdvmcall_info.leaf =1 ; + vcpu->run->exit_reason =KVM_EXIT_TDX_GET_TDVMCALL_INFO ; + vcpu->arch.complete_userspace_io = tdx_complete_get_tdcall_info; +memset (vcpu->run->tdx_get_tdvmcall_info.leaf_output,0 , +sizeof (vcpu->run->tdx_get_tdvmcall_info.leaf_output)); +return 0 ; +default : +tdvmcall_set_return_code (vcpu,TDVMCALL_STATUS_INVALID_OPERAND ); +return 1 ; } -return 1 ; }static int tdx_complete_get_quote (struct kvm_vcpu *vcpu)
include/uapi/linux/kvm.h +7 -0
@@ -179,6 +179,7 @@ struct kvm_xen_exit {#define KVM_EXIT_LOONGARCH_IOCSR 38#define KVM_EXIT_MEMORY_FAULT 39#define KVM_EXIT_TDX_GET_QUOTE 40 +#define KVM_EXIT_TDX_GET_TDVMCALL_INFO 41/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -454,6 +455,12 @@ struct kvm_run {__u64 gpa;__u64 size; } tdx_get_quote; +/* KVM_EXIT_TDX_GET_TDVMCALL_INFO */ +struct { +__u64 ret; +__u64 leaf; +__u64 leaf_output[4 ]; + } tdx_get_tdvmcall_info;/* Fix the size of the union. */ char padding[256 ]; };
Signature · 1 line
2.46.0
Re: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 26 lines · Binbin Wu
Exit to userspace for TDG.VP.VMCALL<GetTdVmCallInfo> via a new KVM exit reason to allow userspace to provide information about the support of TDVMCALLs when r12 is 1 for the TDVMCALLs beyond the GHCI base API.
GHCI spec defines the GHCI base TDVMCALLs: <GetTdVmCallInfo>, <MapGPA>, <ReportFatalError>, <Instruction.CPUID>, <#VE.RequestMMIO>, <Instruction.HLT>, <Instruction.IO>, <Instruction.RDMSR> and <Instruction.WRMSR>. They must be supported by VMM to support TDX guests.
For GetTdVmCallInfo
- When leaf (r12) to enumerate TDVMCALL functionality is set to 0, successful execution indicates all GHCI base TDVMCALLs listed above are supported.Update the KVM TDX document with the set of the GHCI base APIs.
- When leaf (r12) to enumerate TDVMCALL functionality is set to 1, it indicates the TDX guest is querying the supported TDVMCALLs beyond the GHCI base TDVMCALLs.
Exit to userspace to let userspace set the TDVMCALL sub-function bit(s) accordingly to the leaf outputs. KVM could set the TDVMCALL bit(s) supported by itself when the TDVMCALLs don't need support from userspace after returning from userspace and before entering guest. Currently, no such TDVMCALLs implemented, KVM just sets the values returned from userspace.
Quoted 3 lines
A new KVM exit reason
KVM_EXIT_TDX_GET_TDVMCALL_INFOand its structure are added. Userspace is required to handle the exit reason as the initial support for TDX.
It doesn't look like a good and correct design.
Consider the case that userspace supports SetupEventNotifyInterrupt and returns bit 1 of leaf_output[0] as 1 to KVM, and KVM returns it to TD guest for TDVMCALL_GET_TD_VM_CALL_INFO. So TD guest treats it as SetupEventNotifyInterrupt is support. But when TD guest issues this TDVMCALL, KVM doesn't support the exit to userspace for this specific leaf and userspace doesn't have chance to handle it.
I have a proposal:
Implement KVM_CAP_TDX_USER_EXIT_TDVMCALL
- on check of this cap, KVM returns the bitmap of TDVMCALL leafs that allows userspace to opt-in the user exit; e.g.,
#define TDX_USER_EXIT_TDVMCALL_GETQUOTE BIT_ULL(0)
#define TDX_USER_EXIT_TDVMCALL_SetupEventNotifyInterrupt BIT_ULL(1)
...
- on enable of this cap, KVM allows userspace to opt-in which leaf will exit to usersapce for handling;
- KVM returns the result for leaf(r12)==1 of <GetTdVmCallInfo> based on the KVM_CAP_TDX_USER_EXIT_TDVMCALL enabled by userspace and its own capability;
If a non-base GHCI TDVMCALL is supported by KVM itself and no need exit to userspace, it's not reported in KVM_CAP_TDX_USER_EXIT_TDVMCALL so that usersapce cannot enable the user exit of this leaf. But KVM will still return the corresponding bit of this leaf as 1 for leaf(r12)==1 of <GetTdVmCallInfo> since it's supported by KVM itself.
If a non-base GHCI TDVMCALL is not supported by KVM itself but KVM allows/supports userspace to opt-in the user exit. e.g., <Getquote>. check of KVM_CAP_TDX_USER_EXIT_TDVMCALL will have the bit of this leaf set.
- only when usersapce enable the corresponding bit by calling enable_cap(KVM_CAP_TDX_USER_EXIT_TDVMCALL), will KVM return the corresponding bit of this leaf as 1 for leaf(r12)==1 of <GetTdVmCallInfo> and exit to userspace for such leaf.
- if userspace doesn't enable the corresponding bit with enable_cap(KVM_CAP_TDX_USER_EXIT_TDVMCALL). KVM will return 0 for the corresponding bit of this leaf for leaf(r12)==1 of <GetTdVmCallInfo> and return TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED when TDX guest issues such leaf;
If a non-base GHCI TDVMCALL is not supported by KVM and KVM doesn't allow/support userspace to opt-in the user exit. e.g., <SetupEventNotifyInterrupt>. check of KVM_CAP_TDX_USER_EXIT_TDVMCALL will not have the bit of this leaf set, so that userspace cannot opt-in the user exit of such leaf. leaf(r12)==1 of <GetTdVmCallInfo> return the bit of such leaf as 0 and TDX guest request of such leaf gets TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 5 lines · Xiaoyao Li
Quoted 4 lines
A new KVM exit reason
KVM_EXIT_TDX_GET_TDVMCALL_INFOand its structure are added. Userspace is required to handle the exit reason as the initial
support for TDX.
Quoted 8 lines
It doesn't look like a good and correct design.
Consider the case that userspace supports SetupEventNotifyInterrupt and returns bit 1 of leaf_output[0] as 1 to KVM, and KVM returns it to TD guest for
TDVMCALL_GET_TD_VM_CALL_INFO. So TD guest treats it as SetupEventNotifyInterrupt is support. But when TD guest issues this TDVMCALL, KVM doesn't support the exit to userspace for this specific leaf and userspace doesn't have chance to handle it.
Why do we need an opt-in interface instead of a way to expose which exit's are supported by KVM? I would think the need for a TDVMCALL opt-in interface would only come up if there was a bad guest that was making TDVMCALLs that it did not see in GetTdVmCallInfo. So that we would actually require an opt-in is not guaranteed.
Another consideration could be how to handle GetQuote for an eventual TDVMCALL opt-in interface, should it be needed. The problem would be GetQuote would be opted in by default and make the interface weird. But we may not want to have a TDVMCall specific opt-in interface. There could be other TDX behaviors that we need to opt-in around. In which case the opt-in interface could be more generic, and by implementing the TDVMCall opt-in interface ahead of time we would end up with two opt-in interfaces instead of one.
So how about just adding a field to struct kvm_tdx_capabilities to describe the KVM TDVMcalls? Or some other place? But don't invent an opt-in interface until/if we need it.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 14 lines · Rick Edgecombe
Why do we need an opt-in interface instead of a way to expose which exit's are supported by KVM? I would think the need for a TDVMCALL opt-in interface would only come up if there was a bad guest that was making TDVMCALLs that it did not see in GetTdVmCallInfo. So that we would actually require an opt-in is not guaranteed.
Another consideration could be how to handle GetQuote for an eventual TDVMCALL opt-in interface, should it be needed. The problem would be GetQuote would be opted in by default and make the interface weird. But we may not want to have a TDVMCall specific opt-in interface. There could be other TDX behaviors that we need to opt-in around. In which case the opt-in interface could be more generic, and by implementing the TDVMCall opt-in interface ahead of time we would end up with two opt-in interfaces instead of one.
Quoted 3 lines
So how about just adding a field to struct
kvm_tdx_capabilitiesto describe the KVM TDVMcalls? Or some other place? But don't invent an opt-in interface until/if we need it.
Oh, and there already is a hypercall exit opt-in interface, so KVM_CAP_TDX_USER_EXIT_TDVMCALL would overlap with it, right?
Re: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 5 lines · Rick P
Quoted 4 lines · Rick Edgecombe
Why do we need an opt-in interface instead of a way to expose which exit's are supported by KVM? I would think the need for a TDVMCALL opt-in interface would only come up if there was a bad guest that was making TDVMCALLs that it did not see in GetTdVmCallInfo.
The opt-in interface can eliminate some requirements for userspace. E.g, for GetQuote, this patch set enforces userspace to handle the exit reason due to GetQuote as the initial support, because KVM doesn't know if userspace is able to handle the exit reason or not without userspace's opt-in, unless it's handled by default in userspace.
Quoted 10 lines
Quoted 10 lines
So that we would actually require an opt-in is not
guaranteed.Another consideration could be how to handle GetQuote for an eventual TDVMCALL opt-in interface, should it be needed. The problem would be GetQuote would be opted in by default and make the interface weird. But we may not want to have a TDVMCall specific opt-in interface. There could be other TDX behaviors that we need to opt-in around. In which case the opt-in interface could be more generic, and by implementing the TDVMCall opt-in interface ahead of time we would end up with two opt-in interfaces instead of one.
Maybe we can use a TDX specific opt-in interface instead of TDVMCALL specific interface.
But not sure we should add it now or later.
Quoted 6 lines
Quoted 4 lines
So how about just adding a field to struct
kvm_tdx_capabilitiesto describe the KVM TDVMcalls? Or some other place? But don't invent an opt-in interface until/if we need it.Oh, and there already is a hypercall exit opt-in interface, so
KVM_CAP_TDX_USER_EXIT_TDVMCALLwould overlap with it, right?
Not sure what does "overlap" mean here.
They have different namespaces, so they don't impact each other.
Or did you mean it's a duplication both having KVM_CAP_EXIT_HYPERCALL and KVM_CAP_TDX_USER_EXIT_TDVMCALL?
Re: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 9 lines · Binbin Wu
Quoted 5 lines · Rick P
Quoted 4 lines · Rick Edgecombe
Why do we need an opt-in interface instead of a way to expose which exit's are supported by KVM? I would think the need for a TDVMCALL opt-in interface would only come up if there was a bad guest that was making TDVMCALLs that it did not see in GetTdVmCallInfo.
Quoted 5 lines
The opt-in interface can eliminate some requirements for userspace. E.g, for GetQuote, this patch set enforces userspace to handle the exit reason due to GetQuote as the initial support, because KVM doesn't know if userspace is able to handle the exit reason or not without userspace's opt-in, unless it's handled by default in userspace.
Beside it.
opt-in is not needed if we only care about <getquote> for now since KVM makes <getquote> exit to userspace unconditionally.
But to support any new TDVMCALL leaf that needs to exit userspace, we will have to use opt-in. So to me, implement the opt-in at the first place as a common interface for all the optional tdvmcall leafs instead of making <getquote> specific is not a bad idea to me, especially the opt-in implementation doesn't look complicated.
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
So that we would actually require an opt-in is not guaranteed.
Another consideration could be how to handle GetQuote for an eventual TDVMCALL opt-in interface, should it be needed. The problem would be GetQuote would be opted in by default and make the interface weird. But we may not want to have a TDVMCall specific opt-in interface. There could be other TDX behaviors that we need to opt-in around. In which case the opt-in interface could be more generic, and by implementing the TDVMCall opt-in interface ahead of time we would end up with two opt-in interfaces instead of one.
Quoted 3 lines
Maybe we can use a TDX specific opt-in interface instead of TDVMCALL specific interface.
But not sure we should add it now or later.
For simplicity, I prefer separate opt-in interfaces, it makes code simpler.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 6 lines · Xiaoyao Li
Quoted 4 lines
Maybe we can use a TDX specific opt-in interface instead of TDVMCALL specific
interface.
But not sure we should add it now or later.For simplicity, I prefer separate opt-in interfaces, it makes code simpler.
What is the problem with using the existing exit opt-in interface?
Re: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 9 lines · Rick P
Quoted 6 lines · Xiaoyao Li
Quoted 4 lines
Maybe we can use a TDX specific opt-in interface instead of TDVMCALL specific
interface.
But not sure we should add it now or later.For simplicity, I prefer separate opt-in interfaces, it makes code simpler.
What is the problem with using the existing exit opt-in interface?
It mixes up common KVM defined hypercall leafs (KVM_HC_*) with TDX specific TDVMCALL leafs. Surely it can work but it just doesn't look clean to me.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 5 lines · Xiaoyao Li
Quoted 1 line
What is the problem with using the existing exit opt-in interface?
It mixes up common KVM defined hypercall leafs (KVM_HC_*) with TDX specific TDVMCALL leafs. Surely it can work but it just doesn't look clean to me.
- TDVMCallInfo is used to expose which TDVMCalls are expose to the guest
(clean)
- KVM HC opt-in is used to expose which exits KVM should fwd to userspace
(seems clean to me)
There is an asymmetry in that what is returned in TDVMCallInfo can't be blindly dumped into a KVM opt-in interface. But the logic matching TDVMCallInfo leaf <-> KVM HC needs to be somewhere. The default place should be userspace.
Did I miss anything?
Re: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
In reply to Xiaoyao LiQuoted 27 lines · Xiaoyao Li
Quoted 26 lines · Binbin Wu
Exit to userspace for TDG.VP.VMCALL<GetTdVmCallInfo> via a new KVM exit reason to allow userspace to provide information about the support of TDVMCALLs when r12 is 1 for the TDVMCALLs beyond the GHCI base API.
GHCI spec defines the GHCI base TDVMCALLs: <GetTdVmCallInfo>, <MapGPA>, <ReportFatalError>, <Instruction.CPUID>, <#VE.RequestMMIO>, <Instruction.HLT>, <Instruction.IO>, <Instruction.RDMSR> and <Instruction.WRMSR>. They must be supported by VMM to support TDX guests.
For GetTdVmCallInfo
- When leaf (r12) to enumerate TDVMCALL functionality is set to 0, successful execution indicates all GHCI base TDVMCALLs listed above are supported.Update the KVM TDX document with the set of the GHCI base APIs.
- When leaf (r12) to enumerate TDVMCALL functionality is set to 1, it indicates the TDX guest is querying the supported TDVMCALLs beyond the GHCI base TDVMCALLs.
Exit to userspace to let userspace set the TDVMCALL sub-function bit(s) accordingly to the leaf outputs. KVM could set the TDVMCALL bit(s) supported by itself when the TDVMCALLs don't need support from userspace after returning from userspace and before entering guest. Currently, no such TDVMCALLs implemented, KVM just sets the values returned from userspace.
Quoted 7 lines
Quoted 3 lines
A new KVM exit reason
KVM_EXIT_TDX_GET_TDVMCALL_INFOand its structure are added. Userspace is required to handle the exit reason as the initial support for TDX.It doesn't look like a good and correct design.
Consider the case that userspace supports SetupEventNotifyInterrupt and returns bit 1 of leaf_output[0] as 1 to KVM, and KVM returns it to TD guest for
TDVMCALL_GET_TD_VM_CALL_INFO. So TD guest treats it as SetupEventNotifyInterrupt is support. But when TD guest issues this TDVMCALL, KVM doesn't support the exit to userspace for this specific leaf and userspace doesn't have chance to handle it.
Previously, I also had the idea of setting the information based on userspace's opt-ins.
But for simplicity, this patch set doesn't adopt the opt-in mechanism for KVM exit reasons due to TDVMCALLs.
To resolve the issue you mentions that userspace could set a bit that KVM doesn't support the exit to userspace, KVM could mask off the bit(s) not supported by KVM before returning back to guest.
If we agree that it's the right time to have the opt-in, we could go the opt-in way and KVM could set the information based on userspace's opt-ins without exit to userspace for GetTdVmCallInfo.
Re: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 40 lines · Binbin Wu
Quoted 27 lines · Xiaoyao Li
Quoted 26 lines · Binbin Wu
Exit to userspace for TDG.VP.VMCALL<GetTdVmCallInfo> via a new KVM exit reason to allow userspace to provide information about the support of TDVMCALLs when r12 is 1 for the TDVMCALLs beyond the GHCI base API.
GHCI spec defines the GHCI base TDVMCALLs: <GetTdVmCallInfo>, <MapGPA>, <ReportFatalError>, <Instruction.CPUID>, <#VE.RequestMMIO>, <Instruction.HLT>, <Instruction.IO>, <Instruction.RDMSR> and <Instruction.WRMSR>. They must be supported by VMM to support TDX guests.
For GetTdVmCallInfo
- When leaf (r12) to enumerate TDVMCALL functionality is set to 0, successful execution indicates all GHCI base TDVMCALLs listed above are supported.Update the KVM TDX document with the set of the GHCI base APIs.
- When leaf (r12) to enumerate TDVMCALL functionality is set to 1, it indicates the TDX guest is querying the supported TDVMCALLs beyond the GHCI base TDVMCALLs.
Exit to userspace to let userspace set the TDVMCALL sub-function bit(s) accordingly to the leaf outputs. KVM could set the TDVMCALL bit(s) supported by itself when the TDVMCALLs don't need support from userspace after returning from userspace and before entering guest. Currently, no such TDVMCALLs implemented, KVM just sets the values returned from userspace.Quoted 7 lines
Quoted 3 lines
A new KVM exit reason
KVM_EXIT_TDX_GET_TDVMCALL_INFOand its structure are added. Userspace is required to handle the exit reason as the initial support for TDX.It doesn't look like a good and correct design.
Consider the case that userspace supports SetupEventNotifyInterrupt and returns bit 1 of leaf_output[0] as 1 to KVM, and KVM returns it to TD guest for
TDVMCALL_GET_TD_VM_CALL_INFO. So TD guest treats it as SetupEventNotifyInterrupt is support. But when TD guest issues this TDVMCALL, KVM doesn't support the exit to userspace for this specific leaf and userspace doesn't have chance to handle it.Previously, I also had the idea of setting the information based on userspace's opt-ins.
Quoted 6 lines
But for simplicity, this patch set doesn't adopt the opt-in mechanism for KVM exit reasons due to TDVMCALLs.
To resolve the issue you mentions that userspace could set a bit that KVM doesn't support the exit to userspace, KVM could mask off the bit(s) not supported by KVM before returning back to guest.
silently mask off the value provided from userspace is not a stable ABI from userspace perspective. A kvm cap to report what value is allowed/supported is still helpful.
Quoted 3 lines
If we agree that it's the right time to have the opt-in, we could go the opt-in way and KVM could set the information based on userspace's opt-ins without exit to userspace for GetTdVmCallInfo.
Let's see what Paolo and Sean will say.
Sean Christopherson
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 62 lines · Xiaoyao Li
Quoted 50 lines · Binbin Wu
Quoted 38 lines · Xiaoyao Li
Quoted 31 lines · Binbin Wu
Exit to userspace for TDG.VP.VMCALL<GetTdVmCallInfo> via a new KVM exit reason to allow userspace to provide information about the support of TDVMCALLs when r12 is 1 for the TDVMCALLs beyond the GHCI base API.
GHCI spec defines the GHCI base TDVMCALLs: <GetTdVmCallInfo>, <MapGPA>, <ReportFatalError>, <Instruction.CPUID>, <#VE.RequestMMIO>, <Instruction.HLT>, <Instruction.IO>, <Instruction.RDMSR> and <Instruction.WRMSR>. They must be supported by VMM to support TDX guests.
For GetTdVmCallInfo
- When leaf (r12) to enumerate TDVMCALL functionality is set to 0, successful execution indicates all GHCI base TDVMCALLs listed above are
supported.Update the KVM TDX document with the set of the GHCI base APIs.
- When leaf (r12) to enumerate TDVMCALL functionality is set to 1, it indicates the TDX guest is querying the supported TDVMCALLs beyond the GHCI base TDVMCALLs.
Exit to userspace to let userspace set the TDVMCALL sub-function bit(s)
accordingly to the leaf outputs. KVM could set the TDVMCALL bit(s) supported by itself when the TDVMCALLs don't need support from userspace
after returning from userspace and before entering guest. Currently, no
such TDVMCALLs implemented, KVM just sets the values returned from userspace.Quoted 5 lines
A new KVM exit reason
KVM_EXIT_TDX_GET_TDVMCALL_INFOand its structure
are added. Userspace is required to handle the exit reason as the initial
support for TDX.Quoted 8 lines
It doesn't look like a good and correct design.
Consider the case that userspace supports SetupEventNotifyInterrupt and returns bit 1 of leaf_output[0] as 1 to KVM, and KVM returns it to TD guest for
TDVMCALL_GET_TD_VM_CALL_INFO. So TD guest treats it as SetupEventNotifyInterrupt is support. But when TD guest issues this TDVMCALL, KVM doesn't support the exit to userspace for this specific leaf and userspace doesn't have chance to handle it.Previously, I also had the idea of setting the information based on userspace's opt-ins.
Quoted 6 lines
But for simplicity, this patch set doesn't adopt the opt-in mechanism for KVM exit reasons due to TDVMCALLs.
To resolve the issue you mentions that userspace could set a bit that KVM doesn't support the exit to userspace, KVM could mask off the bit(s) not supported by KVM before returning back to guest.
silently mask off the value provided from userspace is not a stable ABI from userspace perspective. A kvm cap to report what value is allowed/supported is still helpful.
Quoted 5 lines
Quoted 3 lines
If we agree that it's the right time to have the opt-in, we could go the opt-in way and KVM could set the information based on userspace's opt-ins without exit to userspace for GetTdVmCallInfo.
Let's see what Paolo and Sean will say.
Kicking this to userspace seems premature. AIUI, no "optional" VMCALL features are defined at this time, i.e. there's nothing to enumerate. And there's no guarantee that there will ever be capabilties that require enumeration from *userspace*. E.g. if fancy feature XYZ requires enumeration, but that feature requires explicit KVM support, then forcing userspace will be messy.
So I don't see why KVM should anything other than return '0' to the guest (or whatever value says "there's nothing here").
Edgecombe, Rick P
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 11 lines · Sean Christopherson
Quoted 1 line
Let's see what Paolo and Sean will say.
Kicking this to userspace seems premature. AIUI, no "optional" VMCALL features
are defined at this time, i.e. there's nothing to enumerate. And there's no guarantee that there will ever be capabilties that require enumeration from *userspace*. E.g. if fancy feature XYZ requires enumeration, but that feature requires explicit KVM support, then forcing userspace will be messy.So I don't see why KVM should anything other than return '0' to the guest (or whatever value says "there's nothing here").
GetQuote is not part of the "Base" TDVMCALLs and so has a bit in GetTdVmCallInfo. We could move it to base?
Paolo seemed keen on GetTdVmCallInfo exiting to userspace, but this was before the spec overhaul.
Sean Christopherson
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 10 lines · Rick P Edgecombe
Quoted 9 lines · Sean Christopherson
Quoted 1 line
Let's see what Paolo and Sean will say.
Kicking this to userspace seems premature. AIUI, no "optional" VMCALL features
are defined at this time, i.e. there's nothing to enumerate. And there's no guarantee that there will ever be capabilties that require enumeration from *userspace*. E.g. if fancy feature XYZ requires enumeration, but that feature requires explicit KVM support, then forcing userspace will be messy.
Quoted 5 lines
Quoted 2 lines
So I don't see why KVM should anything other than return '0' to the guest (or whatever value says "there's nothing here").
GetQuote is not part of the "Base" TDVMCALLs and so has a bit in GetTdVmCallInfo. We could move it to base?
Is GetQuote actually optional? TDX without attestation seems rather pointless.
Quoted 2 lines
Paolo seemed keen on GetTdVmCallInfo exiting to userspace, but this was before the spec overhaul.
If GetQuote is truly optional, then exiting to userspace makes sense. But as above, that seems odd to me.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 5 lines · Sean Christopherson
Quoted 2 lines
GetQuote is not part of the "Base" TDVMCALLs and so has a bit in GetTdVmCallInfo. We could move it to base?
Is GetQuote actually optional? TDX without attestation seems rather pointless.
I don't know if that was a consideration for why it got added to the optional category. The inputs were gathered from more than just Linux.
Quoted 7 lines
Quoted 4 lines
Paolo seemed keen on GetTdVmCallInfo exiting to userspace, but this was before
the spec overhaul.If GetQuote is truly optional, then exiting to userspace makes sense. But as above, that seems odd to me.
Let us check on adding it to the "base" TDVMCALLs. If there is some good reason, we can post v2 with GetTdVmCallInfo exiting. This will probably be Friday due to the required people being out. Does it work for the timelines?
Under the change to add GetQuote to the base TDVMCALLs, patch 1 and 2 would be the only changes anyway. It starts to feel a little fast and loose, but another option is just to take patches 1 and 2, and we can finalize GetTdVmCallInfo at whatever pace is best.
Xiaoyao, Binbin do you see any pitfalls to that? If we add GetTdVmCallInfo as a new exit, and we could add the supported TDVMCALLs to struct kvm_tdx_capabilities at that point. If it doesn't make it for 6.16, it gets an opt-in.
Sean Christopherson
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 9 lines · Rick P Edgecombe
Quoted 5 lines · Sean Christopherson
Quoted 2 lines
GetQuote is not part of the "Base" TDVMCALLs and so has a bit in GetTdVmCallInfo. We could move it to base?
Is GetQuote actually optional? TDX without attestation seems rather pointless.
I don't know if that was a consideration for why it got added to the optional category. The inputs were gathered from more than just Linux.
If there's an actual use case for TDX without attestation, then by all means, make it optional. I'm genuinely curious if there's a hypervisor that plans on productizing TDX without supporting attestation. It's entirely possible (likely?) I'm missing or forgetting something.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 9 lines · Sean Christopherson
Quoted 3 lines
I don't know if that was a consideration for why it got added to the optional
category. The inputs were gathered from more than just Linux.If there's an actual use case for TDX without attestation, then by all means, make it optional. I'm genuinely curious if there's a hypervisor that plans on productizing TDX without supporting attestation. It's entirely possible (likely?)
I'm missing or forgetting something.
Ok, will check back in with the story.
The only things I could think of are:
1. TDX usage as a hardening thing, similar to unmapping guest memory for all page tables in the host.
2. Some highly coupled guest/VMM has an alternate attestation scheme.
More likely it was to retroactively bring the initial KVM PR into spec. We got some pretty specific direction from Paolo to explore GetTdVmCallInfo exiting, so it didn't make much of a difference one way or the other until now.
Re: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
In reply to Sean ChristophersonQuoted 8 lines · Sean Christopherson
Quoted 7 lines · Rick P Edgecombe
Quoted 5 lines · Sean Christopherson
Quoted 2 lines
GetQuote is not part of the "Base" TDVMCALLs and so has a bit in GetTdVmCallInfo. We could move it to base?
Is GetQuote actually optional? TDX without attestation seems rather pointless.
Quoted 7 lines
Quoted 2 lines
I don't know if that was a consideration for why it got added to the optional category. The inputs were gathered from more than just Linux.
If there's an actual use case for TDX without attestation, then by all means, make it optional. I'm genuinely curious if there's a hypervisor that plans on productizing TDX without supporting attestation. It's entirely possible (likely?) I'm missing or forgetting something.
With no intention to disrupt this discussion, but even w/o GetQuote TDX can also support attestation, because TD can just get the TDREPORT and send to remote Quoting Enclave to get it signed, via whatever communication channel available (vsock, TCP/IP etc). :-)
It's just not all TDX guests have those communication channels available in CSP's deployment, and GetQuote can fill up the hole as a last resort.
Of course now TD userspace may choose to only support GetQuote simply because kernel supports "unified ABI" to return remotely verifiable blob across vendors, but still ...
Edgecombe, Rick P
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
Quoted 11 lines · Kai
With no intention to disrupt this discussion, but even w/o GetQuote TDX can also support attestation, because TD can just get the TDREPORT and send to remote Quoting Enclave to get it signed, via whatever communication channel available (vsock, TCP/IP etc). :-)
It's just not all TDX guests have those communication channels available in CSP's deployment, and GetQuote can fill up the hole as a last resort.
Of course now TD userspace may choose to only support GetQuote simply because kernel supports "unified ABI" to return remotely verifiable blob across vendors, but still ...
This was what I was getting at by "Some highly coupled guest/VMM has an alternate attestation scheme". If you don't care about running on any VMM, you could invent your own communication channel. But the same is true for the other TDVMCALLs as well.
I guess the way to look at it is what is the MVP for a TD that expects to run on general VMMs. The others could also invent their own enumeration API and return failure for GetTdVmCallInfo.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 3/4] KVM: TDX: Exit to userspace for GetTdVmCallInfo
In reply to Sean ChristophersonQuoted 8 lines · Sean Christopherson
Quoted 7 lines · Rick P Edgecombe
Quoted 5 lines · Sean Christopherson
Quoted 2 lines
GetQuote is not part of the "Base" TDVMCALLs and so has a bit in GetTdVmCallInfo. We could move it to base?
Is GetQuote actually optional? TDX without attestation seems rather pointless.
Quoted 7 lines
Quoted 2 lines
I don't know if that was a consideration for why it got added to the optional category. The inputs were gathered from more than just Linux.
If there's an actual use case for TDX without attestation, then by all means, make it optional. I'm genuinely curious if there's a hypervisor that plans on productizing TDX without supporting attestation. It's entirely possible (likely?) I'm missing or forgetting something.
It turns out there is a VMM that has a proprietary interface that they use instead of GetQuote.
RFC PATCH 4/4 KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
In reply to Binbin WuCheck userspace has enabled KVM exit on KVM_HC_MAP_GPA_RANGE during KVM_TDX_FINALIZE_VM.
TDVMCALL_MAP_GPA is one of the GHCI base TDVMCALLs, so it must be implemented by VMM to support TDX guests. KVM converts TDVMCALL_MAP_GPA to KVM_HC_MAP_GPA_RANGE, which requires userspace to enable KVM_CAP_EXIT_HYPERCALL with KVM_HC_MAP_GPA_RANGE bit set. Check it when userspace requests KVM_TDX_FINALIZE_VM, so that there is no need to check it during TDX guests running.
--- Documentation/virt/kvm/x86/intel-tdx.rst | 3 +++ arch/x86/kvm/vmx/tdx.c | 20 +++++++++----------- 2 files changed, 12 insertions(+), 11 deletions(-)
Documentation/virt/kvm/x86/intel-tdx.rst +3 -0
@@ -246,6 +246,9 @@ control flow is as follows: * Prepare content of initial guest memory. * KVM_TDX_INIT_MEM_REGION: Add initial guest memory. * KVM_TDX_FINALIZE_VM: Finalize the measurement of the TDX guest. + Note: To support TDVMCALL_MAP_GPA, userspace must opt-in + KVM_CAP_EXIT_HYPERCALL with KVM_HC_MAP_GPA_RANGE before calling + KVM_TDX_FINALIZE_VM. #. Run VCPU
arch/x86/kvm/vmx/tdx.c +9 -11
@@ -1209,17 +1209,6 @@ static int tdx_map_gpa(struct kvm_vcpu *vcpu)u64 size = tdx->vp_enter_args.r13;u64 ret; -/* -* Converting TDVMCALL_MAP_GPA to KVM_HC_MAP_GPA_RANGE requires -* userspace to enable KVM_CAP_EXIT_HYPERCALL with KVM_HC_MAP_GPA_RANGE -* bit set. If not, the error code is not defined in GHCI for TDX, use -* TDVMCALL_STATUS_INVALID_OPERAND for this case. -*/ -if (!user_exit_on_hypercall (vcpu->kvm,KVM_HC_MAP_GPA_RANGE )) { - ret =TDVMCALL_STATUS_INVALID_OPERAND ; - goto error; - } - if (gpa + size <= gpa || !kvm_vcpu_is_legal_gpa(vcpu, gpa) || !kvm_vcpu_is_legal_gpa(vcpu, gpa + size - 1) || (vt_is_tdx_private_gpa(vcpu->kvm, gpa) !=
@@ -2821,6 +2810,15 @@ static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +/* +* TDVMCALL_MAP_GPA is one of the GHCI base TDVMCALLs, so it must be +* implemented by VMM to support TDX guests. KVM converts +* TDVMCALL_MAP_GPA to KVM_HC_MAP_GPA_RANGE, which requires userspace to +* enable KVM_CAP_EXIT_HYPERCALL with KVM_HC_MAP_GPA_RANGE bit set. +*/ +if (!user_exit_on_hypercall (kvm,KVM_HC_MAP_GPA_RANGE )) +return -EINVAL ; +guard (mutex)(&kvm->slots_lock);if (!is_hkid_assigned (kvm_tdx) || kvm_tdx->state ==TD_STATE_RUNNABLE )
Signature · 1 line
2.46.0
Edgecombe, Rick P
maintainerRe: [RFC PATCH 4/4] KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
Quoted 11 lines · Binbin Wu
Check userspace has enabled KVM exit on
KVM_HC_MAP_GPA_RANGEduringKVM_TDX_FINALIZE_VM.
TDVMCALL_MAP_GPAis one of the GHCI base TDVMCALLs, so it must be implemented by VMM to support TDX guests. KVM convertsTDVMCALL_MAP_GPAtoKVM_HC_MAP_GPA_RANGE, which requires userspace to enableKVM_CAP_EXIT_HYPERCALLwithKVM_HC_MAP_GPA_RANGEbit set. Check it when userspace requestsKVM_TDX_FINALIZE_VM, so that there is no need to check it during TDX guests running.Signed-off-by Binbin Wu <[email protected]>
Do we need this change? It seems reasonable, but I don't think we need KVM to ensure that userspace creates a TD that meets the GHCI spec. So I'm not sure about the justification.
It seems like the reasoning could be just to shrink the possible configurations KVM has to think about, and that we only have the option to do this now before the ABI becomes harder to change.
Did you need any QEMU changes as a result of this patch?
Wait, actually I think the patch is wrong, because KVM_CAP_EXIT_HYPERCALL could be called again after KVM_TDX_FINALIZE_VM. In which case userspace could get an exit unexpectedly. So should we drop this patch?
Sean Christopherson
maintainerRe: [RFC PATCH 4/4] KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
Quoted 11 lines · Rick P Edgecombe
Quoted 10 lines · Binbin Wu
Check userspace has enabled KVM exit on
KVM_HC_MAP_GPA_RANGEduringKVM_TDX_FINALIZE_VM.
TDVMCALL_MAP_GPAis one of the GHCI base TDVMCALLs, so it must be implemented by VMM to support TDX guests. KVM convertsTDVMCALL_MAP_GPAtoKVM_HC_MAP_GPA_RANGE, which requires userspace to enableKVM_CAP_EXIT_HYPERCALLwithKVM_HC_MAP_GPA_RANGEbit set. Check it when userspace requestsKVM_TDX_FINALIZE_VM, so that there is no need to check it during TDX guests running.
Quoted 4 lines
Quoted 1 line
Signed-off-by Binbin Wu <[email protected]>Do we need this change? It seems reasonable, but I don't think we need KVM to ensure that userspace creates a TD that meets the GHCI spec.
+1. We do need to be careful about unintentionally creating ABI, but generally speaking KVM shouldn't police userspace.
Quoted 11 lines
So I'm not sure about the justification.
It seems like the reasoning could be just to shrink the possible configurations KVM has to think about, and that we only have the option to do this now before the ABI becomes harder to change.
Did you need any QEMU changes as a result of this patch?
Wait, actually I think the patch is wrong, because
KVM_CAP_EXIT_HYPERCALLcould be called again afterKVM_TDX_FINALIZE_VM. In which case userspace could get an exit unexpectedly. So should we drop this patch?
Yes, drop it.
Re: [RFC PATCH 4/4] KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
Quoted 12 lines · Sean Christopherson
Quoted 11 lines · Rick P Edgecombe
Quoted 10 lines · Binbin Wu
Check userspace has enabled KVM exit on
KVM_HC_MAP_GPA_RANGEduringKVM_TDX_FINALIZE_VM.
TDVMCALL_MAP_GPAis one of the GHCI base TDVMCALLs, so it must be implemented by VMM to support TDX guests. KVM convertsTDVMCALL_MAP_GPAtoKVM_HC_MAP_GPA_RANGE, which requires userspace to enableKVM_CAP_EXIT_HYPERCALLwithKVM_HC_MAP_GPA_RANGEbit set. Check it when userspace requestsKVM_TDX_FINALIZE_VM, so that there is no need to check it during TDX guests running.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Binbin Wu <[email protected]>Do we need this change? It seems reasonable, but I don't think we need KVM to ensure that userspace creates a TD that meets the GHCI spec.
+1. We do need to be careful about unintentionally creating ABI, but generally speaking KVM shouldn't police userspace.
OK.
Quoted 6 lines
Quoted 6 lines
So I'm not sure about the justification.
It seems like the reasoning could be just to shrink the possible configurations KVM has to think about, and that we only have the option to do this now before the ABI becomes harder to change.
Quoted 7 lines
Quoted 5 lines
Did you need any QEMU changes as a result of this patch?
Wait, actually I think the patch is wrong, because
KVM_CAP_EXIT_HYPERCALLcould be called again afterKVM_TDX_FINALIZE_VM. In which case userspace could get an exit unexpectedly. So should we drop this patch?Yes, drop it.
So, when the TDX guest calls MapGPA and KVM finds userspace doesn't opt-in KVM_HC_MAP_GPA_RANGE, just return error to userspace?
Sean Christopherson
maintainerRe: [RFC PATCH 4/4] KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
Quoted 8 lines · Binbin Wu
Quoted 7 lines · Sean Christopherson
Quoted 6 lines · Rick P Edgecombe
It seems like the reasoning could be just to shrink the possible configurations KVM has to think about, and that we only have the option to do this now before the ABI becomes harder to change.
Did you need any QEMU changes as a result of this patch?
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Wait, actually I think the patch is wrong, because
KVM_CAP_EXIT_HYPERCALLcould be called again afterKVM_TDX_FINALIZE_VM. In which case userspace could get an exit unexpectedly. So should we drop this patch?Yes, drop it.
So, when the TDX guest calls MapGPA and KVM finds userspace doesn't opt-in
KVM_HC_MAP_GPA_RANGE, just return error to userspace?
Why can't KVM just do what it already does, and return an error to the guest?
if (!user_exit_on_hypercall(vcpu->kvm, KVM_HC_MAP_GPA_RANGE)) {
ret = TDVMCALL_STATUS_INVALID_OPERAND;
goto error;
}Re: [RFC PATCH 4/4] KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
Quoted 12 lines · Sean Christopherson
Quoted 11 lines · Binbin Wu
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Rick P Edgecombe
It seems like the reasoning could be just to shrink the possible configurations KVM has to think about, and that we only have the option to do this now before the ABI becomes harder to change.
Did you need any QEMU changes as a result of this patch?
Wait, actually I think the patch is wrong, because
KVM_CAP_EXIT_HYPERCALLcould be called again afterKVM_TDX_FINALIZE_VM. In which case userspace could get an exit unexpectedly. So should we drop this patch?
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Yes, drop it.
So, when the TDX guest calls MapGPA and KVM finds userspace doesn't opt-in
KVM_HC_MAP_GPA_RANGE, just return error to userspace?Why can't KVM just do what it already does, and return an error to the guest?
Because GHCI requires it must be supported. No matter with the old GHCI that only allows <GetTdVmCallInfo> to succeed and the success of <GetTdVmCallInfo> means all the TDVMCALL leafs are support, or the proposed updated GHCI that defines <MapGpa> as one of the base API/leaf, and the base API must be supported by VMM.
Binbin wants to honor it.
Quoted 4 lines
if (!user_exit_on_hypercall(vcpu->kvm, KVM_HC_MAP_GPA_RANGE)) {
ret = TDVMCALL_STATUS_INVALID_OPERAND;
goto error;
}Edgecombe, Rick P
maintainerRe: [RFC PATCH 4/4] KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
Quoted 6 lines · Xiaoyao Li
Quoted 5 lines
Quoted 2 lines
So, when the TDX guest calls MapGPA and KVM finds userspace doesn't opt-in
KVM_HC_MAP_GPA_RANGE, just return error to userspace?Why can't KVM just do what it already does, and return an error to the guest?
Quoted 7 lines
Because GHCI requires it must be supported. No matter with the old GHCI that only allows <GetTdVmCallInfo> to succeed and the success of <GetTdVmCallInfo> means all the TDVMCALL leafs are support, or the proposed updated GHCI that defines <MapGpa> as one of the base API/leaf, and the base API must be supported by VMM.
Binbin wants to honor it.
But KVM doesn't need to support all ways that userspace could meet the GHCI spec. If userspace opts-in to the exit, they will meet the spec. If they configure KVM differently then they wont, but this is their decision.
Re: [RFC PATCH 4/4] KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
Quoted 13 lines · Rick P
Quoted 12 lines · Xiaoyao Li
Quoted 5 lines
Quoted 2 lines
So, when the TDX guest calls MapGPA and KVM finds userspace doesn't opt-in
KVM_HC_MAP_GPA_RANGE, just return error to userspace?Why can't KVM just do what it already does, and return an error to the guest?
Because GHCI requires it must be supported. No matter with the old GHCI that only allows <GetTdVmCallInfo> to succeed and the success of <GetTdVmCallInfo> means all the TDVMCALL leafs are support, or the proposed updated GHCI that defines <MapGpa> as one of the base API/leaf, and the base API must be supported by VMM.
Quoted 5 lines
Quoted 1 line
Binbin wants to honor it.
But KVM doesn't need to support all ways that userspace could meet the GHCI spec. If userspace opts-in to the exit, they will meet the spec. If they configure KVM differently then they wont, but this is their decision.
I agree with you and Sean. And I'm trying to answer Sean's question on behalf of Binbin.
Strictly speaking, KVM can be blamed for some reason. Because it is KVM that returns success for <GetTdVmCallInfo> unconditionally when r12 == 0
to report that all the (base) leafs are supported.
But I totally agree with KVM cannot guarantee userspace will behave correctly. Even with this patch that KVM mandates the userspace to enable user exit of KVM_HC_MAP_GPA_RANGE, it's still possible for a misbehaved userspace to error to TD guest on KVM_HC_MAP_GPA_RANGE and breaks the semantics of successful <GetTdVmCallInfo>.
So I'm with you and Sean.
Re: [RFC PATCH 4/4] KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
Quoted 14 lines · Xiaoyao Li
Quoted 13 lines · Rick P
Quoted 12 lines · Xiaoyao Li
Quoted 5 lines
Quoted 2 lines
So, when the TDX guest calls MapGPA and KVM finds userspace doesn't opt-in
KVM_HC_MAP_GPA_RANGE, just return error to userspace?Why can't KVM just do what it already does, and return an error to the guest?
Because GHCI requires it must be supported. No matter with the old GHCI that only allows <GetTdVmCallInfo> to succeed and the success of <GetTdVmCallInfo> means all the TDVMCALL leafs are support, or the proposed updated GHCI that defines <MapGpa> as one of the base API/leaf, and the base API must be supported by VMM.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Binbin wants to honor it.
But KVM doesn't need to support all ways that userspace could meet the GHCI spec. If userspace opts-in to the exit, they will meet the spec. If they configure KVM differently then they wont, but this is their decision.
I agree with you and Sean. And I'm trying to answer Sean's question on behalf of Binbin.
Yes, it was my thought.
Quoted 7 lines
Strictly speaking, KVM can be blamed for some reason. Because it is KVM that returns success for <GetTdVmCallInfo> unconditionally when r12 == 0 to report that all the (base) leafs are supported.
But I totally agree with KVM cannot guarantee userspace will behave correctly. Even with this patch that KVM mandates the userspace to enable user exit of
KVM_HC_MAP_GPA_RANGE, it's still possible for a misbehaved userspace to error to TD guest onKVM_HC_MAP_GPA_RANGEand breaks the semantics of successful <GetTdVmCallInfo>.So I'm with you and Sean.
Also see my reply
https://lore.kernel.org/kvm/[email protected]/
In the next version, how about combining this patch to patch 1 to use TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED if userspace doesn't opt-in KVM_HC_MAP_GPA_RANGE?
Re: [RFC PATCH 4/4] KVM: TDX: Check KVM exit on KVM_HC_MAP_GPA_RANGE when TD finalize
In reply to Sean ChristophersonQuoted 14 lines · Sean Christopherson
Quoted 13 lines · Binbin Wu
Quoted 12 lines · Sean Christopherson
Quoted 9 lines · Rick P Edgecombe
It seems like the reasoning could be just to shrink the possible configurations KVM has to think about, and that we only have the option to do this now before the ABI becomes harder to change.
Did you need any QEMU changes as a result of this patch?
Wait, actually I think the patch is wrong, because
KVM_CAP_EXIT_HYPERCALLcould be called again afterKVM_TDX_FINALIZE_VM. In which case userspace could get an exit unexpectedly. So should we drop this patch?Yes, drop it.
Quoted 9 lines
Quoted 2 lines
So, when the TDX guest calls MapGPA and KVM finds userspace doesn't opt-in
KVM_HC_MAP_GPA_RANGE, just return error to userspace?Why can't KVM just do what it already does, and return an error to the guest?
if (!user_exit_on_hypercall(vcpu->kvm, KVM_HC_MAP_GPA_RANGE)) { ret = TDVMCALL_STATUS_INVALID_OPERAND; goto error; }
My previous thought was MapGpa is in base GHIC API.
Userspace is required to opt-in KVM_HC_MAP_GPA_RANGE.
If not, it's userspace's responsibility, so I thought exit to userspace with error may be better.
If return an error code is preferred, now it has a new status code TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED to use.
Basically, if the MapGpa is not support, either choice will stop VM from execution. But had a second thought, returning an error code to guest allows guest to choose to continue or not if MapGpa failed, though I can't imagine what case it is.