KVM TDX basic feature support
Subsystems (6)
v12 cover KVM TDX basic feature support
From: Isaku Yamahata <[email protected]>
KVM TDX basic feature support
Hello. This is v12 the patch series vof KVM TDX support. This is based on v6.2-rc7 + the following patch series.
Related patch series This patch is based on:
- fd-based approach for supporing KVM v10 and
https://lore.kernel.org/lkml/[email protected]/
[email protected]:sean-jc/linux.git x86/upm_base_support
- TDX host kernel support v9
https://lore.kernel.org/lkml/[email protected]/T/#m2562c7f18eabae171793ed7ea81b3544d7af09c4
The tree can be found at https://github.com/intel/tdx/tree/kvm-upstream
How to run/test: It's describe at https://github.com/intel/tdx/wiki/TDX-KVM
This patch still adopts freezing sept entry instead of detecting error of TDX SEAMCALL because it's fragile to detect race of SEAMCALL on same Secure-EPT. The error code of SEAMCALL doesn't necessarily provides information so that the caller, KVM TDX code, can't reliable determin if it's due to race or bug.
Changes from v11:
- ABI change of KVM_TDX_VM_INIT
- Split the hook of TDP MMU to not modify handle_changed_spte()
- Enhanced commit message on mtrr patch
- Made KVM_CAP_MAX_VCPUS to x86 specific
Thanks,
Isaku Yamahata
Changes from v10:
- rebased to v6.2-rc3
- support mtrr with its own patches
- Integrated fd-based private page v10
- Integrated TDX host kernel support v8
- Integrated kvm_init rework v2
- removed struct tdx_td_page and its initialization logic
- cleaned up mmio spte and require enable_mmio_caching=true for TDX
- removed dubious WARN_ON_ONCE()
- split a patch adding methods as nop into several patches
Changes from v9:
- rebased to v6.1-rc2
- Integrated fd-based private page v9 as prerequisite.
- Integrated TDX host kernel support v6
- TDP MMU: Make handle_change_spte() return value.
- TDX: removed seamcall_lock and return -EAGAIN so that TDP MMU can retry
Changes from v8:
- rebased to v6.0-rc7
- Integrated with kvm hardware initialization. Check all packages has at least one online CPU when creating guest TD and refuse cpu offline during guest TDs are running.
- Integrated fd-based private page v8 as prerequisite.
- TDP MMU: Introduced more callbacks instead of single callback.
Changes from v7:
- Use xarray to track whether GFN is private or shared. Drop SPTE_SHARED_MASK. The complex state machine with SPTE_SHARED_MASK was ditched.
- Large page support is implemented. But will be posted as independent RFC patch.
- fd-based private page v7 is integrated. This is mostly same to Chao's patches. It's in github.
Changes from v6:
- rebased to v5.19
Changes from v5:
- export __seamcall and use it
- move mutex lock from callee function of smp_call_on_cpu to the caller.
- rename mmu_prezap => flush_shadow_all_private() and tdx_mmu_release_hkid
- updated comment
- drop the use of tdh_mng_key.reclaimid(): as the function is for backward compatibility to only return success
- struct kvm_tdx_cmd: metadata => flags, added __u64 error.
- make this ioctl systemwide ioctl
- ABI change to struct kvm_init_vm
- guest_tsc_khz: use kvm->arch.default_tsc_khz
- rename BUILD_BUG_ON_MEMCPY to MEMCPY_SAME_SIZE
- drop exporting kvm_set_tsc_khz().
- fix kvm_tdp_page_fault() for mtrr emulation
- rename it to kvm_gfn_shared_mask(), dropped kvm_gpa_shared_mask()
- drop kvm_is_private_gfn(), kept kvm_is_private_gpa()
keep kvm_{gfn, gpa}_private(), kvm_gpa_private()
- update commit message
- rename shadow_init_value => shadow_nonprsent_value
- added ept_violation_ve_test mode
- shadow_nonpresent_value => SHADOW_NONPRESENT_VALUE in tdp_mmu.c
- legacy MMU case
=> - mmu_topup_shadow_page_cache(), kvm_mmu_create()
- FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)
- #VE warning:
- rename: REMOVED_SPTE => __REMOVED_SPTE, SHADOW_REMOVED_SPTE => REMOVED_SPTE
- merge into Like we discussed, this patch should be merged with patch "KVM: x86/mmu: Allow non-zero init value for shadow PTE".
- fix pointed by Sagi. check !is_private check => (kvm_gfn_shared_mask && !is_private)
- introduce kvm_gfn_for_root(kvm, root, gfn)
- add only_shared argument to kvm_tdp_mmu_handle_gfn()
- use kvm_arch_dirty_log_supported()
- rename SPTE_PRIVATE_PROHIBIT to SPTE_SHARED_MASK.
- rename: is_private_prohibit_spte() => spte_shared_mask()
- fix: shadow_nonpresent_value => SHADOW_NONPRESENT_VALUE in comment
- dropped this patch as the change was merged into kvm/queue
- update vt_apicv_post_state_restore()
- use is_64_bit_hypercall()
- comment: expand MSMI -> Machine Check System Management Interrupt
- fixed TDX_SEPT_PFERR
- tdvmcall_p[1234]_{write, read}() => tdvmcall_a[0123]_{read,write}()
- rename tdmvcall_exit_readon() => tdvmcall_leaf()
- remove optional zero check of argument.
- do a check for static_call(kvm_x86_has_emulated_msr)(kvm, MSR_IA32_SMBASE) in kvm_vcpu_ioctl_smi and __apic_accept_irq.
- WARN_ON_ONCE in tdx_smi_allowed and tdx_enable_smi_window.
- introduce vcpu_deliver_init to x86_ops
- sprinkeled KVM_BUG_ON()
Changes from v4:
- rebased to TDX host kernel patch series.
- include all the patches to make this patch series working.
- add [MARKER] patches to mark the patch layer clear.
---
* What's TDX?
TDX stands for Trust Domain Extensions, which extends Intel Virtual Machines Extensions (VMX) to introduce a kind of virtual machine guest called a Trust Domain (TD) for confidential computing.
A TD runs in a CPU mode that is designed to protect the confidentiality of its memory contents and its CPU state from any other software, including the hosting Virtual Machine Monitor (VMM), unless explicitly shared by the TD itself.
We have more detailed explanations below (***). We have the high-level design of TDX KVM below (****).
In this patch series, we use "TD" or "guest TD" to differentiate it from the current "VM" (Virtual Machine), which is supported by KVM today.
* The organization of this patch series
This patch series is on top of the patches series "TDX host kernel support":
https://lore.kernel.org/lkml/[email protected]/
This patch series is available at
https://github.com/intel/tdx/tree/kvm-upstream
The related repositories (TDX qemu, TDX OVMF(tdvf) etc) are described at
https://github.com/intel/tdx/wiki/TDX-KVM
The relations of the layers are depicted as follows.
The arrows below show the order of patch reviews we would like to have.
The below layers are chosen so that the device model, for example, qemu can exercise each layering step by step. Check if TDX is supported, create TD VM, create TD vcpu, allow vcpu running, populate TD guest private memory, and handle vcpu exits/hypercalls/interrupts to run TD fully.
TDX vcpu
interrupt/exits/hypercall<------------\
^ |
| |
TD finalization |
^ |
| |
TDX EPT violation<------------\ |
^ | |
| | |
TD vcpu enter/exit | |
^ | |
| | |
TD vcpu creation/destruction | \-------KVM TDP MMU MapGPA
^ | ^
| | |
TD VM creation/destruction \---------------KVM TDP MMU hooks ^ ^
| |
TDX architectural definitions KVM TDP refactoring for TDX
^ ^
| |
TDX, VMX <--------TDX host kernel KVM MMU GPA stolen bits
coexistence support
The followings are explanations of each layer. Each layer has a dummy commit that starts with [MARKER] in subject. It is intended to help to identify where each layer starts.
TDX host kernel support:
https://lore.kernel.org/lkml/[email protected]/
The guts of system-wide initialization of TDX module. There is an independent patch series for host x86. TDX KVM patches call functions this patch series provides to initialize the TDX module.
TDX, VMX coexistence:
Infrastructure to allow TDX to coexist with VMX and trigger the initialization of the TDX module.
This layer starts with
"KVM: VMX: Move out vmx_x86_ops to 'main.c' to wrap VMX and TDX"
TDX architectural definitions:
Add TDX architectural definitions and helper functions
This layer starts with
"[MARKER] The start of TDX KVM patch series: TDX architectural definitions".
TD VM creation/destruction:
Guest TD creation/destroy allocation and releasing of TDX specific vm and vcpu structure. Create an initial guest memory image with TDX measurement.
This layer starts with
"[MARKER] The start of TDX KVM patch series: TD VM creation/destruction".
TD vcpu creation/destruction:
guest TD creation/destroy Allocation and releasing of TDX specific vm and vcpu structure. Create an initial guest memory image with TDX measurement.
This layer starts with
"[MARKER] The start of TDX KVM patch series: TD vcpu creation/destruction"
TDX EPT violation:
Create an initial guest memory image with TDX measurement. Handle secure EPT violations to populate guest pages with TDX SEAMCALLs. This layer starts with
"[MARKER] The start of TDX KVM patch series: TDX EPT violation"
TD vcpu enter/exit:
Allow TDX vcpu to enter into TD and exit from TD. Save CPU state before entering into TD. Restore CPU state after exiting from TD.
This layer starts with
"[MARKER] The start of TDX KVM patch series: TD vcpu enter/exit"
TD vcpu interrupts/exit/hypercall:
Handle various exits/hypercalls and allow interrupts to be injected so that TD vcpu can continue running.
This layer starts with
"[MARKER] The start of TDX KVM patch series: TD vcpu exits/interrupts/hypercalls"
KVM MMU GPA shared bit:
Introduce framework to handle shared bit repurposed bit of GPA TDX repurposed a bit of GPA to indicate shared or private. If it's shared, it's the same as the conventional VMX EPT case. VMM can access shared guest pages. If it's private, it's handled by Secure-EPT and the guest page is encrypted.
This layer starts with
"[MARKER] The start of TDX KVM patch series: KVM MMU GPA stolen bits"
KVM TDP refactoring for TDX:
TDX Secure EPT requires different constants. e.g. initial value EPT entry value etc. Various refactoring for those differences. This layer starts with
"[MARKER] The start of TDX KVM patch series: KVM TDP refactoring for TDX"
KVM TDP MMU hooks:
Introduce framework to TDP MMU to add hooks in addition to direct EPT access TDX added Secure EPT which is an enhancement to VMX EPT. Unlike conventional VMX EPT, CPU can't directly read/write Secure EPT. Instead, use TDX SEAMCALLs to operate on Secure EPT.
This layer starts with
"[MARKER] The start of TDX KVM patch series: KVM TDP MMU hooks"
KVM TDP MMU MapGPA:
Introduce framework to handle switching guest pages from private/shared to shared/private. For a given GPA, a guest page can be assigned to a private GPA or a shared GPA exclusively. With TDX MapGPA hypercall, guest TD converts GPA assignments from private (or shared) to shared (or private).
This layer starts with
"[MARKER] The start of TDX KVM patch series: KVM TDP MMU MapGPA "
KVM guest private memory: (not shown in the above diagram)
[PATCH v4 00/12] KVM: mm: fd-based approach for supporting KVM guest private
memory: https://lkml.org/lkml/2022/1/18/395
Guest private memory requires different memory management in KVM. The patch proposes a way for it. Integration with TDX KVM.
(***)
* TDX module
A CPU-attested software module called the "TDX module" is designed to implement the TDX architecture, and it is loaded by the UEFI firmware today. It can be loaded by the kernel or driver at runtime, but in this patch series we assume that the TDX module is already loaded and initialized.
The TDX module provides two main new logical modes of operation built upon the new SEAM (Secure Arbitration Mode) root and non-root CPU modes added to the VMX architecture. TDX root mode is mostly identical to the VMX root operation mode, and the TDX functions (described later) are triggered by the new SEAMCALL instruction with the desired interface function selected by an input operand (leaf number, in RAX). TDX non-root mode is used for TD guest operation. TDX non-root operation (i.e. "guest TD" mode) is similar to the VMX non-root operation (i.e. guest VM), with changes and restrictions to better assure that no other software or hardware has direct visibility of the TD memory and state.
TDX transitions between TDX root operation and TDX non-root operation include TD Entries, from TDX root to TDX non-root mode, and TD Exits from TDX non-root to TDX root mode. A TD Exit might be asynchronous, triggered by some external event (e.g., external interrupt or SMI) or an exception, or it might be synchronous, triggered by a TDCALL (TDG.VP.VMCALL) function.
TD VCPUs can be entered using SEAMCALL(TDH.VP.ENTER) by KVM. TDH.VP.ENTER is one of the TDX interface functions as mentioned above, and "TDH" stands for Trust Domain Host. Those host-side TDX interface functions are categorized into various areas just for better organization, such as SYS (TDX module management), MNG (TD management), VP (VCPU), PHYSMEM (physical memory), MEM (private memory), etc. For example, SEAMCALL(TDH.SYS.INFO) returns the TDX module information.
TDCS (Trust Domain Control Structure) is the main control structure of a guest TD, and encrypted (using the guest TD's ephemeral private key). At a high level, TDCS holds information for controlling TD operation as a whole, execution, EPTP, MSR bitmaps, etc that KVM needs to set it up. Note that MSR bitmaps are held as part of TDCS (unlike VMX) because they are meant to have the same value for all VCPUs of the same TD.
Trust Domain Virtual Processor State (TDVPS) is the root control structure of a TD VCPU. It helps the TDX module control the operation of the VCPU, and holds the VCPU state while the VCPU is not running. TDVPS is opaque to software and DMA access, accessible only by using the TDX module interface functions (such as TDH.VP.RD, TDH.VP.WR). TDVPS includes TD VMCS, and TD VMCS auxiliary structures, such as virtual APIC page, virtualization exception information, etc.
Several VMX control structures (such as Shared EPT and Posted interrupt descriptor) are directly managed and accessed by the host VMM. These control structures are pointed to by fields in the TD VMCS.
The above means that 1) KVM needs to allocate different data structures for TDs,
2) KVM can reuse the existing code for TDs for some operations, 3) it needs to define TD-specific handling for others. 3) Redirect operations to . 3) Redirect operations to the TDX specific callbacks, like "if (is_td_vcpu(vcpu)) tdx_callback() else vmx_callback();".
*TD Private Memory
TD private memory is designed to hold TD private content, encrypted by the CPU using the TD ephemeral key. An encryption engine holds a table of encryption keys, and an encryption key is selected for each memory transaction based on a Host Key Identifier (HKID). By design, the host VMM does not have access to the encryption keys.
In the first generation of MKTME, HKID is "stolen" from the physical address by allocating a configurable number of bits from the top of the physical address. The HKID space is partitioned into shared HKIDs for legacy MKTME accesses and private HKIDs for SEAM-mode-only accesses. We use 0 for the shared HKID on the host so that MKTME can be opaque or bypassed on the host.
During TDX non-root operation (i.e. guest TD), memory accesses can be qualified as either shared or private, based on the value of a new SHARED bit in the Guest Physical Address (GPA). The CPU translates shared GPAs using the usual VMX EPT (Extended Page Table) or "Shared EPT" (in this document), which resides in host VMM memory. The Shared EPT is directly managed by the host VMM - the same as with the current VMX. Since guest TDs usually require I/O, and the data exchange needs to be done via shared memory, thus KVM needs to use the current EPT functionality even for TDs.
* Secure EPT and Minoring using the TDP code
The CPU translates private GPAs using a separate Secure EPT. The Secure EPT pages are encrypted and integrity-protected with the TD's ephemeral private key. Secure EPT can be managed _indirectly_ by the host VMM, using the TDX interface functions, and thus conceptually Secure EPT is a subset of EPT (why "subset"). Since execution of such interface functions takes much longer time than accessing memory directly, in KVM we use the existing TDP code to minor the Secure EPT for the TD.
This way, we can effectively walk Secure EPT without using the TDX interface functions.
* VM life cycle and TDX specific operations
The userspace VMM, such as QEMU, needs to build and treat TDs differently. For example, a TD needs to boot in private memory, and the host software cannot copy the initial image to private memory.
* TSC Virtualization
The TDX module helps TDs maintain reliable TSC (Time Stamp Counter) values (e.g. consistent among the TD VCPUs) and the virtual TSC frequency is determined by TD configuration, i.e. when the TD is created, not per VCPU. The current KVM owns TSC virtualization for VMs, but the TDX module does for TDs.
* MCE support for TDs
The TDX module doesn't allow VMM to inject MCE. Instead PV way is needed for TD to communicate with VMM. For now, KVM silently ignores MCE request by VMM. MSRs related to MCE (e.g, MCE bank registers) can be naturally emulated by paravirtualizing MSR access.
[1] For details, the specifications, [2], [3], [4], [5], [6], [7], are available.
* Restrictions or future work
Some features are not included to reduce patch size. Those features are addressed as future independent patch series.
- large page (2M, 1G)
- qemu gdb stub
- guest PMU
- and more
* Prerequisites
It's required to load the TDX module and initialize it. It's out of the scope of this patch series. Another independent patch for the common x86 code is planned. It defines CONFIG_INTEL_TDX_HOST and this patch series uses CONFIG_INTEL_TDX_HOST. It's assumed that With CONFIG_INTEL_TDX_HOST=y, the TDX module is initialized and ready for KVM to use the TDX module APIs for TDX guest life cycle like tdh.mng.init are ready to use.
Concretely Global initialization, LP (Logical Processor) initialization, global configuration, the key configuration, and TDMR and PAMT initialization are done. The state of the TDX module is SYS_READY. Please refer to the TDX module specification, the chapter Intel TDX Module Lifecycle State Machine
** Detecting the TDX module readiness.
TDX host patch series implements the detection of the TDX module availability and its initialization so that KVM can use it. Also it manages Host KeyID (HKID) assigned to guest TD.
The assumed APIs the TDX host patch series provides are
- const struct tdsysinfo_struct *tdx_get_sysinfo(void);
Return the system wide information about the TDX module. NULL if the TDX isn't initialized.
- int tdx_enable(void);
Initialization of TDX module so that the TDX module is ready for KVM to use.
- extern u32 tdx_global_keyid __read_mostly;
global host key id that is used for the TDX module itself.
- u32 tdx_get_num_keyid(void);
return the number of available TDX private host key id.
- int tdx_keyid_alloc(void);
Allocate HKID for guest TD.
- void tdx_keyid_free(int keyid);
Free HKID for guest TD.
(****)
* TDX KVM high-level design
- Host key ID management
Host Key ID (HKID) needs to be assigned to each TDX guest for memory encryption. It is assumed The TDX host patch series implements necessary functions, u32 tdx_get_global_keyid(void), int tdx_keyid_alloc(void) and,
void tdx_keyid_free(int keyid).
- Data structures and VM type
Because TDX is different from VMX, define its own VM/VCPU structures, struct kvm_tdx and struct vcpu_tdx instead of struct kvm_vmx and struct vcpu_vmx. To identify the VM, introduce VM-type to specify which VM type, VMX (default) or TDX, is used.
- VM life cycle and TDX specific operations
Re-purpose the existing KVM_MEMORY_ENCRYPT_OP to add TDX specific operations. New commands are used to get the TDX system parameters, set TDX specific VM/VCPU parameters, set initial guest memory and measurement.
The creation of TDX VM requires five additional operations in addition to the conventional VM creation.
- Get KVM system capability to check if TDX VM type is supported
- VM creation (KVM_CREATE_VM)
- New: Get the TDX specific system parameters. KVM_TDX_GET_CAPABILITY.
- New: Set TDX specific VM parameters. KVM_TDX_INIT_VM.
- VCPU creation (KVM_CREATE_VCPU)
- New: Set TDX specific VCPU parameters. KVM_TDX_INIT_VCPU.
- New: Initialize guest memory as boot state and extend the measurement with the memory. KVM_TDX_INIT_MEM_REGION.
- New: Finalize VM. KVM_TDX_FINALIZE. Complete measurement of the initial TDX VM contents.
- VCPU RUN (KVM_VCPU_RUN)
- Protected guest state
Because the guest state (CPU state and guest memory) is protected, the KVM VMM can't operate on them. For example, accessing CPU registers, injecting exceptions, and accessing guest memory. Those operations are handled as silently ignored, returning zero or initial reset value when it's requested via KVM API ioctls.
VM/VCPU state and callbacks for TDX specific operations. Define tdx specific VM state and VCPU state instead of VMX ones. Redirect operations to TDX specific callbacks. "if (tdx) tdx_op() else vmx_op()".
Operations on the CPU state
silently ignore operations on the guest state. For example, the write to CPU registers is ignored and the read from CPU registers returns 0.
. ignore access to CPU registers except for allowed ones.
. TSC: add a check if tsc is immutable and return an error. Because the KVM
implementation updates the internal tsc state and it's difficult to back out those changes. Instead, skip the logic.
. dirty logging: add check if dirty logging is supported.
. exceptions/SMI/MCE/SIPI/INIT: silently ignore
Note: virtual external interrupt and NMI can be injected into TDX guests.
- KVM MMU integration
One bit of the guest physical address (bit 51 or 47) is repurposed to indicate if the guest physical address is private (the bit is cleared) or shared (the bit is set). The bits are called stolen bits.
- Stolen bits framework
systematically tracks which guest physical address, shared or private, is used.
- Shared EPT and secure EPT
There are two EPTs. Shared EPT (the conventional one) and Secure EPT(the new one). Shared EPT is handled the same for the stolen bit set. Secure EPT points to private guest pages. To resolve EPT violation, KVM walks one of two EPTs based on faulted GPA. Because it's costly to access secure EPT during walking EPTs with SEAMCALLs for the private guest physical address, another private EPT is used as a shadow of Secure-EPT with the existing logic at the cost of extra memory.
The following depicts the relationship.
KVM | TDX module
| | |
-------------+---------- | |
| | | |
V V | |
shared GPA private GPA | |
CPU shared EPT pointer KVM private EPT pointer | CPU secure EPT pointer
| | | |
| | | |
V V | V
shared EPT private EPT--------mirror----->Secure EPT
| | | |
| \--------------------+------\ |
| | | |
V | V V
shared guest page | private guest page
|
|
non-encrypted memory | encrypted memory
| - Operating on Secure EPT
Use the TDX module APIs to operate on Secure EPT. To call the TDX API during resolving EPT violation, add hooks to additional operation and wiring it to TDX backend.
* References
[1] TDX specification
https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html
[2] Intel Trust Domain Extensions (Intel TDX)
https://cdrdv2.intel.com/v1/dl/getContent/726790
[3] Intel CPU Architectural Extensions Specification
https://www.intel.com/content/dam/develop/external/us/en/documents-tps/intel-tdx-cpu-architectural-specification.pdf
[4] Intel TDX Module 1.0 Specification
https://www.intel.com/content/dam/develop/external/us/en/documents/tdx-module-1.0-public-spec-v0.931.pdf
[5] Intel TDX Loader Interface Specification
https://www.intel.com/content/dam/develop/external/us/en/documents-tps/intel-tdx-seamldr-interface-specification.pdf
[6] Intel TDX Guest-Hypervisor Communication Interface
https://cdrdv2.intel.com/v1/dl/getContent/726790
[7] Intel TDX Virtual Firmware Design Guide
https://www.intel.com/content/dam/develop/external/us/en/documents/tdx-virtual-firmware-design-guide-rev-1.01.pdf
[8] intel public github
kvm TDX branch: https://github.com/intel/tdx/tree/kvm
TDX guest branch: https://github.com/intel/tdx/tree/guest
qemu TDX https://github.com/intel/qemu-tdx
[9] TDVF
https://github.com/tianocore/edk2-staging/tree/TDVF
This was merged into EDK2 main branch. https://github.com/tianocore/edk2
Chao Gao (1):
KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o
wrmsr
Isaku Yamahata (84):
[MARKER] The start of TDX KVM patch series: TDX architectural
definitions
KVM: TDX: Define TDX architectural definitions
KVM: TDX: Add C wrapper functions for SEAMCALLs to the TDX module
KVM: TDX: Add helper functions to print TDX SEAMCALL error [MARKER] The start of TDX KVM patch series: TD VM creation/destruction x86/cpu: Add helper functions to allocate/free TDX private host key id x86/virt/tdx: Add a helper function to return system wide info about
TDX module
KVM: TDX: Add place holder for TDX VM specific mem_enc_op ioctl
KVM: x86, tdx: Make KVM_CAP_MAX_VCPUS backend specific
KVM: x86/vmx, tdx: Allow VMX, TDX to override KVM_ENABLE_CAP
KVM: TDX: create/destroy VM structure
KVM: TDX: initialize VM with TDX specific parameters
KVM: TDX: Make pmu_intel.c ignore guest TD case
KVM: TDX: Refuse to unplug the last cpu on the package [MARKER] The start of TDX KVM patch series: TD vcpu
creation/destruction
KVM: TDX: allocate/free TDX vcpu structure
KVM: TDX: Do TDX specific vcpu initialization
[MARKER] The start of TDX KVM patch series: KVM MMU GPA shared bits
KVM: x86/mmu: introduce config for PRIVATE KVM MMU
KVM: x86/mmu: Add address conversion functions for TDX shared bit of
GPA
[MARKER] The start of TDX KVM patch series: KVM TDP refactoring for
TDX
KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE
KVM: x86/mmu: Add Suppress VE bit to
shadow_mmio_mask/shadow_present_mask
KVM: x86/mmu: Track shadow MMIO value on a per-VM basis
KVM: x86/mmu: Disallow fast page fault on private GPA
KVM: VMX: Introduce test mode related to EPT violation VE [MARKER] The start of TDX KVM patch series: KVM TDP MMU hooks
KVM: x86/tdp_mmu: Init role member of struct kvm_mmu_page at
allocation
KVM: x86/mmu: Require TDP MMU and mmio caching for TDX
KVM: x86/mmu: Add a new is_private member for union kvm_mmu_page_role
KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page
KVM: Add flags to struct kvm_gfn_range
KVM: x86/tdp_mmu: Sprinkle __must_check
KVM: x86/tdp_mmu: Support TDX private mapping for TDP MMU [MARKER] The start of TDX KVM patch series: TDX EPT violation
KVM: x86/mmu: Disallow dirty logging for x86 TDX
KVM: TDX: TDP MMU TDX support
KVM: TDX: MTRR: implement get_mt_mask() for TDX
[MARKER] The start of TDX KVM patch series: TD finalization
KVM: TDX: Create initial guest memory
KVM: TDX: Finalize VM initialization
[MARKER] The start of TDX KVM patch series: TD vcpu enter/exit
KVM: TDX: Add helper assembly function to TDX vcpu
KVM: TDX: Implement TDX vcpu enter/exit path
KVM: TDX: vcpu_run: save/restore host state(host kernel gs)
KVM: TDX: restore host xsave state when exit from the guest TD
KVM: TDX: restore user ret MSRs
[MARKER] The start of TDX KVM patch series: TD vcpu
exits/interrupts/hypercalls
KVM: TDX: complete interrupts after tdexit
KVM: TDX: restore debug store when TD exit
KVM: TDX: handle vcpu migration over logical processor
KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched
behavior
KVM: TDX: remove use of struct vcpu_vmx from posted_interrupt.c
KVM: TDX: Implement interrupt injection
KVM: TDX: Implements vcpu request_immediate_exit
KVM: TDX: Implement methods to inject NMI
KVM: TDX: Add a place holder to handle TDX VM exit
KVM: TDX: handle EXIT_REASON_OTHER_SMI
KVM: TDX: handle ept violation/misconfig exit
KVM: TDX: handle EXCEPTION_NMI and EXTERNAL_INTERRUPT
KVM: TDX: Add a place holder for handler of TDX hypercalls
(TDG.VP.VMCALL)
KVM: TDX: handle KVM hypercall with TDG.VP.VMCALL
KVM: TDX: Add KVM Exit for TDX TDG.VP.VMCALL
KVM: TDX: Handle TDX PV CPUID hypercall
KVM: TDX: Handle TDX PV HLT hypercall
KVM: TDX: Handle TDX PV port io hypercall
KVM: TDX: Implement callbacks for MSR operations for TDX
KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall
KVM: TDX: Handle TDX PV report fatal error hypercall
KVM: TDX: Handle TDX PV map_gpa hypercall
KVM: TDX: Handle TDG.VP.VMCALL<GetTdVmCallInfo> hypercall
KVM: TDX: Silently discard SMI request
KVM: TDX: Silently ignore INIT/SIPI
KVM: TDX: Add methods to ignore guest instruction emulation
KVM: TDX: Add a method to ignore dirty logging
KVM: TDX: Add methods to ignore VMX preemption timer
KVM: TDX: Add methods to ignore accesses to TSC
KVM: TDX: Ignore setting up mce
KVM: TDX: Add a method to ignore for TDX to ignore hypercall patch
KVM: TDX: Add methods to ignore virtual apic related operation Documentation/virt/kvm: Document on Trust Domain Extensions(TDX)
KVM: x86: design documentation on TDX support of x86 KVM TDP MMU
RFC: KVM: TDX: Make busy with S-EPT on entry bug [MARKER] the end of (the first phase of) TDX KVM patch series
Sean Christopherson (18):
KVM: TDX: Add TDX "architectural" error codes
KVM: TDX: x86: Add ioctl to get TDX systemwide parameters
KVM: Allow page-sized MMU caches to be initialized with custom 64-bit
values
KVM: x86/mmu: Allow non-zero value for non-present SPTE and removed
SPTE
KVM: x86/mmu: Allow per-VM override of the TDP max page level
KVM: x86/tdp_mmu: Don't zap private pages for unsupported cases
KVM: VMX: Split out guts of EPT violation to common/exposed function
KVM: VMX: Move setting of EPT MMU masks to common VT-x code
KVM: TDX: Add accessors VMX VMCS helpers
KVM: TDX: Add load_mmu_pgd method for TDX
KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by TDX
KVM: TDX: Add support for find pending IRQ in a protected local APIC
KVM: x86: Assume timer IRQ was injected if APIC state is proteced
KVM: VMX: Modify NMI and INTR handlers to take intr_info as function
argument
KVM: VMX: Move NMI/exception handler to common helper
KVM: x86: Split core of hypercall emulation to helper function
KVM: TDX: Handle TDX PV MMIO hypercall
KVM: TDX: Add methods to ignore accesses to CPU state
Yan Zhao (1):
KVM: x86/mmu: TDX: Do not enable page track for TD guest
Yao Yuan (1):
KVM: TDX: Handle vmentry failure for INTEL TD guest
Yuan Yao (1):
KVM: TDX: Retry seamcall when TDX_OPERAND_BUSY with operand SEPT
Documentation/virt/kvm/api.rst | 9 +- Documentation/virt/kvm/index.rst | 3 + Documentation/virt/kvm/intel-tdx.rst | 351 +++ Documentation/virt/kvm/tdx-tdp-mmu.rst | 417 ++++ arch/x86/events/intel/ds.c | 1 + arch/x86/include/asm/kvm-x86-ops.h | 14 + arch/x86/include/asm/kvm_host.h | 75 +- arch/x86/include/asm/tdx.h | 74 +- arch/x86/include/asm/vmx.h | 14 + arch/x86/include/uapi/asm/kvm.h | 83 + arch/x86/include/uapi/asm/vmx.h | 5 +- arch/x86/kvm/Kconfig | 4 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/cpuid.c | 7 + arch/x86/kvm/cpuid.h | 2 + arch/x86/kvm/irq.c | 3 + arch/x86/kvm/lapic.c | 33 +- arch/x86/kvm/lapic.h | 2 + arch/x86/kvm/mmu.h | 36 + arch/x86/kvm/mmu/mmu.c | 192 +- arch/x86/kvm/mmu/mmu_internal.h | 111 +- arch/x86/kvm/mmu/page_track.c | 3 + arch/x86/kvm/mmu/paging_tmpl.h | 3 +- arch/x86/kvm/mmu/spte.c | 22 +- arch/x86/kvm/mmu/spte.h | 29 +- arch/x86/kvm/mmu/tdp_iter.h | 14 +- arch/x86/kvm/mmu/tdp_mmu.c | 436 +++- arch/x86/kvm/mmu/tdp_mmu.h | 7 +- arch/x86/kvm/smm.h | 7 +- arch/x86/kvm/svm/svm.c | 1 + arch/x86/kvm/vmx/common.h | 176 ++ arch/x86/kvm/vmx/main.c | 1121 +++++++++- arch/x86/kvm/vmx/pmu_intel.c | 46 +- arch/x86/kvm/vmx/pmu_intel.h | 28 + arch/x86/kvm/vmx/posted_intr.c | 43 +- arch/x86/kvm/vmx/posted_intr.h | 13 + arch/x86/kvm/vmx/tdx.c | 2775 +++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 211 +- arch/x86/kvm/vmx/tdx_arch.h | 168 ++ arch/x86/kvm/vmx/tdx_errno.h | 38 + arch/x86/kvm/vmx/tdx_error.c | 21 + arch/x86/kvm/vmx/tdx_ops.h | 218 ++ arch/x86/kvm/vmx/vmcs.h | 5 + arch/x86/kvm/vmx/vmenter.S | 156 ++ arch/x86/kvm/vmx/vmx.c | 252 +-- arch/x86/kvm/vmx/vmx.h | 52 +- arch/x86/kvm/vmx/x86_ops.h | 111 +- arch/x86/kvm/x86.c | 136 +- arch/x86/kvm/x86.h | 2 + arch/x86/virt/vmx/tdx/seamcall.S | 2 + arch/x86/virt/vmx/tdx/tdx.c | 55 +- arch/x86/virt/vmx/tdx/tdx.h | 51 - include/linux/kvm_host.h | 12 +- include/linux/kvm_types.h | 1 + include/uapi/linux/kvm.h | 57 + tools/arch/x86/include/uapi/asm/kvm.h | 92 + virt/kvm/kvm_main.c | 46 +- 57 files changed, 7303 insertions(+), 545 deletions(-) create mode 100644 Documentation/virt/kvm/intel-tdx.rst create mode 100644 Documentation/virt/kvm/tdx-tdp-mmu.rst create mode 100644 arch/x86/kvm/vmx/common.h create mode 100644 arch/x86/kvm/vmx/pmu_intel.h create mode 100644 arch/x86/kvm/vmx/tdx_arch.h create mode 100644 arch/x86/kvm/vmx/tdx_errno.h create mode 100644 arch/x86/kvm/vmx/tdx_error.c create mode 100644 arch/x86/kvm/vmx/tdx_ops.h
base-commit: 513c1a3d3f1982fb850c910937099525b0d35e24
prerequisite-patch-id: 0711427e09d394052f47d629a05fff9139cf45f9
prerequisite-patch-id: f4e4bba285caf9235ffd723a9dea5d4bb6f091bb
prerequisite-patch-id: ae77e481e68d80051823abb20ffc0a11a7e7519f
prerequisite-patch-id: 447db73f329f75f973a5535dd2a831b16adb7e29
prerequisite-patch-id: 0918dcd86d4b3a738fe1d2bd5fbb018daf7fedb3
prerequisite-patch-id: 835f8c91d0bcbaa467531b0c673591dcbac93aab
prerequisite-patch-id: 70a64b2a969263a662235e004f5d165987cdee7b
prerequisite-patch-id: 9e02516dfb181273064b90d9e649ab710faa8327
prerequisite-patch-id: c6c86136ad5fe3f4f028aca2ae576f626a609d50
prerequisite-patch-id: d74e0cff593b7482b4aa9adcb404e63335aa7998
prerequisite-patch-id: 05ae7b8b946fb40fdcca626170937521745087a2
prerequisite-patch-id: 23484d65b2ca3d1dc3e6038fe634966ca926c1fd
prerequisite-patch-id: dc57166c8b1002b0d523c36a567a25c7d97f99ad
prerequisite-patch-id: bcb7f7c4f9afa44b5261c55803bcd6096f80b283
prerequisite-patch-id: 5d184cb4d4658388188e347c328da8778bdf9e29
prerequisite-patch-id: 6b25bde8b3ed7ae0b759fa69e931220d64d2e522
prerequisite-patch-id: 253c15673e595bd69c0b584ceabe29a25dfe4d52
prerequisite-patch-id: 2345b717dbc9088f51de2430ff2a8cf48bc41c7e
prerequisite-patch-id: 2e84c58128ab60d53dcf17e456396df62536d162
prerequisite-patch-id: b141fb4c84b6a6399c5a0e399a5a7c5279892aa4
prerequisite-patch-id: 97b3b271be3bd8541c102f73e8491bf95abad365
prerequisite-patch-id: f8dd2855bd255866b8c3670f3a826061fdf689be
prerequisite-patch-id: 4066b6b99abaf69a4b4f02fd623e0d3c896f9dd5
prerequisite-patch-id: 23113c11b5d43ba44f50a434b6f16c6cd1115305
prerequisite-patch-id: 9831f77bab0eb4320e60a8b7b9a5846069260afd
prerequisite-patch-id: 88aa762b68fe99342b101d476847905451507172
prerequisite-patch-id: 9b117dd0273de675bb20d43256bbfb89f7b7575c
prerequisite-patch-id: c07e2331876dc8ab4100655d5e2a33b0395a0a14
prerequisite-patch-id: 7b6a3d3a3695e8cc31877e27588e4a0aa55f4587
prerequisite-patch-id: 0eee33e1c8fb1ba97f3f7f8d9cca2e5ec8cd6c70
prerequisite-patch-id: 93b4dcfa25d777a4db8711cbb3e806abbcf348d3
prerequisite-patch-id: 7ef859d08678020338b04daaae0abb42821fc559
prerequisite-patch-id: 13c5aae87d890b4c856ec2718252616851001a7b
prerequisite-patch-id: 00524c037d3084b8e8af731030e3d47d146d2e2b
prerequisite-patch-id: bb5bcbe9e50bc33565612543865478e001fa8fee
prerequisite-patch-id: b9f0ecb32b8a48f6b975b3f2ba4c059f489c32aa
prerequisite-patch-id: 6f38d6cea1fcf7df86677b75c73905c3855ad57f
prerequisite-patch-id: e23f0c75c61f46d8e801465cfa05fdcb722e6caa
prerequisite-patch-id: e457d47d9e2fdceaf5088e627b6e4cc216b5e85d
prerequisite-patch-id: a62d991592c1ba08d928e923e0c6c03530bd63dd
prerequisite-patch-id: 9bd5f96a907a709aae8af549fa10484c4347782b
prerequisite-patch-id: 972d15ad44b9b44f390dea6d037e15421a897fd8
prerequisite-patch-id: e01e352a0a5545a93f5bc9af9ddc9ebb6062b703
prerequisite-patch-id: 6a7fcb0f2364deaf2514f8a23ff782be02fe5026
prerequisite-patch-id: 15608c8c226ee426ee976cc4410bff8ae865d920
prerequisite-patch-id: 76b053ab857a63c265df686024ff1fda0a2b6e7d
prerequisite-patch-id: 32a45051ddb8b785a0389814b73e38d8d7d7a6ae
prerequisite-patch-id: 5acae7bf571007f6c9b021fb69c500a7110b0ae3
prerequisite-patch-id: 14f83dfadef9dd8da6268ab6d73046a199a57116
prerequisite-patch-id: c2888053763858ee8ef39b30a1c8005c26a2378c
prerequisite-patch-id: 5bf7a07a4e19850fee9cd56d15a421b2cb2802b8
prerequisite-patch-id: 34ba41a3d0b4ad565c387bba9eef51e0556ff4ee
prerequisite-patch-id: 94ae4f32ad3f3693b9700f08d80da5a369d2cdaa
prerequisite-patch-id: 068d0c78244299a6cd9182c130bcb81d1c3ffdf1
prerequisite-patch-id: 9c38fc7292d5526e00148635229eedbbcb788583
prerequisite-patch-id: a244b94ee9b65b5bbba29f1eae311177adc3cefd
prerequisite-patch-id: acf875a766868d3969446897742f154b8e364a80
prerequisite-patch-id: c3d704436879601a54b8d42b5e565ca934c48195
prerequisite-patch-id: dc172265f050831b96447522dc8fab4231fd50d9
prerequisite-patch-id: 8d1ef7b82aba942d60171ed0877479d2a4297404
prerequisite-patch-id: e78505575e9e9f08d0166fa3eaf05c140929e319
prerequisite-patch-id: bcd4a65da6528d07f1edd628ec7f1d8df66222a0
prerequisite-patch-id: 2597d4d142448ff4796f4b9fcc5ab2e9873ae391
prerequisite-patch-id: 73023e0f63805ae226ffffe6af0798adb14c9e63
prerequisite-patch-id: daaff2e774b9e2aa139fbf89486650acc815cfa4
prerequisite-patch-id: 027b97fe2a2d10188ad1b18154a5b3a494fcdb3b
prerequisite-patch-id: 945c0546972666678bfb2b5c7737370e9f0897cb
prerequisite-patch-id: 94475f3a196844c29c9ffcfb728cecbc2baa0900
prerequisite-patch-id: 4eb88b47e0ed46c9034008bb5c9350c9e940c454
prerequisite-patch-id: 261cfb768e11e24317a4113f3cec3e65ba021696
prerequisite-patch-id: 8fbfa7cdd01d715cb8bbfe20041d0fecc691f72d
prerequisite-patch-id: c0a4c61d4e210aa0c691b26b3d4a0ab06efa617d
prerequisite-patch-id: 66322a79e5a22623c2398e9663ad23597b11347d
prerequisite-patch-id: 569bc795ecfd8c434c75bbc0202fa058c423e40e
prerequisite-patch-id: bb6bb2b65db2bc477c4b2e0e4144b9606db713a6
prerequisite-patch-id: 982f8129cad940137f6cf2c6234300b2e7d60c80
prerequisite-patch-id: f4fa117498ff3dd92db3b47c2d92e42a3d5c20f1
prerequisite-patch-id: 887655947c99ab6e822e7518ca779db5f27b86a1
prerequisite-patch-id: 8cc565c48b64291afb9cc817671d046cdc28a4ac
prerequisite-patch-id: 27f2d18b5ce2af402462117b522ad5a827ba3e82
prerequisite-patch-id: 40e5e3850a9402a43c31d4305f9c2e8051026328
prerequisite-patch-id: 871f2d5e5df13094f62fb8a52cb7d88ebb6146ba
prerequisite-patch-id: 176eac82fd72836b18ae8d50460f194421a00a02
prerequisite-patch-id: d38e46840d508967d1566c02ca5d1fecc39ad4ef
prerequisite-patch-id: 3160d1d6c295d51a610c71f8f9b8a0d83ee6d66c
prerequisite-patch-id: eb533cad65d1717efc48978ea6bc1c7a00cecfd2
prerequisite-patch-id: 1b4777af5019256652cf46a9719b645f35ad3545
prerequisite-patch-id: ab67f6fbf39573bf1926ed47f0f377d1907d93f9
prerequisite-patch-id: e53b862a3f19bfae6bad83a14e65399317e8f412
prerequisite-patch-id: f8c467f752f8ae76e4371637fc7f7f29ebba44c2
prerequisite-patch-id: 7746a8aef667b0d76e867bb3fe84c37e18ba9cf5
prerequisite-patch-id: 78d7b3b721d433f62393de3b12169c4b462fa939
prerequisite-patch-id: 24d59ae1238fb43f7337a0d08478f394441b04dc
prerequisite-patch-id: 42287c3913d32bf80cc28be41d8bf9972139dc33
prerequisite-patch-id: be3e7f787d9912862b913df033f4087d25ec5aa4
prerequisite-patch-id: a0a7ae7f18baf2ba244a5f2c38375c3352c933b8
prerequisite-patch-id: 7f4f230b3739ec80e426fb42bd7e80795ab97f20
prerequisite-patch-id: 103d70d6a171ec88afbe7b79dacc38f45141e1f8
prerequisite-patch-id: c02ee58512800188946f1a4aecdee9c8fd9783bc
prerequisite-patch-id: 64aadbfc41067b73b942410fe4900d83e8b2fa34
prerequisite-patch-id: 97871a9f8b43b5b45c39008fe90a461eacb4eade
prerequisite-patch-id: f2be318daa505a3ec4d310438484bd27c47f08dd
prerequisite-patch-id: 4d458c35e2988d7ffa25f1b7945e84a43efffb5e
prerequisite-patch-id: 1c2f09dc37a0b4ddd186f474207d98dafa8de411
prerequisite-patch-id: c3813c0f10f83c0c0b93a84e1c31c304ff5b48e0
prerequisite-patch-id: 06060a0723d48ca605a960d819b9e3c7fe0a2bb1
prerequisite-patch-id: 3023cf5f10af5d6d129d8483ce0b28b3c2ebff57
prerequisite-patch-id: 27d601d1cceccc045e79567b7ca4e9c9bbfff071
prerequisite-patch-id: 7d0b40a3a2a480541704e429bbbec763482a802c
prerequisite-patch-id: f15d178f9000430e0089c546756ab1d8d29341a7
prerequisite-patch-id: e162f31c8556dc9b1eebb52b153d2d4ae76d663f
prerequisite-patch-id: 81f0aa03c369838b739657547210249097aa3429
prerequisite-patch-id: 308ddf8aa7d3260ce21f687c0a39d21a8a7e421e
prerequisite-patch-id: 9d77fb0e8ce8c8c21e22ff3f26bd168eb5446df0
prerequisite-patch-id: 6aa44589d9a85a39af2224817c4f857ac3a04695
prerequisite-patch-id: 498e20857392d3ab0b290335f6bc4a646a9d8797
prerequisite-patch-id: 773bf14343711b6df1790f3c0e0bceec952be19c
prerequisite-patch-id: c1ff9950424897e617e2be2c96881714fb4aa2d2
prerequisite-patch-id: 8a420307f338a7a496e87649f200227816226880
prerequisite-patch-id: 94420af3725fc8b7d9b0bea2549f71c106dc6c78
prerequisite-patch-id: 52e7b497758cc689a2717c4483e64a3b0562f3f7
prerequisite-patch-id: c771edd362e6306753f058b0061f3218158ce717
prerequisite-patch-id: 9db94eaed64451494088ffc13fca31d8bf0144e6
prerequisite-patch-id: 68bc54f6dbee57d34ec6397df8bd2d76f0e5b15e
prerequisite-patch-id: 301dbdf8448175ea609664c890a3694750ecf740
prerequisite-patch-id: ba8e6068bcef7865bb5523065e19edd49fbc02de
prerequisite-patch-id: 7a11e08c6b7b73cd3fd29809c155af5d5400a24f
prerequisite-patch-id: a123ed868b3ed76e35dfc4830e4c3e903464db64
prerequisite-patch-id: cb5d2a167bed556fe3b2aa26452ee53d5e7c4428
prerequisite-patch-id: 5618d2414a1ef641b4c247b5e28076f67a765b24
prerequisite-patch-id: 338234d708730ff22c40b3a8ffbcf6820e9e6381
prerequisite-patch-id: 265913c78c61a3f84975f2a427bc895319a64b64
prerequisite-patch-id: bda07e82c47928bff9f1fcc8d003c526a5a42b6b
prerequisite-patch-id: 909d82383aac39075f4a80325d7a0d09f060ffb6
prerequisite-patch-id: 7ae6c571e61a2dc52aa1470b088d57b87267047f
prerequisite-patch-id: 2457494a366e0571189f98814b531435e656bc9d
prerequisite-patch-id: b7fc334127598465b74056593ee5ec1ceb67cee3
prerequisite-patch-id: b5eb477c22f1d4e9311760be7dcb4155f7bd01a0
prerequisite-patch-id: 0577b5bae65761382b14df238a107c24c5589f22
prerequisite-patch-id: 20f4cee4163cdbae5311cb8ee253ef3db48e10ad
prerequisite-patch-id: 8ea0ca56650455427780838fbd3f3a3fe412add5
prerequisite-patch-id: 4cfa225b52663f3308346df86c9f33120982fffa
prerequisite-patch-id: 66b406c400fb1ada0bae998b933621ed77557173
prerequisite-patch-id: cb32f271215dd2080619b4a794331c8b1c518ae6
prerequisite-patch-id: ffb4d5a65b230e2a6cfbedc4341287d520a215ee
prerequisite-patch-id: 3c93e412ef811eb92d0c9e7442108e57f4c0161d
prerequisite-patch-id: 8d5eff028fb5b925f659364f5aa41102a693fec3
prerequisite-patch-id: 90af14973f31856b7af0a44c33059729a91547c3
prerequisite-patch-id: aa3b25f154c23e996118c42521a77b9dc211256c
prerequisite-patch-id: f4c167fb86fb2fa6ff05f9f7932d154e1eaa4223
prerequisite-patch-id: 0b2bc7d01b85a07d441b56a7c55b3e383e8fe433
prerequisite-patch-id: 12d16cfb3118cc4cdbafa192c8458bd1103ed704
prerequisite-patch-id: dec14f3d6c1f4243b23bcbe0ecca83ae6ab703de
prerequisite-patch-id: 4451413560f58c6ce2f752b05085e3226878b256
prerequisite-patch-id: 82897b015f65fc920f5fdb6ffcd60d5de0fe09d2
prerequisite-patch-id: 716544cdd2e1ccec5efb4968a87f560689cebbca
prerequisite-patch-id: 4399516d31260b164e5bdebaea0f566e0508e318
prerequisite-patch-id: 78579fdc8bec4a6e3d576ac01c97124b6b2503a3
prerequisite-patch-id: e998ddb2d9d31a0b3e968d7195cbc4d2372bef36
prerequisite-patch-id: df55ce2450aa405f904d1a5085ffe33b0ec6e399
prerequisite-patch-id: 16d63052069b5079502e6e8ee7a0e38111471af5
prerequisite-patch-id: bf5fd74736424429cee5bf645f85d09bb93a26be
prerequisite-patch-id: d0a71fc6ae91999a93703727e682532fe5d12c9a
prerequisite-patch-id: a8774091ff1e54f65c926c443703a776c288916d
prerequisite-patch-id: b139f94a8d2458d3d2a4d81e0bdf77cd2eaacc05
prerequisite-patch-id: b1aed667f5cf78a947491280cffd6fa85c5d310c
prerequisite-patch-id: 8a4fcdc6aefb233e60f6be919d9d94f7192b7fee
prerequisite-patch-id: 4f0cbb8eeddc992d22b4b0cdd7eda53aed85e9f8
prerequisite-patch-id: 0dd9f6c8dc9cebd06a7203ec29d5d4d2256929ab
Signature · 1 line
2.25.1
PATCH v12 1/106 The start of TDX KVM patch series: TDX architectural definitions
From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TDX architectural definitions.
--- .../virt/kvm/intel-tdx-layer-status.rst | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 Documentation/virt/kvm/intel-tdx-layer-status.rst
Documentation/virt/kvm/intel-tdx-layer-status.rst new file +28 -0
@@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================== +Intel Trust Dodmain Extensions(TDX) +=================================== + +Layer status +============ +What qemu can do +---------------- +- TDX VM TYPE is exposed to Qemu. +- Qemu can try to create VM of TDX VM type and then fails. + +Patch Layer status +------------------ + Patch layer Status +* TDX, VMX coexistence: Applied +* TDX architectural definitions: Applying +* TD VM creation/destruction: Not yet +* TD vcpu creation/destruction: Not yet +* TDX EPT violation: Not yet +* TD finalization: Not yet +* TD vcpu enter/exit: Not yet +* TD vcpu interrupts/exit/hypercall: Not yet + +* KVM MMU GPA shared bits: Not yet +* KVM TDP refactoring for TDX: Not yet +* KVM TDP MMU hooks: Not yet
Signature · 1 line
2.25.1
PATCH v12 2/106 KVM: TDX: Define TDX architectural definitions
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Define architectural definitions for KVM to issue the TDX SEAMCALLs.
Structures and values that are architecturally defined in the TDX module specifications the chapter of ABI Reference.
--- arch/x86/kvm/vmx/tdx_arch.h | 168 ++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 arch/x86/kvm/vmx/tdx_arch.h
arch/x86/kvm/vmx/tdx_arch.h new file +168 -0
@@ -0,0 +1,168 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* architectural constants/data definitions for TDX SEAMCALLs */ + +#ifndef __KVM_X86_TDX_ARCH_H +#define __KVM_X86_TDX_ARCH_H + +#include <linux/types.h> + +/* +* TDX SEAMCALL API function leaves +*/ +#define TDH_VP_ENTER 0 +#define TDH_MNG_ADDCX 1 +#define TDH_MEM_PAGE_ADD 2 +#define TDH_MEM_SEPT_ADD 3 +#define TDH_VP_ADDCX 4 +#define TDH_MEM_PAGE_RELOCATE 5 +#define TDH_MEM_PAGE_AUG 6 +#define TDH_MEM_RANGE_BLOCK 7 +#define TDH_MNG_KEY_CONFIG 8 +#define TDH_MNG_CREATE 9 +#define TDH_VP_CREATE 10 +#define TDH_MNG_RD 11 +#define TDH_MR_EXTEND 16 +#define TDH_MR_FINALIZE 17 +#define TDH_VP_FLUSH 18 +#define TDH_MNG_VPFLUSHDONE 19 +#define TDH_MNG_KEY_FREEID 20 +#define TDH_MNG_INIT 21 +#define TDH_VP_INIT 22 +#define TDH_VP_RD 26 +#define TDH_MNG_KEY_RECLAIMID 27 +#define TDH_PHYMEM_PAGE_RECLAIM 28 +#define TDH_MEM_PAGE_REMOVE 29 +#define TDH_MEM_SEPT_REMOVE 30 +#define TDH_MEM_TRACK 38 +#define TDH_MEM_RANGE_UNBLOCK 39 +#define TDH_PHYMEM_CACHE_WB 40 +#define TDH_PHYMEM_PAGE_WBINVD 41 +#define TDH_VP_WR 43 +#define TDH_SYS_LP_SHUTDOWN 44 + +#define TDG_VP_VMCALL_GET_TD_VM_CALL_INFO 0x10000 +#define TDG_VP_VMCALL_MAP_GPA 0x10001 +#define TDG_VP_VMCALL_GET_QUOTE 0x10002 +#define TDG_VP_VMCALL_REPORT_FATAL_ERROR 0x10003 +#define TDG_VP_VMCALL_SETUP_EVENT_NOTIFY_INTERRUPT 0x10004 + +/* TDX control structure (TDR/TDCS/TDVPS) field access codes */ +#define TDX_NON_ARCH BIT_ULL(63) +#define TDX_CLASS_SHIFT 56 +#define TDX_FIELD_MASK GENMASK_ULL(31, 0) + +#define __BUILD_TDX_FIELD (non_arch, class, field) \ + (((non_arch) ? TDX_NON_ARCH : 0) | \ + ((u64)(class) << TDX_CLASS_SHIFT) | \ + ((u64)(field) & TDX_FIELD_MASK)) + +#define BUILD_TDX_FIELD (class, field) \ + __BUILD_TDX_FIELD(false, (class), (field)) + +#define BUILD_TDX_FIELD_NON_ARCH (class, field) \ + __BUILD_TDX_FIELD(true, (class), (field)) + + +/* Class code for TD */ +#define TD_CLASS_EXECUTION_CONTROLS 17ULL + +/* Class code for TDVPS */ +#define TDVPS_CLASS_VMCS 0ULL +#define TDVPS_CLASS_GUEST_GPR 16ULL +#define TDVPS_CLASS_OTHER_GUEST 17ULL +#define TDVPS_CLASS_MANAGEMENT 32ULL + +enum tdx_tdcs_execution_control { +TD_TDCS_EXEC_TSC_OFFSET =10 , +}; + +/* @field is any of enum tdx_tdcs_execution_control */ +#define TDCS_EXEC (field) BUILD_TDX_FIELD(TD_CLASS_EXECUTION_CONTROLS, (field)) + +/* @field is the VMCS field encoding */ +#define TDVPS_VMCS (field) BUILD_TDX_FIELD(TDVPS_CLASS_VMCS, (field)) + +enum tdx_vcpu_guest_other_state { +TD_VCPU_STATE_DETAILS_NON_ARCH =0x100 , +}; + +union tdx_vcpu_state_details { +struct { +u64 vmxip :1 ; +u64 reserved :63 ; + }; +u64 full; +}; + +/* @field is any of enum tdx_guest_other_state */ +#define TDVPS_STATE (field) BUILD_TDX_FIELD(TDVPS_CLASS_OTHER_GUEST, (field)) +#define TDVPS_STATE_NON_ARCH (field) BUILD_TDX_FIELD_NON_ARCH(TDVPS_CLASS_OTHER_GUEST, (field)) + +/* Management class fields */ +enum tdx_vcpu_guest_management { +TD_VCPU_PEND_NMI =11 , +}; + +/* @field is any of enum tdx_vcpu_guest_management */ +#define TDVPS_MANAGEMENT (field) BUILD_TDX_FIELD(TDVPS_CLASS_MANAGEMENT, (field)) + +#define TDX_EXTENDMR_CHUNKSIZE 256 + +struct tdx_cpuid_value { +u32 eax; +u32 ebx; +u32 ecx; +u32 edx; +} __packed; + +#define TDX_TD_ATTRIBUTE_DEBUG BIT_ULL(0) +#define TDX_TD_ATTRIBUTE_PKS BIT_ULL(30) +#define TDX_TD_ATTRIBUTE_KL BIT_ULL(31) +#define TDX_TD_ATTRIBUTE_PERFMON BIT_ULL(63) + +/* +* TD_PARAMS is provided as an input to TDH_MNG_INIT, the size of which is 1024B. +*/ +#define TDX_MAX_VCPUS (~(u16)0) + +struct td_params { +u64 attributes; +u64 xfam; +u16 max_vcpus; +u8 reserved0[6 ]; + +u64 eptp_controls; +u64 exec_controls; +u16 tsc_frequency; +u8 reserved1[38 ]; + +u64 mrconfigid[6 ]; +u64 mrowner[6 ]; +u64 mrownerconfig[6 ]; +u64 reserved2[4 ]; + +union { +struct tdx_cpuid_value cpuid_values[0 ]; +u8 reserved3[768 ]; + }; +} __packed__aligned(1024) ; + +/* +* Guest uses MAX_PA for GPAW when set. +* 0: GPA.SHARED bit is GPA[47] +* 1: GPA.SHARED bit is GPA[51] +*/ +#define TDX_EXEC_CONTROL_MAX_GPAW BIT_ULL(0) + +/* +* TDX requires the frequency to be defined in units of 25MHz, which is the +* frequency of the core crystal clock on TDX-capable platforms, i.e. the TDX +* module can only program frequencies that are multiples of 25MHz. The +* frequency must be between 100mhz and 10ghz (inclusive). +*/ +#define TDX_TSC_KHZ_TO_25MHZ (tsc_in_khz) ((tsc_in_khz) / (25 * 1000)) +#define TDX_TSC_25MHZ_TO_KHZ (tsc_in_25mhz) ((tsc_in_25mhz) * (25 * 1000)) +#define TDX_MIN_TSC_FREQUENCY_KHZ (100 * 1000) +#define TDX_MAX_TSC_FREQUENCY_KHZ (10 * 1000 * 1000) + +#endif /* __KVM_X86_TDX_ARCH_H */
Signature · 1 line
2.25.1
PATCH v12 3/106 KVM: TDX: Add TDX "architectural" error codes
In reply to [email protected]From: Sean Christopherson <[email protected]>
Add error codes for the TDX SEAMCALLs both for TDX VMM side for TDH SEAMCALL and TDX guest side for TDG.VP.VMCALL. KVM issues the TDX SEAMCALLs and checks its error code. KVM handles hypercall from the TDX guest and may return an error. So error code for the TDX guest is also needed.
TDX SEAMCALL uses bits 31:0 to return more information, so these error codes will only exactly match RAX[63:32]. Error codes for TDG.VP.VMCALL is defined by TDX Guest-Host-Communication interface spec.
--- arch/x86/kvm/vmx/tdx_errno.h | 38 ++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 arch/x86/kvm/vmx/tdx_errno.h
arch/x86/kvm/vmx/tdx_errno.h new file +38 -0
@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* architectural status code for SEAMCALL */ + +#ifndef __KVM_X86_TDX_ERRNO_H +#define __KVM_X86_TDX_ERRNO_H + +#define TDX_SEAMCALL_STATUS_MASK 0xFFFFFFFF00000000ULL + +/* +* TDX SEAMCALL Status Codes (returned in RAX) +*/ +#define TDX_SUCCESS 0x0000000000000000ULL +#define TDX_NON_RECOVERABLE_VCPU 0x4000000100000000ULL +#define TDX_INTERRUPTED_RESUMABLE 0x8000000300000000ULL +#define TDX_OPERAND_BUSY 0x8000020000000000ULL +#define TDX_VCPU_NOT_ASSOCIATED 0x8000070200000000ULL +#define TDX_KEY_GENERATION_FAILED 0x8000080000000000ULL +#define TDX_KEY_STATE_INCORRECT 0xC000081100000000ULL +#define TDX_KEY_CONFIGURED 0x0000081500000000ULL +#define TDX_NO_HKID_READY_TO_WBCACHE 0x0000082100000000ULL +#define TDX_EPT_WALK_FAILED 0xC0000B0000000000ULL + +/* +* TDG.VP.VMCALL Status Codes (returned in R10) +*/ +#define TDG_VP_VMCALL_SUCCESS 0x0000000000000000ULL +#define TDG_VP_VMCALL_RETRY 0x0000000000000001ULL +#define TDG_VP_VMCALL_INVALID_OPERAND 0x8000000000000000ULL +#define TDG_VP_VMCALL_TDREPORT_FAILED 0x8000000000000001ULL + +/* +* TDX module operand ID, appears in 31:0 part of error code as +* detail information +*/ +#define TDX_OPERAND_ID_RCX 0x01 +#define TDX_OPERAND_ID_SEPT 0x92 + +#endif /* __KVM_X86_TDX_ERRNO_H */
Signature · 1 line
2.25.1
PATCH v12 4/106 KVM: TDX: Add C wrapper functions for SEAMCALLs to the TDX module
In reply to [email protected]From: Isaku Yamahata <[email protected]>
A VMM interacts with the TDX module using a new instruction (SEAMCALL). A TDX VMM uses SEAMCALLs where a VMX VMM would have directly interacted with VMX instructions. For instance, a TDX VMM does not have full access to the VM control structure corresponding to VMX VMCS. Instead, a VMM induces the TDX module to act on behalf via SEAMCALLs.
Export __seamcall and define C wrapper functions for SEAMCALLs for readability.
Some SEAMCALL APIs donates host pages to TDX module or guest TD and the donated pages are encrypted. Some of such SEAMCALLs flush cache lines (typically by movdir64b instruction), some don't. Those that doesn't clear cache lines require the VMM to flush the cache lines to avoid cache line alias.
--- arch/x86/include/asm/tdx.h | 2 + arch/x86/kvm/vmx/tdx_ops.h | 185 +++++++++++++++++++++++++++++++ arch/x86/virt/vmx/tdx/seamcall.S | 2 + 3 files changed, 189 insertions(+) create mode 100644 arch/x86/kvm/vmx/tdx_ops.h
arch/x86/include/asm/tdx.h +2 -0
@@ -108,6 +108,8 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,bool platform_tdx_enabled (void );int tdx_enable (void );int tdx_cpu_online (unsigned int cpu); +u64 __seamcall (u64 op,u64 rcx,u64 rdx,u64 r8,u64 r9, +struct tdx_module_output *out);#else /* !CONFIG_INTEL_TDX_HOST */ static inline bool platform_tdx_enabled (void ) {return false; }static inline int tdx_enable (void ) {return -EINVAL ; }
arch/x86/kvm/vmx/tdx_ops.h new file +185 -0
185 lines not on this page · raw message
arch/x86/virt/vmx/tdx/seamcall.S +2 -0
@@ -1,5 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include <linux/linkage.h> +#include <asm/export.h> #include <asm/frame.h> #include "tdxcall.S"
@@ -50,3 +51,4 @@ SYM_FUNC_START(__seamcall) FRAME_END RET SYM_FUNC_END(__seamcall) +EXPORT_SYMBOL_GPL(__seamcall)
Signature · 1 line
2.25.1
PATCH v12 5/106 KVM: TDX: Add helper functions to print TDX SEAMCALL error
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add helper functions to print out errors from the TDX module in a uniform manner.
--- arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/vmx/tdx_error.c | 21 +++++++++++++++++++++ arch/x86/kvm/vmx/tdx_ops.h | 3 +++ 3 files changed, 25 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kvm/vmx/tdx_error.c
arch/x86/kvm/Makefile +1 -1
@@ -25,7 +25,7 @@ kvm-$(CONFIG_KVM_SMM) += smm.o kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \ vmx/hyperv.o vmx/nested.o vmx/posted_intr.o vmx/main.o kvm-intel-$(CONFIG_X86_SGX_KVM) += vmx/sgx.o -kvm-intel-$(CONFIG_INTEL_TDX_HOST) += vmx/tdx.o +kvm-intel-$(CONFIG_INTEL_TDX_HOST) += vmx/tdx.o vmx/tdx_error.o kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o \ svm/sev.o svm/hyperv.o
arch/x86/kvm/vmx/tdx_error.c new file +21 -0
@@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +/* functions to record TDX SEAMCALL error */ + +#include <linux/kernel.h> +#include <linux/bug.h> + +#include "tdx_ops.h" + +void pr_tdx_error (u64 op,u64 error_code,const struct tdx_module_output *out) +{ +if (!out) { +pr_err_ratelimited ("SEAMCALL[%lld] failed: 0x%llx\n" , + op, error_code); +return ; + } + +pr_err_ratelimited ("SEAMCALL[%lld] failed: 0x%llx RCX 0x%llx, RDX 0x%llx," +" R8 0x%llx, R9 0x%llx, R10 0x%llx, R11 0x%llx\n" , + op, error_code, + out->rcx, out->rdx, out->r8, out->r9, out->r10, out->r11); +}
arch/x86/kvm/vmx/tdx_ops.h +3 -0
@@ -9,12 +9,15 @@#include <asm/cacheflush.h> #include <asm/asm.h> #include <asm/kvm_host.h> +#include <asm/tdx.h> #include "tdx_errno.h" #include "tdx_arch.h" #ifdef CONFIG_INTEL_TDX_HOST +void pr_tdx_error (u64 op,u64 error_code,const struct tdx_module_output *out); +static inline u64 tdh_mng_addcx (hpa_t tdr,hpa_t addr) {clflush_cache_range (__va (addr),PAGE_SIZE );
Signature · 1 line
2.25.1
PATCH v12 6/106 The start of TDX KVM patch series: TD VM creation/destruction
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD VM creation/destruction.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2
@@ -15,8 +15,8 @@ Patch Layer status ------------------ Patch layer Status * TDX, VMX coexistence: Applied -* TDX architectural definitions: Applying -* TD VM creation/destruction: Not yet +* TDX architectural definitions: Applied +* TD VM creation/destruction: Applying * TD vcpu creation/destruction: Not yet * TDX EPT violation: Not yet * TD finalization: Not yet
Signature · 1 line
2.25.1
PATCH v12 7/106 x86/cpu: Add helper functions to allocate/free TDX private host key id
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX private host key id (HKID) is assigned to guest TD. The memory controller encrypts guest TD memory with the assigned TDX HKID. Add helper functions to allocate/free TDX private HKID so that TDX KVM can manage it.
Also export the global TDX private HKID that is used to encrypt TDX module, its memory and some dynamic data (TDR). When VMM releasing encrypted page to reuse it, the page needs to be flushed with the used HKID. VMM needs the global TDX private HKID to flush such pages.
--- arch/x86/include/asm/tdx.h | 12 ++++++++++++ arch/x86/virt/vmx/tdx/tdx.c | 34 +++++++++++++++++++++++++++++++++- 2 files changed, 45 insertions(+), 1 deletion(-)
arch/x86/include/asm/tdx.h +12 -0
@@ -108,12 +108,24 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,bool platform_tdx_enabled (void );int tdx_enable (void );int tdx_cpu_online (unsigned int cpu); +/* +* Key id globally used by TDX module: TDX module maps TDR with this TDX global +* key id. TDR includes key id assigned to the TD. Then TDX module maps other +* TD-related pages with the assigned key id. TDR requires this TDX global key +* id for cache flush unlike other TD-related pages. +*/ +extern u32 tdx_global_keyid__ro_after_init ; +int tdx_guest_keyid_alloc (void ); +void tdx_guest_keyid_free (int keyid); +u64 __seamcall (u64 op,u64 rcx,u64 rdx,u64 r8,u64 r9,struct tdx_module_output *out);#else /* !CONFIG_INTEL_TDX_HOST */ static inline bool platform_tdx_enabled (void ) {return false; }static inline int tdx_enable (void ) {return -EINVAL ; }static inline int tdx_cpu_online (unsigned int cpu) {return 0 ; } +static inline int tdx_guest_keyid_alloc (void ) {return -EOPNOTSUPP ; } +static inline void tdx_guest_keyid_free (int keyid) { }#endif /* CONFIG_INTEL_TDX_HOST */ #endif /* !__ASSEMBLY__ */
arch/x86/virt/vmx/tdx/tdx.c +33 -1
@@ -31,7 +31,8 @@#include <asm/tdx.h> #include "tdx.h" -static u32 tdx_global_keyid__ro_after_init ; +u32 tdx_global_keyid__ro_after_init ; +EXPORT_SYMBOL_GPL (tdx_global_keyid);static u32 tdx_guest_keyid_start__ro_after_init ;static u32 tdx_nr_guest_keyids__ro_after_init ;
@@ -132,6 +133,31 @@ static struct notifier_block tdx_memory_nb = { .notifier_call = tdx_memory_notifier, }; +/* TDX KeyID pool */ +static DEFINE_IDA (tdx_guest_keyid_pool); + +int tdx_guest_keyid_alloc (void ) +{ +if (WARN_ON_ONCE (!tdx_guest_keyid_start || !tdx_nr_guest_keyids)) +return -EINVAL ; + +/* The first keyID is reserved for the global key. */ +return ida_alloc_range (&tdx_guest_keyid_pool, tdx_guest_keyid_start +1 , + tdx_guest_keyid_start + tdx_nr_guest_keyids -1 , +GFP_KERNEL ); +} +EXPORT_SYMBOL_GPL (tdx_guest_keyid_alloc); + +void tdx_guest_keyid_free (int keyid) +{ +/* keyid = 0 is reserved. */ +if (WARN_ON_ONCE (keyid <=0 )) +return ; + +ida_free (&tdx_guest_keyid_pool, keyid); +} +EXPORT_SYMBOL_GPL (tdx_guest_keyid_free); +static int __init tdx_init (void ) {u32 tdx_keyid_start, nr_tdx_keyids;
@@ -1220,6 +1246,12 @@ static int init_tdx_module(void)if (ret) goto out_free_pamts; +/* +* Reserve the first TDX KeyID as global KeyID to protect +* TDX module metadata. +*/ + tdx_global_keyid = tdx_keyid_start; +/* Initialize TDMRs to complete the TDX module initialization */ ret =init_tdmrs (&tdx_tdmr_list);
Signature · 1 line
2.25.1
PATCH v12 8/106 x86/virt/tdx: Add a helper function to return system wide info about TDX module
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX KVM needs system-wide information about the TDX module, struct tdsysinfo_struct. Add a helper function tdx_get_sysinfo() to return it instead of KVM getting it with various error checks. Make KVM call the function and stash the info. Move out the struct definition about it to common place arch/x86/include/asm/tdx.h.
--- arch/x86/include/asm/tdx.h | 54 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.c | 15 ++++++++++- arch/x86/virt/vmx/tdx/tdx.c | 21 ++++++++++++--- arch/x86/virt/vmx/tdx/tdx.h | 51 ----------------------------------- 4 files changed, 85 insertions(+), 56 deletions(-)
arch/x86/include/asm/tdx.h +54 -0
@@ -105,6 +105,58 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,#endif /* CONFIG_INTEL_TDX_GUEST && CONFIG_KVM_GUEST */ #ifdef CONFIG_INTEL_TDX_HOST +struct tdx_cpuid_config { +u32 leaf; +u32 sub_leaf; +u32 eax; +u32 ebx; +u32 ecx; +u32 edx; +} __packed; + +#define TDSYSINFO_STRUCT_SIZE 1024 +#define TDSYSINFO_STRUCT_ALIGNMENT 1024 + +/* +* The size of this structure itself is flexible. The actual structure +* passed to TDH.SYS.INFO must be padded to TDSYSINFO_STRUCT_SIZE and be +* aligned to TDSYSINFO_STRUCT_ALIGNMENT using DECLARE_PADDED_STRUCT(). +*/ +struct tdsysinfo_struct { +/* TDX-SEAM Module Info */ +u32 attributes; +u32 vendor_id; +u32 build_date; +u16 build_num; +u16 minor_version; +u16 major_version; +u8 reserved0[14 ]; +/* Memory Info */ +u16 max_tdmrs; +u16 max_reserved_per_tdmr; +u16 pamt_entry_size; +u8 reserved1[10 ]; +/* Control Struct Info */ +u16 tdcs_base_size; +u8 reserved2[2 ]; +u16 tdvps_base_size; +u8 tdvps_xfam_dependent_size; +u8 reserved3[9 ]; +/* TD Capabilities */ +u64 attributes_fixed0; +u64 attributes_fixed1; +u64 xfam_fixed0; +u64 xfam_fixed1; +u8 reserved4[32 ]; +u32 num_cpuid_config; +/* +* The actual number of CPUID_CONFIG depends on above +* 'num_cpuid_config'. +*/ + DECLARE_FLEX_ARRAY(struct tdx_cpuid_config, cpuid_configs); +} __packed; + +const struct tdsysinfo_struct *tdx_get_sysinfo (void );bool platform_tdx_enabled (void );int tdx_enable (void );int tdx_cpu_online (unsigned int cpu);
@@ -121,6 +173,8 @@ void tdx_guest_keyid_free(int keyid);u64 __seamcall (u64 op,u64 rcx,u64 rdx,u64 r8,u64 r9,struct tdx_module_output *out);#else /* !CONFIG_INTEL_TDX_HOST */ +struct tdsysinfo_struct ; +static inline const struct tdsysinfo_struct *tdx_get_sysinfo (void ) {return NULL ; }static inline bool platform_tdx_enabled (void ) {return false; }static inline int tdx_enable (void ) {return -EINVAL ; }static inline int tdx_cpu_online (unsigned int cpu) {return 0 ; }
arch/x86/kvm/vmx/tdx.c +14 -1
@@ -11,9 +11,18 @@#undef pr_fmt#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#define TDX_MAX_NR_CPUID_CONFIGS \ + ((TDSYSINFO_STRUCT_SIZE - \ + offsetof(struct tdsysinfo_struct, cpuid_configs)) \ + / sizeof(struct tdx_cpuid_config)) +static int __init tdx_module_setup (void ) { -int ret; +const struct tdsysinfo_struct *tdsysinfo; +int ret =0 ; + +BUILD_BUG_ON (sizeof (*tdsysinfo) >TDSYSINFO_STRUCT_SIZE ); +BUILD_BUG_ON (TDX_MAX_NR_CPUID_CONFIGS !=37 ); ret =tdx_enable ();if (ret) {
@@ -21,6 +30,10 @@ static int __init tdx_module_setup(void)return ret; } +/* Sanitary check just in case. */ + tdsysinfo =tdx_get_sysinfo (); +WARN_ON (tdsysinfo->num_cpuid_config >TDX_MAX_NR_CPUID_CONFIGS ); +pr_info ("TDX is supported.\n" );return 0 ; }
arch/x86/virt/vmx/tdx/tdx.c +17 -4
@@ -357,7 +357,7 @@ static void print_cmrs(struct cmr_info *cmr_array, int nr_cmrs)* CMRs, and save them to @sysinfo and @cmr_array. @sysinfo must have * been padded to have enough room to save the TDSYSINFO_STRUCT. */ -static int tdx_get_sysinfo (struct tdsysinfo_struct *sysinfo, +static int __tdx_get_sysinfo (struct tdsysinfo_struct *sysinfo,struct cmr_info *cmr_array) {struct tdx_module_output out;
@@ -382,6 +382,21 @@ static int tdx_get_sysinfo(struct tdsysinfo_struct *sysinfo,return 0 ; } +static DECLARE_PADDED_STRUCT(tdsysinfo_struct, tdsysinfo, + TDSYSINFO_STRUCT_SIZE, TDSYSINFO_STRUCT_ALIGNMENT); + +const struct tdsysinfo_struct *tdx_get_sysinfo (void ) +{ +const struct tdsysinfo_struct *r =NULL ; + +mutex_lock (&tdx_module_lock); +if (tdx_module_status ==TDX_MODULE_INITIALIZED ) + r = &PADDED_STRUCT (tdsysinfo); +mutex_unlock (&tdx_module_lock); +return r; +} +EXPORT_SYMBOL_GPL (tdx_get_sysinfo); +/* * Add a memory region as a TDX memory block. The caller must make sure * all memory regions are added in address ascending order and don't
@@ -1164,8 +1179,6 @@ static int init_tdmrs(struct tdmr_info_list *tdmr_list)static int init_tdx_module (void ) { - static DECLARE_PADDED_STRUCT(tdsysinfo_struct, tdsysinfo, - TDSYSINFO_STRUCT_SIZE, TDSYSINFO_STRUCT_ALIGNMENT);static struct cmr_info cmr_array[MAX_CMRS ]__aligned(CMR_INFO_ARRAY_ALIGNMENT) ;struct tdsysinfo_struct *sysinfo = &PADDED_STRUCT (tdsysinfo);
@@ -1196,7 +1209,7 @@ static int init_tdx_module(void)if (ret) goto out; - ret =tdx_get_sysinfo (sysinfo, cmr_array); + ret =__tdx_get_sysinfo (sysinfo, cmr_array);if (ret) goto out;
arch/x86/virt/vmx/tdx/tdx.h +0 -51
63 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 9/106 KVM: TDX: x86: Add ioctl to get TDX systemwide parameters
In reply to [email protected]From: Sean Christopherson <[email protected]>
Implement a system-scoped ioctl to get system-wide parameters for TDX.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/uapi/asm/kvm.h | 48 +++++++++++++++++++++++++ arch/x86/kvm/vmx/main.c | 2 ++ arch/x86/kvm/vmx/tdx.c | 51 +++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ arch/x86/kvm/x86.c | 6 ++++ tools/arch/x86/include/uapi/asm/kvm.h | 48 +++++++++++++++++++++++++ 8 files changed, 159 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -117,6 +117,7 @@ KVM_X86_OP(enter_smm) KVM_X86_OP(leave_smm) KVM_X86_OP(enable_smi_window) #endif +KVM_X86_OP_OPTIONAL(dev_mem_enc_ioctl) KVM_X86_OP_OPTIONAL(mem_enc_ioctl) KVM_X86_OP_OPTIONAL(mem_enc_register_region) KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1702,6 +1702,7 @@ struct kvm_x86_ops {void (*enable_smi_window)(struct kvm_vcpu *vcpu);#endif +int (*dev_mem_enc_ioctl)(void __user *argp);int (*mem_enc_ioctl)(struct kvm *kvm,void __user *argp);int (*mem_enc_register_region)(struct kvm *kvm,struct kvm_enc_region *argp);int (*mem_enc_unregister_region)(struct kvm *kvm,struct kvm_enc_region *argp);
arch/x86/include/uapi/asm/kvm.h +48 -0
52 lines not on this page · raw message
arch/x86/kvm/vmx/main.c +2 -0
@@ -188,6 +188,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .complete_emulated_msr = kvm_complete_insn_gp, .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector, + + .dev_mem_enc_ioctl = tdx_dev_ioctl, };struct kvm_x86_init_ops vt_init_ops__initdata = {
arch/x86/kvm/vmx/tdx.c +51 -0
57 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -140,9 +140,11 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);#ifdef CONFIG_INTEL_TDX_HOST int __init tdx_hardware_setup (struct kvm_x86_ops *x86_ops);bool tdx_is_vm_type_supported (unsigned long type); +int tdx_dev_ioctl (void __user *argp);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -ENOSYS ; }static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; } +static inline int tdx_dev_ioctl (void __user *argp) {return -EOPNOTSUPP ; };#endif #endif /* __KVM_X86_VMX_X86_OPS_H */
arch/x86/kvm/x86.c +6 -0
@@ -4708,6 +4708,12 @@ long kvm_arch_dev_ioctl(struct file *filp, r =kvm_x86_dev_has_attr (&attr);break ; } +case KVM_MEMORY_ENCRYPT_OP : + r = -EINVAL ; +if (!kvm_x86_ops.dev_mem_enc_ioctl) + goto out; + r =static_call (kvm_x86_dev_mem_enc_ioctl)(argp); +break ;default : r = -EINVAL ;break ;
tools/arch/x86/include/uapi/asm/kvm.h +48 -0
52 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 10/106 KVM: TDX: Add place holder for TDX VM specific mem_enc_op ioctl
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add a place holder function for TDX specific VM-scoped ioctl as mem_enc_op. TDX specific sub-commands will be added to retrieve/pass TDX specific parameters. Make mem_enc_ioctl non-optional as it's not optional now.
KVM_MEMORY_ENCRYPT_OP was introduced for VM-scoped operations specific for guest state-protected VM. It defined subcommands for technology-specific operations under KVM_MEMORY_ENCRYPT_OP. Despite its name, the subcommands are not limited to memory encryption, but various technology-specific operations are defined. It's natural to repurpose KVM_MEMORY_ENCRYPT_OP for TDX specific operations and define subcommands.
TDX requires VM-scoped TDX-specific operations for device model, for example, qemu. Getting system-wide parameters, TDX-specific VM initialization.
--- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/kvm/vmx/main.c | 9 +++++++++ arch/x86/kvm/vmx/tdx.c | 26 ++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ arch/x86/kvm/x86.c | 4 ---- 5 files changed, 40 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -1
@@ -117,7 +117,7 @@ KVM_X86_OP(enter_smm) KVM_X86_OP(leave_smm) KVM_X86_OP(enable_smi_window) #endif -KVM_X86_OP_OPTIONAL(dev_mem_enc_ioctl) +KVM_X86_OP(dev_mem_enc_ioctl) KVM_X86_OP_OPTIONAL(mem_enc_ioctl) KVM_X86_OP_OPTIONAL(mem_enc_register_region) KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)
arch/x86/kvm/vmx/main.c +9 -0
@@ -37,6 +37,14 @@ static int vt_vm_init(struct kvm *kvm)return vmx_vm_init (kvm); } +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) +{ +if (!is_td (kvm)) +return -ENOTTY ; + +return tdx_vm_ioctl (kvm, argp); +} +#define VMX_REQUIRED_APICV_INHIBITS \ ( \ BIT(APICV_INHIBIT_REASON_DISABLE)| \
@@ -190,6 +198,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector, .dev_mem_enc_ioctl = tdx_dev_ioctl, + .mem_enc_ioctl = vt_mem_enc_ioctl, };struct kvm_x86_init_ops vt_init_ops__initdata = {
arch/x86/kvm/vmx/tdx.c +26 -0
@@ -67,6 +67,32 @@ int tdx_dev_ioctl(void __user *argp)return 0 ; } +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) +{ +struct kvm_tdx_cmd tdx_cmd; +int r; + +if (copy_from_user (&tdx_cmd, argp,sizeof (struct kvm_tdx_cmd ))) +return -EFAULT ; +if (tdx_cmd.error || tdx_cmd.unused) +return -EINVAL ; + +mutex_lock (&kvm->lock); + +switch (tdx_cmd.id) { +default : + r = -EINVAL ; + goto out; + } + +if (copy_to_user (argp, &tdx_cmd,sizeof (struct kvm_tdx_cmd ))) + r = -EFAULT ; + +out: +mutex_unlock (&kvm->lock); +return r; +} +static int __init tdx_module_setup (void ) {const struct tdsysinfo_struct *tdsysinfo;
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -141,10 +141,14 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);int __init tdx_hardware_setup (struct kvm_x86_ops *x86_ops);bool tdx_is_vm_type_supported (unsigned long type);int tdx_dev_ioctl (void __user *argp); + +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -ENOSYS ; }static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; }static inline int tdx_dev_ioctl (void __user *argp) {return -EOPNOTSUPP ; }; + +static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }#endif #endif /* __KVM_X86_VMX_X86_OPS_H */
arch/x86/kvm/x86.c +0 -4
@@ -6961,10 +6961,6 @@ long kvm_arch_vm_ioctl(struct file *filp, goto out; }case KVM_MEMORY_ENCRYPT_OP : { - r = -ENOTTY ; -if (!kvm_x86_ops.mem_enc_ioctl) - goto out; - r =static_call (kvm_x86_mem_enc_ioctl)(kvm, argp);break ; }
Signature · 1 line
2.25.1
PATCH v12 11/106 KVM: x86, tdx: Make KVM_CAP_MAX_VCPUS backend specific
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX has its own limitation on the maximum number of vcpus. Make it backend specific and return TDX specific value for KVM_CAP_MAX_VCPUS.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx/main.c | 13 +++++++++++++ arch/x86/kvm/x86.c | 2 ++ 4 files changed, 17 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -21,6 +21,7 @@ KVM_X86_OP(hardware_unsetup) KVM_X86_OP(has_emulated_msr) KVM_X86_OP(vcpu_after_set_cpuid) KVM_X86_OP(is_vm_type_supported) +KVM_X86_OP_OPTIONAL(max_vcpus); KVM_X86_OP(vm_init) KVM_X86_OP_OPTIONAL(vm_destroy) KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1547,6 +1547,7 @@ struct kvm_x86_ops {void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);bool (*is_vm_type_supported)(unsigned long vm_type); +int (*max_vcpus)(struct kvm *kvm);unsigned int vm_size;int (*vm_init)(struct kvm *kvm);void (*vm_destroy)(struct kvm *kvm);
arch/x86/kvm/vmx/main.c +13 -0
@@ -6,6 +6,7 @@#include "nested.h" #include "pmu.h" #include "tdx.h" +#include "tdx_arch.h" static bool enable_tdx__ro_after_init ;module_param_named (tdx, enable_tdx, bool,0444 );
@@ -16,6 +17,17 @@ static bool vt_is_vm_type_supported(unsigned long type) (enable_tdx &&tdx_is_vm_type_supported (type)); } +static int vt_max_vcpus (struct kvm *kvm) +{ +if (!kvm) +return KVM_MAX_VCPUS ; + +if (is_td (kvm)) +return min3 (kvm->max_vcpus,KVM_MAX_VCPUS ,TDX_MAX_VCPUS ); + +return kvm->max_vcpus; +} +static __init int vt_hardware_setup (void ) {int ret;
@@ -68,6 +80,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .has_emulated_msr = vmx_has_emulated_msr, .is_vm_type_supported = vt_is_vm_type_supported, + .max_vcpus = vt_max_vcpus, .vm_size = sizeof(struct kvm_vmx), .vm_init = vt_vm_init, .vm_destroy = vmx_vm_destroy,
arch/x86/kvm/x86.c +2 -0
@@ -4490,6 +4490,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)break ;case KVM_CAP_MAX_VCPUS : r =KVM_MAX_VCPUS ; +if (kvm_x86_ops.max_vcpus) + r =static_call (kvm_x86_max_vcpus)(kvm);break ;case KVM_CAP_MAX_VCPU_ID : r =KVM_MAX_VCPU_IDS ;
Signature · 1 line
2.25.1
PATCH v12 12/106 KVM: x86/vmx, tdx: Allow VMX, TDX to override KVM_ENABLE_CAP
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX has its own control of maximum number of vcpus. Not KVM_MAX_VCPUS. Allow vmx, tdx to override KVM_CAP_MAX_CPUS so that it can specify its own maximum number of cpus instead of KVM_MAX_VCPUS.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx/main.c | 9 +++++++++ arch/x86/kvm/vmx/tdx.c | 30 ++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 3 +++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ arch/x86/kvm/x86.c | 2 ++ 7 files changed, 48 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -22,6 +22,7 @@ KVM_X86_OP(has_emulated_msr) KVM_X86_OP(vcpu_after_set_cpuid) KVM_X86_OP(is_vm_type_supported) KVM_X86_OP_OPTIONAL(max_vcpus); +KVM_X86_OP_OPTIONAL(vm_enable_cap) KVM_X86_OP(vm_init) KVM_X86_OP_OPTIONAL(vm_destroy) KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1549,6 +1549,7 @@ struct kvm_x86_ops {bool (*is_vm_type_supported)(unsigned long vm_type);int (*max_vcpus)(struct kvm *kvm);unsigned int vm_size; +int (*vm_enable_cap)(struct kvm *kvm,struct kvm_enable_cap *cap);int (*vm_init)(struct kvm *kvm);void (*vm_destroy)(struct kvm *kvm);
arch/x86/kvm/vmx/main.c +9 -0
@@ -41,6 +41,14 @@ static __init int vt_hardware_setup(void)return 0 ; } +static int vt_vm_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap) +{ +if (is_td (kvm)) +return tdx_vm_enable_cap (kvm, cap); + +return -EINVAL ; +} +static int vt_vm_init (struct kvm *kvm) {if (is_td (kvm))
@@ -82,6 +90,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .is_vm_type_supported = vt_is_vm_type_supported, .max_vcpus = vt_max_vcpus, .vm_size = sizeof(struct kvm_vmx), + .vm_enable_cap = vt_vm_enable_cap, .vm_init = vt_vm_init, .vm_destroy = vmx_vm_destroy,
arch/x86/kvm/vmx/tdx.c +30 -0
@@ -16,6 +16,36 @@ offsetof(struct tdsysinfo_struct, cpuid_configs)) \ / sizeof(struct tdx_cpuid_config)) +int tdx_vm_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap) +{ +int r; + +switch (cap->cap) { +case KVM_CAP_MAX_VCPUS : { +if (cap->flags || cap->args[0 ] ==0 ) +return -EINVAL ; +if (cap->args[0 ] >KVM_MAX_VCPUS ) +return -E2BIG ; +if (cap->args[0 ] >TDX_MAX_VCPUS ) +return -E2BIG ; + +mutex_lock (&kvm->lock); +if (kvm->created_vcpus) + r = -EBUSY ; +else { + kvm->max_vcpus = cap->args[0 ]; + r =0 ; + } +mutex_unlock (&kvm->lock); +break ; + } +default : + r = -EINVAL ; +break ; + } +return r; +} +int tdx_dev_ioctl (void __user *argp) {struct kvm_tdx_capabilities __user *user_caps;
arch/x86/kvm/vmx/tdx.h +3 -0
@@ -3,6 +3,9 @@#define __KVM_X86_TDX_H #ifdef CONFIG_INTEL_TDX_HOST + +#include "tdx_ops.h" + struct kvm_tdx { struct kvm kvm; /* TDX specific members follow. */
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -142,12 +142,14 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);bool tdx_is_vm_type_supported (unsigned long type);int tdx_dev_ioctl (void __user *argp); +int tdx_vm_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -ENOSYS ; }static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; }static inline int tdx_dev_ioctl (void __user *argp) {return -EOPNOTSUPP ; }; +static inline int tdx_vm_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap) {return -EINVAL ; };static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }#endif
arch/x86/kvm/x86.c +2 -0
@@ -6421,6 +6421,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,break ;default : r = -EINVAL ; +if (kvm_x86_ops.vm_enable_cap) + r =static_call (kvm_x86_vm_enable_cap)(kvm, cap);break ; }return r;
Signature · 1 line
2.25.1
PATCH v12 13/106 KVM: TDX: create/destroy VM structure
In reply to [email protected]From: Isaku Yamahata <[email protected]>
As the first step to create TDX guest, create/destroy VM struct. Assign TDX private Host Key ID (HKID) to the TDX guest for memory encryption and allocate extra pages for the TDX guest. On destruction, free allocated pages, and HKID.
Before tearing down private page tables, TDX requires some resources of the guest TD to be destroyed (i.e. HKID must have been reclaimed, etc). Add flush_shadow_all_private callback before tearing down private page tables for it.
Add vm_free() of kvm_x86_ops hook at the end of kvm_arch_destroy_vm() because some per-VM TDX resources, e.g. TDR, need to be freed after other TDX resources, e.g. HKID, were freed.
---
Changes v11 -> v12:
- use cpu_feature_enabled().
Changes v10 -> v11:
- Fix doule free in tdx_vm_free() by setting NULL.
- replace struct tdx_td_page tdr and tdcs from struct kvm_tdx with unsigned long
--- arch/x86/include/asm/kvm-x86-ops.h | 2 + arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/vmx/main.c | 34 ++- arch/x86/kvm/vmx/tdx.c | 440 ++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 6 +- arch/x86/kvm/vmx/x86_ops.h | 9 + arch/x86/kvm/x86.c | 8 + 7 files changed, 496 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +2 -0
@@ -24,7 +24,9 @@ KVM_X86_OP(is_vm_type_supported)KVM_X86_OP_OPTIONAL (max_vcpus); KVM_X86_OP_OPTIONAL(vm_enable_cap) KVM_X86_OP(vm_init) +KVM_X86_OP_OPTIONAL(flush_shadow_all_private) KVM_X86_OP_OPTIONAL(vm_destroy) +KVM_X86_OP_OPTIONAL(vm_free) KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate) KVM_X86_OP(vcpu_create) KVM_X86_OP(vcpu_free)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1551,7 +1551,9 @@ struct kvm_x86_ops {unsigned int vm_size;int (*vm_enable_cap)(struct kvm *kvm,struct kvm_enable_cap *cap);int (*vm_init)(struct kvm *kvm); +void (*flush_shadow_all_private)(struct kvm *kvm);void (*vm_destroy)(struct kvm *kvm); +void (*vm_free)(struct kvm *kvm);/* Create, but do not attach this VCPU */ int (*vcpu_precreate)(struct kvm *kvm);
arch/x86/kvm/vmx/main.c +31 -3
@@ -49,14 +49,40 @@ static int vt_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap)return -EINVAL ; } +static void vt_hardware_unsetup (void ) +{ +tdx_hardware_unsetup (); +vmx_hardware_unsetup (); +} +static int vt_vm_init (struct kvm *kvm) {if (is_td (kvm)) -return -EOPNOTSUPP ;/* Not ready to create guest TD yet. */ +return tdx_vm_init (kvm);return vmx_vm_init (kvm); } +static void vt_flush_shadow_all_private (struct kvm *kvm) +{ +if (is_td (kvm)) +tdx_mmu_release_hkid (kvm); +} + +static void vt_vm_destroy (struct kvm *kvm) +{ +if (is_td (kvm)) +return ; + +vmx_vm_destroy (kvm); +} + +static void vt_vm_free (struct kvm *kvm) +{ +if (is_td (kvm)) +tdx_vm_free (kvm); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -81,7 +107,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .check_processor_compatibility = vmx_check_processor_compat, - .hardware_unsetup = vmx_hardware_unsetup, + .hardware_unsetup = vt_hardware_unsetup, .hardware_enable = vmx_hardware_enable, .hardware_disable = vmx_hardware_disable,
@@ -92,7 +118,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vm_size = sizeof(struct kvm_vmx), .vm_enable_cap = vt_vm_enable_cap, .vm_init = vt_vm_init, - .vm_destroy = vmx_vm_destroy, + .flush_shadow_all_private = vt_flush_shadow_all_private, + .vm_destroy = vt_vm_destroy, + .vm_free = vt_vm_free, .vcpu_precreate = vmx_vcpu_precreate, .vcpu_create = vmx_vcpu_create,
arch/x86/kvm/vmx/tdx.c +439 -1
481 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +5 -1
@@ -8,7 +8,11 @@struct kvm_tdx {struct kvm kvm; -/* TDX specific members follow. */ + +unsigned long tdr_pa; +unsigned long *tdcs_pa; + +int hkid; };struct vcpu_tdx {
arch/x86/kvm/vmx/x86_ops.h +9 -0
@@ -139,17 +139,26 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);#ifdef CONFIG_INTEL_TDX_HOST int __init tdx_hardware_setup (struct kvm_x86_ops *x86_ops); +void tdx_hardware_unsetup (void );bool tdx_is_vm_type_supported (unsigned long type);int tdx_dev_ioctl (void __user *argp);int tdx_vm_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap); +int tdx_vm_init (struct kvm *kvm); +void tdx_mmu_release_hkid (struct kvm *kvm); +void tdx_vm_free (struct kvm *kvm);int tdx_vm_ioctl (struct kvm *kvm,void __user *argp);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -ENOSYS ; } +static inline void tdx_hardware_unsetup (void ) {}static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; }static inline int tdx_dev_ioctl (void __user *argp) {return -EOPNOTSUPP ; };static inline int tdx_vm_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap) {return -EINVAL ; }; +static inline int tdx_vm_init (struct kvm *kvm) {return -EOPNOTSUPP ; } +static inline void tdx_mmu_release_hkid (struct kvm *kvm) {} +static inline void tdx_flush_shadow_all_private (struct kvm *kvm) {} +static inline void tdx_vm_free (struct kvm *kvm) {}static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; }#endif
arch/x86/kvm/x86.c +8 -0
@@ -12356,6 +12356,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm)kvm_page_track_cleanup (kvm);kvm_xen_destroy_vm (kvm);kvm_hv_destroy_vm (kvm); +static_call_cond (kvm_x86_vm_free)(kvm); }static void memslot_rmap_free (struct kvm_memory_slot *slot)
@@ -12670,6 +12671,13 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,void kvm_arch_flush_shadow_all (struct kvm *kvm) { +/* +* kvm_mmu_zap_all() zaps both private and shared page tables. Before +* tearing down private page tables, TDX requires some TD resources to +* be destroyed (i.e. keyID must have been reclaimed, etc). Invoke +* kvm_x86_flush_shadow_all_private() for this. +*/ +static_call_cond (kvm_x86_flush_shadow_all_private)(kvm);kvm_mmu_zap_all (kvm); }
Signature · 1 line
2.25.1
Re: [PATCH v12 013/106] KVM: TDX: create/destroy VM structure
Quoted 600 lines · On 2/27/2023 4:22 PM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
As the first step to create TDX guest, create/destroy VM struct. Assign TDX private Host Key ID (HKID) to the TDX guest for memory encryption and allocate extra pages for the TDX guest. On destruction, free allocated pages, and HKID.
Before tearing down private page tables, TDX requires some resources of the guest TD to be destroyed (i.e. HKID must have been reclaimed, etc). Add
flush_shadow_all_privatecallback before tearing down private page tables for it.Add
vm_free()ofkvm_x86_opshook at the end ofkvm_arch_destroy_vm()because some per-VM TDX resources, e.g. TDR, need to be freed after other TDX resources, e.g. HKID, were freed.Co-developed-by Kai Huang <[email protected]>Signed-off-by Kai Huang <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>---
Changes v11 -> v12:
- usecpu_feature_enabled().Changes v10 -> v11:
- Fix doule free intdx_vm_free()by setting NULL.
- replace structtdx_td_pagetdr and tdcs from structkvm_tdxwith unsigned long--- arch/x86/include/asm/kvm-x86-ops.h | 2 + arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/vmx/main.c | 34 ++- arch/x86/kvm/vmx/tdx.c | 440 ++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 6 +- arch/x86/kvm/vmx/x86_ops.h | 9 + arch/x86/kvm/x86.c | 8 + 7 files changed, 496 insertions(+), 5 deletions(-)arch/x86/include/asm/kvm-x86-ops.h +2
-0@@ -24,7 +24,9 @@ KVM_X86_OP(is_vm_type_supported)KVM_X86_OP_OPTIONAL (max_vcpus); KVM_X86_OP_OPTIONAL(vm_enable_cap) KVM_X86_OP(vm_init) +KVM_X86_OP_OPTIONAL(flush_shadow_all_private) KVM_X86_OP_OPTIONAL(vm_destroy) +KVM_X86_OP_OPTIONAL(vm_free) KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate) KVM_X86_OP(vcpu_create) KVM_X86_OP(vcpu_free)arch/x86/include/asm/kvm_host.h +2
-0@@ -1551,7 +1551,9 @@ struct kvm_x86_ops {unsigned int vm_size;int (*vm_enable_cap)(struct kvm *kvm,struct kvm_enable_cap *cap);int (*vm_init)(struct kvm *kvm); +void (*flush_shadow_all_private)(struct kvm *kvm);void (*vm_destroy)(struct kvm *kvm); +void (*vm_free)(struct kvm *kvm);/* Create, but do not attach this VCPU */ int (*vcpu_precreate)(struct kvm *kvm);arch/x86/kvm/vmx/main.c +31
-3@@ -49,14 +49,40 @@ static int vt_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap)return -EINVAL ; } +static void vt_hardware_unsetup (void ) +{ +tdx_hardware_unsetup (); +vmx_hardware_unsetup (); +} +static int vt_vm_init (struct kvm *kvm) {if (is_td (kvm)) -return -EOPNOTSUPP ;/* Not ready to create guest TD yet. */ +return tdx_vm_init (kvm);return vmx_vm_init (kvm); } +static void vt_flush_shadow_all_private (struct kvm *kvm) +{ +if (is_td (kvm)) +tdx_mmu_release_hkid (kvm); +} + +static void vt_vm_destroy (struct kvm *kvm) +{ +if (is_td (kvm)) +return ; + +vmx_vm_destroy (kvm); +} + +static void vt_vm_free (struct kvm *kvm) +{ +if (is_td (kvm)) +tdx_vm_free (kvm); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))@@ -81,7 +107,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .check_processor_compatibility = vmx_check_processor_compat, - .hardware_unsetup = vmx_hardware_unsetup, + .hardware_unsetup = vt_hardware_unsetup, .hardware_enable = vmx_hardware_enable, .hardware_disable = vmx_hardware_disable,@@ -92,7 +118,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vm_size = sizeof(struct kvm_vmx), .vm_enable_cap = vt_vm_enable_cap, .vm_init = vt_vm_init, - .vm_destroy = vmx_vm_destroy, + .flush_shadow_all_private = vt_flush_shadow_all_private, + .vm_destroy = vt_vm_destroy, + .vm_free = vt_vm_free, .vcpu_precreate = vmx_vcpu_precreate, .vcpu_create = vmx_vcpu_create,arch/x86/kvm/vmx/tdx.c +425
-1@@ -6,6 +6,7 @@#include "capabilities.h" #include "x86_ops.h" #include "tdx.h" +#include "tdx_ops.h" #include "x86.h" #undef pr_fmt@@ -46,6 +47,271 @@ int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap)return r; } +struct tdx_info { +u8 nr_tdcs_pages; +}; + +/* Info about the TDX module. */ +static struct tdx_info tdx_info; + +/* +* Some TDX SEAMCALLs (TDH.MNG.CREATE, TDH.PHYMEM.CACHE.WB, +* TDH.MNG.KEY.RECLAIMID, TDH.MNG.KEY.FREEID etc) tries to acquire a global lock +* internally in TDX module. If failed, TDX_OPERAND_BUSY is returned without +* spinning or waiting due to a constraint on execution time. It's caller's +* responsibility to avoid race (or retry on TDX_OPERAND_BUSY). Use this mutex +* to avoid race in TDX module because the kernel knows better about scheduling. +*/ +static DEFINE_MUTEX (tdx_lock); +static struct mutex *tdx_mng_key_config_lock; + +static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid) +{ +return pa | ((hpa_t )hkid << boot_cpu_data.x86_phys_bits); +} + +static inline bool is_td_created (struct kvm_tdx *kvm_tdx) +{ +return kvm_tdx->tdr_pa; +} + +static inline void tdx_hkid_free (struct kvm_tdx *kvm_tdx) +{ +tdx_guest_keyid_free (kvm_tdx->hkid); + kvm_tdx->hkid =0 ; +} + +static inline bool is_hkid_assigned (struct kvm_tdx *kvm_tdx) +{ +return kvm_tdx->hkid >0 ; +} + +static void tdx_clear_page (unsigned long page_pa) +{ +const void *zero_page = (const void *)__va (page_to_phys (ZERO_PAGE (0 ))); +void *page =__va (page_pa); +unsigned long i; + +if (!cpu_feature_enabled (X86_FEATURE_MOVDIR64B )) { +clear_page (page); +return ; + } + +/* +* Zeroing the page is only necessary for systems with MKTME-i: +* when re-assign one page from old keyid to a new keyid, MOVDIR64B is +* required to clear/write the page with new keyid to prevent integrity +* error when read on the page with new keyid. +* +* clflush doesn't flush cache with HKID set. +* The cache line could be poisoned (even without MKTME-i), clear the +* poison bit. +*/ +for (i =0 ; i <PAGE_SIZE ; i +=64 ) +movdir64b (page + i, zero_page); +/* +* MOVDIR64B store uses WC buffer. Prevent following memory reads +* from seeing potentially poisoned cache. +*/ +__mb (); +} + +static int tdx_reclaim_page (hpa_t pa,bool do_wb,u16 hkid) +{ +struct tdx_module_output out; +u64 err; + +do { + err =tdh_phymem_page_reclaim (pa, &out); +/* +* TDH.PHYMEM.PAGE.RECLAIM is allowed only when TD is shutdown. +* state. i.e. destructing TD. +* TDH.PHYMEM.PAGE.RECLAIM requires TDR and target page. +* Because we're destructing TD, it's rare to contend with TDR. +*/ + }while (err == (TDX_OPERAND_BUSY |TDX_OPERAND_ID_RCX )); +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_PHYMEM_PAGE_RECLAIM , err, &out); +return -EIO ; + } + +if (do_wb) { +/* +* Only TDR page gets into this path. No contention is expected +* because of the last page of TD. +*/ + err =tdh_phymem_page_wbinvd (set_hkid_to_hpa (pa, hkid)); +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err,NULL ); +return -EIO ; + } + } + +tdx_clear_page (pa); +return 0 ; +} + +static void tdx_reclaim_td_page (unsigned long td_page_pa) +{ +if (!td_page_pa) +return ; +/* +* TDCX are being reclaimed. TDX module maps TDCX with HKID +* assigned to the TD. Here the cache associated to the TD +* was already flushed by TDH.PHYMEM.CACHE.WB before here, So +* cache doesn't need to be flushed again. +*/ +if (tdx_reclaim_page (td_page_pa, false,0 )) +/* +* Leak the page on failure: +* tdx_reclaim_page() returns an error if and only if there's an +* unexpected, fatal error, e.g. a SEAMCALL with bad params, +* incorrect concurrency in KVM, a TDX Module bug, etc. +* Retrying at a later point is highly unlikely to be +* successful. +* No log here as tdx_reclaim_page() already did. +*/ +return ; +free_page ((unsigned long )__va (td_page_pa)); +} + +static int tdx_do_tdh_phymem_cache_wb (void *param) +{ +u64 err =0 ; + +do { + err =tdh_phymem_cache_wb (!!err); + }while (err ==TDX_INTERRUPTED_RESUMABLE ); + +/* Other thread may have done for us. */ +if (err ==TDX_NO_HKID_READY_TO_WBCACHE ) + err =TDX_SUCCESS ; +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_PHYMEM_CACHE_WB , err,NULL ); +return -EIO ; + } + +return 0 ; +} + +void tdx_mmu_release_hkid (struct kvm *kvm) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +cpumask_var_t packages; +bool cpumask_allocated; +u64 err; +int ret; +int i; + +if (!is_hkid_assigned (kvm_tdx)) +return ; + +if (!is_td_created (kvm_tdx)) + goto free_hkid; + + cpumask_allocated =zalloc_cpumask_var (&packages,GFP_KERNEL ); +cpus_read_lock (); +for_each_online_cpu (i) { +if (cpumask_allocated && +cpumask_test_and_set_cpu (topology_physical_package_id (i), + packages)) +continue ; + +/* +* We can destroy multiple the guest TDs simultaneously. +* Prevent tdh_phymem_cache_wb from returning TDX_BUSY by +* serialization. +*/ +mutex_lock (&tdx_lock); + ret =smp_call_on_cpu (i, tdx_do_tdh_phymem_cache_wb,NULL ,1 ); +mutex_unlock (&tdx_lock); +if (ret) +break ; + } +cpus_read_unlock (); +free_cpumask_var (packages); + +mutex_lock (&tdx_lock); + err =tdh_mng_key_freeid (kvm_tdx->tdr_pa); +mutex_unlock (&tdx_lock); +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_MNG_KEY_FREEID , err,NULL ); +pr_err ("tdh_mng_key_freeid failed. HKID %d is leaked.\n" , + kvm_tdx->hkid); +return ; + } + +free_hkid: +tdx_hkid_free (kvm_tdx); +} + +void tdx_vm_free (struct kvm *kvm) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +int i; + +/* Can't reclaim or free TD pages if teardown failed. */ +if (is_hkid_assigned (kvm_tdx)) +return ; + +if (kvm_tdx->tdcs_pa) { +for (i =0 ; i < tdx_info.nr_tdcs_pages; i++) +tdx_reclaim_td_page (kvm_tdx->tdcs_pa[i]); +kfree (kvm_tdx->tdcs_pa); + kvm_tdx->tdcs_pa =NULL ; + } + +if (!kvm_tdx->tdr_pa) +return ; +/* +* TDX module maps TDR with TDX global HKID. TDX module may access TDR +* while operating on TD (Especially reclaiming TDCS). Cache flush with +* TDX global HKID is needed. +*/ +if (tdx_reclaim_page (kvm_tdx->tdr_pa, true, tdx_global_keyid)) +return ; + +free_page ((unsigned long )__va (kvm_tdx->tdr_pa)); + kvm_tdx->tdr_pa =0 ; +} + +static int tdx_do_tdh_mng_key_config (void *param) +{ +hpa_t *tdr_p = param; +u64 err; + +do { + err =tdh_mng_key_config (*tdr_p); + +/* +* If it failed to generate a random key, retry it because this +* is typically caused by an entropy error of the CPU's random +* number generator. +*/ + }while (err ==TDX_KEY_GENERATION_FAILED ); + +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_MNG_KEY_CONFIG , err,NULL ); +return -EIO ; + } + +return 0 ; +} + +static int __tdx_td_init (struct kvm *kvm); + +int tdx_vm_init (struct kvm *kvm) +{ +/* +* TDX has its own limit of the number of vcpus in addition to +* KVM_MAX_VCPUS. +*/ + kvm->max_vcpus =min (kvm->max_vcpus,TDX_MAX_VCPUS ); + +/* Place holder for TDX specific logic. */ +return __tdx_td_init (kvm); +} +int tdx_dev_ioctl (void __user *argp) {struct kvm_tdx_capabilities __user *user_caps;@@ -97,6 +363,160 @@ int tdx_dev_ioctl(void __user *argp)return 0 ; } +static int __tdx_td_init (struct kvm *kvm) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +cpumask_var_t packages; +unsigned long *tdcs_pa =NULL ; +unsigned long tdr_pa =0 ; +unsigned long va; +int ret, i; +u64 err; + + ret =tdx_guest_keyid_alloc (); +if (ret <0 ) +return ret; + kvm_tdx->hkid = ret; + + va =__get_free_page (GFP_KERNEL_ACCOUNT ); +if (!va) + goto free_hkid; + tdr_pa =__pa (va); + + tdcs_pa =kcalloc (tdx_info.nr_tdcs_pages,sizeof (*kvm_tdx->tdcs_pa), +GFP_KERNEL_ACCOUNT | __GFP_ZERO); +if (!tdcs_pa) + goto free_tdr; +for (i =0 ; i < tdx_info.nr_tdcs_pages; i++) { + va =__get_free_page (GFP_KERNEL_ACCOUNT ); +if (!va) + goto free_tdcs; + tdcs_pa[i] =__pa (va); + } + +if (!zalloc_cpumask_var (&packages,GFP_KERNEL )) { + ret = -ENOMEM ; + goto free_tdcs; + } +cpus_read_lock (); +/* +* Need at least one CPU of the package to be online in order to +* program all packages for host key id. Check it. +*/ +for_each_present_cpu (i) +cpumask_set_cpu (topology_physical_package_id (i), packages); +for_each_online_cpu (i) +cpumask_clear_cpu (topology_physical_package_id (i), packages); +if (!cpumask_empty (packages)) { + ret = -EIO ; +/* +* Because it's hard for human operator to figure out the +* reason, warn it. +*/ +pr_warn ("All packages need to have online CPU to create TD. Online CPU and retry.\n" ); + goto free_packages; + } + +/* +* Acquire global lock to avoid TDX_OPERAND_BUSY: +* TDH.MNG.CREATE and other APIs try to lock the global Key Owner +* Table (KOT) to track the assigned TDX private HKID. It doesn't spin +* to acquire the lock, returns TDX_OPERAND_BUSY instead, and let the +* caller to handle the contention. This is because of time limitation +* usable inside the TDX module and OS/VMM knows better about process +* scheduling. +* +* APIs to acquire the lock of KOT: +* TDH.MNG.CREATE, TDH.MNG.KEY.FREEID, TDH.MNG.VPFLUSHDONE, and +* TDH.PHYMEM.CACHE.WB. +*/ +mutex_lock (&tdx_lock); + err =tdh_mng_create (tdr_pa, kvm_tdx->hkid); +mutex_unlock (&tdx_lock); +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_MNG_CREATE , err,NULL ); + ret = -EIO ; + goto free_packages; + } + kvm_tdx->tdr_pa = tdr_pa; + +for_each_online_cpu (i) { +int pkg =topology_physical_package_id (i); + +if (cpumask_test_and_set_cpu (pkg, packages)) +continue ; + +/* +* Program the memory controller in the package with an +* encryption key associated to a TDX private host key id +* assigned to this TDR. Concurrent operations on same memory +* controller results in TDX_OPERAND_BUSY. Avoid this race by +* mutex. +*/ +mutex_lock (&tdx_mng_key_config_lock[pkg]); + ret =smp_call_on_cpu (i, tdx_do_tdh_mng_key_config, + &kvm_tdx->tdr_pa, true); +mutex_unlock (&tdx_mng_key_config_lock[pkg]); +if (ret) +break ; + } +cpus_read_unlock (); +free_cpumask_var (packages); +if (ret) + goto teardown; + + kvm_tdx->tdcs_pa = tdcs_pa; +for (i =0 ; i < tdx_info.nr_tdcs_pages; i++) { + err =tdh_mng_addcx (kvm_tdx->tdr_pa, tdcs_pa[i]); +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_MNG_ADDCX , err,NULL ); +for (i++; i < tdx_info.nr_tdcs_pages; i++) { +free_page ((unsigned long )__va (tdcs_pa[i])); + tdcs_pa[i] =0 ; + } + ret = -EIO ; + goto teardown; + } + } + +/* +* Note, TDH_MNG_INIT cannot be invoked here. TDH_MNG_INIT requires a dedicated +* ioctl() to define the configure CPUID values for the TD. +*/ +return 0 ; + +/* +* The sequence for freeing resources from a partially initialized TD +* varies based on where in the initialization flow failure occurred. +* Simply use the full teardown and destroy, which naturally play nice +* with partial initialization. +*/ +teardown: +tdx_mmu_release_hkid (kvm); +tdx_vm_free (kvm); +return ret; + +free_packages: +cpus_read_unlock (); +free_cpumask_var (packages); +free_tdcs: +for (i =0 ; i < tdx_info.nr_tdcs_pages; i++) { +if (tdcs_pa[i]) +free_page ((unsigned long )__va (tdcs_pa[i])); + } +kfree (tdcs_pa); + kvm_tdx->tdcs_pa =NULL ; + +free_tdr: +if (tdr_pa) +free_page ((unsigned long )__va (tdr_pa)); + kvm_tdx->tdr_pa =0 ; +free_hkid: +if (is_hkid_assigned (kvm_tdx)) +tdx_hkid_free (kvm_tdx); +return ret; +} +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {struct kvm_tdx_cmd tdx_cmd;@@ -137,9 +557,11 @@ static int __init tdx_module_setup(void)return ret; } -/* Sanitary check just in case. */ tdsysinfo =tdx_get_sysinfo ();WARN_ON (tdsysinfo->num_cpuid_config >TDX_MAX_NR_CPUID_CONFIGS ); + tdx_info = (struct tdx_info ) { + .nr_tdcs_pages = tdsysinfo->tdcs_base_size /PAGE_SIZE , + };pr_info ("TDX is supported.\n" );return 0 ;@@ -153,6 +575,8 @@ bool tdx_is_vm_type_supported(unsigned long type)int __init tdx_hardware_setup (struct kvm_x86_ops *x86_ops) { +int max_pkgs; +int i;int r;if (!enable_ept) {
...
Quoted 16 lines
arch/x86/kvm/vmx/tdx.c +8
-0@@ -160,6 +584,14 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return -EINVAL ; } + max_pkgs =topology_max_packages (); + tdx_mng_key_config_lock =kcalloc (max_pkgs,sizeof (*tdx_mng_key_config_lock), +GFP_KERNEL ); +if (!tdx_mng_key_config_lock) +return -ENOMEM ; +for (i =0 ; i < max_pkgs; i++) +mutex_init (&tdx_mng_key_config_lock[i]); +/* TDX requires VMX. */ r =vmxon_all ();if (!r)
arch/x86/kvm/vmx/tdx.c int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops) · line 605
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +3
-0@@ -168,3 +600,9 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return r; } + +void tdx_hardware_unsetup (void ) +{
To align with tdx_hardware_setup(), add some check like:
if (!enable_ept || !enable_tdx)
return;
Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +3
-0+/* kfree accepts NULL. */ +kfree (tdx_mng_key_config_lock); +}
Re: [PATCH v12 013/106] KVM: TDX: create/destroy VM structure
Quoted 24 lines · Chenyi Qiang
Quoted 15 lines
arch/x86/kvm/vmx/tdx.c +8
-0return -EINVAL ; } + max_pkgs =topology_max_packages (); + tdx_mng_key_config_lock =kcalloc (max_pkgs,sizeof (*tdx_mng_key_config_lock), +GFP_KERNEL ); +if (!tdx_mng_key_config_lock) +return -ENOMEM ; +for (i =0 ; i < max_pkgs; i++) +mutex_init (&tdx_mng_key_config_lock[i]); +/* TDX requires VMX. */ r =vmxon_all ();if (!r)arch/x86/kvm/vmx/tdx.c int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops) · line 605
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +3
-0@@ -168,3 +600,9 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return r; } + +void tdx_hardware_unsetup (void ) +{To align with
tdx_hardware_setup(), add some check like:
Quoted 3 lines
if (!enable_ept || !enable_tdx)
return;
Oh yes, thanks for catching it.
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v12 14/106 KVM: TDX: initialize VM with TDX specific parameters
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX requires additional parameters for TDX VM for confidential execution to protect its confidentiality of its memory contents and its CPU state from any other software, including VMM. When creating guest TD VM before creating vcpu, the number of vcpu, TSC frequency (that is same among vcpus. and it can't be changed.) CPUIDs which is emulated by the TDX module. It means guest can trust those CPUIDs. and sha384 values for measurement.
Add new subcommand, KVM_TDX_INIT_VM, to pass parameters for TDX guest. It assigns encryption key to the TDX guest for memory encryption. TDX encrypts memory per-guest bases. It assigns device model passes per-VM parameters for the TDX guest. The maximum number of vcpus, tsc frequency (TDX guest has fised VM-wide TSC frequency. not per-vcpu. The TDX guest can not change it.), attributes (production or debug), available extended features (which is reflected into guest XCR0, IA32_XSS MSR), cpuids, sha384 measurements, and etc.
This subcommand is called before creating vcpu and KVM_SET_CPUID2, i.e. cpuids configurations aren't available yet. So CPUIDs configuration values needs to be passed in struct kvm_tdx_init_vm. It's device model responsibility to make this cpuid config for KVM_TDX_INIT_VM and KVM_SET_CPUID2.
---
Changes from v11 to v12
- ABI change. Changes struct kvm_tdx_init_vm layout
--- arch/x86/include/asm/tdx.h | 3 + arch/x86/include/uapi/asm/kvm.h | 24 +++ arch/x86/kvm/cpuid.c | 7 + arch/x86/kvm/cpuid.h | 2 + arch/x86/kvm/vmx/tdx.c | 247 ++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.h | 18 ++ tools/arch/x86/include/uapi/asm/kvm.h | 33 ++++ 7 files changed, 324 insertions(+), 10 deletions(-)
arch/x86/include/asm/tdx.h +3 -0
@@ -105,6 +105,9 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,#endif /* CONFIG_INTEL_TDX_GUEST && CONFIG_KVM_GUEST */ #ifdef CONFIG_INTEL_TDX_HOST + +/* -1 indicates CPUID leaf with no sub-leaves. */ +#define TDX_CPUID_NO_SUBLEAF ((u32)-1) struct tdx_cpuid_config { u32 leaf; u32 sub_leaf;
arch/x86/include/uapi/asm/kvm.h +24 -0
@@ -535,6 +535,7 @@ struct kvm_pmu_event_filter {/* Trust Domain eXtension sub-ioctl() commands. */ enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 , +KVM_TDX_INIT_VM ,KVM_TDX_CMD_NR_MAX , };
@@ -580,4 +581,27 @@ struct kvm_tdx_capabilities { struct kvm_tdx_cpuid_config cpuid_configs[0]; }; +struct kvm_tdx_init_vm { +__u64 attributes; +__u64 mrconfigid[6 ];/* sha384 digest */ +__u64 mrowner[6 ];/* sha384 digest */ +__u64 mrownerconfig[6 ];/* sha348 digest */ +/* +* For future extensibility to make sizeof(struct kvm_tdx_init_vm) = 8KB. +* This should be enough given sizeof(TD_PARAMS) = 1024. +* 8KB was chosen given because +* sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(=256) = 8KB. +*/ +__u64 reserved[1004 ]; + +/* +* KVM_TDX_INIT_VM is called before vcpu creation, thus before +* KVM_SET_CPUID2. +* This configuration supersedes KVM_SET_CPUID2s for VCPUs. The user +* space VMM, e.g. qemu, should make KVM_SET_CPUID2 consistent with this +* values. +*/ +struct kvm_cpuid2 cpuid; +}; +#endif /* _ASM_X86_KVM_H */
arch/x86/kvm/cpuid.c +7 -0
@@ -1378,6 +1378,13 @@ int kvm_dev_ioctl_get_cpuid(struct kvm_cpuid2 *cpuid,return r; } +struct kvm_cpuid_entry2 *kvm_find_cpuid_entry2 (struct kvm_cpuid2 *cpuid, +u32 function,u32 index) +{ +return cpuid_entry2_find (cpuid->entries, cpuid->nent, function, index); +} +EXPORT_SYMBOL_GPL (kvm_find_cpuid_entry2); +struct kvm_cpuid_entry2 *kvm_find_cpuid_entry_index (struct kvm_vcpu *vcpu,u32 function,u32 index) {
arch/x86/kvm/cpuid.h +2 -0
@@ -13,6 +13,8 @@ void kvm_set_cpu_caps(void);void kvm_update_cpuid_runtime (struct kvm_vcpu *vcpu);void kvm_update_pv_runtime (struct kvm_vcpu *vcpu); +struct kvm_cpuid_entry2 *kvm_find_cpuid_entry2 (struct kvm_cpuid2 *cpuid, +u32 function,u32 index);struct kvm_cpuid_entry2 *kvm_find_cpuid_entry_index (struct kvm_vcpu *vcpu,u32 function,u32 index); struct kvm_cpuid_entry2 *kvm_find_cpuid_entry(struct kvm_vcpu *vcpu,
arch/x86/kvm/vmx/tdx.c +237 -10
293 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +18 -0
@@ -12,7 +12,11 @@ struct kvm_tdx {unsigned long tdr_pa;unsigned long *tdcs_pa; +u64 attributes; +u64 xfam;int hkid; + +u64 tsc_offset; };struct vcpu_tdx {
@@ -39,6 +43,20 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) { return container_of(vcpu, struct vcpu_tdx, vcpu); } + +static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field) +{ + struct tdx_module_output out; + u64 err; + + err = tdh_mng_rd(kvm_tdx->tdr_pa, TDCS_EXEC(field), &out); + if (unlikely(err)) { + pr_err("TDH_MNG_RD[EXEC.0x%x] failed: 0x%llx\n", field, err); + return 0; + } + return out.r8; +} + #else struct kvm_tdx { struct kvm kvm;
tools/arch/x86/include/uapi/asm/kvm.h +33 -0
@@ -535,6 +535,7 @@ struct kvm_pmu_event_filter {/* Trust Domain eXtension sub-ioctl() commands. */ enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 , +KVM_TDX_INIT_VM ,KVM_TDX_CMD_NR_MAX , };
@@ -580,4 +581,36 @@ struct kvm_tdx_capabilities { struct kvm_tdx_cpuid_config cpuid_configs[0]; }; +struct kvm_tdx_init_vm { +__u64 attributes; +__u32 max_vcpus; +__u32 padding; +__u64 mrconfigid[6 ];/* sha384 digest */ +__u64 mrowner[6 ];/* sha384 digest */ +__u64 mrownerconfig[6 ];/* sha348 digest */ +union { +/* +* KVM_TDX_INIT_VM is called before vcpu creation, thus before +* KVM_SET_CPUID2. CPUID configurations needs to be passed. +* +* This configuration supersedes KVM_SET_CPUID{,2}. +* The user space VMM, e.g. qemu, should make them consistent +* with this values. +* sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(256) +* = 8KB. +*/ +struct { +struct kvm_cpuid2 cpuid; +/* 8KB with KVM_MAX_CPUID_ENTRIES. */ +struct kvm_cpuid_entry2 entries[]; + }; +/* +* For future extensibility. +* The size(struct kvm_tdx_init_vm) = 16KB. +* This should be enough given sizeof(TD_PARAMS) = 1024 +*/ +__u64 reserved[2028 ]; + }; +}; +#endif /* _ASM_X86_KVM_H */
Signature · 1 line
2.25.1
PATCH v12 15/106 KVM: TDX: Make pmu_intel.c ignore guest TD case
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because TDX KVM doesn't support PMU yet (it's future work of TDX KVM support as another patch series) and pmu_intel.c touches vmx specific structure in vcpu initialization, as workaround add dummy structure to struct vcpu_tdx and pmu_intel.c can ignore TDX case.
--- arch/x86/kvm/vmx/pmu_intel.c | 46 +++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/pmu_intel.h | 28 ++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 8 ++++++- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/vmx/vmx.h | 32 +------------------------ 5 files changed, 82 insertions(+), 34 deletions(-) create mode 100644 arch/x86/kvm/vmx/pmu_intel.h
arch/x86/kvm/vmx/pmu_intel.c +45 -1
@@ -19,6 +19,7 @@#include "lapic.h" #include "nested.h" #include "pmu.h" +#include "tdx.h" #define MSR_PMC_FULL_WIDTH_BIT (MSR_IA32_PMC0 - MSR_IA32_PERFCTR0)
@@ -37,6 +38,26 @@ static struct kvm_event_hw_type_mapping intel_arch_events[] = {/* mapping between fixed pmc index and intel_arch_events array */ static int fixed_pmc_events[] = {1 ,0 ,7 }; +struct lbr_desc *vcpu_to_lbr_desc (struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_INTEL_TDX_HOST +if (is_td_vcpu (vcpu)) +return &to_tdx (vcpu)->lbr_desc; +#endif + +return &to_vmx (vcpu)->lbr_desc; +} + +struct x86_pmu_lbr *vcpu_to_lbr_records (struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_INTEL_TDX_HOST +if (is_td_vcpu (vcpu)) +return &to_tdx (vcpu)->lbr_desc.records; +#endif + +return &to_vmx (vcpu)->lbr_desc.records; +} +static void reprogram_fixed_counters (struct kvm_pmu *pmu,u64 data) {struct kvm_pmc *pmc;
@@ -169,6 +190,23 @@ static inline struct kvm_pmc *get_fw_gp_pmc(struct kvm_pmu *pmu, u32 msr)return get_gp_pmc (pmu, msr,MSR_IA32_PMC0 ); } +bool intel_pmu_lbr_is_compatible (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return false; +return cpuid_model_is_consistent (vcpu); +} + +bool intel_pmu_lbr_is_enabled (struct kvm_vcpu *vcpu) +{ +struct x86_pmu_lbr *lbr =vcpu_to_lbr_records (vcpu); + +if (is_td_vcpu (vcpu)) +return false; + +return lbr->nr && (vcpu_get_perf_capabilities (vcpu) &PMU_CAP_LBR_FMT ); +} +static bool intel_pmu_is_valid_lbr_msr (struct kvm_vcpu *vcpu,u32 index) {struct x86_pmu_lbr *records =vcpu_to_lbr_records (vcpu);
@@ -279,6 +317,9 @@ int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu) PERF_SAMPLE_BRANCH_USER, }; +if (WARN_ON_ONCE (is_td_vcpu (vcpu))) +return 0 ; +if (unlikely (lbr_desc->event)) {__set_bit (INTEL_PMC_IDX_FIXED_VLBR , pmu->pmc_in_use);return 0 ;
@@ -588,7 +629,7 @@ static void intel_pmu_refresh(struct kvm_vcpu *vcpu) INTEL_PMC_MAX_GENERIC, pmu->nr_arch_fixed_counters); perf_capabilities =vcpu_get_perf_capabilities (vcpu); -if (cpuid_model_is_consistent (vcpu) && +if (intel_pmu_lbr_is_compatible (vcpu) && (perf_capabilities &PMU_CAP_LBR_FMT ))x86_perf_get_lbr (&lbr_desc->records);else
@@ -644,6 +685,9 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu)struct kvm_pmc *pmc =NULL ;int i; +if (is_td_vcpu (vcpu)) +return ; +for (i =0 ; i <KVM_INTEL_PMC_MAX_GENERIC ; i++) { pmc = &pmu->gp_counters[i];
arch/x86/kvm/vmx/pmu_intel.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_VMX_PMU_INTEL_H +#define __KVM_X86_VMX_PMU_INTEL_H + +struct lbr_desc *vcpu_to_lbr_desc (struct kvm_vcpu *vcpu); +struct x86_pmu_lbr *vcpu_to_lbr_records (struct kvm_vcpu *vcpu); + +bool intel_pmu_lbr_is_compatible (struct kvm_vcpu *vcpu); +bool intel_pmu_lbr_is_enabled (struct kvm_vcpu *vcpu); +int intel_pmu_create_guest_lbr_event (struct kvm_vcpu *vcpu); + +struct lbr_desc { +/* Basic info about guest LBR records. */ +struct x86_pmu_lbr records; + +/* +* Emulate LBR feature via passthrough LBR registers when the +* per-vcpu guest LBR event is scheduled on the current pcpu. +* +* The records may be inaccurate if the host reclaims the LBR. +*/ +struct perf_event *event; + +/* True if LBRs are marked as not intercepted in the MSR bitmap */ +bool msr_passthrough; +}; + +#endif /* __KVM_X86_VMX_PMU_INTEL_H */
arch/x86/kvm/vmx/tdx.h +7 -1
@@ -4,6 +4,7 @@#ifdef CONFIG_INTEL_TDX_HOST +#include "pmu_intel.h" #include "tdx_ops.h" struct kvm_tdx {
@@ -21,7 +22,12 @@ struct kvm_tdx {struct vcpu_tdx {struct kvm_vcpu vcpu; -/* TDX specific members follow. */ + +/* +* Dummy to make pmu_intel not corrupt memory. +* TODO: Support PMU for TDX. Future work. +*/ +struct lbr_desc lbr_desc; };static inline bool is_td (struct kvm *kvm)
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -2434,7 +2434,7 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if ((data &PMU_CAP_LBR_FMT ) != (kvm_caps.supported_perf_cap &PMU_CAP_LBR_FMT ))return 1 ; -if (!cpuid_model_is_consistent (vcpu)) +if (!intel_pmu_lbr_is_compatible (vcpu))return 1 ; }if (data &PERF_CAP_PEBS_FORMAT ) {
arch/x86/kvm/vmx/vmx.h +1 -31
@@ -11,6 +11,7 @@#include "capabilities.h" #include "../kvm_cache_regs.h" #include "posted_intr.h" +#include "pmu_intel.h" #include "vmcs.h" #include "vmx_ops.h" #include "../cpuid.h"
@@ -105,22 +106,6 @@ static inline bool intel_pmu_has_perf_global_ctrl(struct kvm_pmu *pmu)return pmu->version >1 ; } -struct lbr_desc { -/* Basic info about guest LBR records. */ -struct x86_pmu_lbr records; - -/* -* Emulate LBR feature via passthrough LBR registers when the -* per-vcpu guest LBR event is scheduled on the current pcpu. -* -* The records may be inaccurate if the host reclaims the LBR. -*/ -struct perf_event *event; - -/* True if LBRs are marked as not intercepted in the MSR bitmap */ -bool msr_passthrough; -}; -/* * The nested_vmx structure is part of vcpu_vmx, and holds information we need * for correct emulation of VMX (i.e., nested VMX) on this vcpu.
@@ -650,21 +635,6 @@ static inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu) return container_of(vcpu, struct vcpu_vmx, vcpu); } -static inline struct lbr_desc *vcpu_to_lbr_desc (struct kvm_vcpu *vcpu) -{ -return &to_vmx (vcpu)->lbr_desc; -} - -static inline struct x86_pmu_lbr *vcpu_to_lbr_records (struct kvm_vcpu *vcpu) -{ -return &vcpu_to_lbr_desc (vcpu)->records; -} - -static inline bool intel_pmu_lbr_is_enabled (struct kvm_vcpu *vcpu) -{ -return !!vcpu_to_lbr_records (vcpu)->nr; -} -void intel_pmu_cross_mapped_check (struct kvm_pmu *pmu);int intel_pmu_create_guest_lbr_event (struct kvm_vcpu *vcpu);void vmx_passthrough_lbr_msrs (struct kvm_vcpu *vcpu);
Signature · 1 line
2.25.1
PATCH v12 16/106 KVM: TDX: Refuse to unplug the last cpu on the package
In reply to [email protected]From: Isaku Yamahata <[email protected]>
In order to reclaim TDX HKID, (i.e. when deleting guest TD), needs to call TDH.PHYMEM.PAGE.WBINVD on all packages. If we have active TDX HKID, refuse to offline the last online cpu to guarantee at least one CPU online per package. Add arch callback for cpu offline.
Because TDX doesn't support suspend by the TDX 1.0 spec, this also refuses suspend if TDs are running. If no TD is running, suspend is allowed.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx/main.c | 1 + arch/x86/kvm/vmx/tdx.c | 43 +++++++++++++++++++++++++++++- arch/x86/kvm/vmx/x86_ops.h | 2 ++ arch/x86/kvm/x86.c | 5 ++++ include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 12 +++++++-- 8 files changed, 63 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -18,6 +18,7 @@ KVM_X86_OP(check_processor_compatibility) KVM_X86_OP(hardware_enable) KVM_X86_OP(hardware_disable) KVM_X86_OP(hardware_unsetup) +KVM_X86_OP_OPTIONAL_RET0(offline_cpu) KVM_X86_OP(has_emulated_msr) KVM_X86_OP(vcpu_after_set_cpuid) KVM_X86_OP(is_vm_type_supported)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1543,6 +1543,7 @@ struct kvm_x86_ops {int (*hardware_enable)(void );void (*hardware_disable)(void );void (*hardware_unsetup)(void ); +int (*offline_cpu)(void );bool (*has_emulated_msr)(struct kvm *kvm,u32 index);void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);
arch/x86/kvm/vmx/main.c +1 -0
@@ -108,6 +108,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .check_processor_compatibility = vmx_check_processor_compat, .hardware_unsetup = vt_hardware_unsetup, + .offline_cpu = tdx_offline_cpu, .hardware_enable = vmx_hardware_enable, .hardware_disable = vmx_hardware_disable,
arch/x86/kvm/vmx/tdx.c +42 -1
64 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -142,6 +142,7 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);void tdx_hardware_unsetup (void );bool tdx_is_vm_type_supported (unsigned long type);int tdx_dev_ioctl (void __user *argp); +int tdx_offline_cpu (void );int tdx_vm_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap);int tdx_vm_init (struct kvm *kvm);
@@ -153,6 +154,7 @@ static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return -ENOSstatic inline void tdx_hardware_unsetup (void ) {}static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; }static inline int tdx_dev_ioctl (void __user *argp) {return -EOPNOTSUPP ; }; +static inline int tdx_offline_cpu (void ) {return 0 ; }static inline int tdx_vm_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap) {return -EINVAL ; };static inline int tdx_vm_init (struct kvm *kvm) {return -EOPNOTSUPP ; }
arch/x86/kvm/x86.c +5 -0
@@ -12119,6 +12119,11 @@ void kvm_arch_hardware_disable(void)drop_user_return_notifiers (); } +int kvm_arch_offline_cpu (unsigned int cpu) +{ +return static_call (kvm_x86_offline_cpu)(); +} +bool kvm_vcpu_is_reset_bsp (struct kvm_vcpu *vcpu) {return vcpu->kvm->arch.bsp_vcpu_id == vcpu->vcpu_id;
include/linux/kvm_host.h +1 -0
@@ -1478,6 +1478,7 @@ static inline void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu) {}int kvm_arch_hardware_enable (void );void kvm_arch_hardware_disable (void );#endif +int kvm_arch_offline_cpu (unsigned int cpu);int kvm_arch_vcpu_runnable (struct kvm_vcpu *vcpu);bool kvm_arch_vcpu_in_kernel (struct kvm_vcpu *vcpu);int kvm_arch_vcpu_should_kick (struct kvm_vcpu *vcpu);
virt/kvm/kvm_main.c +10 -2
@@ -5509,13 +5509,21 @@ static void hardware_disable_nolock(void *junk)__this_cpu_write (hardware_enabled, false); } +__weak int kvm_arch_offline_cpu (unsigned int cpu) +{ +return 0 ; +} +static int kvm_offline_cpu (unsigned int cpu) { +int r =0 ; +mutex_lock (&kvm_lock); -if (kvm_usage_count) + r =kvm_arch_offline_cpu (cpu); +if (!r && kvm_usage_count)hardware_disable_nolock (NULL );mutex_unlock (&kvm_lock); -return 0 ; +return r; }static void hardware_disable_all_nolock (void )
Signature · 1 line
2.25.1
PATCH v12 17/106 The start of TDX KVM patch series: TD vcpu creation/destruction
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD vcpu creation/destruction.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +3 -3
@@ -9,15 +9,15 @@ Layer status What qemu can do ---------------- - TDX VM TYPE is exposed to Qemu. -- Qemu can try to create VM of TDX VM type and then fails. +- Qemu can create/destroy guest of TDX vm type. Patch Layer status ------------------ Patch layer Status * TDX, VMX coexistence: Applied * TDX architectural definitions: Applied -* TD VM creation/destruction: Applying -* TD vcpu creation/destruction: Not yet +* TD VM creation/destruction: Applied +* TD vcpu creation/destruction: Applying * TDX EPT violation: Not yet * TD finalization: Not yet * TD vcpu enter/exit: Not yet
Signature · 1 line
2.25.1
PATCH v12 18/106 KVM: TDX: allocate/free TDX vcpu structure
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The next step of TDX guest creation is to create vcpu. Allocate TDX vcpu structures, initialize it that doesn't require TDX SEAMCALL. Allocate pages of TDX vcpu for the TDX module. Actual donation TDX vcpu pages to the TDX module is not done yet.
In the case of the conventional case, cpuid is empty at the initialization. and cpuid is configured after the vcpu initialization. Because TDX supports only X2APIC mode, cpuid is forcibly initialized to support X2APIC on the vcpu initialization in vcpu_reset method. because kvm_arch_vcpu_create() also initializes kvm MMU that depends on local apic settings. So x2apic needs to be initialized to X2APIC mode by vcpu_reset method.
---
Changes v11 -> v12:
- add more comments in tdx_vcpu_reset().
- use KVM_BUG_ON()
Changes v10 -> v11:
- NULL check of kvmalloc_array() in tdx_vcpu_reset. Move it to tdx_vcpu_create()
--- arch/x86/kvm/vmx/main.c | 44 ++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 82 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 10 +++++ arch/x86/kvm/x86.c | 2 + 4 files changed, 134 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/main.c +40 -4
@@ -83,6 +83,42 @@ static void vt_vm_free(struct kvm *kvm)tdx_vm_free (kvm); } +static int vt_vcpu_precreate (struct kvm *kvm) +{ +if (is_td (kvm)) +return 0 ; + +return vmx_vcpu_precreate (kvm); +} + +static int vt_vcpu_create (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_vcpu_create (vcpu); + +return vmx_vcpu_create (vcpu); +} + +static void vt_vcpu_free (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +tdx_vcpu_free (vcpu); +return ; + } + +vmx_vcpu_free (vcpu); +} + +static void vt_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) +{ +if (is_td_vcpu (vcpu)) { +tdx_vcpu_reset (vcpu, init_event); +return ; + } + +vmx_vcpu_reset (vcpu, init_event); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -123,10 +159,10 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vm_destroy = vt_vm_destroy, .vm_free = vt_vm_free, - .vcpu_precreate = vmx_vcpu_precreate, - .vcpu_create = vmx_vcpu_create, - .vcpu_free = vmx_vcpu_free, - .vcpu_reset = vmx_vcpu_reset, + .vcpu_precreate = vt_vcpu_precreate, + .vcpu_create = vt_vcpu_create, + .vcpu_free = vt_vcpu_free, + .vcpu_reset = vt_vcpu_reset, .prepare_switch_to_guest = vmx_prepare_switch_to_guest, .vcpu_load = vmx_vcpu_load,
arch/x86/kvm/vmx/tdx.c +82 -0
88 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +10 -0
@@ -148,7 +148,12 @@ int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap);int tdx_vm_init (struct kvm *kvm);void tdx_mmu_release_hkid (struct kvm *kvm);void tdx_vm_free (struct kvm *kvm); +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp); + +int tdx_vcpu_create (struct kvm_vcpu *vcpu); +void tdx_vcpu_free (struct kvm_vcpu *vcpu); +void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -ENOSYS ; }static inline void tdx_hardware_unsetup (void ) {}
@@ -161,7 +166,12 @@ static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; }static inline void tdx_mmu_release_hkid (struct kvm *kvm) {}static inline void tdx_flush_shadow_all_private (struct kvm *kvm) {}static inline void tdx_vm_free (struct kvm *kvm) {} +static inline int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {return -EOPNOTSUPP ; } + +static inline int tdx_vcpu_create (struct kvm_vcpu *vcpu) {return -EOPNOTSUPP ; } +static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {}#endif #endif /* __KVM_X86_VMX_X86_OPS_H */
arch/x86/kvm/x86.c +2 -0
@@ -492,6 +492,7 @@ int kvm_set_apic_base(struct kvm_vcpu *vcpu, struct msr_data *msr_info)kvm_recalculate_apic_map (vcpu->kvm);return 0 ; } +EXPORT_SYMBOL_GPL (kvm_set_apic_base);/* * Handle a fault on a hardware virtualization (VMX or SVM) instruction.
@@ -12128,6 +12129,7 @@ bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu) {return vcpu->kvm->arch.bsp_vcpu_id == vcpu->vcpu_id; } +EXPORT_SYMBOL_GPL (kvm_vcpu_is_reset_bsp);bool kvm_vcpu_is_bsp (struct kvm_vcpu *vcpu) {
Signature · 1 line
2.25.1
PATCH v12 19/106 KVM: TDX: Do TDX specific vcpu initialization
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TD guest vcpu needs TDX specific initialization before running. Repurpose KVM_MEMORY_ENCRYPT_OP to vcpu-scope, add a new sub-command KVM_TDX_INIT_VCPU, and implement the callback for it.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/vmx/main.c | 9 ++ arch/x86/kvm/vmx/tdx.c | 154 +++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 7 ++ arch/x86/kvm/vmx/x86_ops.h | 4 + arch/x86/kvm/x86.c | 6 + tools/arch/x86/include/uapi/asm/kvm.h | 1 + 9 files changed, 183 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -124,6 +124,7 @@ KVM_X86_OP(enable_smi_window)#endif KVM_X86_OP(dev_mem_enc_ioctl) KVM_X86_OP_OPTIONAL(mem_enc_ioctl) +KVM_X86_OP_OPTIONAL(vcpu_mem_enc_ioctl) KVM_X86_OP_OPTIONAL(mem_enc_register_region) KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1709,6 +1709,7 @@ struct kvm_x86_ops {int (*dev_mem_enc_ioctl)(void __user *argp);int (*mem_enc_ioctl)(struct kvm *kvm,void __user *argp); +int (*vcpu_mem_enc_ioctl)(struct kvm_vcpu *vcpu,void __user *argp);int (*mem_enc_register_region)(struct kvm *kvm,struct kvm_enc_region *argp);int (*mem_enc_unregister_region)(struct kvm *kvm,struct kvm_enc_region *argp);int (*vm_copy_enc_context_from)(struct kvm *kvm,unsigned int source_fd);
arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -536,6 +536,7 @@ struct kvm_pmu_event_filter {enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 ,KVM_TDX_INIT_VM , +KVM_TDX_INIT_VCPU ,KVM_TDX_CMD_NR_MAX , };
arch/x86/kvm/vmx/main.c +9 -0
@@ -127,6 +127,14 @@ static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp)return tdx_vm_ioctl (kvm, argp); } +static int vt_vcpu_mem_enc_ioctl (struct kvm_vcpu *vcpu,void __user *argp) +{ +if (!is_td_vcpu (vcpu)) +return -EINVAL ; + +return tdx_vcpu_ioctl (vcpu, argp); +} +#define VMX_REQUIRED_APICV_INHIBITS \ ( \ BIT(APICV_INHIBIT_REASON_DISABLE)| \
@@ -286,6 +294,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .dev_mem_enc_ioctl = tdx_dev_ioctl, .mem_enc_ioctl = vt_mem_enc_ioctl, + .vcpu_mem_enc_ioctl = vt_vcpu_mem_enc_ioctl, };struct kvm_x86_init_ops vt_init_ops__initdata = {
arch/x86/kvm/vmx/tdx.c +153 -1
196 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +7 -0
@@ -17,12 +17,19 @@ struct kvm_tdx {u64 xfam;int hkid; +bool finalized; +u64 tsc_offset; };struct vcpu_tdx {struct kvm_vcpu vcpu; +unsigned long tdvpr_pa; +unsigned long *tdvpx_pa; + +bool initialized; +/* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work.
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -154,6 +154,8 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);int tdx_vcpu_create (struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event); + +int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -ENOSYS ; }static inline void tdx_hardware_unsetup (void ) {}
@@ -172,6 +174,8 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPstatic inline int tdx_vcpu_create (struct kvm_vcpu *vcpu) {return -EOPNOTSUPP ; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {} + +static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }#endif #endif /* __KVM_X86_VMX_X86_OPS_H */
arch/x86/kvm/x86.c +6 -0
@@ -5993,6 +5993,12 @@ long kvm_arch_vcpu_ioctl(struct file *filp,case KVM_SET_DEVICE_ATTR : r =kvm_vcpu_ioctl_device_attr (vcpu, ioctl, argp);break ; +case KVM_MEMORY_ENCRYPT_OP : + r = -ENOTTY ; +if (!kvm_x86_ops.vcpu_mem_enc_ioctl) + goto out; + r = kvm_x86_ops.vcpu_mem_enc_ioctl (vcpu, argp); +break ;default : r = -EINVAL ; }
tools/arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -536,6 +536,7 @@ struct kvm_pmu_event_filter {enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 ,KVM_TDX_INIT_VM , +KVM_TDX_INIT_VCPU ,KVM_TDX_CMD_NR_MAX , };
Signature · 1 line
2.25.1
PATCH v12 20/106 The start of TDX KVM patch series: KVM MMU GPA shared bits
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of KVM MMU GPA shared bits.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +3 -2
@@ -10,6 +10,7 @@ What qemu can do ---------------- - TDX VM TYPE is exposed to Qemu. - Qemu can create/destroy guest of TDX vm type. +- Qemu can create/destroy vcpu of TDX vm type. Patch Layer status ------------------
@@ -17,12 +18,12 @@ Patch Layer status * TDX, VMX coexistence: Applied * TDX architectural definitions: Applied * TD VM creation/destruction: Applied -* TD vcpu creation/destruction: Applying +* TD vcpu creation/destruction: Applied * TDX EPT violation: Not yet * TD finalization: Not yet * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet -* KVM MMU GPA shared bits: Not yet +* KVM MMU GPA shared bits: Applying * KVM TDP refactoring for TDX: Not yet * KVM TDP MMU hooks: Not yet
Signature · 1 line
2.25.1
PATCH v12 21/106 KVM: x86/mmu: introduce config for PRIVATE KVM MMU
In reply to [email protected]From: Isaku Yamahata <[email protected]>
To keep the case of non TDX intact, introduce a new config option for private KVM MMU support. At the moment, this is synonym for CONFIG_INTEL_TDX_HOST && CONFIG_KVM_INTEL. The config makes it clear that the config is only for x86 KVM MMU.
--- arch/x86/kvm/Kconfig | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/kvm/Kconfig +4 -0
@@ -153,4 +153,8 @@ config KVM_XEN config KVM_EXTERNAL_WRITE_TRACKING bool +config KVM_MMU_PRIVATE + def_bool y + depends on INTEL_TDX_HOST && KVM_INTEL + endif # VIRTUALIZATION
Signature · 1 line
2.25.1
PATCH v12 22/106 KVM: x86/mmu: Add address conversion functions for TDX shared bit of GPA
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX repurposes one GPA bit (51 bit or 47 bit based on configuration) to indicate the GPA is private(if cleared) or shared (if set) with VMM. If GPA.shared is set, GPA is covered by the existing conventional EPT pointed by EPTP. If GPA.shared bit is cleared, GPA is covered by TDX module. VMM has to issue SEAMCALLs to operate.
Add a member to remember GPA shared bit for each guest TDs, add address conversion functions between private GPA and shared GPA and test if GPA is private.
Because struct kvm_arch (or struct kvm which includes struct kvm_arch. See kvm_arch_alloc_vm() that passes __GPF_ZERO) is zero-cleared when allocated, the new member to remember GPA shared bit is guaranteed to be zero with this patch unless it's initialized explicitly.
--- arch/x86/include/asm/kvm_host.h | 4 ++++ arch/x86/kvm/mmu.h | 32 ++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.c | 5 +++++ 3 files changed, 41 insertions(+)
arch/x86/include/asm/kvm_host.h +4 -0
@@ -1448,6 +1448,10 @@ struct kvm_arch {*/ #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)struct kvm_mmu_memory_cache split_desc_cache; + +#ifdef CONFIG_KVM_MMU_PRIVATE +gfn_t gfn_shared_mask; +#endif };struct kvm_vm_stat {
arch/x86/kvm/mmu.h +32 -0
@@ -278,4 +278,36 @@ static inline gpa_t kvm_translate_gpa(struct kvm_vcpu *vcpu,return gpa;return translate_nested_gpa (vcpu, gpa, access, exception); } + +static inline gfn_t kvm_gfn_shared_mask (const struct kvm *kvm) +{ +#ifdef CONFIG_KVM_MMU_PRIVATE +return kvm->arch.gfn_shared_mask; +#else +return 0 ; +#endif +} + +static inline gfn_t kvm_gfn_shared (const struct kvm *kvm,gfn_t gfn) +{ +return gfn |kvm_gfn_shared_mask (kvm); +} + +static inline gfn_t kvm_gfn_private (const struct kvm *kvm,gfn_t gfn) +{ +return gfn & ~kvm_gfn_shared_mask (kvm); +} + +static inline gpa_t kvm_gpa_private (const struct kvm *kvm,gpa_t gpa) +{ +return gpa & ~gfn_to_gpa (kvm_gfn_shared_mask (kvm)); +} + +static inline bool kvm_is_private_gpa (const struct kvm *kvm,gpa_t gpa) +{ +gfn_t mask =kvm_gfn_shared_mask (kvm); + +return mask && !(gpa_to_gfn (gpa) & mask); +} +#endif
arch/x86/kvm/vmx/tdx.c +5 -0
@@ -851,6 +851,11 @@ static int tdx_td_init(struct kvm *kvm, struct kvm_tdx_cmd *cmd) kvm_tdx->attributes = td_params->attributes; kvm_tdx->xfam = td_params->xfam; +if (td_params->exec_controls &TDX_EXEC_CONTROL_MAX_GPAW ) + kvm->arch.gfn_shared_mask =gpa_to_gfn (BIT_ULL (51 )); +else + kvm->arch.gfn_shared_mask =gpa_to_gfn (BIT_ULL (47 )); + out:/* kfree() accepts NULL. */ kfree (init_vm);
Signature · 1 line
2.25.1
PATCH v12 23/106 The start of TDX KVM patch series: KVM TDP refactoring for TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of KVM TDP refactoring for TDX.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2
@@ -24,6 +24,6 @@ Patch Layer status * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet -* KVM MMU GPA shared bits: Applying -* KVM TDP refactoring for TDX: Not yet +* KVM MMU GPA shared bits: Applied +* KVM TDP refactoring for TDX: Applying * KVM TDP MMU hooks: Not yet
Signature · 1 line
2.25.1
PATCH v12 24/106 KVM: Allow page-sized MMU caches to be initialized with custom 64-bit values
In reply to [email protected]From: Sean Christopherson <[email protected]>
Add support to MMU caches for initializing a page with a custom 64-bit value, e.g. to pre-fill an entire page table with non-zero PTE values. The functionality will be used by x86 to support Intel's TDX, which needs to set bit 63 in all non-present PTEs in order to prevent !PRESENT page faults from getting reflected into the guest (Intel's EPT Violation #VE architecture made the less than brilliant decision of having the per-PTE behavior be opt-out instead of opt-in).
--- include/linux/kvm_types.h | 1 + virt/kvm/kvm_main.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-)
include/linux/kvm_types.h +1 -0
@@ -94,6 +94,7 @@ struct kvm_mmu_memory_cache {int nobjs;gfp_t gfp_zero;gfp_t gfp_custom; +u64 init_value;struct kmem_cache *kmem_cache;int capacity;void **objects;
virt/kvm/kvm_main.c +14 -2
@@ -381,12 +381,17 @@ static void kvm_flush_shadow_all(struct kvm *kvm)static inline void *mmu_memory_cache_alloc_obj (struct kvm_mmu_memory_cache *mc,gfp_t gfp_flags) { +void *page; + gfp_flags |= mc->gfp_zero;if (mc->kmem_cache)return kmem_cache_alloc (mc->kmem_cache, gfp_flags); -else -return (void *)__get_free_page (gfp_flags); + + page = (void *)__get_free_page (gfp_flags); +if (page && mc->init_value) +memset64 (page, mc->init_value,PAGE_SIZE /sizeof (mc->init_value)); +return page; }int __kvm_mmu_topup_memory_cache (struct kvm_mmu_memory_cache *mc,int capacity,int min)
@@ -401,6 +406,13 @@ int __kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int capacity,if (WARN_ON_ONCE (!capacity))return -EIO ; +/* +* Custom init values can be used only for page allocations, +* and obviously conflict with __GFP_ZERO. +*/ +if (WARN_ON_ONCE (mc->init_value && (mc->kmem_cache || mc->gfp_zero))) +return -EIO ; + mc->objects =kvmalloc_array (sizeof (void *), capacity, gfp);if (!mc->objects)return -ENOMEM ;
Signature · 1 line
2.25.1
PATCH v12 25/106 KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX support will need the "suppress #VE" bit (bit 63) set as the initial value for SPTE. To reduce code change size, introduce a new macro SHADOW_NONPRESENT_VALUE for the initial value for the shadow page table entry (SPTE) and replace hard-coded value 0 for it. Initialize shadow page tables with their value.
The plan is to unconditionally set the "suppress #VE" bit for both AMD and Intel as: 1) AMD hardware uses the bit 63 as NX for present SPTE and ignored for non-present SPTE; 2) for conventional VMX guests, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware.
--- arch/x86/kvm/mmu/mmu.c | 17 +++++++++++++---- arch/x86/kvm/mmu/paging_tmpl.h | 3 ++- arch/x86/kvm/mmu/spte.h | 2 ++ arch/x86/kvm/mmu/tdp_mmu.c | 15 ++++++++------- 4 files changed, 25 insertions(+), 12 deletions(-)
arch/x86/kvm/mmu/mmu.c +13 -4
@@ -547,9 +547,9 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep)if (!is_shadow_present_pte (old_spte) || !spte_has_volatile_bits (old_spte)) -__update_clear_spte_fast (sptep,0ull ); +__update_clear_spte_fast (sptep,SHADOW_NONPRESENT_VALUE );else - old_spte =__update_clear_spte_slow (sptep,0ull ); + old_spte =__update_clear_spte_slow (sptep,SHADOW_NONPRESENT_VALUE );if (!is_shadow_present_pte (old_spte))return old_spte;
@@ -583,7 +583,7 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep)*/ static void mmu_spte_clear_no_track (u64 *sptep) { -__update_clear_spte_fast (sptep,0ull ); +__update_clear_spte_fast (sptep,SHADOW_NONPRESENT_VALUE ); }static u64 mmu_spte_get_lockless (u64 *sptep)
@@ -6001,7 +6001,16 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu) vcpu->arch.mmu_page_header_cache.kmem_cache = mmu_page_header_cache; vcpu->arch.mmu_page_header_cache.gfp_zero = __GFP_ZERO; - vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO; +/* +* When X86_64, initial SEPT entries are initialized with +* SHADOW_NONPRESENT_VALUE. Otherwise zeroed. See +* mmu_memory_cache_alloc_obj(). +*/ +if (IS_ENABLED (CONFIG_X86_64 )) + vcpu->arch.mmu_shadow_page_cache.init_value = +SHADOW_NONPRESENT_VALUE ; +if (!vcpu->arch.mmu_shadow_page_cache.init_value) + vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO; vcpu->arch.mmu = &vcpu->arch.root_mmu; vcpu->arch.walk_mmu = &vcpu->arch.root_mmu;
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
@@ -1028,7 +1028,8 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)gpa_t pte_gpa;gfn_t gfn; -if (!sp->spt[i]) +/* spt[i] has initial value of shadow page table allocation */ +if (sp->spt[i] ==SHADOW_NONPRESENT_VALUE )continue ; pte_gpa = first_pte_gpa + i *sizeof (pt_element_t);
arch/x86/kvm/mmu/spte.h +2 -0
@@ -148,6 +148,8 @@ static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11);#define MMIO_SPTE_GEN_MASK GENMASK_ULL(MMIO_SPTE_GEN_LOW_BITS + MMIO_SPTE_GEN_HIGH_BITS - 1, 0) +#define SHADOW_NONPRESENT_VALUE 0ULL +extern u64 __read_mostly shadow_host_writable_mask;extern u64 __read_mostly shadow_mmu_writable_mask;extern u64 __read_mostly shadow_nx_mask;
arch/x86/kvm/mmu/tdp_mmu.c +8 -7
@@ -690,7 +690,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm,* here since the SPTE is going from non-present to non-present. Use * the raw write helper to avoid an unnecessary check on volatile bits. */ -__kvm_tdp_mmu_write_spte (iter->sptep,0 ); +__kvm_tdp_mmu_write_spte (iter->sptep,SHADOW_NONPRESENT_VALUE );return 0 ; }
@@ -867,8 +867,8 @@ static void __tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root,continue ;if (!shared) -tdp_mmu_set_spte (kvm, &iter,0 ); -else if (tdp_mmu_set_spte_atomic (kvm, &iter,0 )) +tdp_mmu_set_spte (kvm, &iter,SHADOW_NONPRESENT_VALUE ); +else if (tdp_mmu_set_spte_atomic (kvm, &iter,SHADOW_NONPRESENT_VALUE )) goto retry; } }
@@ -924,8 +924,9 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)if (WARN_ON_ONCE (!is_shadow_present_pte (old_spte)))return false; -__tdp_mmu_set_spte (kvm,kvm_mmu_page_as_id (sp), sp->ptep, old_spte,0 , - sp->gfn, sp->role.level +1 , true, true); +__tdp_mmu_set_spte (kvm,kvm_mmu_page_as_id (sp), sp->ptep, old_spte, +SHADOW_NONPRESENT_VALUE , sp->gfn, sp->role.level +1 , + true, true);return true; }
@@ -959,7 +960,7 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root, !is_last_spte(iter.old_spte, iter.level))continue ; -tdp_mmu_set_spte (kvm, &iter,0 ); +tdp_mmu_set_spte (kvm, &iter,SHADOW_NONPRESENT_VALUE ); flush = true; }
@@ -1328,7 +1329,7 @@ static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter,* invariant that the PFN of a present * leaf SPTE can never change. * See __handle_changed_spte(). */ -tdp_mmu_set_spte (kvm, iter,0 ); +tdp_mmu_set_spte (kvm, iter,SHADOW_NONPRESENT_VALUE );if (!pte_write (range->pte)) { new_spte = kvm_mmu_changed_pte_notifier_make_spte(iter->old_spte,
Signature · 1 line
2.25.1
PATCH v12 26/106 KVM: x86/mmu: Allow non-zero value for non-present SPTE and removed SPTE
In reply to [email protected]From: Sean Christopherson <[email protected]>
For TD guest, the current way to emulate MMIO doesn't work any more, as KVM is not able to access the private memory of TD guest and do the emulation. Instead, TD guest expects to receive #VE when it accesses the MMIO and then it can explicitly make hypercall to KVM to get the expected information.
To achieve this, the TDX module always enables "EPT-violation #VE" in the VMCS control. And accordingly, for the MMIO spte for the shared GPA,
1. KVM needs to set "suppress #VE" bit for the non-present SPTE so that EPT violation happens on TD accessing MMIO range. 2. On EPT violation, KVM sets the MMIO spte to clear "suppress #VE" bit so the TD guest can receive the #VE instead of EPT misconfigration unlike VMX case. For the shared GPA that is not populated yet, EPT violation need to be triggered when TD guest accesses such shared GPA. The non-present SPTE value for shared GPA should set "suppress #VE" bit.
Add "suppress #VE" bit (bit 63) to SHADOW_NONPRESENT_VALUE and REMOVED_SPTE. Unconditionally set the "suppress #VE" bit (which is bit 63) for both AMD and Intel as: 1) AMD hardware doesn't use this bit when present bit is off; 2) for normal VMX guest, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware.
--- arch/x86/kvm/mmu/spte.h | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/spte.h +14 -1
@@ -148,7 +148,20 @@ static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11);#define MMIO_SPTE_GEN_MASK GENMASK_ULL(MMIO_SPTE_GEN_LOW_BITS + MMIO_SPTE_GEN_HIGH_BITS - 1, 0) +/* +* Non-present SPTE value for both VMX and SVM for TDP MMU. +* For SVM NPT, for non-present spte (bit 0 = 0), other bits are ignored. +* For VMX EPT, bit 63 is ignored if #VE is disabled. (EPT_VIOLATION_VE=0) +* bit 63 is #VE suppress if #VE is enabled. (EPT_VIOLATION_VE=1) +* For TDX: +* TDX module sets EPT_VIOLATION_VE for Secure-EPT and conventional EPT +*/ +#ifdef CONFIG_X86_64 +#define SHADOW_NONPRESENT_VALUE BIT_ULL(63) +static_assert (!(SHADOW_NONPRESENT_VALUE &SPTE_MMU_PRESENT_MASK )); +#else #define SHADOW_NONPRESENT_VALUE 0ULL +#endif extern u64 __read_mostly shadow_host_writable_mask;extern u64 __read_mostly shadow_mmu_writable_mask;
@@ -195,7 +208,7 @@ extern u64 __read_mostly shadow_nonpresent_or_rsvd_mask;* * Only used by the TDP MMU. */ -#define REMOVED_SPTE 0x5a0ULL +#define REMOVED_SPTE (SHADOW_NONPRESENT_VALUE | 0x5a0ULL)/* Removed SPTEs must not be misconstrued as shadow present PTEs. */ static_assert (!(REMOVED_SPTE &SPTE_MMU_PRESENT_MASK ));
Signature · 1 line
2.25.1
PATCH v12 27/106 KVM: x86/mmu: Add Suppress VE bit to shadow_mmio_mask/shadow_present_mask
In reply to [email protected]From: Isaku Yamahata <[email protected]>
To make use of the same value of shadow_mmio_mask and shadow_present_mask for TDX and VMX, add Suppress-VE bit to shadow_mmio_mask and shadow_present_mask so that they can be common for both VMX and TDX.
TDX will require shadow_mmio_mask and shadow_present_mask to include VMX_SUPPRESS_VE for shared GPA so that EPT violation is triggered for shared GPA. For VMX, VMX_SUPPRESS_VE doesn't matter for MMIO because the spte value is required to cause EPT misconfig. the additional bit doesn't affect VMX logic to add the bit to shadow_mmio_{value, mask}.
--- arch/x86/include/asm/vmx.h | 1 + arch/x86/kvm/mmu/spte.c | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-)
arch/x86/include/asm/vmx.h +1 -0
@@ -511,6 +511,7 @@ enum vmcs_field {#define VMX_EPT_IPAT_BIT (1ull << 6)#define VMX_EPT_ACCESS_BIT (1ull << 8)#define VMX_EPT_DIRTY_BIT (1ull << 9) +#define VMX_EPT_SUPPRESS_VE_BIT (1ull << 63)#define VMX_EPT_RWX_MASK (VMX_EPT_READABLE_MASK | \ VMX_EPT_WRITABLE_MASK | \ VMX_EPT_EXECUTABLE_MASK)
arch/x86/kvm/mmu/spte.c +4 -2
@@ -431,7 +431,9 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) shadow_dirty_mask = has_ad_bits ?VMX_EPT_DIRTY_BIT :0ull ; shadow_nx_mask =0ull ; shadow_x_mask =VMX_EPT_EXECUTABLE_MASK ; - shadow_present_mask = has_exec_only ?0ull :VMX_EPT_READABLE_MASK ; +/* VMX_EPT_SUPPRESS_VE_BIT is needed for W or X violation. */ + shadow_present_mask = + (has_exec_only ?0ull :VMX_EPT_READABLE_MASK ) |VMX_EPT_SUPPRESS_VE_BIT ;/* * EPT overrides the host MTRRs, and so KVM must program the desired * memtype directly into the SPTEs. Note, this mask is just the mask
@@ -448,7 +450,7 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only)* of an EPT paging-structure entry is 110b (write/execute). */ kvm_mmu_set_mmio_spte_mask (VMX_EPT_MISCONFIG_WX_VALUE , -VMX_EPT_RWX_MASK ,0 ); +VMX_EPT_RWX_MASK |VMX_EPT_SUPPRESS_VE_BIT ,0 ); }EXPORT_SYMBOL_GPL (kvm_mmu_set_ept_masks);
Signature · 1 line
2.25.1
PATCH v12 28/106 KVM: x86/mmu: Track shadow MMIO value on a per-VM basis
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX will use a different shadow PTE entry value for MMIO from VMX. Add members to kvm_arch and track value for MMIO per-VM instead of global variables. By using the per-VM EPT entry value for MMIO, the existing VMX logic is kept working. Introduce a separate setter function so that guest TD can override later.
Also require mmio spte cachcing for TDX. Actually this is true case because TDX require EPT and KVM EPT allows mmio spte caching.
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 7 ++++--- arch/x86/kvm/mmu/spte.c | 10 ++++++++-- arch/x86/kvm/mmu/spte.h | 4 ++-- arch/x86/kvm/mmu/tdp_mmu.c | 6 +++--- 6 files changed, 20 insertions(+), 10 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1260,6 +1260,8 @@ struct kvm_arch {*/ spinlock_t mmu_unsync_pages_lock; +u64 shadow_mmio_value; +struct list_head assigned_dev_head;struct iommu_domain *iommu_domain;bool iommu_noncoherent;
arch/x86/kvm/mmu.h +1 -0
@@ -101,6 +101,7 @@ static inline u8 kvm_get_shadow_phys_bits(void) }void kvm_mmu_set_mmio_spte_mask (u64 mmio_value,u64 mmio_mask,u64 access_mask); +void kvm_mmu_set_mmio_spte_value (struct kvm *kvm,u64 mmio_value);void kvm_mmu_set_me_spte_mask (u64 me_value,u64 me_mask);void kvm_mmu_set_ept_masks (bool has_ad_bits,bool has_exec_only);
arch/x86/kvm/mmu/mmu.c +4 -3
@@ -2445,7 +2445,7 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp,return kvm_mmu_prepare_zap_page (kvm, child, invalid_list); } - }else if (is_mmio_spte (pte)) { + }else if (is_mmio_spte (kvm, pte)) {mmu_spte_clear_no_track (spte); }return 0 ;
@@ -4126,7 +4126,7 @@ static int handle_mmio_page_fault(struct kvm_vcpu *vcpu, u64 addr, bool direct)if (WARN_ON (reserved))return -EINVAL ; -if (is_mmio_spte (spte)) { +if (is_mmio_spte (vcpu->kvm, spte)) {gfn_t gfn =get_mmio_spte_gfn (spte);unsigned int access =get_mmio_spte_access (spte);
@@ -4674,7 +4674,7 @@ static unsigned long get_cr3(struct kvm_vcpu *vcpu)static bool sync_mmio_spte (struct kvm_vcpu *vcpu,u64 *sptep,gfn_t gfn,unsigned int access) { -if (unlikely (is_mmio_spte (*sptep))) { +if (unlikely (is_mmio_spte (vcpu->kvm, *sptep))) {if (gfn !=get_mmio_spte_gfn (*sptep)) {mmu_spte_clear_no_track (sptep);return true;
@@ -6163,6 +6163,7 @@ int kvm_mmu_init_vm(struct kvm *kvm)struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;int r; + kvm->arch.shadow_mmio_value = shadow_mmio_value;INIT_LIST_HEAD (&kvm->arch.active_mmu_pages);INIT_LIST_HEAD (&kvm->arch.zapped_obsolete_pages);INIT_LIST_HEAD (&kvm->arch.possible_nx_huge_pages);
arch/x86/kvm/mmu/spte.c +8 -2
@@ -74,10 +74,10 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)u64 spte =generation_mmio_spte_mask (gen);u64 gpa = gfn <<PAGE_SHIFT ; -WARN_ON_ONCE (!shadow_mmio_value); +WARN_ON_ONCE (!vcpu->kvm->arch.shadow_mmio_value); access &= shadow_mmio_access_mask; - spte |= shadow_mmio_value | access; + spte |= vcpu->kvm->arch.shadow_mmio_value | access; spte |= gpa | shadow_nonpresent_or_rsvd_mask; spte |= (gpa & shadow_nonpresent_or_rsvd_mask) <<SHADOW_NONPRESENT_OR_RSVD_MASK_LEN ;
@@ -413,6 +413,12 @@ void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask) }EXPORT_SYMBOL_GPL (kvm_mmu_set_mmio_spte_mask); +void kvm_mmu_set_mmio_spte_value (struct kvm *kvm,u64 mmio_value) +{ + kvm->arch.shadow_mmio_value = mmio_value; +} +EXPORT_SYMBOL_GPL (kvm_mmu_set_mmio_spte_value); +void kvm_mmu_set_me_spte_mask (u64 me_value,u64 me_mask) {/* shadow_me_value must be a subset of shadow_me_mask */
arch/x86/kvm/mmu/spte.h +2 -2
@@ -251,9 +251,9 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep)return to_shadow_page (__pa (sptep)); } -static inline bool is_mmio_spte (u64 spte) +static inline bool is_mmio_spte (struct kvm *kvm,u64 spte) { -return (spte & shadow_mmio_mask) == shadow_mmio_value && +return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&likely (enable_mmio_caching); }
arch/x86/kvm/mmu/tdp_mmu.c +3 -3
@@ -576,8 +576,8 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,* impact the guest since both the former and current SPTEs * are nonpresent. */ -if (WARN_ON (!is_mmio_spte (old_spte) && - !is_mmio_spte (new_spte) && +if (WARN_ON (!is_mmio_spte (kvm, old_spte) && + !is_mmio_spte (kvm, new_spte) && !is_removed_spte (new_spte)))pr_err ("Unexpected SPTE change! Nonpresent SPTEs\n" "should not be replaced with another,\n"
@@ -1095,7 +1095,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, }/* If a MMIO SPTE is installed, the MMIO will need to be emulated. */ -if (unlikely (is_mmio_spte (new_spte))) { +if (unlikely (is_mmio_spte (vcpu->kvm, new_spte))) { vcpu->stat.pf_mmio_spte_created++;trace_mark_mmio_spte (rcu_dereference (iter->sptep), iter->gfn, new_spte);
Signature · 1 line
2.25.1
PATCH v12 29/106 KVM: x86/mmu: Disallow fast page fault on private GPA
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX requires TDX SEAMCALL to operate Secure EPT instead of direct memory access and TDX SEAMCALL is heavy operation. Fast page fault on private GPA doesn't make sense. Disallow fast page fault on private GPA.
--- arch/x86/kvm/mmu/mmu.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +10 -2
@@ -3280,8 +3280,16 @@ static int kvm_handle_noslot_fault(struct kvm_vcpu *vcpu,return RET_PF_CONTINUE ; } -static bool page_fault_can_be_fast (struct kvm_page_fault *fault) +static bool page_fault_can_be_fast (struct kvm *kvm,struct kvm_page_fault *fault) { +/* +* TDX private mapping doesn't support fast page fault because the EPT +* entry is read/written with TDX SEAMCALLs instead of direct memory +* access. +*/ +if (kvm_is_private_gpa (kvm, fault->addr)) +return false; +/* * Page faults with reserved bits set, i.e. faults on MMIO SPTEs, only * reach the common page fault handler if the SPTE has an invalid MMIO
@@ -3391,7 +3399,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)u64 *sptep =NULL ;uint retry_count =0 ; -if (!page_fault_can_be_fast (fault)) +if (!page_fault_can_be_fast (vcpu->kvm, fault))return ret;walk_shadow_page_lockless_begin (vcpu);
Signature · 1 line
2.25.1
PATCH v12 30/106 KVM: x86/mmu: Allow per-VM override of the TDP max page level
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX requires special handling to support large private page. For simplicity, only support 4K page for TD guest for now. Add per-VM maximum page level support to support different maximum page sizes for TD guest and conventional VMX guest.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/mmu/mmu_internal.h | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1234,6 +1234,7 @@ struct kvm_arch {unsigned long n_requested_mmu_pages;unsigned long n_max_mmu_pages;unsigned int indirect_shadow_pages; +int tdp_max_page_level;u8 mmu_valid_gen;struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES ];struct list_head active_mmu_pages;
arch/x86/kvm/mmu/mmu.c +1 -0
@@ -6195,6 +6195,7 @@ int kvm_mmu_init_vm(struct kvm *kvm) kvm->arch.split_desc_cache.kmem_cache = pte_list_desc_cache; kvm->arch.split_desc_cache.gfp_zero = __GFP_ZERO; + kvm->arch.tdp_max_page_level =KVM_MAX_HUGEPAGE_LEVEL ;return 0 ; }
arch/x86/kvm/mmu/mmu_internal.h +1 -1
@@ -277,7 +277,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, .nx_huge_page_workaround_enabled = is_nx_huge_page_enabled(vcpu->kvm), - .max_level = KVM_MAX_HUGEPAGE_LEVEL, + .max_level = vcpu->kvm->arch.tdp_max_page_level, .req_level = PG_LEVEL_4K, .goal_level = PG_LEVEL_4K, .is_private = kvm_mem_is_private(vcpu->kvm, cr2_or_gpa >> PAGE_SHIFT),
Signature · 1 line
2.25.1
PATCH v12 31/106 KVM: VMX: Introduce test mode related to EPT violation VE
In reply to [email protected]From: Isaku Yamahata <[email protected]>
To support TDX, KVM is enhanced to operate with #VE. For TDX, KVM programs to inject #VE conditionally and set #VE suppress bit in EPT entry. For VMX case, #VE isn't used. If #VE happens for VMX, it's a bug. To be defensive (test that VMX case isn't broken), introduce option ept_violation_ve_test and when it's set, set error.
--- arch/x86/include/asm/vmx.h | 12 +++++++ arch/x86/kvm/vmx/vmcs.h | 5 +++ arch/x86/kvm/vmx/vmx.c | 69 +++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/vmx.h | 6 +++- 4 files changed, 90 insertions(+), 2 deletions(-)
arch/x86/include/asm/vmx.h +12 -0
@@ -68,6 +68,7 @@#define SECONDARY_EXEC_ENCLS_EXITING VMCS_CONTROL_BIT(ENCLS_EXITING)#define SECONDARY_EXEC_RDSEED_EXITING VMCS_CONTROL_BIT(RDSEED_EXITING)#define SECONDARY_EXEC_ENABLE_PML VMCS_CONTROL_BIT(PAGE_MOD_LOGGING) +#define SECONDARY_EXEC_EPT_VIOLATION_VE VMCS_CONTROL_BIT(EPT_VIOLATION_VE)#define SECONDARY_EXEC_PT_CONCEAL_VMX VMCS_CONTROL_BIT(PT_CONCEAL_VMX)#define SECONDARY_EXEC_XSAVES VMCS_CONTROL_BIT(XSAVES)#define SECONDARY_EXEC_MODE_BASED_EPT_EXEC VMCS_CONTROL_BIT(MODE_BASED_EPT_EXEC)
@@ -223,6 +224,8 @@ enum vmcs_field {VMREAD_BITMAP_HIGH =0x00002027 ,VMWRITE_BITMAP =0x00002028 ,VMWRITE_BITMAP_HIGH =0x00002029 , +VE_INFORMATION_ADDRESS =0x0000202A , +VE_INFORMATION_ADDRESS_HIGH =0x0000202B ,XSS_EXIT_BITMAP =0x0000202C ,XSS_EXIT_BITMAP_HIGH =0x0000202D , ENCLS_EXITING_BITMAP = 0x0000202E,
@@ -628,4 +631,13 @@ enum vmx_l1d_flush_state {extern enum vmx_l1d_flush_state l1tf_vmx_mitigation; +struct vmx_ve_information { +u32 exit_reason; +u32 delivery; +u64 exit_qualification; +u64 guest_linear_address; +u64 guest_physical_address; +u16 eptp_index; +}; +#endif
arch/x86/kvm/vmx/vmcs.h +5 -0
@@ -140,6 +140,11 @@ static inline bool is_nm_fault(u32 intr_info)return is_exception_n (intr_info,NM_VECTOR ); } +static inline bool is_ve_fault (u32 intr_info) +{ +return is_exception_n (intr_info,VE_VECTOR ); +} +/* Undocumented: icebp/int1 */ static inline bool is_icebp (u32 intr_info) {
arch/x86/kvm/vmx/vmx.c +68 -1
124 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.h +5 -1
@@ -359,6 +359,9 @@ struct vcpu_vmx {DECLARE_BITMAP (read,MAX_POSSIBLE_PASSTHROUGH_MSRS );DECLARE_BITMAP (write,MAX_POSSIBLE_PASSTHROUGH_MSRS ); } shadow_msr_intercept; + +/* ve_info must be page aligned. */ +struct vmx_ve_information *ve_info; };struct kvm_vmx {
@@ -570,7 +573,8 @@ static inline u8 vmx_get_rvi(void) SECONDARY_EXEC_ENABLE_VMFUNC | \ SECONDARY_EXEC_BUS_LOCK_DETECTION | \ SECONDARY_EXEC_NOTIFY_VM_EXITING | \ - SECONDARY_EXEC_ENCLS_EXITING) + SECONDARY_EXEC_ENCLS_EXITING | \ + SECONDARY_EXEC_EPT_VIOLATION_VE)#define KVM_REQUIRED_VMX_TERTIARY_VM_EXEC_CONTROL 0#define KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL \
Signature · 1 line
2.25.1
PATCH v12 32/106 The start of TDX KVM patch series: KVM TDP MMU hooks
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of KVM TDP MMU hooks.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2
@@ -25,5 +25,5 @@ Patch Layer status * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied -* KVM TDP refactoring for TDX: Applying -* KVM TDP MMU hooks: Not yet +* KVM TDP refactoring for TDX: Applied +* KVM TDP MMU hooks: Applying
Signature · 1 line
2.25.1
PATCH v12 33/106 KVM: x86/tdp_mmu: Init role member of struct kvm_mmu_page at allocation
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Refactor tdp_mmu_alloc_sp() and tdp_mmu_init_sp and eliminate tdp_mmu_init_child_sp(). Currently tdp_mmu_init_sp() (or tdp_mmu_init_child_sp()) sets kvm_mmu_page.role after tdp_mmu_alloc_sp() allocating struct kvm_mmu_page and its page table page. This patch makes tdp_mmu_alloc_sp() initialize kvm_mmu_page.role instead of tdp_mmu_init_sp().
To handle private page tables, argument of is_private needs to be passed down. Given that already page level is passed down, it would be cumbersome to add one more parameter about sp. Instead replace the level argument with union kvm_mmu_page_role. Thus the number of argument won't be increased and more info about sp can be passed down.
For private sp, secure page table will be also allocated in addition to struct kvm_mmu_page and page table (spt member). The allocation functions (tdp_mmu_alloc_sp() and __tdp_mmu_alloc_sp_for_split()) need to know if the allocation is for the conventional page table or private page table. Pass union kvm_mmu_role to those functions and initialize role member of struct kvm_mmu_page.
--- arch/x86/kvm/mmu/tdp_iter.h | 12 ++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 44 ++++++++++++++++--------------------- 2 files changed, 31 insertions(+), 25 deletions(-)
arch/x86/kvm/mmu/tdp_iter.h +12 -0
@@ -115,4 +115,16 @@ void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root,void tdp_iter_next (struct tdp_iter *iter);void tdp_iter_restart (struct tdp_iter *iter); +static inline union kvm_mmu_page_role tdp_iter_child_role (struct tdp_iter *iter) +{ +union kvm_mmu_page_role child_role; +struct kvm_mmu_page *parent_sp; + + parent_sp =sptep_to_sp (rcu_dereference (iter->sptep)); + + child_role = parent_sp->role; + child_role.level--; +return child_role; +} +#endif /* __KVM_X86_MMU_TDP_ITER_H */
arch/x86/kvm/mmu/tdp_mmu.c +19 -25
119 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 34/106 KVM: x86/mmu: Require TDP MMU and mmio caching for TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
As TDP MMU is becoming main stream than the legacy MMU, the legacy MMU support for TDX isn't implemented. TDX requires KVM mmio caching. Disable TDX support when TDP MMU or mmio caching aren't supported.
--- arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/vmx/main.c | 6 ++++++ 2 files changed, 7 insertions(+)
arch/x86/kvm/mmu/mmu.c +1 -0
@@ -99,6 +99,7 @@ module_param_named(flush_on_reuse, force_flush_and_sync_on_reuse, bool, 0644);* If the hardware supports that we don't need to do shadow paging. */ bool tdp_enabled = false; +EXPORT_SYMBOL_GPL (tdp_enabled);bool __ro_after_init tdp_mmu_allowed;
arch/x86/kvm/vmx/main.c +6 -0
@@ -2,6 +2,7 @@#include <linux/moduleparam.h> #include "x86_ops.h" +#include "mmu.h" #include "vmx.h" #include "nested.h" #include "pmu.h"
@@ -38,6 +39,11 @@ static __init int vt_hardware_setup(void) enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops); +/* TDX requires KVM TDP MMU and MMIO caching. */ +if (enable_tdx && (!tdp_enabled || !enable_mmio_caching)) { + enable_tdx = false; +pr_warn_ratelimited ("tdp mmu and mmio caching need to be enabled.\n" ); + }return 0 ; }
Signature · 1 line
2.25.1
PATCH v12 35/106 KVM: x86/mmu: Add a new is_private member for union kvm_mmu_page_role
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because TDX support introduces private mapping, add a new member in union kvm_mmu_page_role with access functions to check the member.
--- arch/x86/include/asm/kvm_host.h | 27 +++++++++++++++++++++++++++ arch/x86/kvm/mmu/mmu_internal.h | 5 +++++ arch/x86/kvm/mmu/spte.h | 6 ++++++ 3 files changed, 38 insertions(+)
arch/x86/include/asm/kvm_host.h +27 -0
@@ -338,7 +338,12 @@ union kvm_mmu_page_role { unsigned ad_disabled:1; unsigned guest_mode:1; unsigned passthrough:1; +#ifdef CONFIG_KVM_MMU_PRIVATE + unsigned is_private:1; + unsigned :4; +#else unsigned :5; +#endif /* * This is left at the top of the word so that
@@ -350,6 +355,28 @@ union kvm_mmu_page_role { }; }; +#ifdef CONFIG_KVM_MMU_PRIVATE +static inline bool kvm_mmu_page_role_is_private (union kvm_mmu_page_role role) +{ +return !!role.is_private; +} + +static inline void kvm_mmu_page_role_set_private (union kvm_mmu_page_role *role) +{ + role->is_private =1 ; +} +#else +static inline bool kvm_mmu_page_role_is_private (union kvm_mmu_page_role role) +{ +return false; +} + +static inline void kvm_mmu_page_role_set_private (union kvm_mmu_page_role *role) +{ +WARN_ON_ONCE (1 ); +} +#endif +/* * kvm_mmu_extended_role complements kvm_mmu_page_role, tracking properties * relevant to the current MMU configuration. When loading CR0, CR4, or EFER,
arch/x86/kvm/mmu/mmu_internal.h +5 -0
@@ -143,6 +143,11 @@ static inline int kvm_mmu_page_as_id(struct kvm_mmu_page *sp)return kvm_mmu_role_as_id (sp->role); } +static inline bool is_private_sp (const struct kvm_mmu_page *sp) +{ +return kvm_mmu_page_role_is_private (sp->role); +} +static inline bool kvm_mmu_page_ad_need_write_protect (struct kvm_mmu_page *sp) {/*
arch/x86/kvm/mmu/spte.h +6 -0
@@ -251,6 +251,12 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep)return to_shadow_page (__pa (sptep)); } +static inline bool is_private_sptep (u64 *sptep) +{ +WARN_ON_ONCE (!sptep); +return is_private_sp (sptep_to_sp (sptep)); +} +static inline bool is_mmio_spte (struct kvm *kvm,u64 spte) { return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&
Signature · 1 line
2.25.1
PATCH v12 36/106 KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page
In reply to [email protected]From: Isaku Yamahata <[email protected]>
For private GPA, CPU refers a private page table whose contents are encrypted. The dedicated APIs to operate on it (e.g. updating/reading its PTE entry) are used and their cost is expensive.
When KVM resolves KVM page fault, it walks the page tables. To reuse the existing KVM MMU code and mitigate the heavy cost to directly walk private page table, allocate one more page to copy the dummy page table for KVM MMU code to directly walk. Resolve KVM page fault with the existing code, and do additional operations necessary for the private page table. To distinguish such cases, the existing KVM page table is called a shared page table (i.e. not associated with private page table), and the page table with private page table is called a private page table. The relationship is depicted below.
Add a private pointer to struct kvm_mmu_page for private page table and add helper functions to allocate/initialize/free a private page table page.
KVM page fault |
| |
V |
-------------+---------- |
| | |
V V |
shared GPA private GPA |
| | |
V V |
shared PT root dummy PT root | private PT root
| | | |
V V | V
shared PT dummy PT ----propagate----> private PT
| | | |
| \-----------------+------\ |
| | | |
V | V V
shared guest page | private guest page
|
non-encrypted memory | encrypted memory
|PT: page table
- Shared PT is visible to KVM and it is used by CPU.
- Private PT is used by CPU but it is invisible to KVM.
- Dummy PT is visible to KVM but not used by CPU. It is used to propagate PT change to the actual private PT which is used by CPU.
--- arch/x86/include/asm/kvm_host.h | 5 ++ arch/x86/kvm/mmu/mmu.c | 7 +++ arch/x86/kvm/mmu/mmu_internal.h | 83 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 1 + 4 files changed, 92 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +5 -0
@@ -810,6 +810,11 @@ struct kvm_vcpu_arch {struct kvm_mmu_memory_cache mmu_shadow_page_cache;struct kvm_mmu_memory_cache mmu_shadowed_info_cache;struct kvm_mmu_memory_cache mmu_page_header_cache; +/* +* This cache is to allocate private page table. E.g. Secure-EPT used +* by the TDX module. +*/ +struct kvm_mmu_memory_cache mmu_private_spt_cache;/* * QEMU userspace and the guest each have their own FPU state.
arch/x86/kvm/mmu/mmu.c +7 -0
@@ -666,6 +666,12 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect) 1 + PT64_ROOT_MAX_LEVEL + PTE_PREFETCH_NUM);if (r)return r; +if (kvm_gfn_shared_mask (vcpu->kvm)) { + r =kvm_mmu_topup_memory_cache (&vcpu->arch.mmu_private_spt_cache, +PT64_ROOT_MAX_LEVEL ); +if (r) +return r; + } r =kvm_mmu_topup_memory_cache (&vcpu->arch.mmu_shadow_page_cache,PT64_ROOT_MAX_LEVEL );if (r)
@@ -685,6 +691,7 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)kvm_mmu_free_memory_cache (&vcpu->arch.mmu_pte_list_desc_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadow_page_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadowed_info_cache); +kvm_mmu_free_memory_cache (&vcpu->arch.mmu_private_spt_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_page_header_cache); }
arch/x86/kvm/mmu/mmu_internal.h +79 -4
101 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
@@ -56,6 +56,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)static void tdp_mmu_free_sp (struct kvm_mmu_page *sp) { +kvm_mmu_free_private_spt (sp);free_page ((unsigned long )sp->spt);kmem_cache_free (mmu_page_header_cache, sp); }
Signature · 1 line
2.25.1
PATCH v12 37/106 KVM: Add flags to struct kvm_gfn_range
In reply to [email protected]From: Isaku Yamahata <[email protected]>
kvm_unmap_gfn_range() needs to know the reason of the callback for TDX. mmu notifier, set memattr ioctl or restrictedmem notifier. Based on the reason, TDX changes the behavior. For mmu notifier, it's the operation on shared memory slot to zap shared PTE. For set memattr, it's the operation of private<->shared conversion, zap the original PTE. For restrictedmem, it's punching a hole of the range, zap the corresponding PTE.
--- include/linux/kvm_host.h | 10 +++++++++- virt/kvm/kvm_main.c | 5 ++++- 2 files changed, 13 insertions(+), 2 deletions(-)
include/linux/kvm_host.h +9 -1
@@ -257,12 +257,20 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);#endif #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER + +#define KVM_GFN_RANGE_FLAGS_RESTRICTED_MEM BIT(0) +#define KVM_GFN_RANGE_FLAGS_SET_MEM_ATTR BIT(1) +struct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;gfn_t end; -pte_t pte; +union { +pte_t pte; +u64 attrs; + };bool may_block; +unsigned int flags; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);
virt/kvm/kvm_main.c +4 -1
@@ -626,6 +626,7 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start =hva_to_gfn_memslot (hva_start, slot); gfn_range.end =hva_to_gfn_memslot (hva_end +PAGE_SIZE -1 , slot); gfn_range.slot = slot; + gfn_range.flags =0 ;if (!locked) { locked = true;
@@ -957,6 +958,7 @@ static int restrictedmem_get_gfn_range(struct kvm_memory_slot *slot, range->slot = slot; range->pte =__pte (0 ); range->may_block = true; + range->flags =KVM_GFN_RANGE_FLAGS_RESTRICTED_MEM ;return 0 ; }
@@ -2557,8 +2559,9 @@ static void kvm_mem_attrs_changed(struct kvm *kvm, unsigned long attrs,bool flush = false;int i; - gfn_range.pte =__pte (0 ); + gfn_range.attrs = attrs; gfn_range.may_block = true; + gfn_range.flags =KVM_GFN_RANGE_FLAGS_SET_MEM_ATTR ;for (i =0 ; i <kvm_arch_nr_memslot_as_ids (kvm); i++) { slots =__kvm_memslots (kvm, i);
Signature · 1 line
2.25.1
PATCH v12 38/106 KVM: x86/tdp_mmu: Don't zap private pages for unsupported cases
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX supports only write-back(WB) memory type for private memory architecturally so that (virtualized) memory type change doesn't make sense for private memory. Also currently, page migration isn't supported for TDX yet. (TDX architecturally supports page migration. it's KVM and kernel implementation issue.)
Regarding memory type change (mtrr virtualization and lapic page mapping change), pages are zapped by kvm_zap_gfn_range(). On the next KVM page fault, the SPTE entry with a new memory type for the page is populated. Regarding page migration, pages are zapped by the mmu notifier. On the next KVM page fault, the new migrated page is populated. Don't zap private pages on unmapping for those two cases.
When deleting/moving a KVM memory slot, zap private pages. Typically tearing down VM. Don't invalidate private page tables. i.e. zap only leaf SPTEs for KVM mmu that has a shared bit mask. The existing kvm_tdp_mmu_invalidate_all_roots() depends on role.invalid with read-lock of mmu_lock so that other vcpu can operate on KVM mmu concurrently. It marks the root page table invalid and zaps SPTEs of the root page tables. The TDX module doesn't allow to unlink a protected root page table from the hardware and then allocate a new one for it. i.e. replacing a protected root page table. Instead, zap only leaf SPTEs for KVM mmu with a shared bit mask set.
--- arch/x86/kvm/mmu/mmu.c | 81 ++++++++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 24 ++++++++--- arch/x86/kvm/mmu/tdp_mmu.h | 5 ++- 3 files changed, 99 insertions(+), 11 deletions(-)
arch/x86/kvm/mmu/mmu.c +77 -4
@@ -1590,8 +1590,28 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)if (kvm_memslots_have_rmaps (kvm)) flush =kvm_handle_gfn_range (kvm, range, kvm_zap_rmap); -if (tdp_mmu_enabled) - flush =kvm_tdp_mmu_unmap_gfn_range (kvm, range, flush); +if (tdp_mmu_enabled) { +bool zap_private; + +if (range->flags &KVM_GFN_RANGE_FLAGS_RESTRICTED_MEM ) { +WARN_ON_ONCE (!kvm_slot_can_be_private (range->slot)); +/* +* For private slot, the callback is triggered +* via PUNCH_HOLE (fallocate(PUNCH_HOLE) or truncate). +* private-shared conversion is done by +* KVM_SET_MEMORY_ATTRIBUTES. +*/ + zap_private = true; + }else if (range->flags &KVM_GFN_RANGE_FLAGS_SET_MEM_ATTR ) + zap_private = !(range->attrs &KVM_MEMORY_ATTRIBUTE_PRIVATE ); +else +/* +* For now private pages are pinned during VM's life +* time. +*/ + zap_private = false; + flush =kvm_tdp_mmu_unmap_gfn_range (kvm, range, flush, zap_private); + }return flush; }
@@ -6167,11 +6187,54 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)return unlikely (!list_empty_careful (&kvm->arch.zapped_obsolete_pages)); } +static void kvm_mmu_zap_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) +{ +bool flush = false; + +write_lock (&kvm->mmu_lock); + +/* +* Zapping non-leaf SPTEs, a.k.a. not-last SPTEs, isn't required, worst +* case scenario we'll have unused shadow pages lying around until they +* are recycled due to age or when the VM is destroyed. +*/ +if (tdp_mmu_enabled) { +struct kvm_gfn_range range = { + .slot = slot, + .start = slot->base_gfn, + .end = slot->base_gfn + slot->npages, + .may_block = true, + }; + +/* +* this handles both private gfn and shared gfn. +* All private page should be zapped on memslot deletion. +*/ + flush =kvm_tdp_mmu_unmap_gfn_range (kvm, &range, flush, true); + }else { +/* TDX supports only TDP-MMU case. */ +WARN_ON_ONCE (1 ); + flush = true; + } +if (flush) +kvm_flush_remote_tlbs (kvm); + +write_unlock (&kvm->mmu_lock); +} +static void kvm_mmu_invalidate_zap_pages_in_memslot (struct kvm *kvm,struct kvm_memory_slot *slot,struct kvm_page_track_notifier_node *node) { -kvm_mmu_zap_all_fast (kvm); +if (kvm_gfn_shared_mask (kvm)) +/* +* Secure-EPT requires to release PTs from the leaf. The +* optimization to zap root PT first with child PT doesn't +* work. +*/ +kvm_mmu_zap_memslot (kvm, slot); +else +kvm_mmu_zap_all_fast (kvm); }int kvm_mmu_init_vm (struct kvm *kvm)
@@ -6279,8 +6342,18 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)if (tdp_mmu_enabled) {for (i =0 ; i <kvm_arch_nr_memslot_as_ids (kvm); i++) +/* +* zap_private = true. Zap both private/shared pages. +* +* kvm_zap_gfn_range() is used when MTRR or PAT memory +* type was changed. Later on the next kvm page fault, +* populate it with updated spte entry. +* Because only WB is supported for private pages, don't +* care of private pages. +*/ flush =kvm_tdp_mmu_zap_leafs (kvm, i, gfn_start, - gfn_end, true, flush); + gfn_end, true, flush, + false); }if (flush)
arch/x86/kvm/mmu/tdp_mmu.c +19 -5
@@ -932,7 +932,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)* operation can cause a soft lockup. */ static bool tdp_mmu_zap_leafs (struct kvm *kvm,struct kvm_mmu_page *root, -gfn_t start,gfn_t end,bool can_yield,bool flush) +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool zap_private) {struct tdp_iter iter;
@@ -940,6 +941,10 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,lockdep_assert_held_write (&kvm->mmu_lock); +WARN_ON_ONCE (zap_private && !is_private_sp (root)); +if (!zap_private &&is_private_sp (root)) +return false; +rcu_read_lock ();for_each_tdp_pte_min_level (iter, root,PG_LEVEL_4K , start, end) {
@@ -972,12 +977,13 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,* more SPTEs were zapped since the MMU lock was last acquired. */ bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,int as_id,gfn_t start,gfn_t end, -bool can_yield,bool flush) +bool can_yield,bool flush,bool zap_private) {struct kvm_mmu_page *root;for_each_tdp_mmu_root_yield_safe (kvm, root, as_id) - flush =tdp_mmu_zap_leafs (kvm, root, start, end, can_yield, flush); + flush =tdp_mmu_zap_leafs (kvm, root, start, end, can_yield, flush, + zap_private &&is_private_sp (root));return flush; }
@@ -1037,6 +1043,12 @@ void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm)lockdep_assert_held_write (&kvm->mmu_lock);list_for_each_entry (root, &kvm->arch.tdp_mmu_roots, link) { +/* +* Skip private root since private page table +* is only torn down when VM is destroyed. +*/ +if (is_private_sp (root)) +continue ;if (!root->role.invalid && !WARN_ON_ONCE (!kvm_tdp_mmu_get_root (root))) { root->role.invalid = true;
@@ -1221,11 +1233,13 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return ret; } +/* Used by mmu notifier via kvm_unmap_gfn_range() */ bool kvm_tdp_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range, -bool flush) +bool flush,bool zap_private) {return kvm_tdp_mmu_zap_leafs (kvm, range->slot->as_id, range->start, - range->end, range->may_block, flush); + range->end, range->may_block, flush, + zap_private); } typedef bool (*tdp_handler_t)(struct kvm *kvm, struct tdp_iter *iter,
arch/x86/kvm/mmu/tdp_mmu.h +3 -2
@@ -21,7 +21,8 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root, bool shared);bool kvm_tdp_mmu_zap_leafs (struct kvm *kvm,int as_id,gfn_t start, -gfn_t end,bool can_yield,bool flush); +gfn_t end,bool can_yield,bool flush, +bool zap_private);bool kvm_tdp_mmu_zap_sp (struct kvm *kvm,struct kvm_mmu_page *sp);void kvm_tdp_mmu_zap_all (struct kvm *kvm);void kvm_tdp_mmu_invalidate_all_roots (struct kvm *kvm);
@@ -30,7 +31,7 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm);int kvm_tdp_mmu_map (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);bool kvm_tdp_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range, -bool flush); +bool flush,bool zap_private);bool kvm_tdp_mmu_age_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_tdp_mmu_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_tdp_mmu_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);
Signature · 1 line
2.25.1
PATCH v12 39/106 KVM: x86/tdp_mmu: Sprinkle __must_check
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDP MMU allows tdp_mmu_set_spte_atomic() and tdp_mmu_zap_spte_atomic() to return -EBUSY or -EAGAIN error. The caller must check the return value and retry. Sprinkle __must_check to guarantee it.
--- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +5 -5
@@ -628,9 +628,9 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,* no side-effects other than setting iter->old_spte to the last * known value of the spte. */ -static inline int tdp_mmu_set_spte_atomic (struct kvm *kvm, -struct tdp_iter *iter, -u64 new_spte) +static inline int __must_check tdp_mmu_set_spte_atomic (struct kvm *kvm, +struct tdp_iter *iter, +u64 new_spte) {u64 *sptep =rcu_dereference (iter->sptep);
@@ -658,8 +658,8 @@ static inline int tdp_mmu_set_spte_atomic(struct kvm *kvm,return 0 ; } -static inline int tdp_mmu_zap_spte_atomic (struct kvm *kvm, -struct tdp_iter *iter) +static inline int __must_check tdp_mmu_zap_spte_atomic (struct kvm *kvm, +struct tdp_iter *iter) {int ret;
Signature · 1 line
2.25.1
PATCH v12 40/106 KVM: x86/tdp_mmu: Support TDX private mapping for TDP MMU
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Allocate protected page table for private page table, and add hooks to operate on protected page table. This patch adds allocation/free of protected page tables and hooks. When calling hooks to update SPTE entry, freeze the entry, call hooks and unfreeze the entry to allow concurrent updates on page tables. Which is the advantage of TDP MMU. As kvm_gfn_shared_mask() returns false always, those hooks aren't called yet with this patch.
When the faulting GPA is private, the KVM fault is called private. When resolving private KVM fault, allocate protected page table and call hooks to operate on protected page table. On the change of the private PTE entry, invoke kvm_x86_ops hook in __handle_changed_spte() to propagate the change to protected page table. The following depicts the relationship.
private KVM page fault |
| |
V |
private GPA | CPU protected EPTP
| | |
V | V
private PT root | protected PT root
| | |
V | V
private PT --hook to propagate-->protected PT
| | |
\--------------------+------\ |
| | |
| V V
| private guest page
|
|
non-encrypted memory | encrypted memory
|PT: page table
The existing KVM TDP MMU code uses atomic update of SPTE. On populating the EPT entry, atomically set the entry. However, it requires TLB shootdown to zap SPTE. To address it, the entry is frozen with the special SPTE value that clears the present bit. After the TLB shootdown, the entry is set to the eventual value (unfreeze).
For protected page table, hooks are called to update protected page table in addition to direct access to the private SPTE. For the zapping case, it works to freeze the SPTE. It can call hooks in addition to TLB shootdown. For populating the private SPTE entry, there can be a race condition without further protection
vcpu 1: populating 2M private SPTE
vcpu 2: populating 4K private SPTE
vcpu 2: TDX SEAMCALL to update 4K protected SPTE => error vcpu 1: TDX SEAMCALL to update 2M protected SPTE
To avoid the race, the frozen SPTE is utilized. Instead of atomic update of the private entry, freeze the entry, call the hook that update protected SPTE, set the entry to the final value.
Support 4K page only at this stage. 2M page support can be done in future patches.
---
Changes v11 -> v12
- Split tdp mmu hooks at tdp_mmu_set_spte_atomic() for populating and __handle_changed_spte() for zapping.
--- arch/x86/include/asm/kvm-x86-ops.h | 5 + arch/x86/include/asm/kvm_host.h | 11 ++ arch/x86/kvm/mmu/mmu.c | 13 +- arch/x86/kvm/mmu/mmu_internal.h | 21 +- arch/x86/kvm/mmu/tdp_iter.h | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 300 +++++++++++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- virt/kvm/kvm_main.c | 1 + 8 files changed, 313 insertions(+), 42 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +5 -0
@@ -96,6 +96,11 @@ KVM_X86_OP_OPTIONAL_RET0(set_tss_addr) KVM_X86_OP_OPTIONAL_RET0(set_identity_map_addr) KVM_X86_OP_OPTIONAL_RET0(get_mt_mask) KVM_X86_OP(load_mmu_pgd) +KVM_X86_OP_OPTIONAL(link_private_spt) +KVM_X86_OP_OPTIONAL(free_private_spt) +KVM_X86_OP_OPTIONAL(set_private_spte) +KVM_X86_OP_OPTIONAL(remove_private_spte) +KVM_X86_OP_OPTIONAL(zap_private_spte) KVM_X86_OP(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier)
arch/x86/include/asm/kvm_host.h +11 -0
@@ -470,6 +470,7 @@ struct kvm_mmu { struct kvm_mmu_page *sp);void (*invlpg)(struct kvm_vcpu *vcpu,gva_t gva,hpa_t root_hpa);struct kvm_mmu_root_info root; +hpa_t private_root_hpa;union kvm_cpu_role cpu_role;union kvm_mmu_page_role root_role;
@@ -1690,6 +1691,16 @@ struct kvm_x86_ops {void (*load_mmu_pgd)(struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level); +int (*link_private_spt)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +void *private_spt); +int (*free_private_spt)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +void *private_spt); +int (*set_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +kvm_pfn_t pfn); +int (*remove_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level, +kvm_pfn_t pfn); +int (*zap_private_spte)(struct kvm *kvm,gfn_t gfn,enum pg_level level); +bool (*has_wbinvd_exit)(void );u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu);
arch/x86/kvm/mmu/mmu.c +11 -2
@@ -3681,7 +3681,12 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) goto out_unlock;if (tdp_mmu_enabled) { - root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu); +if (kvm_gfn_shared_mask (vcpu->kvm) && + !VALID_PAGE (mmu->private_root_hpa)) { + root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu, true); + mmu->private_root_hpa = root; + } + root =kvm_tdp_mmu_get_vcpu_root_hpa (vcpu, false); mmu->root.hpa = root; }else if (shadow_root_level >=PT64_ROOT_4LEVEL ) { root =mmu_alloc_root (vcpu,0 ,0 , shadow_root_level);
@@ -5981,6 +5986,7 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) mmu->root.hpa =INVALID_PAGE ; mmu->root.pgd =0 ; + mmu->private_root_hpa =INVALID_PAGE ;for (i =0 ; i <KVM_MMU_NUM_PREV_ROOTS ; i++) mmu->prev_roots[i] =KVM_MMU_ROOT_INFO_INVALID ;
@@ -6207,7 +6213,7 @@ static void kvm_mmu_zap_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) };/* -* this handles both private gfn and shared gfn. +* This handles both private gfn and shared gfn. * All private page should be zapped on memslot deletion. */ flush =kvm_tdp_mmu_unmap_gfn_range (kvm, &range, flush, true);
@@ -7027,6 +7033,9 @@ int kvm_mmu_vendor_module_init(void)void kvm_mmu_destroy (struct kvm_vcpu *vcpu) {kvm_mmu_unload (vcpu); +if (tdp_mmu_enabled) +mmu_free_root_page (vcpu->kvm, &vcpu->arch.mmu->private_root_hpa, +NULL );free_mmu_pages (&vcpu->arch.root_mmu);free_mmu_pages (&vcpu->arch.guest_mmu);mmu_free_memory_caches (vcpu);
arch/x86/kvm/mmu/mmu_internal.h +19 -2
@@ -6,6 +6,8 @@#include <linux/kvm_host.h> #include <asm/kvm_host.h> +#include "mmu.h" +#undef MMU_DEBUG#ifdef MMU_DEBUG
@@ -204,6 +206,15 @@ static inline void kvm_mmu_free_private_spt(struct kvm_mmu_page *sp)if (sp->private_spt) free_page((unsigned long)sp->private_spt); } + +static inline gfn_t kvm_gfn_for_root (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t gfn) +{ +if (is_private_sp (root)) +return kvm_gfn_private (kvm, gfn); +else +return kvm_gfn_shared (kvm, gfn); +}#else static inline void *kvm_mmu_private_spt (struct kvm_mmu_page *sp) {
@@ -221,6 +232,12 @@ static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu, struct kvm_mstatic inline void kvm_mmu_free_private_spt (struct kvm_mmu_page *sp) { } + +static inline gfn_t kvm_gfn_for_root (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t gfn) +{ +return gfn; +}#endif static inline bool kvm_mmu_page_ad_need_write_protect (struct kvm_mmu_page *sp)
@@ -360,12 +377,12 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, .max_level = vcpu->kvm->arch.tdp_max_page_level, .req_level = PG_LEVEL_4K, .goal_level = PG_LEVEL_4K, - .is_private = kvm_mem_is_private(vcpu->kvm, cr2_or_gpa >> PAGE_SHIFT), + .is_private = kvm_is_private_gpa(vcpu->kvm, cr2_or_gpa), };int r;if (vcpu->arch.mmu->root_role.direct) { - fault.gfn = fault.addr >>PAGE_SHIFT ; + fault.gfn =gpa_to_gfn (fault.addr) & ~kvm_gfn_shared_mask (vcpu->kvm); fault.slot =kvm_vcpu_gfn_to_memslot (vcpu, fault.gfn); }
arch/x86/kvm/mmu/tdp_iter.h +1 -1
@@ -71,7 +71,7 @@ struct tdp_iter {tdp_ptep_t pt_path[PT64_ROOT_MAX_LEVEL ];/* A pointer to the current SPTE */ tdp_ptep_t sptep; -/* The lowest GFN mapped by the current SPTE */ +/* The lowest GFN (shared bits included) mapped by the current SPTE */ gfn_t gfn;/* The level of the root page given to the iterator */ int root_level;
arch/x86/kvm/mmu/tdp_mmu.c +264 -36
468 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.h +1 -1
@@ -10,7 +10,7 @@int kvm_mmu_init_tdp_mmu (struct kvm *kvm);void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm); -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu); +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu,bool private);__must_check static inline bool kvm_tdp_mmu_get_root (struct kvm_mmu_page *root) {
virt/kvm/kvm_main.c +1 -0
@@ -206,6 +206,7 @@ struct page *kvm_pfn_to_refcounted_page(kvm_pfn_t pfn)return NULL ; } +EXPORT_SYMBOL_GPL (kvm_pfn_to_refcounted_page);/* * Switches to specified vcpu, until a matching vcpu_put()
Signature · 1 line
2.25.1
PATCH v12 41/106 The start of TDX KVM patch series: TDX EPT violation
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TDX EPT violation.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2
@@ -19,11 +19,11 @@ Patch Layer status * TDX architectural definitions: Applied * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied -* TDX EPT violation: Not yet +* TDX EPT violation: Applying * TD finalization: Not yet * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied * KVM TDP refactoring for TDX: Applied -* KVM TDP MMU hooks: Applying +* KVM TDP MMU hooks: Applied
Signature · 1 line
2.25.1
PATCH v12 42/106 KVM: x86/mmu: Disallow dirty logging for x86 TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX doesn't support dirty logging. Report dirty logging isn't supported so that device model, for example qemu, can properly handle it. Silently ignore on dirty logging on private GFNs of TDX.
--- arch/x86/kvm/mmu/mmu.c | 3 +++ arch/x86/kvm/mmu/tdp_mmu.c | 36 +++++++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 8 ++++++++ include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 10 +++++++++- 5 files changed, 56 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -6690,6 +6690,9 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,for_each_rmap_spte (rmap_head, &iter, sptep) { sp =sptep_to_sp (sptep); +/* Private page dirty logging is not supported yet. */ +KVM_BUG_ON (is_private_sptep (sptep), kvm); +/* * We cannot do huge page mapping for indirect shadow pages, * which are found on the last rmap (level = 1) when not using
arch/x86/kvm/mmu/tdp_mmu.c +35 -1
@@ -1476,9 +1476,27 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm,* into this helper allow blocking; it'd be dead, wasteful code. */ for_each_tdp_mmu_root (kvm, root, range->slot->as_id) { +gfn_t start; +gfn_t end; + +/* +* For now, operation on private GPA, e.g. dirty page logging, +* isn't supported yet. +*/ +if (is_private_sp (root)) +continue ; +rcu_read_lock (); -tdp_root_for_each_leaf_pte (iter, root, range->start, range->end) +/* +* For TDX shared mapping, set GFN shared bit to the range, +* so the handler() doesn't need to set it, to avoid duplicated +* code in multiple handler()s. +*/ + start =kvm_gfn_shared (kvm, range->start); + end =kvm_gfn_shared (kvm, range->end); + +tdp_root_for_each_leaf_pte (iter, root, start, end) ret |=handler (kvm, &iter, range);rcu_read_unlock ();
@@ -1961,6 +1979,13 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm,struct kvm_mmu_page *root;lockdep_assert_held_write (&kvm->mmu_lock); +/* +* First TDX generation doesn't support clearing dirty bit, +* since there's no secure EPT API to support it. For now silently +* ignore KVM_CLEAR_DIRTY_LOG. +*/ +if (!kvm_arch_dirty_log_supported (kvm)) +return ;for_each_tdp_mmu_root (kvm, root, slot->as_id)clear_dirty_pt_masked (kvm, root, gfn, mask, wrprot); }
@@ -2080,6 +2105,15 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm,bool spte_set = false;lockdep_assert_held_write (&kvm->mmu_lock); + +/* +* First TDX generation doesn't support write protecting private +* mappings, silently ignore the request. KVM_GET_DIRTY_LOG etc +* can reach here, no warning. +*/ +if (!kvm_arch_dirty_log_supported (kvm)) +return false; +for_each_tdp_mmu_root (kvm, root, slot->as_id) spte_set |=write_protect_gfn (kvm, root, gfn, min_level);
arch/x86/kvm/x86.c +8 -0
@@ -12549,6 +12549,9 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,u32 new_flags = new ? new->flags :0 ;bool log_dirty_pages = new_flags &KVM_MEM_LOG_DIRTY_PAGES ; +if (!kvm_arch_dirty_log_supported (kvm) && log_dirty_pages) +return ; +/* * Update CPU dirty logging if dirty logging is being toggled. This * applies to all operations.
@@ -13521,6 +13524,11 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, }EXPORT_SYMBOL_GPL (kvm_sev_es_string_io); +bool kvm_arch_dirty_log_supported (struct kvm *kvm) +{ +return kvm->arch.vm_type !=KVM_X86_PROTECTED_VM ; +} +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_entry);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_exit);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_fast_mmio);
include/linux/kvm_host.h +1 -0
@@ -1495,6 +1495,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_dirty_log_supported (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/*
virt/kvm/kvm_main.c +9 -1
@@ -1700,10 +1700,18 @@ static void kvm_replace_memslot(struct kvm *kvm, } } +bool __weak kvm_arch_dirty_log_supported (struct kvm *kvm) +{ +return true; +} +static int check_memory_region_flags (struct kvm *kvm,const struct kvm_userspace_memory_region2 *mem) { -u32 valid_flags =KVM_MEM_LOG_DIRTY_PAGES ; +u32 valid_flags =0 ; + +if (kvm_arch_dirty_log_supported (kvm)) + valid_flags |=KVM_MEM_LOG_DIRTY_PAGES ;if (kvm_arch_has_private_mem (kvm)) valid_flags |=KVM_MEM_PRIVATE ;
Signature · 1 line
2.25.1
PATCH v12 43/106 KVM: x86/mmu: TDX: Do not enable page track for TD guest
In reply to [email protected]From: Yan Zhao <[email protected]>
TDX does not support write protection and hence page track. Though !tdp_enabled and kvm_shadow_root_allocated(kvm) are always false for TD guest, should also return false when external write tracking is enabled.
--- arch/x86/kvm/mmu/page_track.c | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/page_track.c +3 -0
@@ -22,6 +22,9 @@bool kvm_page_track_write_tracking_enabled (struct kvm *kvm) { +if (kvm->arch.vm_type ==KVM_X86_PROTECTED_VM ) +return false; +return IS_ENABLED (CONFIG_KVM_EXTERNAL_WRITE_TRACKING ) || !tdp_enabled ||kvm_shadow_root_allocated (kvm); }
Signature · 1 line
2.25.1
PATCH v12 44/106 KVM: VMX: Split out guts of EPT violation to common/exposed function
In reply to [email protected]From: Sean Christopherson <[email protected]>
The difference of TDX EPT violation is how to retrieve information, GPA, and exit qualification. To share the code to handle EPT violation, split out the guts of EPT violation handler so that VMX/TDX exit handler can call it after retrieving GPA and exit qualification.
--- arch/x86/kvm/vmx/common.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 25 +++---------------------- 2 files changed, 36 insertions(+), 22 deletions(-) create mode 100644 arch/x86/kvm/vmx/common.h
arch/x86/kvm/vmx/common.h new file +33 -0
@@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_X86_VMX_COMMON_H +#define __KVM_X86_VMX_COMMON_H + +#include <linux/kvm_host.h> + +#include "mmu.h" + +static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa, +unsigned long exit_qualification) +{ +u64 error_code; + +/* Is it a read fault? */ + error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) + ?PFERR_USER_MASK :0 ; +/* Is it a write fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) + ?PFERR_WRITE_MASK :0 ; +/* Is it a fetch fault? */ + error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) + ?PFERR_FETCH_MASK :0 ; +/* ept page table entry is present? */ + error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) + ?PFERR_PRESENT_MASK :0 ; + + error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? +PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; + +return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +} + +#endif /* __KVM_X86_VMX_COMMON_H */
arch/x86/kvm/vmx/vmx.c +3 -22
@@ -51,6 +51,7 @@#include <asm/vmx.h> #include "capabilities.h" +#include "common.h" #include "cpuid.h" #include "hyperv.h" #include "kvm_onhyperv.h"
@@ -5760,11 +5761,8 @@ static int handle_task_switch(struct kvm_vcpu *vcpu)static int handle_ept_violation (struct kvm_vcpu *vcpu) { -unsigned long exit_qualification; +unsigned long exit_qualification =vmx_get_exit_qual (vcpu);gpa_t gpa; -u64 error_code; - - exit_qualification =vmx_get_exit_qual (vcpu);/* * EPT violation happened while executing iret from NMI,
@@ -5779,23 +5777,6 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu) gpa =vmcs_read64 (GUEST_PHYSICAL_ADDRESS );trace_kvm_page_fault (vcpu, gpa, exit_qualification); - -/* Is it a read fault? */ - error_code = (exit_qualification &EPT_VIOLATION_ACC_READ ) - ?PFERR_USER_MASK :0 ; -/* Is it a write fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_WRITE ) - ?PFERR_WRITE_MASK :0 ; -/* Is it a fetch fault? */ - error_code |= (exit_qualification &EPT_VIOLATION_ACC_INSTR ) - ?PFERR_FETCH_MASK :0 ; -/* ept page table entry is present? */ - error_code |= (exit_qualification &EPT_VIOLATION_RWX_MASK ) - ?PFERR_PRESENT_MASK :0 ; - - error_code |= (exit_qualification &EPT_VIOLATION_GVA_TRANSLATED ) !=0 ? -PFERR_GUEST_FINAL_MASK :PFERR_GUEST_PAGE_MASK ; - vcpu->arch.exit_qualification = exit_qualification;/*
@@ -5809,7 +5790,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)if (unlikely (allow_smaller_maxphyaddr &&kvm_vcpu_is_illegal_gpa (vcpu, gpa)))return kvm_emulate_instruction (vcpu,0 ); -return kvm_mmu_page_fault (vcpu, gpa, error_code,NULL ,0 ); +return __vmx_handle_ept_violation (vcpu, gpa, exit_qualification); }static int handle_ept_misconfig (struct kvm_vcpu *vcpu)
Signature · 1 line
2.25.1
PATCH v12 45/106 KVM: VMX: Move setting of EPT MMU masks to common VT-x code
In reply to [email protected]From: Sean Christopherson <[email protected]>
EPT MMU masks are used commonly for VMX and TDX. The value needs to be initialized in common code before both VMX/TDX-specific initialization code.
--- arch/x86/kvm/vmx/main.c | 9 +++++++++ arch/x86/kvm/vmx/vmx.c | 4 ---- 2 files changed, 9 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/main.c +9 -0
@@ -5,6 +5,7 @@#include "mmu.h" #include "vmx.h" #include "nested.h" +#include "mmu.h" #include "pmu.h" #include "tdx.h" #include "tdx_arch.h"
@@ -39,6 +40,14 @@ static __init int vt_hardware_setup(void) enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops); +/* +* As kvm_mmu_set_ept_masks() updates enable_mmio_caching, call it +* before checking enable_mmio_caching. +*/ +if (enable_ept) +kvm_mmu_set_ept_masks (enable_ept_ad_bits, +cpu_has_vmx_ept_execute_only ()); +/* TDX requires KVM TDP MMU and MMIO caching. */ if (enable_tdx && (!tdp_enabled || !enable_mmio_caching)) { enable_tdx = false;
arch/x86/kvm/vmx/vmx.c +0 -4
@@ -8351,10 +8351,6 @@ __init int vmx_hardware_setup(void)set_bit (0 , vmx_vpid_bitmap);/* 0 is reserved for host */ -if (enable_ept) -kvm_mmu_set_ept_masks (enable_ept_ad_bits, -cpu_has_vmx_ept_execute_only ()); -/* * Setup shadow_me_value/shadow_me_mask to include MKTME KeyID * bits to shadow_zero_check.
Signature · 1 line
2.25.1
PATCH v12 46/106 KVM: TDX: Add accessors VMX VMCS helpers
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX defines SEAMCALL APIs to access TDX control structures corresponding to the VMX VMCS. Introduce helper accessors to hide its SEAMCALL ABI details.
--- arch/x86/kvm/vmx/tdx.h | 95 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)
arch/x86/kvm/vmx/tdx.h +95 -0
@@ -57,6 +57,101 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) return container_of(vcpu, struct vcpu_tdx, vcpu); } +static __always_inline void tdvps_vmcs_check(u32 field, u8 bits) +{ +#define VMCS_ENC_ACCESS_TYPE_MASK 0x1UL +#define VMCS_ENC_ACCESS_TYPE_FULL 0x0UL +#define VMCS_ENC_ACCESS_TYPE_HIGH 0x1UL +#define VMCS_ENC_ACCESS_TYPE (field) ((field) & VMCS_ENC_ACCESS_TYPE_MASK) + +/* TDX is 64bit only. HIGH field isn't supported. */ +BUILD_BUG_ON_MSG (__builtin_constant_p (field) && +VMCS_ENC_ACCESS_TYPE (field) ==VMCS_ENC_ACCESS_TYPE_HIGH , +"Read/Write to TD VMCS *_HIGH fields not supported" ); + +BUILD_BUG_ON (bits !=16 && bits !=32 && bits !=64 ); + +#define VMCS_ENC_WIDTH_MASK GENMASK(14, 13) +#define VMCS_ENC_WIDTH_16BIT (0UL << 13) +#define VMCS_ENC_WIDTH_64BIT (1UL << 13) +#define VMCS_ENC_WIDTH_32BIT (2UL << 13) +#define VMCS_ENC_WIDTH_NATURAL (3UL << 13) +#define VMCS_ENC_WIDTH (field) ((field) & VMCS_ENC_WIDTH_MASK) + +/* TDX is 64bit only. i.e. natural width = 64bit. */ +BUILD_BUG_ON_MSG (bits !=64 &&__builtin_constant_p (field) && + (VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_64BIT || +VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_NATURAL ), +"Invalid TD VMCS access for 64-bit field" ); +BUILD_BUG_ON_MSG (bits !=32 &&__builtin_constant_p (field) && +VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_32BIT , +"Invalid TD VMCS access for 32-bit field" ); +BUILD_BUG_ON_MSG (bits !=16 &&__builtin_constant_p (field) && +VMCS_ENC_WIDTH (field) ==VMCS_ENC_WIDTH_16BIT , +"Invalid TD VMCS access for 16-bit field" ); +} + +static __always_inline void tdvps_state_non_arch_check(u64 field, u8 bits) {} +static __always_inline void tdvps_management_check(u64 field, u8 bits) {} + +#define TDX_BUILD_TDVPS_ACCESSORS (bits, uclass, lclass) \ +static __always_inline u##bits td_##lclass##_read##bits(struct vcpu_tdx *tdx, \ + u32 field) \ +{ \ + struct tdx_module_output out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_rd(tdx->tdvpr_pa, TDVPS_##uclass(field), &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) { \ + pr_err("TDH_VP_RD["#uclass".0x%x] failed: 0x%llx\n", \ + field, err); \ + return 0; \ + } \ + return (u##bits)out.r8; \ +} \ +static __always_inline void td_##lclass##_write##bits(struct vcpu_tdx *tdx, \ + u32 field, u##bits val) \ +{ \ + struct tdx_module_output out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), val, \ + GENMASK_ULL(bits - 1, 0), &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] = 0x%llx failed: 0x%llx\n", \ + field, (u64)val, err); \ +} \ +static __always_inline void td_##lclass##_setbit##bits(struct vcpu_tdx *tdx, \ + u32 field, u64 bit) \ +{ \ + struct tdx_module_output out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), bit, bit, &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] |= 0x%llx failed: 0x%llx\n", \ + field, bit, err); \ +} \ +static __always_inline void td_##lclass##_clearbit##bits(struct vcpu_tdx *tdx, \ + u32 field, u64 bit) \ +{ \ + struct tdx_module_output out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err = tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), 0, bit, &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] &= ~0x%llx failed: 0x%llx\n", \ + field, bit, err); \ +} + +TDX_BUILD_TDVPS_ACCESSORS (16 ,VMCS , vmcs); +TDX_BUILD_TDVPS_ACCESSORS (32 ,VMCS , vmcs); +TDX_BUILD_TDVPS_ACCESSORS (64 ,VMCS , vmcs); + static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field) {struct tdx_module_output out;
Signature · 1 line
2.25.1
PATCH v12 47/106 KVM: TDX: Add load_mmu_pgd method for TDX
In reply to [email protected]From: Sean Christopherson <[email protected]>
For virtual IO, the guest TD shares guest pages with VMM without encryption. Shared EPT is used to map guest pages in unprotected way.
Add the VMCS field encoding for the shared EPTP, which will be used by TDX to have separate EPT walks for private GPAs (existing EPTP) versus shared GPAs (new shared EPTP).
Set shared EPT pointer value for the TDX guest to initialize TDX MMU.
--- arch/x86/include/asm/vmx.h | 1 + arch/x86/kvm/vmx/main.c | 13 ++++++++++++- arch/x86/kvm/vmx/tdx.c | 5 +++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 22 insertions(+), 1 deletion(-)
arch/x86/include/asm/vmx.h +1 -0
@@ -234,6 +234,7 @@ enum vmcs_field {TSC_MULTIPLIER_HIGH =0x00002033 ,TERTIARY_VM_EXEC_CONTROL =0x00002034 ,TERTIARY_VM_EXEC_CONTROL_HIGH =0x00002035 , +SHARED_EPT_POINTER =0x0000203C ,PID_POINTER_TABLE =0x00002042 ,PID_POINTER_TABLE_HIGH =0x00002043 , GUEST_PHYSICAL_ADDRESS = 0x00002400,
arch/x86/kvm/vmx/main.c +12 -1
@@ -134,6 +134,17 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)vmx_vcpu_reset (vcpu, init_event); } +static void vt_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa, +int pgd_level) +{ +if (is_td_vcpu (vcpu)) { +tdx_load_mmu_pgd (vcpu, root_hpa, pgd_level); +return ; + } + +vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -267,7 +278,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .write_tsc_offset = vmx_write_tsc_offset, .write_tsc_multiplier = vmx_write_tsc_multiplier, - .load_mmu_pgd = vmx_load_mmu_pgd, + .load_mmu_pgd = vt_load_mmu_pgd, .check_intercept = vmx_check_intercept, .handle_exit_irqoff = vmx_handle_exit_irqoff,
arch/x86/kvm/vmx/tdx.c +5 -0
@@ -429,6 +429,11 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)return ; } +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) +{ +td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK ); +} +int tdx_dev_ioctl (void __user *argp) {struct kvm_tdx_capabilities __user *user_caps;
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -156,6 +156,8 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp); + +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -ENOSYS ; }static inline void tdx_hardware_unsetup (void ) {}
@@ -176,6 +178,8 @@ static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {}static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; } + +static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif #endif /* __KVM_X86_VMX_X86_OPS_H */
Signature · 1 line
2.25.1
PATCH v12 48/106 KVM: TDX: Retry seamcall when TDX_OPERAND_BUSY with operand SEPT
In reply to [email protected]From: Yuan Yao <[email protected]>
TDX module internally uses locks to protect internal resources. It tries to acquire the locks. If it fails to obtain the lock, it returns TDX_OPERAND_BUSY error without spin because its execution time limitation.
TDX SEAMCALL API reference describes what resources are used. It's known which TDX SEAMCALL can cause contention with which resources. VMM can avoid contention inside the TDX module by avoiding contentious TDX SEAMCALL with, for example, spinlock. Because OS knows better its process scheduling and its scalability, a lock at OS/VMM layer would work better than simply retrying TDX SEAMCALLs.
TDH.MEM.* API except for TDH.MEM.TRACK operates on a secure EPT tree and the TDX module internally tries to acquire the lock of the secure EPT tree. They return TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT in case of failure to get the lock. TDX KVM allows sept callbacks to return error so that TDP MMU layer can retry.
TDH.VP.ENTER is an exception with zero-step attack mitigation. Normally TDH.VP.ENTER uses only TD vcpu resources and it doesn't cause contention. When a zero-step attack is suspected, it obtains a secure EPT tree lock and tracks the GPAs causing a secure EPT fault. Thus TDG.VP.ENTER may result in TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT. Also TDH.MEM.* SEAMCALLs may result in TDX_OPERAN_BUSY | TDX_OPERAND_ID_SEPT.
Retry TDX TDH.MEM.* API and TDH.VP.ENTER on the error because the error is a rare event caused by zero-step attack mitigation and spinlock can not be used for TDH.VP.ENTER due to indefinite time execution.
--- arch/x86/kvm/vmx/tdx_ops.h | 42 ++++++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx_ops.h +36 -6
@@ -18,6 +18,36 @@void pr_tdx_error (u64 op,u64 error_code,const struct tdx_module_output *out); +/* +* TDX module acquires its internal lock for resources. It doesn't spin to get +* locks because of its restrictions of allowed execution time. Instead, it +* returns TDX_OPERAND_BUSY with an operand id. +* +* Multiple VCPUs can operate on SEPT. Also with zero-step attack mitigation, +* TDH.VP.ENTER may rarely acquire SEPT lock and release it when zero-step +* attack is suspected. It results in TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT +* with TDH.MEM.* operation. Note: TDH.MEM.TRACK is an exception. +* +* Because TDP MMU uses read lock for scalability, spin lock around SEAMCALL +* spoils TDP MMU effort. Retry several times with the assumption that SEPT +* lock contention is rare. But don't loop forever to avoid lockup. Let TDP +* MMU retry. +*/ +#define TDX_ERROR_SEPT_BUSY (TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT) + +static inline u64 seamcall_sept (u64 op,u64 rcx,u64 rdx,u64 r8,u64 r9, +struct tdx_module_output *out) +{ +#define SEAMCALL_RETRY_MAX 16 +int retry =SEAMCALL_RETRY_MAX ; +u64 ret; + +do { + ret =__seamcall (op, rcx, rdx, r8, r9, out); + }while (ret ==TDX_ERROR_SEPT_BUSY && retry-- >0 ); +return ret; +} +static inline u64 tdh_mng_addcx (hpa_t tdr,hpa_t addr) {clflush_cache_range (__va (addr),PAGE_SIZE );
@@ -28,14 +58,14 @@ static inline u64 tdh_mem_page_add(hpa_t tdr, gpa_t gpa, hpa_t hpa, hpa_t source struct tdx_module_output *out) {clflush_cache_range (__va (hpa),PAGE_SIZE ); -return __seamcall (TDH_MEM_PAGE_ADD , gpa, tdr, hpa, source, out); +return seamcall_sept (TDH_MEM_PAGE_ADD , gpa, tdr, hpa, source, out); }static inline u64 tdh_mem_sept_add (hpa_t tdr,gpa_t gpa,int level,hpa_t page,struct tdx_module_output *out) {clflush_cache_range (__va (page),PAGE_SIZE ); -return __seamcall (TDH_MEM_SEPT_ADD , gpa | level, tdr, page,0 , out); +return seamcall_sept (TDH_MEM_SEPT_ADD , gpa | level, tdr, page,0 , out); } static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int level,
@@ -61,13 +91,13 @@ static inline u64 tdh_mem_page_aug(hpa_t tdr, gpa_t gpa, hpa_t hpa, struct tdx_module_output *out) {clflush_cache_range (__va (hpa),PAGE_SIZE ); -return __seamcall (TDH_MEM_PAGE_AUG , gpa, tdr, hpa,0 , out); +return seamcall_sept (TDH_MEM_PAGE_AUG , gpa, tdr, hpa,0 , out); }static inline u64 tdh_mem_range_block (hpa_t tdr,gpa_t gpa,int level,struct tdx_module_output *out) { -return __seamcall (TDH_MEM_RANGE_BLOCK , gpa | level, tdr,0 ,0 , out); +return seamcall_sept (TDH_MEM_RANGE_BLOCK , gpa | level, tdr,0 ,0 , out); }static inline u64 tdh_mng_key_config (hpa_t tdr)
@@ -149,7 +179,7 @@ static inline u64 tdh_phymem_page_reclaim(hpa_t page,static inline u64 tdh_mem_page_remove (hpa_t tdr,gpa_t gpa,int level,struct tdx_module_output *out) { -return __seamcall (TDH_MEM_PAGE_REMOVE , gpa | level, tdr,0 ,0 , out); +return seamcall_sept (TDH_MEM_PAGE_REMOVE , gpa | level, tdr,0 ,0 , out); }static inline u64 tdh_sys_lp_shutdown (void )
@@ -165,7 +195,7 @@ static inline u64 tdh_mem_track(hpa_t tdr)static inline u64 tdh_mem_range_unblock (hpa_t tdr,gpa_t gpa,int level,struct tdx_module_output *out) { -return __seamcall (TDH_MEM_RANGE_UNBLOCK , gpa | level, tdr,0 ,0 , out); +return seamcall_sept (TDH_MEM_RANGE_UNBLOCK , gpa | level, tdr,0 ,0 , out); }static inline u64 tdh_phymem_cache_wb (bool resume)
Signature · 1 line
2.25.1
PATCH v12 49/106 KVM: TDX: TDP MMU TDX support
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implement hooks of TDP MMU for TDX backend. TLB flush, TLB shootdown, propagating the change private EPT entry to Secure EPT and freeing Secure EPT page. TLB flush handles both shared EPT and private EPT. It flushes shared EPT same as VMX. It also waits for the TDX TLB shootdown. For the hook to free Secure EPT page, unlinks the Secure EPT page from the Secure EPT so that the page can be freed to OS.
Propagate the entry change to Secure EPT. The possible entry changes are present -> non-present(zapping) and non-present -> present(population). On population just link the Secure EPT page or the private guest page to the Secure EPT by TDX SEAMCALL. Because TDP MMU allows concurrent zapping/population, zapping requires synchronous TLB shoot down with the frozen EPT entry. It zaps the secure entry, increments TLB counter, sends IPI to remote vcpus to trigger TLB flush, and then unlinks the private guest page from the Secure EPT. For simplicity, batched zapping with exclude lock is handled as concurrent zapping. Although it's inefficient, it can be optimized in the future.
For MMIO SPTE, the spte value changes as follows.
initial value (suppress VE bit is set)
-> Guest issues MMIO and triggers EPT violation
-> KVM updates SPTE value to MMIO value (suppress VE bit is cleared) -> Guest MMIO resumes. It triggers VE exception in guest TD -> Guest VE handler issues TDG.VP.VMCALL<MMIO>
-> KVM handles MMIO
-> Guest VE handler resumes its execution after MMIO instruction
---
Changes v11 to v12
- removed tlb_remote_flush_with_range mothod. Instead disable TDX if hyper-v support populate tlb_remote_flush and tlb_remote_flush_with_range
--- arch/x86/kvm/mmu/spte.c | 3 +- arch/x86/kvm/vmx/main.c | 71 ++++++++- arch/x86/kvm/vmx/tdx.c | 302 ++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 7 + arch/x86/kvm/vmx/x86_ops.h | 4 + 5 files changed, 380 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/spte.c +2 -1
@@ -74,7 +74,8 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)u64 spte =generation_mmio_spte_mask (gen);u64 gpa = gfn <<PAGE_SHIFT ; -WARN_ON_ONCE (!vcpu->kvm->arch.shadow_mmio_value); +WARN_ON_ONCE (!vcpu->kvm->arch.shadow_mmio_value && + !kvm_gfn_shared_mask (vcpu->kvm)); access &= shadow_mmio_access_mask; spte |= vcpu->kvm->arch.shadow_mmio_value | access;
arch/x86/kvm/vmx/main.c +66 -5
96 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.c +301 -1
334 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +7 -0
@@ -18,6 +18,7 @@ struct kvm_tdx {int hkid;bool finalized; +atomic_t tdh_mem_track;u64 tsc_offset; };
@@ -165,6 +166,12 @@ static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 fielreturn out.r8; } +static __always_inline int pg_level_to_tdx_sept_level(enum pg_level level) +{ +WARN_ON_ONCE (level ==PG_LEVEL_NONE ); +return level -1 ; +} +#else struct kvm_tdx { struct kvm kvm;
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -157,6 +157,8 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp); +void tdx_flush_tlb (struct kvm_vcpu *vcpu); +int tdx_sept_tlb_remote_flush (struct kvm *kvm);void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);#else static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -ENOSYS ; }
@@ -179,6 +181,8 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; } +static inline void tdx_flush_tlb (struct kvm_vcpu *vcpu) {} +static inline int tdx_sept_tlb_remote_flush (struct kvm *kvm) {return 0 ; }static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif
Signature · 1 line
2.25.1
PATCH v12 50/106 KVM: TDX: MTRR: implement get_mt_mask() for TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because TDX virtualize cpuid[0x1].EDX[MTRR: bit 12] to fixed 1, guest TD thinks MTRR is supported. Although TDX supports only WB for private GPA, it's desirable to support MTRR for shared GPA. As guest access to MTRR MSRs causes #VE and KVM/x86 tracks the values of MTRR MSRs, the remining part is to implement get_mt_mask method for TDX for shared GPA.
Pass around shared bit from kvm fault handler to get_mt_mask method so that it can determine if the gfn is shared or private. Implement get_mt_mask() following vmx case for shared GPA and return WB for private GPA. the existing vmx_get_mt_mask() can't be directly used as CPU state(CR0.CD) is protected. GFN passed to kvm_mtrr_check_gfn_range_consistency() should include shared bit.
---
Changes from v11 to V12
- Make common function for VMX and TDX
- pass around shared bit from KVM fault handler to get_mt_mask method
- updated commit message
--- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/mmu/spte.c | 5 +++-- arch/x86/kvm/mmu/spte.h | 2 +- arch/x86/kvm/vmx/common.h | 2 ++ arch/x86/kvm/vmx/main.c | 11 ++++++++++- arch/x86/kvm/vmx/tdx.c | 17 +++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 5 +++-- arch/x86/kvm/vmx/x86_ops.h | 2 ++ 8 files changed, 39 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -4555,7 +4555,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)if (shadow_memtype_mask &&kvm_arch_has_noncoherent_dma (vcpu->kvm)) {for ( ; fault->max_level >PG_LEVEL_4K ; --fault->max_level) {int page_num =KVM_PAGES_PER_HPAGE (fault->max_level); -gfn_t base = fault->gfn & ~(page_num -1 ); +gfn_t base = (fault->addr >>PAGE_SHIFT ) & ~(page_num -1 );if (kvm_mtrr_check_gfn_range_consistency (vcpu, base, page_num))break ;
arch/x86/kvm/mmu/spte.c +3 -2
@@ -137,13 +137,14 @@ bool spte_has_volatile_bits(u64 spte)bool make_spte (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp,const struct kvm_memory_slot *slot, -unsigned int pte_access,gfn_t gfn,kvm_pfn_t pfn, +unsigned int pte_access,gfn_t gfn_including_shared,kvm_pfn_t pfn,u64 old_spte,bool prefetch,bool can_unsync,bool host_writable,u64 *new_spte) {int level = sp->role.level;u64 spte =SPTE_MMU_PRESENT_MASK ;bool wrprot = false; +gfn_t gfn = gfn_including_shared & ~kvm_gfn_shared_mask (vcpu->kvm);WARN_ON_ONCE (!pte_access && !shadow_present_mask);
@@ -191,7 +192,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, spte |=PT_PAGE_SIZE_MASK ;if (shadow_memtype_mask) - spte |=static_call (kvm_x86_get_mt_mask)(vcpu, gfn, + spte |=static_call (kvm_x86_get_mt_mask)(vcpu, gfn_including_shared,kvm_is_mmio_pfn (pfn));if (host_writable) spte |= shadow_host_writable_mask;
arch/x86/kvm/mmu/spte.h +1 -1
@@ -481,7 +481,7 @@ bool spte_has_volatile_bits(u64 spte);bool make_spte (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp,const struct kvm_memory_slot *slot, -unsigned int pte_access,gfn_t gfn,kvm_pfn_t pfn, +unsigned int pte_access,gfn_t gfn_including_shared,kvm_pfn_t pfn,u64 old_spte,bool prefetch,bool can_unsync,bool host_writable,u64 *new_spte); u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte,
arch/x86/kvm/vmx/common.h +2 -0
@@ -6,6 +6,8 @@#include "mmu.h" +u8 __vmx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio,bool check_cr0_cd); +static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa,unsigned long exit_qualification) {
arch/x86/kvm/vmx/main.c +10 -1
@@ -3,6 +3,7 @@#include "x86_ops.h" #include "mmu.h" +#include "common.h" #include "vmx.h" #include "nested.h" #include "mmu.h"
@@ -206,6 +207,14 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); } +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_mt_mask (vcpu, gfn, is_mmio); + +return __vmx_get_mt_mask (vcpu, gfn, is_mmio, true); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -326,7 +335,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_tss_addr = vmx_set_tss_addr, .set_identity_map_addr = vmx_set_identity_map_addr, - .get_mt_mask = vmx_get_mt_mask, + .get_mt_mask = vt_get_mt_mask, .get_exit_info = vmx_get_exit_info,
arch/x86/kvm/vmx/tdx.c +17 -0
@@ -5,6 +5,7 @@#include "capabilities.h" #include "x86_ops.h" +#include "common.h" #include "tdx.h" #include "vmx.h" #include "x86.h"
@@ -345,6 +346,22 @@ int tdx_vm_init(struct kvm *kvm)return 0 ; } +u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +/* TDX private GPA is always WB. */ +if (kvm_gfn_private (vcpu->kvm, gfn)) { +/* MMIO is only for shared GPA. */ +WARN_ON_ONCE (is_mmio); +return MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ; + } + +/* Drop shared bit as MTRR doesn't know about shared bit. */ + gfn =kvm_gfn_private (vcpu->kvm, gfn); + +/* As TDX enforces CR0.CD to 0, pass check_cr0_cd = false. */ +return __vmx_get_mt_mask (vcpu, gfn, is_mmio, false); +} +int tdx_vcpu_create (struct kvm_vcpu *vcpu) {struct kvm_cpuid_entry2 *e;
arch/x86/kvm/vmx/vmx.c +3 -2
@@ -7552,7 +7552,8 @@ int vmx_vm_init(struct kvm *kvm)return 0 ; } -u8 vmx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +u8 __vmx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio, +bool check_cr0_cd) {u8 cache;
@@ -7580,7 +7581,7 @@ u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)if (!kvm_arch_has_noncoherent_dma (vcpu->kvm))return (MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ) |VMX_EPT_IPAT_BIT ; -if (kvm_read_cr0 (vcpu) &X86_CR0_CD ) { +if (check_cr0_cd &&kvm_read_cr0 (vcpu) &X86_CR0_CD ) {if (kvm_check_has_quirk (vcpu->kvm,KVM_X86_QUIRK_CD_NW_CLEARED )) cache =MTRR_TYPE_WRBACK ;else
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -154,6 +154,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);int tdx_vcpu_create (struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event); +u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -178,6 +179,7 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPstatic inline int tdx_vcpu_create (struct kvm_vcpu *vcpu) {return -EOPNOTSUPP ; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {} +static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
Re: [PATCH v12 050/106] KVM: TDX: MTRR: implement get_mt_mask() for TDX
Quoted 150 lines · On 2/27/2023 4:22 PM, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Because TDX virtualize cpuid[0x1].EDX[MTRR: bit 12] to fixed 1, guest TD thinks MTRR is supported. Although TDX supports only WB for private GPA, it's desirable to support MTRR for shared GPA. As guest access to MTRR MSRs causes #VE and KVM/x86 tracks the values of MTRR MSRs, the remining part is to implement
get_mt_maskmethod for TDX for shared GPA.Pass around shared bit from kvm fault handler to
get_mt_maskmethod so that it can determine if the gfn is shared or private. Implementget_mt_mask()following vmx case for shared GPA and return WB for private GPA. the existingvmx_get_mt_mask()can't be directly used as CPU state(CR0.CD) is protected. GFN passed tokvm_mtrr_check_gfn_range_consistency()should include shared bit.Suggested-by Kai Huang <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>---
Changes from v11 to V12
- Make common function for VMX and TDX
- pass around shared bit from KVM fault handler toget_mt_maskmethod
- updated commit message--- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/mmu/spte.c | 5 +++-- arch/x86/kvm/mmu/spte.h | 2 +- arch/x86/kvm/vmx/common.h | 2 ++ arch/x86/kvm/vmx/main.c | 11 ++++++++++- arch/x86/kvm/vmx/tdx.c | 17 +++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 5 +++-- arch/x86/kvm/vmx/x86_ops.h | 2 ++ 8 files changed, 39 insertions(+), 7 deletions(-)arch/x86/kvm/mmu/mmu.c +1
-1@@ -4555,7 +4555,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)if (shadow_memtype_mask &&kvm_arch_has_noncoherent_dma (vcpu->kvm)) {for ( ; fault->max_level >PG_LEVEL_4K ; --fault->max_level) {int page_num =KVM_PAGES_PER_HPAGE (fault->max_level); -gfn_t base = fault->gfn & ~(page_num -1 ); +gfn_t base = (fault->addr >>PAGE_SHIFT ) & ~(page_num -1 );if (kvm_mtrr_check_gfn_range_consistency (vcpu, base, page_num))break ;arch/x86/kvm/mmu/spte.c +3
-2@@ -137,13 +137,14 @@ bool spte_has_volatile_bits(u64 spte)bool make_spte (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp,const struct kvm_memory_slot *slot, -unsigned int pte_access,gfn_t gfn,kvm_pfn_t pfn, +unsigned int pte_access,gfn_t gfn_including_shared,kvm_pfn_t pfn,u64 old_spte,bool prefetch,bool can_unsync,bool host_writable,u64 *new_spte) {int level = sp->role.level;u64 spte =SPTE_MMU_PRESENT_MASK ;bool wrprot = false; +gfn_t gfn = gfn_including_shared & ~kvm_gfn_shared_mask (vcpu->kvm);WARN_ON_ONCE (!pte_access && !shadow_present_mask);@@ -191,7 +192,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, spte |=PT_PAGE_SIZE_MASK ;if (shadow_memtype_mask) - spte |=static_call (kvm_x86_get_mt_mask)(vcpu, gfn, + spte |=static_call (kvm_x86_get_mt_mask)(vcpu, gfn_including_shared,kvm_is_mmio_pfn (pfn));if (host_writable) spte |= shadow_host_writable_mask;arch/x86/kvm/mmu/spte.h +1
-1@@ -481,7 +481,7 @@ bool spte_has_volatile_bits(u64 spte);bool make_spte (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp,const struct kvm_memory_slot *slot, -unsigned int pte_access,gfn_t gfn,kvm_pfn_t pfn, +unsigned int pte_access,gfn_t gfn_including_shared,kvm_pfn_t pfn,u64 old_spte,bool prefetch,bool can_unsync,bool host_writable,u64 *new_spte); u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte,arch/x86/kvm/vmx/common.h +2
-0@@ -6,6 +6,8 @@#include "mmu.h" +u8 __vmx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio,bool check_cr0_cd); +static inline int __vmx_handle_ept_violation (struct kvm_vcpu *vcpu,gpa_t gpa,unsigned long exit_qualification) {arch/x86/kvm/vmx/main.c +10
-1@@ -3,6 +3,7 @@#include "x86_ops.h" #include "mmu.h" +#include "common.h" #include "vmx.h" #include "nested.h" #include "mmu.h" @@ -206,6 +207,14 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); } +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_mt_mask (vcpu, gfn, is_mmio); + +return __vmx_get_mt_mask (vcpu, gfn, is_mmio, true); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))@@ -326,7 +335,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_tss_addr = vmx_set_tss_addr, .set_identity_map_addr = vmx_set_identity_map_addr, - .get_mt_mask = vmx_get_mt_mask, + .get_mt_mask = vt_get_mt_mask, .get_exit_info = vmx_get_exit_info,arch/x86/kvm/vmx/tdx.c +1
-0@@ -5,6 +5,7 @@#include "capabilities.h" #include "x86_ops.h" +#include "common.h" #include "tdx.h" #include "vmx.h" #include "x86.h"
arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 352
Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +4
-0@@ -345,6 +346,22 @@ int tdx_vm_init(struct kvm *kvm)return 0 ; } +u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +/* TDX private GPA is always WB. */ +if (kvm_gfn_private (vcpu->kvm, gfn)) {
kvm_gfn_private() only drops shared bit instead of checking if gfn is private of shared. It should be !(gfn & shared mask) as mentioned in your v11 series :)
Quoted 38 lines
arch/x86/kvm/vmx/tdx.c +12
-0+/* MMIO is only for shared GPA. */ +WARN_ON_ONCE (is_mmio); +return MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ; + } + +/* Drop shared bit as MTRR doesn't know about shared bit. */ + gfn =kvm_gfn_private (vcpu->kvm, gfn); + +/* As TDX enforces CR0.CD to 0, pass check_cr0_cd = false. */ +return __vmx_get_mt_mask (vcpu, gfn, is_mmio, false); +} +int tdx_vcpu_create (struct kvm_vcpu *vcpu) {struct kvm_cpuid_entry2 *e;arch/x86/kvm/vmx/vmx.c +3
-2@@ -7552,7 +7552,8 @@ int vmx_vm_init(struct kvm *kvm)return 0 ; } -u8 vmx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +u8 __vmx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio, +bool check_cr0_cd) {u8 cache;@@ -7580,7 +7581,7 @@ u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)if (!kvm_arch_has_noncoherent_dma (vcpu->kvm))return (MTRR_TYPE_WRBACK <<VMX_EPT_MT_EPTE_SHIFT ) |VMX_EPT_IPAT_BIT ; -if (kvm_read_cr0 (vcpu) &X86_CR0_CD ) { +if (check_cr0_cd &&kvm_read_cr0 (vcpu) &X86_CR0_CD ) {if (kvm_check_has_quirk (vcpu->kvm,KVM_X86_QUIRK_CD_NW_CLEARED )) cache =MTRR_TYPE_WRBACK ;else
Re: [PATCH v12 050/106] KVM: TDX: MTRR: implement get_mt_mask() for TDX
Quoted 16 lines · Chenyi Qiang
Quoted 16 lines
arch/x86/kvm/vmx/tdx.c +1
-0@@ -5,6 +5,7 @@#include "capabilities.h" #include "x86_ops.h" +#include "common.h" #include "tdx.h" #include "vmx.h" #include "x86.h" @@ -345,6 +346,22 @@ int tdx_vm_init(struct kvm *kvm)return 0 ; }
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 352
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4
-0+u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) +{ +/* TDX private GPA is always WB. */ +if (kvm_gfn_private (vcpu->kvm, gfn)) {
kvm_gfn_private()only drops shared bit instead of checking if gfn is private of shared. It should be !(gfn & shared mask) as mentioned in your v11 series :)
Oops. Will fix it. thanks for catching it.
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v12 51/106 The start of TDX KVM patch series: TD finalization
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD finalization.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +3 -2
@@ -11,6 +11,7 @@ What qemu can do - TDX VM TYPE is exposed to Qemu. - Qemu can create/destroy guest of TDX vm type. - Qemu can create/destroy vcpu of TDX vm type. +- Qemu can populate initial guest memory image. Patch Layer status ------------------
@@ -19,8 +20,8 @@ Patch Layer status * TDX architectural definitions: Applied * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied -* TDX EPT violation: Applying -* TD finalization: Not yet +* TDX EPT violation: Applied +* TD finalization: Applying * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet
Signature · 1 line
2.25.1
PATCH v12 52/106 KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by TDX
In reply to [email protected]From: Sean Christopherson <[email protected]>
Introduce a helper to directly (pun intended) fault-in a TDP page without having to go through the full page fault path. This allows TDX to get the resulting pfn and also allows the RET_PF_* enums to stay in mmu.c where they belong.
--- arch/x86/kvm/mmu.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 49 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)
arch/x86/kvm/mmu.h +3 -0
@@ -154,6 +154,9 @@ static inline void kvm_mmu_load_pgd(struct kvm_vcpu *vcpu) vcpu->arch.mmu->root_role.level); } +kvm_pfn_t kvm_mmu_map_tdp_page (struct kvm_vcpu *vcpu,gpa_t gpa, +u32 error_code,int max_level); +/* * Check if a given access (described through the I/D, W/R and U/S bits of a * page fault error code pfec) causes a permission fault with the given PTE
arch/x86/kvm/mmu/mmu.c +49 -0
55 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 53/106 KVM: TDX: Create initial guest memory
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because the guest memory is protected in TDX, the creation of the initial guest memory requires a dedicated TDX module API, tdh_mem_page_add, instead of directly copying the memory contents into the guest memory in the case of the default VM type. KVM MMU page fault handler callback, private_page_add, handles it.
Define new subcommand, KVM_TDX_INIT_MEM_REGION, of VM-scoped KVM_MEMORY_ENCRYPT_OP. It assigns the guest page, copies the initial memory contents into the guest memory, encrypts the guest memory. At the same time, optionally it extends memory measurement of the TDX guest. It calls the KVM MMU page fault(EPT-violation) handler to trigger the callbacks for it.
--- arch/x86/include/uapi/asm/kvm.h | 9 ++ arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/vmx/tdx.c | 156 +++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 2 + tools/arch/x86/include/uapi/asm/kvm.h | 9 ++ 5 files changed, 172 insertions(+), 5 deletions(-)
arch/x86/include/uapi/asm/kvm.h +9 -0
@@ -537,6 +537,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 ,KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU , +KVM_TDX_INIT_MEM_REGION , KVM_TDX_CMD_NR_MAX, };
@@ -605,4 +606,12 @@ struct kvm_tdx_init_vm { struct kvm_cpuid2 cpuid; }; +#define KVM_TDX_MEASURE_MEMORY_REGION (1UL << 0) + +struct kvm_tdx_init_mem_region { +__u64 source_addr; +__u64 gpa; +__u64 nr_pages; +}; +#endif /* _ASM_X86_KVM_H */
arch/x86/kvm/mmu/mmu.c +1 -0
@@ -5587,6 +5587,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) out:return r; } +EXPORT_SYMBOL (kvm_mmu_load);void kvm_mmu_unload (struct kvm_vcpu *vcpu) {
arch/x86/kvm/vmx/tdx.c +151 -5
195 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +2 -0
@@ -17,6 +17,8 @@ struct kvm_tdx {u64 xfam;int hkid; +hpa_t source_pa; +bool finalized;atomic_t tdh_mem_track;
tools/arch/x86/include/uapi/asm/kvm.h +9 -0
@@ -537,6 +537,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_CAPABILITIES =0 ,KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU , +KVM_TDX_INIT_MEM_REGION , KVM_TDX_CMD_NR_MAX, };
@@ -614,4 +615,12 @@ struct kvm_tdx_init_vm { }; }; +#define KVM_TDX_MEASURE_MEMORY_REGION (1UL << 0) + +struct kvm_tdx_init_mem_region { +__u64 source_addr; +__u64 gpa; +__u64 nr_pages; +}; +#endif /* _ASM_X86_KVM_H */
Signature · 1 line
2.25.1
PATCH v12 54/106 KVM: TDX: Finalize VM initialization
In reply to [email protected]From: Isaku Yamahata <[email protected]>
To protect the initial contents of the guest TD, the TDX module measures the guest TD during the build process as SHA-384 measurement. The measurement of the guest TD contents needs to be completed to make the guest TD ready to run.
Add a new subcommand, KVM_TDX_FINALIZE_VM, for VM-scoped KVM_MEMORY_ENCRYPT_OP to finalize the measurement and mark the TDX VM ready to run.
--- arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/vmx/tdx.c | 31 +++++++++++++++++++++++++++ tools/arch/x86/include/uapi/asm/kvm.h | 1 + 3 files changed, 33 insertions(+)
arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -538,6 +538,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU ,KVM_TDX_INIT_MEM_REGION , +KVM_TDX_FINALIZE_VM , KVM_TDX_CMD_NR_MAX, };
arch/x86/kvm/vmx/tdx.c +31 -0
@@ -1320,6 +1320,34 @@ static int tdx_init_mem_region(struct kvm *kvm, struct kvm_tdx_cmd *cmd)return ret; } +static int tdx_td_finalizemr (struct kvm *kvm) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +u64 err; + +if (!is_hkid_assigned (kvm_tdx) ||is_td_finalized (kvm_tdx)) +return -EINVAL ; + + err =tdh_mr_finalize (kvm_tdx->tdr_pa); +if (WARN_ON_ONCE (err)) { +pr_tdx_error (TDH_MR_FINALIZE , err,NULL ); +return -EIO ; + } + +/* +* Blindly do TDH_MEM_TRACK after finalizing the measurement to handle +* the case where SEPT entries were zapped/blocked, e.g. from failed +* NUMA balancing, after they were added to the TD via +* tdx_init_mem_region(). TDX module doesn't allow TDH_MEM_TRACK prior +* to TDH.MR.FINALIZE, and conversely requires TDH.MEM.TRACK for entries +* that were TDH.MEM.RANGE.BLOCK'd prior to TDH.MR.FINALIZE. +*/ + (void )tdh_mem_track (to_kvm_tdx (kvm)->tdr_pa); + + kvm_tdx->finalized = true; +return 0 ; +} +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {struct kvm_tdx_cmd tdx_cmd;
@@ -1339,6 +1367,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)case KVM_TDX_INIT_MEM_REGION : r =tdx_init_mem_region (kvm, &tdx_cmd);break ; +case KVM_TDX_FINALIZE_VM : + r =tdx_td_finalizemr (kvm); +break ;default : r = -EINVAL ; goto out;
tools/arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -538,6 +538,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_INIT_VM ,KVM_TDX_INIT_VCPU ,KVM_TDX_INIT_MEM_REGION , +KVM_TDX_FINALIZE_VM , KVM_TDX_CMD_NR_MAX, };
Signature · 1 line
2.25.1
PATCH v12 55/106 The start of TDX KVM patch series: TD vcpu enter/exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD vcpu enter/exit.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +3 -2
@@ -12,6 +12,7 @@ What qemu can do - Qemu can create/destroy guest of TDX vm type. - Qemu can create/destroy vcpu of TDX vm type. - Qemu can populate initial guest memory image. +- Qemu can finalize guest TD. Patch Layer status ------------------
@@ -21,8 +22,8 @@ Patch Layer status * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied * TDX EPT violation: Applied -* TD finalization: Applying -* TD vcpu enter/exit: Not yet +* TD finalization: Applied +* TD vcpu enter/exit: Applying * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied
Signature · 1 line
2.25.1
PATCH v12 56/106 KVM: TDX: Add helper assembly function to TDX vcpu
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX defines an API to run TDX vcpu with its own ABI. Define an assembly helper function to run TDX vcpu to hide the special ABI so that C code can call it with function call ABI.
--- arch/x86/include/asm/tdx.h | 3 +- arch/x86/kvm/vmx/vmenter.S | 156 +++++++++++++++++++++++++++++++++++++ 2 files changed, 158 insertions(+), 1 deletion(-)
arch/x86/include/asm/tdx.h +2 -1
@@ -18,7 +18,8 @@* Bits 47:40 == 0xFF indicate Reserved status code class that never used by * TDX module. */ -#define TDX_ERROR _BITUL(63) +#define TDX_ERROR_BIT 63 +#define TDX_ERROR _BITUL(TDX_ERROR_BIT)#define TDX_SW_ERROR (TDX_ERROR | GENMASK_ULL(47, 40))#define TDX_SEAMCALL_VMFAILINVALID (TDX_SW_ERROR | _UL(0xFFFF0000))
arch/x86/kvm/vmx/vmenter.S +156 -0
171 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 57/106 KVM: TDX: Implement TDX vcpu enter/exit path
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This patch implements running TDX vcpu. Once vcpu runs on the logical processor (LP), the TDX vcpu is associated with it. When the TDX vcpu moves to another LP, the TDX vcpu needs to flush its status on the LP. When destroying TDX vcpu, it needs to complete flush and flush cpu memory cache. Track which LP the TDX vcpu run and flush it as necessary.
Do nothing on sched_in event as TDX doesn't support pause loop.
TDX vcpu execution requires restoring PMU debug store after returning back to KVM because the TDX module unconditionally resets the value. To reuse the existing code, export perf_restore_debug_store.
--- arch/x86/kvm/vmx/main.c | 21 +++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 32 ++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ arch/x86/kvm/x86.c | 1 + 5 files changed, 87 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +19 -2
@@ -148,6 +148,23 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)vmx_vcpu_reset (vcpu, init_event); } +static int vt_vcpu_pre_run (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +/* Unconditionally continue to vcpu_run(). */ +return 1 ; + +return vmx_vcpu_pre_run (vcpu); +} + +static fastpath_t vt_vcpu_run (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_vcpu_run (vcpu); + +return vmx_vcpu_run (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -301,8 +318,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .flush_tlb_gva = vt_flush_tlb_gva, .flush_tlb_guest = vt_flush_tlb_guest, - .vcpu_pre_run = vmx_vcpu_pre_run, - .vcpu_run = vmx_vcpu_run, + .vcpu_pre_run = vt_vcpu_pre_run, + .vcpu_run = vt_vcpu_run, .handle_exit = vmx_handle_exit, .skip_emulated_instruction = vmx_skip_emulated_instruction, .update_emulated_instruction = vmx_update_emulated_instruction,
arch/x86/kvm/vmx/tdx.c +32 -0
@@ -11,6 +11,9 @@#include "x86.h" #include "mmu.h" +#include <trace/events/kvm.h> +#include "trace.h" +#undef pr_fmt#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt
@@ -464,6 +467,35 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)return ; } +u64 __tdx_vcpu_run (hpa_t tdvpr,void *regs,u32 regs_mask); + +static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu, +struct vcpu_tdx *tdx) +{ +guest_enter_irqoff (); + tdx->exit_reason.full =__tdx_vcpu_run (tdx->tdvpr_pa, vcpu->arch.regs,0 ); +guest_exit_irqoff (); +} + +fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + +if (unlikely (vcpu->kvm->vm_bugged)) { + tdx->exit_reason.full =TDX_NON_RECOVERABLE_VCPU ; +return EXIT_FASTPATH_NONE ; + } + +trace_kvm_entry (vcpu); + +tdx_vcpu_enter_exit (vcpu, tdx); + + vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ; +trace_kvm_exit (vcpu,KVM_ISA_VMX ); + +return EXIT_FASTPATH_NONE ; +} +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) {td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK );
arch/x86/kvm/vmx/tdx.h +33 -0
@@ -25,12 +25,45 @@ struct kvm_tdx {u64 tsc_offset; }; +union tdx_exit_reason { +struct { +/* 31:0 mirror the VMX Exit Reason format */ +u64 basic :16 ; +u64 reserved16 :1 ; +u64 reserved17 :1 ; +u64 reserved18 :1 ; +u64 reserved19 :1 ; +u64 reserved20 :1 ; +u64 reserved21 :1 ; +u64 reserved22 :1 ; +u64 reserved23 :1 ; +u64 reserved24 :1 ; +u64 reserved25 :1 ; +u64 bus_lock_detected :1 ; +u64 enclave_mode :1 ; +u64 smi_pending_mtf :1 ; +u64 smi_from_vmx_root :1 ; +u64 reserved30 :1 ; +u64 failed_vmentry :1 ; + +/* 63:32 are TDX specific */ +u64 details_l1 :8 ; +u64 class :8 ; +u64 reserved61_48 :14 ; +u64 non_recoverable :1 ; +u64 error :1 ; + }; +u64 full; +}; +struct vcpu_tdx {struct kvm_vcpu vcpu;unsigned long tdvpr_pa;unsigned long *tdvpx_pa; +union tdx_exit_reason exit_reason; +bool initialized;/*
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -154,6 +154,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);int tdx_vcpu_create (struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event); +fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -179,6 +180,7 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPstatic inline int tdx_vcpu_create (struct kvm_vcpu *vcpu) {return -EOPNOTSUPP ; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {} +static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; }static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
arch/x86/kvm/x86.c +1 -0
@@ -305,6 +305,7 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { };u64 __read_mostly host_xcr0; +EXPORT_SYMBOL_GPL (host_xcr0);static struct kmem_cache *x86_emulator_cache;
Signature · 1 line
2.25.1
PATCH v12 58/106 KVM: TDX: vcpu_run: save/restore host state(host kernel gs)
In reply to [email protected]From: Isaku Yamahata <[email protected]>
On entering/exiting TDX vcpu, Preserved or clobbered CPU state is different from VMX case. Add TDX hooks to save/restore host/guest CPU state. Save/restore kernel GS base MSR.
--- arch/x86/kvm/vmx/main.c | 30 ++++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 40 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 4 ++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 76 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +28 -2
@@ -148,6 +148,32 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)vmx_vcpu_reset (vcpu, init_event); } +static void vt_prepare_switch_to_guest (struct kvm_vcpu *vcpu) +{ +/* +* All host state is saved/restored across SEAMCALL/SEAMRET, and the +* guest state of a TD is obviously off limits. Deferring MSRs and DRs +* is pointless because the TDX module needs to load *something* so as +* not to expose guest state. +*/ +if (is_td_vcpu (vcpu)) { +tdx_prepare_switch_to_guest (vcpu); +return ; + } + +vmx_prepare_switch_to_guest (vcpu); +} + +static void vt_vcpu_put (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +tdx_vcpu_put (vcpu); +return ; + } + +vmx_vcpu_put (vcpu); +} +static int vt_vcpu_pre_run (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -285,9 +311,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_free = vt_vcpu_free, .vcpu_reset = vt_vcpu_reset, - .prepare_switch_to_guest = vmx_prepare_switch_to_guest, + .prepare_switch_to_guest = vt_prepare_switch_to_guest, .vcpu_load = vmx_vcpu_load, - .vcpu_put = vmx_vcpu_put, + .vcpu_put = vt_vcpu_put, .update_exception_bitmap = vmx_update_exception_bitmap, .get_msr_feature = vmx_get_msr_feature,
arch/x86/kvm/vmx/tdx.c +40 -0
@@ -1,5 +1,6 @@// SPDX-License-Identifier: GPL-2.0 #include <linux/cpu.h> +#include <linux/mmu_context.h> #include <asm/tdx.h>
@@ -367,6 +368,7 @@ u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)int tdx_vcpu_create (struct kvm_vcpu *vcpu) { +struct vcpu_tdx *tdx =to_tdx (vcpu);struct kvm_cpuid_entry2 *e;/*
@@ -413,9 +415,45 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.guest_state_protected = !(to_kvm_tdx (vcpu->kvm)->attributes &TDX_TD_ATTRIBUTE_DEBUG ); + tdx->host_state_need_save = true; + tdx->host_state_need_restore = false; +return 0 ; } +void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + +if (!tdx->host_state_need_save) +return ; + +if (likely (is_64bit_mm (current->mm))) + tdx->msr_host_kernel_gs_base = current->thread.gsbase; +else + tdx->msr_host_kernel_gs_base =read_msr (MSR_KERNEL_GS_BASE ); + + tdx->host_state_need_save = false; +} + +static void tdx_prepare_switch_to_host (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + + tdx->host_state_need_save = true; +if (!tdx->host_state_need_restore) +return ; + +wrmsrl (MSR_KERNEL_GS_BASE , tdx->msr_host_kernel_gs_base); + tdx->host_state_need_restore = false; +} + +void tdx_vcpu_put (struct kvm_vcpu *vcpu) +{ +vmx_vcpu_pi_put (vcpu); +tdx_prepare_switch_to_host (vcpu); +} +void tdx_vcpu_free (struct kvm_vcpu *vcpu) {struct vcpu_tdx *tdx =to_tdx (vcpu);
@@ -490,6 +528,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (vcpu, tdx); + tdx->host_state_need_restore = true; + vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;trace_kvm_exit (vcpu,KVM_ISA_VMX );
arch/x86/kvm/vmx/tdx.h +4 -0
@@ -66,6 +66,10 @@ struct vcpu_tdx {bool initialized; +bool host_state_need_save; +bool host_state_need_restore; +u64 msr_host_kernel_gs_base; +/* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work.
arch/x86/kvm/vmx/x86_ops.h +4 -0
@@ -155,6 +155,8 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event);fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu); +void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu); +void tdx_vcpu_put (struct kvm_vcpu *vcpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -181,6 +183,8 @@ static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTSUPP; }static inline void tdx_vcpu_free (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event) {}static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; } +static inline void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {}static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v12 59/106 KVM: TDX: restore host xsave state when exit from the guest TD
In reply to [email protected]From: Isaku Yamahata <[email protected]>
On exiting from the guest TD, xsave state is clobbered. Restore xsave state on TD exit.
--- arch/x86/kvm/vmx/tdx.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
arch/x86/kvm/vmx/tdx.c +18 -0
@@ -2,6 +2,7 @@#include <linux/cpu.h> #include <linux/mmu_context.h> +#include <asm/fpu/xcr.h> #include <asm/tdx.h> #include "capabilities.h"
@@ -505,6 +506,22 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)return ; } +static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm); + +if (static_cpu_has (X86_FEATURE_XSAVE ) && + host_xcr0 != (kvm_tdx->xfam & kvm_caps.supported_xcr0)) +xsetbv (XCR_XFEATURE_ENABLED_MASK , host_xcr0); +if (static_cpu_has (X86_FEATURE_XSAVES ) && +/* PT can be exposed to TD guest regardless of KVM's XSS support */ + host_xss != (kvm_tdx->xfam & (kvm_caps.supported_xss |XFEATURE_MASK_PT ))) +wrmsrl (MSR_IA32_XSS , host_xss); +if (static_cpu_has (X86_FEATURE_PKU ) && + (kvm_tdx->xfam &XFEATURE_MASK_PKRU )) +write_pkru (vcpu->arch.host_pkru); +} +u64 __tdx_vcpu_run (hpa_t tdvpr,void *regs,u32 regs_mask); static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu,
@@ -528,6 +545,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (vcpu, tdx); +tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true; vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;
Signature · 1 line
2.25.1
PATCH v12 60/106 KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr
In reply to [email protected]From: Chao Gao <[email protected]>
Several MSRs are constant and only used in userspace(ring 3). But VMs may have different values. KVM uses kvm_set_user_return_msr() to switch to guest's values and leverages user return notifier to restore them when the kernel is to return to userspace. To eliminate unnecessary wrmsr, KVM also caches the value it wrote to an MSR last time.
TDX module unconditionally resets some of these MSRs to architectural INIT state on TD exit. It makes the cached values in kvm_user_return_msrs are inconsistent with values in hardware. This inconsistency needs to be fixed. Otherwise, it may mislead kvm_on_user_return() to skip restoring some MSRs to the host's values. kvm_set_user_return_msr() can help correct this case, but it is not optimal as it always does a wrmsr. So, introduce a variation of kvm_set_user_return_msr() to update cached values and skip that wrmsr.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -2170,6 +2170,7 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low,int kvm_add_user_return_msr (u32 msr);int kvm_find_user_return_msr (u32 msr);int kvm_set_user_return_msr (unsigned index,u64 val,u64 mask); +void kvm_user_return_update_cache (unsigned int index,u64 val);static inline bool kvm_is_supported_user_return_msr (u32 msr) {
arch/x86/kvm/x86.c +20 -5
@@ -430,6 +430,15 @@ static void kvm_user_return_msr_cpu_online(void) } } +static void kvm_user_return_register_notifier (struct kvm_user_return_msrs *msrs) +{ +if (!msrs->registered) { + msrs->urn.on_user_return = kvm_on_user_return; +user_return_notifier_register (&msrs->urn); + msrs->registered = true; + } +} +int kvm_set_user_return_msr (unsigned slot,u64 value,u64 mask) {unsigned int cpu =smp_processor_id ();
@@ -444,15 +453,21 @@ int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask)return 1 ; msrs->values[slot].curr = value; -if (!msrs->registered) { - msrs->urn.on_user_return = kvm_on_user_return; -user_return_notifier_register (&msrs->urn); - msrs->registered = true; - } +kvm_user_return_register_notifier (msrs);return 0 ; }EXPORT_SYMBOL_GPL (kvm_set_user_return_msr); +/* Update the cache, "curr", and register the notifier */ +void kvm_user_return_update_cache (unsigned int slot,u64 value) +{ +struct kvm_user_return_msrs *msrs =this_cpu_ptr (user_return_msrs); + + msrs->values[slot].curr = value; +kvm_user_return_register_notifier (msrs); +} +EXPORT_SYMBOL_GPL (kvm_user_return_update_cache); +static void drop_user_return_notifiers (void ) {unsigned int cpu =smp_processor_id ();
Signature · 1 line
2.25.1
PATCH v12 61/106 KVM: TDX: restore user ret MSRs
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Several user ret MSRs are clobbered on TD exit. Restore those values on TD exit and before returning to ring 3.
--- arch/x86/kvm/vmx/tdx.c | 43 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/x86/kvm/vmx/tdx.c +43 -0
@@ -506,6 +506,28 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)return ; } +struct tdx_uret_msr { +u32 msr; +unsigned int slot; +u64 defval; +}; + +static struct tdx_uret_msr tdx_uret_msrs[] = { + {.msr =MSR_SYSCALL_MASK ,}, + {.msr =MSR_STAR ,}, + {.msr =MSR_LSTAR ,}, + {.msr =MSR_TSC_AUX ,}, +}; + +static void tdx_user_return_update_cache (void ) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++) +kvm_user_return_update_cache (tdx_uret_msrs[i].slot, + tdx_uret_msrs[i].defval); +} +static void tdx_restore_host_xsave_state (struct kvm_vcpu *vcpu) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (vcpu->kvm);
@@ -545,6 +567,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (vcpu, tdx); +tdx_user_return_update_cache ();tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true;
@@ -1635,6 +1658,26 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)return -EINVAL ; } +for (i =0 ; i <ARRAY_SIZE (tdx_uret_msrs); i++) { +/* +* Here it checks if MSRs (tdx_uret_msrs) can be saved/restored +* before returning to user space. +* +* this_cpu_ptr(user_return_msrs)->registered isn't checked +* because the registration is done at vcpu runtime by +* kvm_set_user_return_msr(). +* Here is setting up cpu feature before running vcpu, +* registered is already false. +*/ + tdx_uret_msrs[i].slot =kvm_find_user_return_msr (tdx_uret_msrs[i].msr); +if (tdx_uret_msrs[i].slot ==-1 ) { +/* If any MSR isn't supported, it is a KVM bug */ +pr_err ("MSR %x isn't included by kvm_find_user_return_msr\n" , + tdx_uret_msrs[i].msr); +return -EIO ; + } + } + max_pkgs =topology_max_packages (); tdx_mng_key_config_lock =kcalloc (max_pkgs,sizeof (*tdx_mng_key_config_lock),GFP_KERNEL );
Signature · 1 line
2.25.1
PATCH v12 62/106 The start of TDX KVM patch series: TD vcpu exits/interrupts/hypercalls
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the start of patch series of TD vcpu exits, interrupts, and hypercalls.
--- Documentation/virt/kvm/intel-tdx-layer-status.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -1
@@ -13,6 +13,7 @@ What qemu can do - Qemu can create/destroy vcpu of TDX vm type. - Qemu can populate initial guest memory image. - Qemu can finalize guest TD. +- Qemu can start to run vcpu. But vcpu can not make progress yet. Patch Layer status ------------------
@@ -23,7 +24,7 @@ Patch Layer status * TD vcpu creation/destruction: Applied * TDX EPT violation: Applied * TD finalization: Applied -* TD vcpu enter/exit: Applying +* TD vcpu enter/exit: Applied * TD vcpu interrupts/exit/hypercall: Not yet * KVM MMU GPA shared bits: Applied
Signature · 1 line
2.25.1
PATCH v12 63/106 KVM: TDX: complete interrupts after tdexit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This corresponds to VMX __vmx_complete_interrupts(). Because TDX virtualize vAPIC, KVM only needs to care NMI injection.
--- arch/x86/kvm/vmx/tdx.c | 10 ++++++++++ arch/x86/kvm/vmx/tdx.h | 2 ++ 2 files changed, 12 insertions(+)
arch/x86/kvm/vmx/tdx.c +10 -0
@@ -506,6 +506,14 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)return ; } +static void tdx_complete_interrupts (struct kvm_vcpu *vcpu) +{ +/* Avoid costly SEAMCALL if no nmi was injected */ +if (vcpu->arch.nmi_injected) + vcpu->arch.nmi_injected =td_management_read8 (to_tdx (vcpu), +TD_VCPU_PEND_NMI ); +} +struct tdx_uret_msr {u32 msr;unsigned int slot;
@@ -574,6 +582,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET ;trace_kvm_exit (vcpu,KVM_ISA_VMX ); +tdx_complete_interrupts (vcpu); +return EXIT_FASTPATH_NONE ; }
arch/x86/kvm/vmx/tdx.h +2 -0
@@ -192,6 +192,8 @@ TDX_BUILD_TDVPS_ACCESSORS(16, VMCS, vmcs);TDX_BUILD_TDVPS_ACCESSORS (32 ,VMCS , vmcs);TDX_BUILD_TDVPS_ACCESSORS (64 ,VMCS , vmcs); +TDX_BUILD_TDVPS_ACCESSORS (8 ,MANAGEMENT , management); + static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field) {struct tdx_module_output out;
Signature · 1 line
2.25.1
PATCH v12 64/106 KVM: TDX: restore debug store when TD exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because debug store is clobbered, restore it on TD exit.
--- arch/x86/events/intel/ds.c | 1 + arch/x86/kvm/vmx/tdx.c | 1 + 2 files changed, 2 insertions(+)
arch/x86/events/intel/ds.c +1 -0
@@ -2350,3 +2350,4 @@ void perf_restore_debug_store(void)wrmsrl (MSR_IA32_DS_AREA , (unsigned long )ds); } +EXPORT_SYMBOL_GPL (perf_restore_debug_store);
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -576,6 +576,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)tdx_vcpu_enter_exit (vcpu, tdx);tdx_user_return_update_cache (); +perf_restore_debug_store ();tdx_restore_host_xsave_state (vcpu); tdx->host_state_need_restore = true;
Signature · 1 line
2.25.1
PATCH v12 65/106 KVM: TDX: handle vcpu migration over logical processor
In reply to [email protected]From: Isaku Yamahata <[email protected]>
For vcpu migration, in the case of VMX, VMCS is flushed on the source pcpu, and load it on the target pcpu. There are corresponding TDX SEAMCALL APIs, call them on vcpu migration. The logic is mostly same as VMX except the TDX SEAMCALLs are used.
When shutting down the machine, (VMX or TDX) vcpus needs to be shutdown on each pcpu. Do the similar for TDX with TDX SEAMCALL APIs.
--- arch/x86/kvm/vmx/main.c | 37 ++++++++- arch/x86/kvm/vmx/tdx.c | 152 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 2 + arch/x86/kvm/vmx/x86_ops.h | 7 ++ 4 files changed, 195 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/main.c +34 -3
@@ -32,6 +32,13 @@ static int vt_max_vcpus(struct kvm *kvm) }static int vt_tlb_remote_flush (struct kvm *kvm); +static void vt_hardware_disable (void ) +{ +/* Note, TDX *and* VMX need to be disabled if TDX is enabled. */ +tdx_hardware_disable (); +vmx_hardware_disable (); +} +static __init int vt_hardware_setup (void ) {int ret;
@@ -191,6 +198,16 @@ static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu)return vmx_vcpu_run (vcpu); } +static void vt_vcpu_load (struct kvm_vcpu *vcpu,int cpu) +{ +if (is_td_vcpu (vcpu)) { +tdx_vcpu_load (vcpu, cpu); +return ; + } + +vmx_vcpu_load (vcpu, cpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -250,6 +267,14 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,vmx_load_mmu_pgd (vcpu, root_hpa, pgd_level); } +static void vt_sched_in (struct kvm_vcpu *vcpu,int cpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_sched_in (vcpu, cpu); +} +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {if (is_td_vcpu (vcpu))
@@ -294,7 +319,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .offline_cpu = tdx_offline_cpu, .hardware_enable = vmx_hardware_enable, - .hardware_disable = vmx_hardware_disable, + .hardware_disable = vt_hardware_disable, .has_emulated_msr = vmx_has_emulated_msr, .is_vm_type_supported = vt_is_vm_type_supported,
@@ -312,7 +337,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_reset = vt_vcpu_reset, .prepare_switch_to_guest = vt_prepare_switch_to_guest, - .vcpu_load = vmx_vcpu_load, + .vcpu_load = vt_vcpu_load, .vcpu_put = vt_vcpu_put, .update_exception_bitmap = vmx_update_exception_bitmap,
@@ -398,7 +423,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .request_immediate_exit = vmx_request_immediate_exit, - .sched_in = vmx_sched_in, + .sched_in = vt_sched_in, .cpu_dirty_log_size = PML_ENTITY_NUM, .update_cpu_dirty_logging = vmx_update_cpu_dirty_logging,
@@ -466,6 +491,10 @@ static int __init vt_init(void)if (r) goto err_vmx_init; + r =tdx_init (); +if (r) + goto err_tdx_init; +/* * Common KVM initialization _must_ come last, after this, /dev/kvm is * exposed to userspace!
@@ -488,6 +517,8 @@ static int __init vt_init(void)return 0 ; err_kvm_init: +/* tdx_exit() is not defined. */ +err_tdx_init:vmx_exit (); err_vmx_init:kvm_x86_vendor_exit ();
arch/x86/kvm/vmx/tdx.c +152 -0
197 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +2 -0
@@ -62,6 +62,8 @@ struct vcpu_tdx {unsigned long tdvpr_pa;unsigned long *tdvpx_pa; +struct list_head cpu_list; +union tdx_exit_reason exit_reason;bool initialized;
arch/x86/kvm/vmx/x86_ops.h +7 -0
@@ -138,8 +138,11 @@ void vmx_cancel_hv_timer(struct kvm_vcpu *vcpu);void vmx_setup_mce (struct kvm_vcpu *vcpu);#ifdef CONFIG_INTEL_TDX_HOST +int __init tdx_init (void );int __init tdx_hardware_setup (struct kvm_x86_ops *x86_ops);void tdx_hardware_unsetup (void ); +void tdx_hardware_enable (void ); +void tdx_hardware_disable (void );bool tdx_is_vm_type_supported (unsigned long type);int tdx_dev_ioctl (void __user *argp);int tdx_offline_cpu (void );
@@ -157,6 +160,7 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu);void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu); +void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -165,8 +169,10 @@ void tdx_flush_tlb(struct kvm_vcpu *vcpu);int tdx_sept_tlb_remote_flush (struct kvm *kvm);void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);#else +static inline int tdx_init (void ) {return 0 ; };static inline int tdx_hardware_setup (struct kvm_x86_ops *x86_ops) {return -ENOSYS ; }static inline void tdx_hardware_unsetup (void ) {} +static inline void tdx_hardware_disable (void ) {}static inline bool tdx_is_vm_type_supported (unsigned long type) {return false; }static inline int tdx_dev_ioctl (void __user *argp) {return -EOPNOTSUPP ; };static inline int tdx_offline_cpu (void ) {return 0 ; }
@@ -185,6 +191,7 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}static inline fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu) {return EXIT_FASTPATH_NONE ; }static inline void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {}static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v12 66/106 KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add a flag, KVM_DEBUGREG_AUTO_SWITCHED_GUEST, to skip saving/restoring DRs irrespective of any other flags. TDX-SEAM unconditionally saves and restores guest DRs and reset to architectural INIT state on TD exit. So, KVM needs to save host DRs before TD enter without restoring guest DRs and restore host DRs after TD exit.
Opportunistically convert the KVM_DEBUGREG_* definitions to use BIT().
--- arch/x86/include/asm/kvm_host.h | 10 ++++++++-- arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/x86.c | 11 ++++++++--- 3 files changed, 17 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -2
@@ -602,8 +602,14 @@ struct kvm_pmu {struct kvm_pmu_ops ;enum { -KVM_DEBUGREG_BP_ENABLED =1 , -KVM_DEBUGREG_WONT_EXIT =2 , +KVM_DEBUGREG_BP_ENABLED =BIT (0 ), +KVM_DEBUGREG_WONT_EXIT =BIT (1 ), +/* +* Guest debug registers (DR0-3 and DR6) are saved/restored by hardware +* on exit from or enter to guest. KVM needn't switch them. Because DR7 +* is cleared on exit from guest, DR7 need to be saved/restored. +*/ +KVM_DEBUGREG_AUTO_SWITCH =BIT (2 ), };struct kvm_mtrr_range {
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -518,6 +518,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.efer =EFER_SCE |EFER_LME |EFER_LMA |EFER_NX ; + vcpu->arch.switch_db_regs =KVM_DEBUGREG_AUTO_SWITCH ; vcpu->arch.cr0_guest_owned_bits =-1ul ; vcpu->arch.cr4_guest_owned_bits =-1ul ;
arch/x86/kvm/x86.c +8 -3
@@ -10632,7 +10632,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (vcpu->arch.guest_fpu.xfd_err)wrmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); -if (unlikely (vcpu->arch.switch_db_regs)) { +if (unlikely (vcpu->arch.switch_db_regs & ~KVM_DEBUGREG_AUTO_SWITCH )) {set_debugreg (0 ,7 );set_debugreg (vcpu->arch.eff_db[0 ],0 );set_debugreg (vcpu->arch.eff_db[1 ],1 );
@@ -10675,6 +10675,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)*/ if (unlikely (vcpu->arch.switch_db_regs &KVM_DEBUGREG_WONT_EXIT )) {WARN_ON (vcpu->guest_debug &KVM_GUESTDBG_USE_HW_BP ); +WARN_ON (vcpu->arch.switch_db_regs &KVM_DEBUGREG_AUTO_SWITCH );static_call (kvm_x86_sync_dirty_debug_regs)(vcpu);kvm_update_dr0123 (vcpu);kvm_update_dr7 (vcpu);
@@ -10687,8 +10688,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)* care about the messed up debug address registers. But if * we have some of them active, restore the old state. */ -if (hw_breakpoint_active ()) -hw_breakpoint_restore (); +if (hw_breakpoint_active ()) { +if (!(vcpu->arch.switch_db_regs &KVM_DEBUGREG_AUTO_SWITCH )) +hw_breakpoint_restore (); +else +set_debugreg (__this_cpu_read (cpu_dr7),7 ); + } vcpu->arch.last_vmentry_cpu = vcpu->cpu; vcpu->arch.last_guest_tsc =kvm_read_l1_tsc (vcpu,rdtsc ());
Signature · 1 line
2.25.1
PATCH v12 67/106 KVM: TDX: Add support for find pending IRQ in a protected local APIC
In reply to [email protected]From: Sean Christopherson <[email protected]>
Add flag and hook to KVM's local APIC management to support determining whether or not a TDX guest as a pending IRQ. For TDX vCPUs, the virtual APIC page is owned by the TDX module and cannot be accessed by KVM. As a result, registers that are virtualized by the CPU, e.g. PPR, cannot be read or written by KVM. To deliver interrupts for TDX guests, KVM must send an IRQ to the CPU on the posted interrupt notification vector. And to determine if TDX vCPU has a pending interrupt, KVM must check if there is an outstanding notification.
Return "no interrupt" in kvm_apic_has_interrupt() if the guest APIC is protected to short-circuit the various other flows that try to pull an IRQ out of the vAPIC, the only valid operation is querying _if_ an IRQ is pending, KVM can't do anything based on _which_ IRQ is pending.
Intentionally omit sanity checks from other flows, e.g. PPR update, so as not to degrade non-TDX guests with unnecessary checks. A well-behaved KVM and userspace will never reach those flows for TDX guests, but reaching them is not fatal if something does go awry.
Note, this doesn't handle interrupts that have been delivered to the vCPU but not yet recognized by the core, i.e. interrupts that are sitting in vmcs.GUEST_INTR_STATUS. Querying that state requires a SEAMCALL and will be supported in a future patch.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/irq.c | 3 +++ arch/x86/kvm/lapic.c | 3 +++ arch/x86/kvm/lapic.h | 2 ++ arch/x86/kvm/vmx/main.c | 11 +++++++++++ arch/x86/kvm/vmx/tdx.c | 6 ++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 8 files changed, 29 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -118,6 +118,7 @@ KVM_X86_OP_OPTIONAL(pi_update_irte) KVM_X86_OP_OPTIONAL(pi_start_assignment) KVM_X86_OP_OPTIONAL(apicv_post_state_restore) KVM_X86_OP_OPTIONAL_RET0(dy_apicv_has_pending_interrupt) +KVM_X86_OP_OPTIONAL(protected_apic_has_interrupt) KVM_X86_OP_OPTIONAL(set_hv_timer) KVM_X86_OP_OPTIONAL(cancel_hv_timer) KVM_X86_OP(setup_mce)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1749,6 +1749,7 @@ struct kvm_x86_ops {void (*pi_start_assignment)(struct kvm *kvm);void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu); +bool (*protected_apic_has_interrupt)(struct kvm_vcpu *vcpu);int (*set_hv_timer)(struct kvm_vcpu *vcpu,u64 guest_deadline_tsc,bool *expired);
arch/x86/kvm/irq.c +3 -0
@@ -100,6 +100,9 @@ int kvm_cpu_has_interrupt(struct kvm_vcpu *v)if (kvm_cpu_has_extint (v))return 1 ; +if (lapic_in_kernel (v) && v->arch.apic->guest_apic_protected) +return static_call (kvm_x86_protected_apic_has_interrupt)(v); +return kvm_apic_has_interrupt (v) !=-1 ;/* LAPIC */ }EXPORT_SYMBOL_GPL (kvm_cpu_has_interrupt);
arch/x86/kvm/lapic.c +3 -0
@@ -2785,6 +2785,9 @@ int kvm_apic_has_interrupt(struct kvm_vcpu *vcpu)if (!kvm_apic_present (vcpu))return -1 ; +if (apic->guest_apic_protected) +return -1 ; +__apic_update_ppr (apic, &ppr);return apic_has_interrupt_for_ppr (apic, ppr); }
arch/x86/kvm/lapic.h +2 -0
@@ -66,6 +66,8 @@ struct kvm_lapic {bool sw_enabled;bool irr_pending;bool lvt0_in_nmi_mode; +/* Select registers in the vAPIC cannot be read/written. */ +bool guest_apic_protected;/* Number of bits set in ISR. */ s16 isr_count;/* The highest vector set in ISR; if -1 - invalid, must scan ISR. */
arch/x86/kvm/vmx/main.c +11 -0
@@ -49,6 +49,9 @@ static __init int vt_hardware_setup(void) enable_tdx = enable_tdx && !tdx_hardware_setup (&vt_x86_ops); +if (!enable_tdx) + vt_x86_ops.protected_apic_has_interrupt =NULL ; +/* * As kvm_mmu_set_ept_masks() updates enable_mmio_caching, call it * before checking enable_mmio_caching.
@@ -208,6 +211,13 @@ static void vt_vcpu_load(struct kvm_vcpu *vcpu, int cpu)vmx_vcpu_load (vcpu, cpu); } +static bool vt_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) +{ +KVM_BUG_ON (!is_td_vcpu (vcpu), vcpu->kvm); + +return tdx_protected_apic_has_interrupt (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -400,6 +410,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .sync_pir_to_irr = vmx_sync_pir_to_irr, .deliver_interrupt = vmx_deliver_interrupt, .dy_apicv_has_pending_interrupt = pi_has_pending_interrupt, + .protected_apic_has_interrupt = vt_protected_apic_has_interrupt, .set_tss_addr = vmx_set_tss_addr, .set_identity_map_addr = vmx_set_identity_map_addr,
arch/x86/kvm/vmx/tdx.c +6 -0
@@ -515,6 +515,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)return -EINVAL ;fpstate_set_confidential (&vcpu->arch.guest_fpu); + vcpu->arch.apic->guest_apic_protected = true; vcpu->arch.efer =EFER_SCE |EFER_LME |EFER_LMA |EFER_NX ;
@@ -553,6 +554,11 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)local_irq_enable (); } +bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) +{ +return pi_has_pending_interrupt (vcpu); +} +void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {struct vcpu_tdx *tdx =to_tdx (vcpu);
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -161,6 +161,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu);void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu);void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu); +bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -192,6 +193,7 @@ static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) { return EXIT_FASTPstatic inline void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {} +static inline bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) {return false; }static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v12 68/106 KVM: x86: Assume timer IRQ was injected if APIC state is proteced
In reply to [email protected]From: Sean Christopherson <[email protected]>
If APIC state is protected, i.e. the vCPU is a TDX guest, assume a timer IRQ was injected when deciding whether or not to busy wait in the "timer advanced" path. The "real" vIRR is not readable/writable, so trying to query for a pending timer IRQ will return garbage.
Note, TDX can scour the PIR if it wants to be more precise and skip the "wait" call entirely.
--- arch/x86/kvm/lapic.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c +10 -1
@@ -1694,8 +1694,17 @@ static void apic_update_lvtt(struct kvm_lapic *apic)static bool lapic_timer_int_injected (struct kvm_vcpu *vcpu) {struct kvm_lapic *apic = vcpu->arch.apic; -u32 reg =kvm_lapic_get_reg (apic,APIC_LVTT ); +u32 reg; +/* +* Assume a timer IRQ was "injected" if the APIC is protected. KVM's +* copy of the vIRR is bogus, it's the responsibility of the caller to +* precisely check whether or not a timer IRQ is pending. +*/ +if (apic->guest_apic_protected) +return true; + + reg =kvm_lapic_get_reg (apic,APIC_LVTT );if (kvm_apic_hw_enabled (apic)) {int vec = reg &APIC_VECTOR_MASK ;void *bitmap = apic->regs +APIC_ISR ;
Signature · 1 line
2.25.1
PATCH v12 69/106 KVM: TDX: remove use of struct vcpu_vmx from posted_interrupt.c
In reply to [email protected]From: Isaku Yamahata <[email protected]>
As TDX will use posted_interrupt.c, the use of struct vcpu_vmx is a blocker. Because the members of struct pi_desc pi_desc and struct list_head pi_wakeup_list are only used in posted_interrupt.c, introduce common structure, struct vcpu_pi, make vcpu_vmx and vcpu_tdx has same layout in the top of structure.
To minimize the diff size, avoid code conversion like, vmx->pi_desc => vmx->common->pi_desc. Instead add compile time check if the layout is expected.
--- arch/x86/kvm/vmx/posted_intr.c | 41 ++++++++++++++++++++++++++-------- arch/x86/kvm/vmx/posted_intr.h | 11 +++++++++ arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/vmx/tdx.h | 8 +++++++ arch/x86/kvm/vmx/vmx.h | 14 +++++++----- 5 files changed, 60 insertions(+), 15 deletions(-)
arch/x86/kvm/vmx/posted_intr.c +32 -9
@@ -11,6 +11,7 @@#include "posted_intr.h" #include "trace.h" #include "vmx.h" +#include "tdx.h" /* * Maintain a per-CPU list of vCPUs that need to be awakened by wakeup_handler()
@@ -31,9 +32,29 @@ static DEFINE_PER_CPU(struct list_head, wakeup_vcpus_on_cpu);*/ static DEFINE_PER_CPU(raw_spinlock_t, wakeup_vcpus_on_cpu_lock); +/* +* The layout of the head of struct vcpu_vmx and struct vcpu_tdx must match with +* struct vcpu_pi. +*/ +static_assert (offsetof(struct vcpu_pi , pi_desc) == + offsetof(struct vcpu_vmx , pi_desc)); +static_assert (offsetof(struct vcpu_pi , pi_wakeup_list) == + offsetof(struct vcpu_vmx , pi_wakeup_list)); +#ifdef CONFIG_INTEL_TDX_HOST +static_assert (offsetof(struct vcpu_pi , pi_desc) == + offsetof(struct vcpu_tdx , pi_desc)); +static_assert (offsetof(struct vcpu_pi , pi_wakeup_list) == + offsetof(struct vcpu_tdx , pi_wakeup_list)); +#endif + +static inline struct vcpu_pi *vcpu_to_pi (struct kvm_vcpu *vcpu) +{ +return (struct vcpu_pi *)vcpu; +} +static inline struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu) { -return &(to_vmx (vcpu)->pi_desc); +return &vcpu_to_pi (vcpu)->pi_desc; }static int pi_try_set_control (struct pi_desc *pi_desc,u64 *pold,u64 new)
@@ -52,8 +73,8 @@ static int pi_try_set_control(struct pi_desc *pi_desc, u64 *pold, u64 new)void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu) { -struct pi_desc *pi_desc =vcpu_to_pi_desc (vcpu); -struct vcpu_vmx *vmx =to_vmx (vcpu); +struct vcpu_pi *vcpu_pi =vcpu_to_pi (vcpu); +struct pi_desc *pi_desc = &vcpu_pi->pi_desc;struct pi_desc old, new;unsigned long flags;unsigned int dest;
@@ -90,7 +111,7 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)*/ if (pi_desc->nv ==POSTED_INTR_WAKEUP_VECTOR ) {raw_spin_lock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); -list_del (&vmx->pi_wakeup_list); +list_del (&vcpu_pi->pi_wakeup_list);raw_spin_unlock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); }
@@ -145,15 +166,15 @@ static bool vmx_can_use_vtd_pi(struct kvm *kvm)*/ static void pi_enable_wakeup_handler (struct kvm_vcpu *vcpu) { -struct pi_desc *pi_desc =vcpu_to_pi_desc (vcpu); -struct vcpu_vmx *vmx =to_vmx (vcpu); +struct vcpu_pi *vcpu_pi =vcpu_to_pi (vcpu); +struct pi_desc *pi_desc = &vcpu_pi->pi_desc;struct pi_desc old, new;unsigned long flags;local_irq_save (flags);raw_spin_lock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); -list_add_tail (&vmx->pi_wakeup_list, +list_add_tail (&vcpu_pi->pi_wakeup_list, &per_cpu (wakeup_vcpus_on_cpu, vcpu->cpu));raw_spin_unlock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu));
@@ -190,7 +211,8 @@ static bool vmx_needs_pi_wakeup(struct kvm_vcpu *vcpu)* notification vector is switched to the one that calls * back to the pi_wakeup_handler() function. */ -return vmx_can_use_ipiv (vcpu) ||vmx_can_use_vtd_pi (vcpu->kvm); +return (vmx_can_use_ipiv (vcpu) && !is_td_vcpu (vcpu)) || +vmx_can_use_vtd_pi (vcpu->kvm); }void vmx_vcpu_pi_put (struct kvm_vcpu *vcpu)
@@ -200,7 +222,8 @@ void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu)if (!vmx_needs_pi_wakeup (vcpu))return ; -if (kvm_vcpu_is_blocking (vcpu) && !vmx_interrupt_blocked (vcpu)) +if (kvm_vcpu_is_blocking (vcpu) && + (is_td_vcpu (vcpu) || !vmx_interrupt_blocked (vcpu)))pi_enable_wakeup_handler (vcpu);/*
arch/x86/kvm/vmx/posted_intr.h +11 -0
@@ -94,6 +94,17 @@ static inline bool pi_test_sn(struct pi_desc *pi_desc) (unsigned long *)&pi_desc->control); } +struct vcpu_pi { +struct kvm_vcpu vcpu; + +/* Posted interrupt descriptor */ +struct pi_desc pi_desc; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; +/* Until here common layout betwwn vcpu_vmx and vcpu_tdx. */ +}; +void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu);void vmx_vcpu_pi_put (struct kvm_vcpu *vcpu);void pi_wakeup_handler (void );
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -516,6 +516,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)fpstate_set_confidential (&vcpu->arch.guest_fpu); vcpu->arch.apic->guest_apic_protected = true; +INIT_LIST_HEAD (&tdx->pi_wakeup_list); vcpu->arch.efer =EFER_SCE |EFER_LME |EFER_LMA |EFER_NX ;
arch/x86/kvm/vmx/tdx.h +8 -0
@@ -4,6 +4,7 @@#ifdef CONFIG_INTEL_TDX_HOST +#include "posted_intr.h" #include "pmu_intel.h" #include "tdx_ops.h"
@@ -59,6 +60,13 @@ union tdx_exit_reason {struct vcpu_tdx {struct kvm_vcpu vcpu; +/* Posted interrupt descriptor */ +struct pi_desc pi_desc; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; +/* Until here same layout to struct vcpu_pi. */ +unsigned long tdvpr_pa;unsigned long *tdvpx_pa;
arch/x86/kvm/vmx/vmx.h +8 -6
@@ -245,6 +245,14 @@ struct nested_vmx {struct vcpu_vmx {struct kvm_vcpu vcpu; + +/* Posted interrupt descriptor */ +struct pi_desc pi_desc; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; +/* Until here same layout to struct vcpu_pi. */ +u8 fail;u8 x2apic_msr_bitmap_mode;
@@ -314,12 +322,6 @@ struct vcpu_vmx {union vmx_exit_reason exit_reason; -/* Posted interrupt descriptor */ -struct pi_desc pi_desc; - -/* Used if this vCPU is waiting for PI notification wakeup. */ -struct list_head pi_wakeup_list; -/* Support for a guest hypervisor (nested VMX) */ struct nested_vmx nested;
Signature · 1 line
2.25.1
PATCH v12 70/106 KVM: TDX: Implement interrupt injection
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX supports interrupt inject into vcpu with posted interrupt. Wire up the corresponding kvm x86 operations to posted interrupt. Move kvm_vcpu_trigger_posted_interrupt() from vmx.c to common.h to share the code.
VMX can inject interrupt by setting interrupt information field, VM_ENTRY_INTR_INFO_FIELD, of VMCS. TDX supports interrupt injection only by posted interrupt. Ignore the execution path to access VM_ENTRY_INTR_INFO_FIELD.
As cpu state is protected and apicv is enabled for the TDX guest, VMM can inject interrupt by updating posted interrupt descriptor. Treat interrupt can be injected always.
--- arch/x86/kvm/vmx/common.h | 71 ++++++++++++++++++++++++++ arch/x86/kvm/vmx/main.c | 93 ++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/posted_intr.c | 2 +- arch/x86/kvm/vmx/posted_intr.h | 2 + arch/x86/kvm/vmx/tdx.c | 24 +++++++++ arch/x86/kvm/vmx/vmx.c | 67 +----------------------- arch/x86/kvm/vmx/x86_ops.h | 7 ++- 7 files changed, 189 insertions(+), 77 deletions(-)
arch/x86/kvm/vmx/common.h +71 -0
@@ -4,6 +4,7 @@#include <linux/kvm_host.h> +#include "posted_intr.h" #include "mmu.h" u8 __vmx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio,bool check_cr0_cd);
@@ -32,4 +33,74 @@ static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t gpa, return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0); } +static inline void kvm_vcpu_trigger_posted_interrupt (struct kvm_vcpu *vcpu, +int pi_vec) +{ +#ifdef CONFIG_SMP +if (vcpu->mode ==IN_GUEST_MODE ) { +/* +* The vector of the virtual has already been set in the PIR. +* Send a notification event to deliver the virtual interrupt +* unless the vCPU is the currently running vCPU, i.e. the +* event is being sent from a fastpath VM-Exit handler, in +* which case the PIR will be synced to the vIRR before +* re-entering the guest. +* +* When the target is not the running vCPU, the following +* possibilities emerge: +* +* Case 1: vCPU stays in non-root mode. Sending a notification +* event posts the interrupt to the vCPU. +* +* Case 2: vCPU exits to root mode and is still runnable. The +* PIR will be synced to the vIRR before re-entering the guest. +* Sending a notification event is ok as the host IRQ handler +* will ignore the spurious event. +* +* Case 3: vCPU exits to root mode and is blocked. vcpu_block() +* has already synced PIR to vIRR and never blocks the vCPU if +* the vIRR is not empty. Therefore, a blocked vCPU here does +* not wait for any requested interrupts in PIR, and sending a +* notification event also results in a benign, spurious event. +*/ + +if (vcpu !=kvm_get_running_vcpu ()) + apic->send_IPI_mask (get_cpu_mask (vcpu->cpu), pi_vec); +return ; + } +#endif +/* +* The vCPU isn't in the guest; wake the vCPU in case it is blocking, +* otherwise do nothing as KVM will grab the highest priority pending +* IRQ via ->sync_pir_to_irr() in vcpu_enter_guest(). +*/ +kvm_vcpu_wake_up (vcpu); +} + +/* +* Send interrupt to vcpu via posted interrupt way. +* 1. If target vcpu is running(non-root mode), send posted interrupt +* notification to vcpu and hardware will sync PIR to vIRR atomically. +* 2. If target vcpu isn't running(root mode), kick it to pick up the +* interrupt from PIR in next vmentry. +*/ +static inline void __vmx_deliver_posted_interrupt (struct kvm_vcpu *vcpu, +struct pi_desc *pi_desc,int vector) +{ +if (pi_test_and_set_pir (vector, pi_desc)) +return ; + +/* If a previous notification has sent the IPI, nothing to do. */ +if (pi_test_and_set_on (pi_desc)) +return ; + +/* +* The implied barrier in pi_test_and_set_on() pairs with the smp_mb_*() +* after setting vcpu->mode in vcpu_enter_guest(), thus the vCPU is +* guaranteed to see PID.ON=1 and sync the PIR to IRR if triggering a +* posted interrupt "fails" because vcpu->mode != IN_GUEST_MODE. +*/ +kvm_vcpu_trigger_posted_interrupt (vcpu,POSTED_INTR_VECTOR ); +} +#endif /* __KVM_X86_VMX_COMMON_H */
arch/x86/kvm/vmx/main.c +84 -9
127 lines not on this page · raw message
arch/x86/kvm/vmx/posted_intr.c +1 -1
@@ -52,7 +52,7 @@ static inline struct vcpu_pi *vcpu_to_pi(struct kvm_vcpu *vcpu)return (struct vcpu_pi *)vcpu; } -static inline struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu) +struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu) {return &vcpu_to_pi (vcpu)->pi_desc; }
arch/x86/kvm/vmx/posted_intr.h +2 -0
@@ -105,6 +105,8 @@ struct vcpu_pi {/* Until here common layout betwwn vcpu_vmx and vcpu_tdx. */ }; +struct pi_desc *vcpu_to_pi_desc (struct kvm_vcpu *vcpu); +void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu);void vmx_vcpu_pi_put (struct kvm_vcpu *vcpu);void pi_wakeup_handler (void );
arch/x86/kvm/vmx/tdx.c +24 -0
@@ -529,6 +529,9 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.guest_state_protected = !(to_kvm_tdx (vcpu->kvm)->attributes &TDX_TD_ATTRIBUTE_DEBUG ); + tdx->pi_desc.nv =POSTED_INTR_VECTOR ; + tdx->pi_desc.sn =1 ; + tdx->host_state_need_save = true; tdx->host_state_need_restore = false;
@@ -539,6 +542,7 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {struct vcpu_tdx *tdx =to_tdx (vcpu); +vmx_vcpu_pi_load (vcpu, cpu);if (vcpu->cpu == cpu)return ;
@@ -724,6 +728,12 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)trace_kvm_entry (vcpu); +if (pi_test_on (&tdx->pi_desc)) { + apic->send_IPI_self (POSTED_INTR_VECTOR ); + +kvm_wait_lapic_expire (vcpu); + } +tdx_vcpu_enter_exit (vcpu, tdx);tdx_user_return_update_cache ();
@@ -1055,6 +1065,16 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,return tdx_sept_drop_private_spte (kvm, gfn, level, pfn); } +void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode, +int trig_mode,int vector) +{ +struct kvm_vcpu *vcpu = apic->vcpu; +struct vcpu_tdx *tdx =to_tdx (vcpu); + +/* TDX supports only posted interrupt. No lapic emulation. */ +__vmx_deliver_posted_interrupt (vcpu, &tdx->pi_desc, vector); +} +int tdx_dev_ioctl (void __user *argp) {struct kvm_tdx_capabilities __user *user_caps;
@@ -1770,6 +1790,10 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)if (ret)return ret; +td_vmcs_write16 (tdx,POSTED_INTR_NV ,POSTED_INTR_VECTOR ); +td_vmcs_write64 (tdx,POSTED_INTR_DESC_ADDR ,__pa (&tdx->pi_desc)); +td_vmcs_setbit32 (tdx,PIN_BASED_VM_EXEC_CONTROL ,PIN_BASED_POSTED_INTR ); + tdx->initialized = true;return 0 ; }
arch/x86/kvm/vmx/vmx.c +1 -66
85 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +6 -1
@@ -62,7 +62,6 @@ int vmx_check_intercept(struct kvm_vcpu *vcpu,bool vmx_apic_init_signal_blocked (struct kvm_vcpu *vcpu);void vmx_migrate_timers (struct kvm_vcpu *vcpu);void vmx_set_virtual_apic_mode (struct kvm_vcpu *vcpu); -void vmx_apicv_post_state_restore (struct kvm_vcpu *vcpu);bool vmx_check_apicv_inhibit_reasons (enum kvm_apicv_inhibit reason);void vmx_hwapic_irr_update (struct kvm_vcpu *vcpu,int max_irr);void vmx_hwapic_isr_update (int max_isr);
@@ -164,6 +163,9 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu);bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio); +void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode, +int trig_mode,int vector); +int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);void tdx_flush_tlb (struct kvm_vcpu *vcpu);
@@ -196,6 +198,9 @@ static inline void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {}static inline bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) {return false; }static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; } +static inline void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode, +int trig_mode,int vector) {} +static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }static inline void tdx_flush_tlb (struct kvm_vcpu *vcpu) {}
Signature · 1 line
2.25.1
PATCH v12 71/106 KVM: TDX: Implements vcpu request_immediate_exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Now we are able to inject interrupts into TDX vcpu, it's ready to block TDX vcpu. Wire up kvm x86 methods for blocking/unblocking vcpu for TDX. To unblock on pending events, request immediate exit methods is also needed.
--- arch/x86/kvm/vmx/main.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +11 -1
@@ -360,6 +360,16 @@ static void vt_enable_irq_window(struct kvm_vcpu *vcpu)vmx_enable_irq_window (vcpu); } +static void vt_request_immediate_exit (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +__kvm_request_immediate_exit (vcpu); +return ; + } + +vmx_request_immediate_exit (vcpu); +} +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {if (is_td_vcpu (vcpu))
@@ -507,7 +517,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .check_intercept = vmx_check_intercept, .handle_exit_irqoff = vmx_handle_exit_irqoff, - .request_immediate_exit = vmx_request_immediate_exit, + .request_immediate_exit = vt_request_immediate_exit, .sched_in = vt_sched_in,
Signature · 1 line
2.25.1
PATCH v12 72/106 KVM: TDX: Implement methods to inject NMI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX vcpu control structure defines one bit for pending NMI for VMM to inject NMI by setting the bit without knowing TDX vcpu NMI states. Because the vcpu state is protected, VMM can't know about NMI states of TDX vcpu. The TDX module handles actual injection and NMI states transition.
Add methods for NMI and treat NMI can be injected always.
--- arch/x86/kvm/vmx/main.c | 64 +++++++++++++++++++++++++++++++++++--- arch/x86/kvm/vmx/tdx.c | 5 +++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 3 files changed, 66 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/main.c +59 -5
@@ -294,6 +294,60 @@ static void vt_flush_tlb_guest(struct kvm_vcpu *vcpu)vmx_flush_tlb_guest (vcpu); } +static void vt_inject_nmi (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +tdx_inject_nmi (vcpu); +return ; + } + +vmx_inject_nmi (vcpu); +} + +static int vt_nmi_allowed (struct kvm_vcpu *vcpu,bool for_injection) +{ +/* +* The TDX module manages NMI windows and NMI reinjection, and hides NMI +* blocking, all KVM can do is throw an NMI over the wall. +*/ +if (is_td_vcpu (vcpu)) +return true; + +return vmx_nmi_allowed (vcpu, for_injection); +} + +static bool vt_get_nmi_mask (struct kvm_vcpu *vcpu) +{ +/* +* Assume NMIs are always unmasked. KVM could query PEND_NMI and treat +* NMIs as masked if a previous NMI is still pending, but SEAMCALLs are +* expensive and the end result is unchanged as the only relevant usage +* of get_nmi_mask() is to limit the number of pending NMIs, i.e. it +* only changes whether KVM or the TDX module drops an NMI. +*/ +if (is_td_vcpu (vcpu)) +return false; + +return vmx_get_nmi_mask (vcpu); +} + +static void vt_set_nmi_mask (struct kvm_vcpu *vcpu,bool masked) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_nmi_mask (vcpu, masked); +} + +static void vt_enable_nmi_window (struct kvm_vcpu *vcpu) +{ +/* Refer the comment in vt_get_nmi_mask(). */ +if (is_td_vcpu (vcpu)) +return ; + +vmx_enable_nmi_window (vcpu); +} +static void vt_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) {
@@ -473,14 +527,14 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .get_interrupt_shadow = vt_get_interrupt_shadow, .patch_hypercall = vmx_patch_hypercall, .inject_irq = vt_inject_irq, - .inject_nmi = vmx_inject_nmi, + .inject_nmi = vt_inject_nmi, .inject_exception = vmx_inject_exception, .cancel_injection = vt_cancel_injection, .interrupt_allowed = vt_interrupt_allowed, - .nmi_allowed = vmx_nmi_allowed, - .get_nmi_mask = vmx_get_nmi_mask, - .set_nmi_mask = vmx_set_nmi_mask, - .enable_nmi_window = vmx_enable_nmi_window, + .nmi_allowed = vt_nmi_allowed, + .get_nmi_mask = vt_get_nmi_mask, + .set_nmi_mask = vt_set_nmi_mask, + .enable_nmi_window = vt_enable_nmi_window, .enable_irq_window = vt_enable_irq_window, .update_cr8_intercept = vmx_update_cr8_intercept, .set_virtual_apic_mode = vmx_set_virtual_apic_mode,
arch/x86/kvm/vmx/tdx.c +5 -0
@@ -749,6 +749,11 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)return EXIT_FASTPATH_NONE ; } +void tdx_inject_nmi (struct kvm_vcpu *vcpu) +{ +td_management_write8 (to_tdx (vcpu),TD_VCPU_PEND_NMI ,1 ); +} +void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level) {td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa &PAGE_MASK );
arch/x86/kvm/vmx/x86_ops.h +2 -0
@@ -165,6 +165,7 @@ u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio);void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector); +void tdx_inject_nmi (struct kvm_vcpu *vcpu);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -200,6 +201,7 @@ static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)static inline void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector) {} +static inline void tdx_inject_nmi (struct kvm_vcpu *vcpu) {}static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v12 73/106 KVM: VMX: Modify NMI and INTR handlers to take intr_info as function argument
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX uses different ABI to get information about VM exit. Pass intr_info to the NMI and INTR handlers instead of pulling it from vcpu_vmx in preparation for sharing the bulk of the handlers with TDX.
When the guest TD exits to VMM, RAX holds status and exit reason, RCX holds exit qualification etc rather than the VMCS fields because VMM doesn't have access to the VMCS. The eventual code will be
VMX:
- get exit reason, intr_info, exit_qualification, and etc from VMCS
- call NMI/INTR handlers (common code)
TDX:
- get exit reason, intr_info, exit_qualification, and etc from guest registers
- call NMI/INTR handlers (common code)
--- arch/x86/kvm/vmx/vmx.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
arch/x86/kvm/vmx/vmx.c +8 -9
@@ -6861,28 +6861,27 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu)rdmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); } -static void handle_exception_nmi_irqoff (struct vcpu_vmx *vmx) +static void handle_exception_nmi_irqoff (struct kvm_vcpu *vcpu,u32 intr_info) {const unsigned long nmi_entry = (unsigned long )asm_exc_nmi_noist; -u32 intr_info =vmx_get_intr_info (&vmx->vcpu);/* if exit due to PF check for async PF */ if (is_page_fault (intr_info)) - vmx->vcpu.arch.apf.host_apf_flags =kvm_read_and_reset_apf_flags (); + vcpu->arch.apf.host_apf_flags =kvm_read_and_reset_apf_flags ();/* if exit due to NM, handle before interrupts are enabled */ else if (is_nm_fault (intr_info)) -handle_nm_fault_irqoff (&vmx->vcpu); +handle_nm_fault_irqoff (vcpu);/* Handle machine checks before interrupts are enabled */ else if (is_machine_check (intr_info))kvm_machine_check ();/* We need to handle NMIs before interrupts are enabled */ else if (is_nmi (intr_info)) -handle_interrupt_nmi_irqoff (&vmx->vcpu, nmi_entry); +handle_interrupt_nmi_irqoff (vcpu, nmi_entry); } -static void handle_external_interrupt_irqoff (struct kvm_vcpu *vcpu) +static void handle_external_interrupt_irqoff (struct kvm_vcpu *vcpu, +u32 intr_info) { -u32 intr_info =vmx_get_intr_info (vcpu);unsigned int vector = intr_info &INTR_INFO_VECTOR_MASK ;gate_desc *desc = (gate_desc *)host_idt_base + vector;
@@ -6902,9 +6901,9 @@ void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu)return ;if (vmx->exit_reason.basic ==EXIT_REASON_EXTERNAL_INTERRUPT ) -handle_external_interrupt_irqoff (vcpu); +handle_external_interrupt_irqoff (vcpu,vmx_get_intr_info (vcpu));else if (vmx->exit_reason.basic ==EXIT_REASON_EXCEPTION_NMI ) -handle_exception_nmi_irqoff (vmx); +handle_exception_nmi_irqoff (vcpu,vmx_get_intr_info (vcpu)); }/*
Signature · 1 line
2.25.1
PATCH v12 74/106 KVM: VMX: Move NMI/exception handler to common helper
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX mostly handles NMI/exception exit mostly the same to VMX case. The difference is how to retrieve exit qualification. To share the code with TDX, move NMI/exception to a common header, common.h.
--- arch/x86/kvm/vmx/common.h | 70 ++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 79 ++++----------------------------------- 2 files changed, 78 insertions(+), 71 deletions(-)
arch/x86/kvm/vmx/common.h +70 -0
@@ -4,8 +4,78 @@#include <linux/kvm_host.h> +#include <asm/traps.h> +#include "posted_intr.h" #include "mmu.h" +#include "vmcs.h" +#include "x86.h" + +extern unsigned long vmx_host_idt_base; +void vmx_do_interrupt_nmi_irqoff (unsigned long entry); + +static inline void vmx_handle_interrupt_nmi_irqoff (struct kvm_vcpu *vcpu, +unsigned long entry) +{ +bool is_nmi = entry == (unsigned long )asm_exc_nmi_noist; + +kvm_before_interrupt (vcpu, is_nmi ?KVM_HANDLING_NMI :KVM_HANDLING_IRQ ); +vmx_do_interrupt_nmi_irqoff (entry); +kvm_after_interrupt (vcpu); +} + +static inline void vmx_handle_nm_fault_irqoff (struct kvm_vcpu *vcpu) +{ +/* +* Save xfd_err to guest_fpu before interrupt is enabled, so the +* MSR value is not clobbered by the host activity before the guest +* has chance to consume it. +* +* Do not blindly read xfd_err here, since this exception might +* be caused by L1 interception on a platform which doesn't +* support xfd at all. +* +* Do it conditionally upon guest_fpu::xfd. xfd_err matters +* only when xfd contains a non-zero value. +* +* Queuing exception is done in vmx_handle_exit. See comment there. +*/ +if (vcpu->arch.guest_fpu.fpstate->xfd) +rdmsrl (MSR_IA32_XFD_ERR , vcpu->arch.guest_fpu.xfd_err); +} + +static inline void vmx_handle_exception_nmi_irqoff (struct kvm_vcpu *vcpu, +u32 intr_info) +{ +const unsigned long nmi_entry = (unsigned long )asm_exc_nmi_noist; + +/* if exit due to PF check for async PF */ +if (is_page_fault (intr_info)) + vcpu->arch.apf.host_apf_flags =kvm_read_and_reset_apf_flags (); +/* if exit due to NM, handle before interrupts are enabled */ +else if (is_nm_fault (intr_info)) +vmx_handle_nm_fault_irqoff (vcpu); +/* Handle machine checks before interrupts are enabled */ +else if (is_machine_check (intr_info)) +kvm_machine_check (); +/* We need to handle NMIs before interrupts are enabled */ +else if (is_nmi (intr_info)) +vmx_handle_interrupt_nmi_irqoff (vcpu, nmi_entry); +} + +static inline void vmx_handle_external_interrupt_irqoff (struct kvm_vcpu *vcpu, +u32 intr_info) +{ +unsigned int vector = intr_info &INTR_INFO_VECTOR_MASK ; +gate_desc *desc = (gate_desc *)vmx_host_idt_base + vector; + +if (KVM_BUG (!is_external_intr (intr_info), vcpu->kvm, +"unexpected VM-Exit interrupt info: 0x%x" , intr_info)) +return ; + +vmx_handle_interrupt_nmi_irqoff (vcpu,gate_offset (desc)); + vcpu->arch.at_instruction_boundary = true; +}u8 __vmx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio,bool check_cr0_cd);
arch/x86/kvm/vmx/vmx.c +8 -71
118 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 75/106 KVM: x86: Split core of hypercall emulation to helper function
In reply to [email protected]From: Sean Christopherson <[email protected]>
By necessity, TDX will use a different register ABI for hypercalls. Break out the core functionality so that it may be reused for TDX.
--- arch/x86/include/asm/kvm_host.h | 4 +++ arch/x86/kvm/x86.c | 54 ++++++++++++++++++++------------- 2 files changed, 37 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm_host.h +4 -0
@@ -2085,6 +2085,10 @@ static inline void kvm_clear_apicv_inhibit(struct kvm *kvm,kvm_set_or_clear_apicv_inhibit (kvm, reason, false); } +unsigned long __kvm_emulate_hypercall (struct kvm_vcpu *vcpu,unsigned long nr, +unsigned long a0,unsigned long a1, +unsigned long a2,unsigned long a3, +int op_64_bit);int kvm_emulate_hypercall (struct kvm_vcpu *vcpu); int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
arch/x86/kvm/x86.c +33 -21
76 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 76/106 KVM: TDX: Add a place holder to handle TDX VM exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up handle_exit and handle_exit_irqoff methods and add a place holder to handle VM exit. Add helper functions to get exit info, exit qualification, etc.
--- arch/x86/kvm/vmx/main.c | 37 ++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 88 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 10 +++++ 3 files changed, 132 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/main.c +34 -3
@@ -218,6 +218,25 @@ static bool vt_protected_apic_has_interrupt(struct kvm_vcpu *vcpu)return tdx_protected_apic_has_interrupt (vcpu); } +static int vt_handle_exit (struct kvm_vcpu *vcpu, +enum exit_fastpath_completion fastpath) +{ +if (is_td_vcpu (vcpu)) +return tdx_handle_exit (vcpu, fastpath); + +return vmx_handle_exit (vcpu, fastpath); +} + +static void vt_handle_exit_irqoff (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +tdx_handle_exit_irqoff (vcpu); +return ; + } + +vmx_handle_exit_irqoff (vcpu); +} +static void vt_apicv_post_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -424,6 +443,18 @@ static void vt_request_immediate_exit(struct kvm_vcpu *vcpu)vmx_request_immediate_exit (vcpu); } +static void vt_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason, +u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code) +{ +if (is_td_vcpu (vcpu)) { +tdx_get_exit_info (vcpu, reason, info1, info2, intr_info, + error_code); +return ; + } + +vmx_get_exit_info (vcpu, reason, info1, info2, intr_info, error_code); +} +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {if (is_td_vcpu (vcpu))
@@ -520,7 +551,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_pre_run = vt_vcpu_pre_run, .vcpu_run = vt_vcpu_run, - .handle_exit = vmx_handle_exit, + .handle_exit = vt_handle_exit, .skip_emulated_instruction = vmx_skip_emulated_instruction, .update_emulated_instruction = vmx_update_emulated_instruction, .set_interrupt_shadow = vt_set_interrupt_shadow,
@@ -555,7 +586,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_identity_map_addr = vmx_set_identity_map_addr, .get_mt_mask = vt_get_mt_mask, - .get_exit_info = vmx_get_exit_info, + .get_exit_info = vt_get_exit_info, .vcpu_after_set_cpuid = vmx_vcpu_after_set_cpuid,
@@ -569,7 +600,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .load_mmu_pgd = vt_load_mmu_pgd, .check_intercept = vmx_check_intercept, - .handle_exit_irqoff = vmx_handle_exit_irqoff, + .handle_exit_irqoff = vt_handle_exit_irqoff, .request_immediate_exit = vt_request_immediate_exit,
arch/x86/kvm/vmx/tdx.c +88 -0
106 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +10 -0
@@ -161,11 +161,16 @@ void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu);void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu); +void tdx_handle_exit_irqoff (struct kvm_vcpu *vcpu); +int tdx_handle_exit (struct kvm_vcpu *vcpu, +enum exit_fastpath_completion fastpath);u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector);void tdx_inject_nmi (struct kvm_vcpu *vcpu); +void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason, +u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -197,11 +202,16 @@ static inline void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_put (struct kvm_vcpu *vcpu) {}static inline void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {}static inline bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) {return false; } +static inline void tdx_handle_exit_irqoff (struct kvm_vcpu *vcpu) {} +static inline int tdx_handle_exit (struct kvm_vcpu *vcpu, +enum exit_fastpath_completion fastpath) {return 0 ; }static inline u8 tdx_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {return 0 ; }static inline void tdx_deliver_interrupt (struct kvm_lapic *apic,int delivery_mode,int trig_mode,int vector) {}static inline void tdx_inject_nmi (struct kvm_vcpu *vcpu) {} +static inline void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1, +u64 *info2,u32 *intr_info,u32 *error_code) {}static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
Signature · 1 line
2.25.1
PATCH v12 77/106 KVM: TDX: Handle vmentry failure for INTEL TD guest
In reply to [email protected]From: Yao Yuan <[email protected]>
TDX module passes control back to VMM if it failed to vmentry for a TD, use same exit reason to notify user space, align with VMX.
If VMM corrupted TD VMCS, machine check during entry can happens. vm exit reason will be EXIT_REASON_MCE_DURING_VMENTRY. If VMM corrupted TD VMCS with debug TD by TDH.VP.WR, the exit reason would be EXIT_REASON_INVALID_STATE or EXIT_REASON_MSR_LOAD_FAIL.
--- arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/x86/kvm/vmx/tdx.c +22 -0
@@ -1138,6 +1138,28 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) goto unhandled_exit; } +/* +* When TDX module saw VMEXIT_REASON_FAILED_VMENTER_MC etc, TDH.VP.ENTER +* returns with TDX_SUCCESS | exit_reason with failed_vmentry = 1. +* Because TDX module maintains TD VMCS correctness, usually vmentry +* failure shouldn't happen. In some corner cases it can happen. For +* example +* - machine check during entry: EXIT_REASON_MCE_DURING_VMENTRY +* - TDH.VP.WR with debug TD. VMM can corrupt TD VMCS +* - EXIT_REASON_INVALID_STATE +* - EXIT_REASON_MSR_LOAD_FAIL +*/ +if (unlikely (exit_reason.failed_vmentry)) { +pr_err ("TDExit: exit_reason 0x%016llx qualification=%016lx ext_qualification=%016lx\n" , + exit_reason.full,tdexit_exit_qual (vcpu),tdexit_ext_exit_qual (vcpu)); + vcpu->run->exit_reason =KVM_EXIT_FAIL_ENTRY ; + vcpu->run->fail_entry.hardware_entry_failure_reason + = exit_reason.full; + vcpu->run->fail_entry.cpu = vcpu->arch.last_vmentry_cpu; + +return 0 ; + } +WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) {
Signature · 1 line
2.25.1
PATCH v12 78/106 KVM: TDX: handle EXIT_REASON_OTHER_SMI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
If the control reaches EXIT_REASON_OTHER_SMI, #SMI is delivered and handled right after returning from the TDX module to KVM nothing needs to be done in KVM. Continue TDX vcpu execution.
--- arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kvm/vmx/tdx.c | 7 +++++++ 2 files changed, 8 insertions(+)
arch/x86/include/uapi/asm/vmx.h +1 -0
@@ -34,6 +34,7 @@#define EXIT_REASON_TRIPLE_FAULT 2#define EXIT_REASON_INIT_SIGNAL 3#define EXIT_REASON_SIPI_SIGNAL 4 +#define EXIT_REASON_OTHER_SMI 6#define EXIT_REASON_INTERRUPT_WINDOW 7#define EXIT_REASON_NMI_WINDOW 8
arch/x86/kvm/vmx/tdx.c +7 -0
@@ -1163,6 +1163,13 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath)WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) { +case EXIT_REASON_OTHER_SMI : +/* +* If reach here, it's not a Machine Check System Management +* Interrupt(MSMI). #SMI is delivered and handled right after +* SEAMRET, nothing needs to be done in KVM. +*/ +return 1 ;default :break ; }
Signature · 1 line
2.25.1
PATCH v12 79/106 KVM: TDX: handle ept violation/misconfig exit
In reply to [email protected]From: Isaku Yamahata <[email protected]>
On EPT violation, call a common function, __vmx_handle_ept_violation() to trigger x86 MMU code. On EPT misconfiguration, exit to ring 3 with KVM_EXIT_UNKNOWN. because EPT misconfiguration can't happen as MMIO is trigged by TDG.VP.VMCALL. No point to set a misconfiguration value for the fast path.
--- arch/x86/kvm/vmx/tdx.c | 46 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)
arch/x86/kvm/vmx/tdx.c +46 -0
@@ -1119,6 +1119,48 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,__vmx_deliver_posted_interrupt (vcpu, &tdx->pi_desc, vector); } +static int tdx_handle_ept_violation (struct kvm_vcpu *vcpu) +{ +unsigned long exit_qual; + +if (kvm_is_private_gpa (vcpu->kvm,tdexit_gpa (vcpu))) { +/* +* Always treat SEPT violations as write faults. Ignore the +* EXIT_QUALIFICATION reported by TDX-SEAM for SEPT violations. +* TD private pages are always RWX in the SEPT tables, +* i.e. they're always mapped writable. Just as importantly, +* treating SEPT violations as write faults is necessary to +* avoid COW allocations, which will cause TDAUGPAGE failures +* due to aliasing a single HPA to multiple GPAs. +*/ +#define TDX_SEPT_VIOLATION_EXIT_QUAL EPT_VIOLATION_ACC_WRITE + exit_qual =TDX_SEPT_VIOLATION_EXIT_QUAL ; + }else { + exit_qual =tdexit_exit_qual (vcpu); +if (exit_qual &EPT_VIOLATION_ACC_INSTR ) { +pr_warn ("kvm: TDX instr fetch to shared GPA = 0x%lx @ RIP = 0x%lx\n" , +tdexit_gpa (vcpu),kvm_rip_read (vcpu)); + vcpu->run->exit_reason =KVM_EXIT_EXCEPTION ; + vcpu->run->ex.exception =PF_VECTOR ; + vcpu->run->ex.error_code = exit_qual; +return 0 ; + } + } + +trace_kvm_page_fault (vcpu,tdexit_gpa (vcpu), exit_qual); +return __vmx_handle_ept_violation (vcpu,tdexit_gpa (vcpu), exit_qual); +} + +static int tdx_handle_ept_misconfig (struct kvm_vcpu *vcpu) +{ +WARN_ON_ONCE (1 ); + + vcpu->run->exit_reason =KVM_EXIT_UNKNOWN ; + vcpu->run->hw.hardware_exit_reason =EXIT_REASON_EPT_MISCONFIG ; + +return 0 ; +} +int tdx_handle_exit (struct kvm_vcpu *vcpu,fastpath_t fastpath) {union tdx_exit_reason exit_reason =to_tdx (vcpu)->exit_reason;
@@ -1163,6 +1205,10 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath)WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) { +case EXIT_REASON_EPT_VIOLATION : +return tdx_handle_ept_violation (vcpu); +case EXIT_REASON_EPT_MISCONFIG : +return tdx_handle_ept_misconfig (vcpu);case EXIT_REASON_OTHER_SMI :/* * If reach here, it's not a Machine Check System Management
Signature · 1 line
2.25.1
PATCH v12 80/106 KVM: TDX: handle EXCEPTION_NMI and EXTERNAL_INTERRUPT
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because guest TD state is protected, exceptions in guest TDs can't be intercepted. TDX VMM doesn't need to handle exceptions. tdx_handle_exit_irqoff() handles NMI and machine check. Ignore NMI and machine check and continue guest TD execution.
For external interrupt, increment stats same to the VMX case.
--- arch/x86/kvm/vmx/tdx.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
arch/x86/kvm/vmx/tdx.c +23 -0
@@ -786,6 +786,25 @@ void tdx_handle_exit_irqoff(struct kvm_vcpu *vcpu) tdexit_intr_info(vcpu)); } +static int tdx_handle_exception (struct kvm_vcpu *vcpu) +{ +u32 intr_info =tdexit_intr_info (vcpu); + +if (is_nmi (intr_info) ||is_machine_check (intr_info)) +return 1 ; + +kvm_pr_unimpl ("unexpected exception 0x%x(exit_reason 0x%llx qual 0x%lx)\n" , + intr_info, +to_tdx (vcpu)->exit_reason.full,tdexit_exit_qual (vcpu)); +return -EFAULT ; +} + +static int tdx_handle_external_interrupt (struct kvm_vcpu *vcpu) +{ + ++vcpu->stat.irq_exits; +return 1 ; +} +static int tdx_handle_triple_fault (struct kvm_vcpu *vcpu) { vcpu->run->exit_reason =KVM_EXIT_SHUTDOWN ;
@@ -1205,6 +1224,10 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath)WARN_ON_ONCE (fastpath !=EXIT_FASTPATH_NONE );switch (exit_reason.basic) { +case EXIT_REASON_EXCEPTION_NMI : +return tdx_handle_exception (vcpu); +case EXIT_REASON_EXTERNAL_INTERRUPT : +return tdx_handle_external_interrupt (vcpu);case EXIT_REASON_EPT_VIOLATION :return tdx_handle_ept_violation (vcpu);case EXIT_REASON_EPT_MISCONFIG :
Signature · 1 line
2.25.1
PATCH v12 81/106 KVM: TDX: Add a place holder for handler of TDX hypercalls (TDG.VP.VMCALL)
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX module specification defines TDG.VP.VMCALL API (TDVMCALL for short) for the guest TD to call hypercall to VMM. When the guest TD issues TDG.VP.VMCALL, the guest TD exits to VMM with a new exit reason of TDVMCALL. The arguments from the guest TD and returned values from the VMM are passed in the guest registers. The guest RCX registers indicates which registers are used. Define helper functions to access those registers as ABI.
Define the TDVMCALL exit reason, which is carved out from the VMX exit reason namespace as the TDVMCALL exit from TDX guest to TDX-SEAM is really just a VM-Exit. Add a place holder to handle TDVMCALL exit.
--- arch/x86/include/uapi/asm/vmx.h | 4 ++- arch/x86/kvm/vmx/tdx.c | 56 ++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 13 ++++++++ 3 files changed, 71 insertions(+), 2 deletions(-)
arch/x86/include/uapi/asm/vmx.h +3 -1
@@ -93,6 +93,7 @@#define EXIT_REASON_TPAUSE 68#define EXIT_REASON_BUS_LOCK 74#define EXIT_REASON_NOTIFY 75 +#define EXIT_REASON_TDCALL 77#define VMX_EXIT_REASONS \ { EXIT_REASON_EXCEPTION_NMI, "EXCEPTION_NMI" }, \
@@ -156,7 +157,8 @@ { EXIT_REASON_UMWAIT, "UMWAIT" }, \ { EXIT_REASON_TPAUSE, "TPAUSE" }, \ { EXIT_REASON_BUS_LOCK, "BUS_LOCK" }, \ - { EXIT_REASON_NOTIFY, "NOTIFY" } + { EXIT_REASON_NOTIFY, "NOTIFY" }, \ + { EXIT_REASON_TDCALL, "TDCALL" } #define VMX_EXIT_REASON_FLAGS \ { VMX_EXIT_REASONS_FAILED_VMENTRY, "FAILED_VMENTRY" }
arch/x86/kvm/vmx/tdx.c +55 -1
86 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +13 -0
@@ -72,6 +72,19 @@ struct vcpu_tdx {struct list_head cpu_list; +union { +struct { +union { +struct { +u16 gpr_mask; +u16 xmm_mask; + }; +u32 regs_mask; + }; +u32 reserved; + }; +u64 rcx; + } tdvmcall;union tdx_exit_reason exit_reason;bool initialized;
Signature · 1 line
2.25.1
PATCH v12 82/106 KVM: TDX: handle KVM hypercall with TDG.VP.VMCALL
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX Guest-Host communication interface (GHCI) specification defines
the ABI for the guest TD to issue hypercall. It reserves vendor specific
arguments for VMM specific use. Use it as KVM hypercall and handle it.
--- arch/x86/kvm/vmx/tdx.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
arch/x86/kvm/vmx/tdx.c +31 -0
@@ -853,8 +853,39 @@ static int tdx_handle_triple_fault(struct kvm_vcpu *vcpu)return 0 ; } +static int tdx_emulate_vmcall (struct kvm_vcpu *vcpu) +{ +unsigned long nr, a0, a1, a2, a3, ret; + +/* +* ABI for KVM tdvmcall argument: +* In Guest-Hypervisor Communication Interface(GHCI) specification, +* Non-zero leaf number (R10 != 0) is defined to indicate +* vendor-specific. KVM uses this for KVM hypercall. NOTE: KVM +* hypercall number starts from one. Zero isn't used for KVM hypercall +* number. +* +* R10: KVM hypercall number +* arguments: R11, R12, R13, R14. +*/ + nr =kvm_r10_read (vcpu); + a0 =kvm_r11_read (vcpu); + a1 =kvm_r12_read (vcpu); + a2 =kvm_r13_read (vcpu); + a3 =kvm_r14_read (vcpu); + + ret =__kvm_emulate_hypercall (vcpu, nr, a0, a1, a2, a3, true); + +tdvmcall_set_return_code (vcpu, ret); + +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) { +if (tdvmcall_exit_type (vcpu)) +return tdx_emulate_vmcall (vcpu); +switch (tdvmcall_leaf (vcpu)) {default :break ;
Signature · 1 line
2.25.1
PATCH v12 83/106 KVM: TDX: Add KVM Exit for TDX TDG.VP.VMCALL
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Some of TDG.VP.VMCALL require device model, for example, qemu, to handle them on behalf of kvm kernel module.
Introduce new kvm exit, KVM_EXIT_TDX, and functions to setup it. TDG_VP_VMCALL_INVALID_OPERAND is set as default return value to avoid random value. Device model should update R10 if necessary.
--- arch/x86/kvm/vmx/tdx.c | 93 +++++++++++++++++++++++++++++++++++++++- include/uapi/linux/kvm.h | 57 ++++++++++++++++++++++++ 2 files changed, 148 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/tdx.c +91 -2
@@ -123,6 +123,18 @@ BUILD_TDVMCALL_ACCESSORS(a1, r13);BUILD_TDVMCALL_ACCESSORS (a2, r14);BUILD_TDVMCALL_ACCESSORS (a3, r15); +#define TDX_VMCALL_REG_MASK_RBX BIT_ULL(2) +#define TDX_VMCALL_REG_MASK_RDX BIT_ULL(3) +#define TDX_VMCALL_REG_MASK_RBP BIT_ULL(5) +#define TDX_VMCALL_REG_MASK_RSI BIT_ULL(6) +#define TDX_VMCALL_REG_MASK_RDI BIT_ULL(7) +#define TDX_VMCALL_REG_MASK_R8 BIT_ULL(8) +#define TDX_VMCALL_REG_MASK_R9 BIT_ULL(9) +#define TDX_VMCALL_REG_MASK_R12 BIT_ULL(12) +#define TDX_VMCALL_REG_MASK_R13 BIT_ULL(13) +#define TDX_VMCALL_REG_MASK_R14 BIT_ULL(14) +#define TDX_VMCALL_REG_MASK_R15 BIT_ULL(15) +static __always_inline unsigned long tdvmcall_exit_type (struct kvm_vcpu *vcpu) {return kvm_r10_read (vcpu);
@@ -881,6 +893,80 @@ static int tdx_emulate_vmcall(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_complete_vp_vmcall (struct kvm_vcpu *vcpu) +{ +struct kvm_tdx_vmcall *tdx_vmcall = &vcpu->run->tdx.u.vmcall; +__u64 reg_mask; + +tdvmcall_set_return_code (vcpu, tdx_vmcall->status_code); +tdvmcall_set_return_val (vcpu, tdx_vmcall->out_r11); + + reg_mask =kvm_rcx_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_R12 ) +kvm_r12_write (vcpu, tdx_vmcall->out_r12); +if (reg_mask &TDX_VMCALL_REG_MASK_R13 ) +kvm_r13_write (vcpu, tdx_vmcall->out_r13); +if (reg_mask &TDX_VMCALL_REG_MASK_R14 ) +kvm_r14_write (vcpu, tdx_vmcall->out_r14); +if (reg_mask &TDX_VMCALL_REG_MASK_R15 ) +kvm_r15_write (vcpu, tdx_vmcall->out_r15); +if (reg_mask &TDX_VMCALL_REG_MASK_RBX ) +kvm_rbx_write (vcpu, tdx_vmcall->out_rbx); +if (reg_mask &TDX_VMCALL_REG_MASK_RDI ) +kvm_rdi_write (vcpu, tdx_vmcall->out_rdi); +if (reg_mask &TDX_VMCALL_REG_MASK_RSI ) +kvm_rsi_write (vcpu, tdx_vmcall->out_rsi); +if (reg_mask &TDX_VMCALL_REG_MASK_R8 ) +kvm_r8_write (vcpu, tdx_vmcall->out_r8); +if (reg_mask &TDX_VMCALL_REG_MASK_R9 ) +kvm_r9_write (vcpu, tdx_vmcall->out_r9); +if (reg_mask &TDX_VMCALL_REG_MASK_RDX ) +kvm_rdx_write (vcpu, tdx_vmcall->out_rdx); + +return 1 ; +} + +static int tdx_vp_vmcall_to_user (struct kvm_vcpu *vcpu) +{ +struct kvm_tdx_vmcall *tdx_vmcall = &vcpu->run->tdx.u.vmcall; +__u64 reg_mask; + + vcpu->arch.complete_userspace_io = tdx_complete_vp_vmcall; +memset (tdx_vmcall,0 ,sizeof (*tdx_vmcall)); + + vcpu->run->exit_reason =KVM_EXIT_TDX ; + vcpu->run->tdx.type =KVM_EXIT_TDX_VMCALL ; + tdx_vmcall->type =tdvmcall_exit_type (vcpu); + tdx_vmcall->subfunction =tdvmcall_leaf (vcpu); + tdx_vmcall->status_code =TDG_VP_VMCALL_INVALID_OPERAND ; + + reg_mask =kvm_rcx_read (vcpu); + tdx_vmcall->reg_mask = reg_mask; +if (reg_mask &TDX_VMCALL_REG_MASK_R12 ) + tdx_vmcall->in_r12 =kvm_r12_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_R13 ) + tdx_vmcall->in_r13 =kvm_r13_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_R14 ) + tdx_vmcall->in_r14 =kvm_r14_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_R15 ) + tdx_vmcall->in_r15 =kvm_r15_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_RBX ) + tdx_vmcall->in_rbx =kvm_rbx_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_RDI ) + tdx_vmcall->in_rdi =kvm_rdi_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_RSI ) + tdx_vmcall->in_rsi =kvm_rsi_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_R8 ) + tdx_vmcall->in_r8 =kvm_r8_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_R9 ) + tdx_vmcall->in_r9 =kvm_r9_read (vcpu); +if (reg_mask &TDX_VMCALL_REG_MASK_RDX ) + tdx_vmcall->in_rdx =kvm_rdx_read (vcpu); + +/* notify userspace to handle the request */ +return 0 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -891,8 +977,11 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)break ; } -tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); -return 1 ; +/* +* Unknown VMCALL. Toss the request to the user space as it may know +* how to handle. +*/ +return tdx_vp_vmcall_to_user (vcpu); }void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int pgd_level)
include/uapi/linux/kvm.h +57 -0
75 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 84/106 KVM: TDX: Handle TDX PV CPUID hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV CPUID hypercall to the KVM backend function.
--- arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/x86/kvm/vmx/tdx.c +22 -0
@@ -967,12 +967,34 @@ static int tdx_vp_vmcall_to_user(struct kvm_vcpu *vcpu)return 0 ; } +static int tdx_emulate_cpuid (struct kvm_vcpu *vcpu) +{ +u32 eax, ebx, ecx, edx; + +/* EAX and ECX for cpuid is stored in R12 and R13. */ + eax =tdvmcall_a0_read (vcpu); + ecx =tdvmcall_a1_read (vcpu); + +kvm_cpuid (vcpu, &eax, &ebx, &ecx, &edx, false); + +tdvmcall_a0_write (vcpu, eax); +tdvmcall_a1_write (vcpu, ebx); +tdvmcall_a2_write (vcpu, ecx); +tdvmcall_a3_write (vcpu, edx); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); + +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))return tdx_emulate_vmcall (vcpu);switch (tdvmcall_leaf (vcpu)) { +case EXIT_REASON_CPUID : +return tdx_emulate_cpuid (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v12 85/106 KVM: TDX: Handle TDX PV HLT hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV HLT hypercall to the KVM backend function.
--- arch/x86/kvm/vmx/tdx.c | 42 +++++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 3 +++ 2 files changed, 44 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/tdx.c +41 -1
@@ -628,7 +628,32 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu) { -return pi_has_pending_interrupt (vcpu); +bool ret =pi_has_pending_interrupt (vcpu); +struct vcpu_tdx *tdx =to_tdx (vcpu); + +if (ret || vcpu->arch.mp_state !=KVM_MP_STATE_HALTED ) +return true; + +if (tdx->interrupt_disabled_hlt) +return false; + +/* +* This is for the case where the virtual interrupt is recognized, +* i.e. set in vmcs.RVI, between the STI and "HLT". KVM doesn't have +* access to RVI and the interrupt is no longer in the PID (because it +* was "recognized". It doesn't get delivered in the guest because the +* TDCALL completes before interrupts are enabled. +* +* TDX modules sets RVI while in an STI interrupt shadow. +* - TDExit(typically TDG.VP.VMCALL<HLT>) from the guest to TDX module. +* The interrupt shadow at this point is gone. +* - It knows that there is an interrupt that can be delivered +* (RVI > PPR && EFLAGS.IF=1, the other conditions of 29.2.2 don't +* matter) +* - It forwards the TDExit nevertheless, to a clueless hypervisor that +* has no way to glean either RVI or PPR. +*/ +return !!xchg (&tdx->buggy_hlt_workaround,0 ); }void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu)
@@ -987,6 +1012,17 @@ static int tdx_emulate_cpuid(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_emulate_hlt (struct kvm_vcpu *vcpu) +{ +struct vcpu_tdx *tdx =to_tdx (vcpu); + +/* See tdx_protected_apic_has_interrupt() to avoid heavy seamcall */ + tdx->interrupt_disabled_hlt =tdvmcall_a0_read (vcpu); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return kvm_emulate_halt_noskip (vcpu); +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -995,6 +1031,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)switch (tdvmcall_leaf (vcpu)) {case EXIT_REASON_CPUID :return tdx_emulate_cpuid (vcpu); +case EXIT_REASON_HLT : +return tdx_emulate_hlt (vcpu);default :break ; }
@@ -1328,6 +1366,8 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,struct kvm_vcpu *vcpu = apic->vcpu;struct vcpu_tdx *tdx =to_tdx (vcpu); +/* See comment in tdx_protected_apic_has_interrupt(). */ + tdx->buggy_hlt_workaround =1 ;/* TDX supports only posted interrupt. No lapic emulation. */ __vmx_deliver_posted_interrupt (vcpu, &tdx->pi_desc, vector); }
arch/x86/kvm/vmx/tdx.h +3 -0
@@ -93,6 +93,9 @@ struct vcpu_tdx {bool host_state_need_restore;u64 msr_host_kernel_gs_base; +bool interrupt_disabled_hlt; +unsigned int buggy_hlt_workaround; +/* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work.
Signature · 1 line
2.25.1
PATCH v12 86/106 KVM: TDX: Handle TDX PV port io hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV port IO hypercall to the KVM backend function.
--- arch/x86/kvm/vmx/tdx.c | 57 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
arch/x86/kvm/vmx/tdx.c +57 -0
@@ -1023,6 +1023,61 @@ static int tdx_emulate_hlt(struct kvm_vcpu *vcpu)return kvm_emulate_halt_noskip (vcpu); } +static int tdx_complete_pio_in (struct kvm_vcpu *vcpu) +{ +struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt; +unsigned long val =0 ; +int ret; + +WARN_ON_ONCE (vcpu->arch.pio.count !=1 ); + + ret = ctxt->ops->pio_in_emulated (ctxt, vcpu->arch.pio.size, + vcpu->arch.pio.port, &val,1 ); +WARN_ON_ONCE (!ret); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +tdvmcall_set_return_val (vcpu, val); + +return 1 ; +} + +static int tdx_emulate_io (struct kvm_vcpu *vcpu) +{ +struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt; +unsigned long val =0 ; +unsigned int port; +int size, ret; +bool write; + + ++vcpu->stat.io_exits; + + size =tdvmcall_a0_read (vcpu); + write =tdvmcall_a1_read (vcpu); + port =tdvmcall_a2_read (vcpu); + +if (size !=1 && size !=2 && size !=4 ) { +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } + +if (write) { + val =tdvmcall_a3_read (vcpu); + ret = ctxt->ops->pio_out_emulated (ctxt, size, port, &val,1 ); + +/* No need for a complete_userspace_io callback. */ + vcpu->arch.pio.count =0 ; + }else { + ret = ctxt->ops->pio_in_emulated (ctxt, size, port, &val,1 ); +if (!ret) + vcpu->arch.complete_userspace_io = tdx_complete_pio_in; +else +tdvmcall_set_return_val (vcpu, val); + } +if (ret) +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return ret; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1033,6 +1088,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_cpuid (vcpu);case EXIT_REASON_HLT :return tdx_emulate_hlt (vcpu); +case EXIT_REASON_IO_INSTRUCTION : +return tdx_emulate_io (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v12 87/106 KVM: TDX: Handle TDX PV MMIO hypercall
In reply to [email protected]From: Sean Christopherson <[email protected]>
Export kvm_io_bus_read and kvm_mmio tracepoint and wire up TDX PV MMIO hypercall to the KVM backend functions.
kvm_io_bus_read/write() searches KVM device emulated in kernel of the given MMIO address and emulates the MMIO. As TDX PV MMIO also needs it, export kvm_io_bus_read(). kvm_io_bus_write() is already exported. TDX PV MMIO emulates some of MMIO itself. To add trace point consistently with x86 kvm, export kvm_mmio tracepoint.
--- arch/x86/kvm/vmx/tdx.c | 114 +++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 1 + virt/kvm/kvm_main.c | 2 + 3 files changed, 117 insertions(+)
arch/x86/kvm/vmx/tdx.c +114 -0
@@ -1078,6 +1078,118 @@ static int tdx_emulate_io(struct kvm_vcpu *vcpu)return ret; } +static int tdx_complete_mmio (struct kvm_vcpu *vcpu) +{ +unsigned long val =0 ; +gpa_t gpa; +int size; + +KVM_BUG_ON (vcpu->mmio_needed !=1 , vcpu->kvm); + vcpu->mmio_needed =0 ; + +if (!vcpu->mmio_is_write) { + gpa = vcpu->mmio_fragments[0 ].gpa; + size = vcpu->mmio_fragments[0 ].len; + +memcpy (&val, vcpu->run->mmio.data, size); +tdvmcall_set_return_val (vcpu, val); +trace_kvm_mmio (KVM_TRACE_MMIO_READ , size, gpa, &val); + } +return 1 ; +} + +static inline int tdx_mmio_write (struct kvm_vcpu *vcpu,gpa_t gpa,int size, +unsigned long val) +{ +if (kvm_iodevice_write (vcpu, &vcpu->arch.apic->dev, gpa, size, &val) && +kvm_io_bus_write (vcpu,KVM_MMIO_BUS , gpa, size, &val)) +return -EOPNOTSUPP ; + +trace_kvm_mmio (KVM_TRACE_MMIO_WRITE , size, gpa, &val); +return 0 ; +} + +static inline int tdx_mmio_read (struct kvm_vcpu *vcpu,gpa_t gpa,int size) +{ +unsigned long val; + +if (kvm_iodevice_read (vcpu, &vcpu->arch.apic->dev, gpa, size, &val) && +kvm_io_bus_read (vcpu,KVM_MMIO_BUS , gpa, size, &val)) +return -EOPNOTSUPP ; + +tdvmcall_set_return_val (vcpu, val); +trace_kvm_mmio (KVM_TRACE_MMIO_READ , size, gpa, &val); +return 0 ; +} + +static int tdx_emulate_mmio (struct kvm_vcpu *vcpu) +{ +struct kvm_memory_slot *slot; +int size, write, r; +unsigned long val; +gpa_t gpa; + +KVM_BUG_ON (vcpu->mmio_needed, vcpu->kvm); + + size =tdvmcall_a0_read (vcpu); + write =tdvmcall_a1_read (vcpu); + gpa =tdvmcall_a2_read (vcpu); + val = write ?tdvmcall_a3_read (vcpu) :0 ; + +if (size !=1 && size !=2 && size !=4 && size !=8 ) + goto error; +if (write !=0 && write !=1 ) + goto error; + +/* Strip the shared bit, allow MMIO with and without it set. */ + gpa = gpa & ~gfn_to_gpa (kvm_gfn_shared_mask (vcpu->kvm)); + +if (size >8u || ((gpa + size -1 ) ^ gpa) &PAGE_MASK ) + goto error; + + slot =kvm_vcpu_gfn_to_memslot (vcpu,gpa_to_gfn (gpa)); +if (slot && !(slot->flags &KVM_MEMSLOT_INVALID )) + goto error; + +if (!kvm_io_bus_write (vcpu,KVM_FAST_MMIO_BUS , gpa,0 ,NULL )) { +trace_kvm_fast_mmio (gpa); +return 1 ; + } + +if (write) + r =tdx_mmio_write (vcpu, gpa, size, val); +else + r =tdx_mmio_read (vcpu, gpa, size); +if (!r) { +/* Kernel completed device emulation. */ +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return 1 ; + } + +/* Request the device emulation to userspace device model. */ + vcpu->mmio_needed =1 ; + vcpu->mmio_is_write = write; + vcpu->arch.complete_userspace_io = tdx_complete_mmio; + + vcpu->run->mmio.phys_addr = gpa; + vcpu->run->mmio.len = size; + vcpu->run->mmio.is_write = write; + vcpu->run->exit_reason =KVM_EXIT_MMIO ; + +if (write) { +memcpy (vcpu->run->mmio.data, &val, size); + }else { + vcpu->mmio_fragments[0 ].gpa = gpa; + vcpu->mmio_fragments[0 ].len = size; +trace_kvm_mmio (KVM_TRACE_MMIO_READ_UNSATISFIED , size, gpa,NULL ); + } +return 0 ; + +error: +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1090,6 +1202,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_hlt (vcpu);case EXIT_REASON_IO_INSTRUCTION :return tdx_emulate_io (vcpu); +case EXIT_REASON_EPT_VIOLATION : +return tdx_emulate_mmio (vcpu);default :break ; }
arch/x86/kvm/x86.c +1 -0
@@ -13564,6 +13564,7 @@ bool kvm_arch_dirty_log_supported(struct kvm *kvm)EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_entry);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_exit); +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_mmio);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_fast_mmio);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_inj_virq);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_page_fault);
virt/kvm/kvm_main.c +2 -0
@@ -2683,6 +2683,7 @@ struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfnreturn NULL ; } +EXPORT_SYMBOL_GPL (kvm_vcpu_gfn_to_memslot);bool kvm_is_visible_gfn (struct kvm *kvm,gfn_t gfn) {
@@ -5839,6 +5840,7 @@ int kvm_io_bus_read(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t addr, r =__kvm_io_bus_read (vcpu, bus, &range, val);return r <0 ? r :0 ; } +EXPORT_SYMBOL_GPL (kvm_io_bus_read);/* Caller must hold slots_lock. */ int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
Signature · 1 line
2.25.1
PATCH v12 88/106 KVM: TDX: Implement callbacks for MSR operations for TDX
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implements set_msr/get_msr/has_emulated_msr methods for TDX to handle hypercall from guest TD for paravirtualized rdmsr and wrmsr. The TDX module virtualizes MSRs. For some MSRs, it injects #VE to the guest TD upon RDMSR or WRMSR. The exact list of such MSRs are defined in the spec.
Upon #VE, the guest TD may execute hypercalls, TDG.VP.VMCALL<INSTRUCTION.RDMSR> and TDG.VP.VMCALL<INSTRUCTION.WRMSR>, which are defined in GHCI (Guest-Host Communication Interface) so that the host VMM (e.g. KVM) can virtualize the MSRs.
There are three classes of MSRs virtualization.
- non-configurable: TDX module directly virtualizes it. VMM can't configure. the value set by KVM_SET_MSR_INDEX_LIST is ignored.
- configurable: TDX module directly virtualizes it. VMM can configure at the VM creation time. The value set by KVM_SET_MSR_INDEX_LIST is used.
- #VE case
Guest TD would issue TDG.VP.VMCALL<INSTRUCTION.{WRMSR,RDMSR> and VMM handles the MSR hypercall. The value set by KVM_SET_MSR_INDEX_LIST is used.
---
Changes v10 -> v11
- added .msr_filter_changed()
--- arch/x86/kvm/vmx/main.c | 44 ++++++++++++++++++++--- arch/x86/kvm/vmx/tdx.c | 74 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 6 ++++ arch/x86/kvm/x86.c | 1 - arch/x86/kvm/x86.h | 2 ++ 5 files changed, 122 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/main.c +40 -4
@@ -237,6 +237,42 @@ static void vt_handle_exit_irqoff(struct kvm_vcpu *vcpu)vmx_handle_exit_irqoff (vcpu); } +static int vt_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr_info) +{ +if (unlikely (is_td_vcpu (vcpu))) +return tdx_set_msr (vcpu, msr_info); + +return vmx_set_msr (vcpu, msr_info); +} + +/* +* The kvm parameter can be NULL (module initialization, or invocation before +* VM creation). Be sure to check the kvm parameter before using it. +*/ +static bool vt_has_emulated_msr (struct kvm *kvm,u32 index) +{ +if (kvm &&is_td (kvm)) +return tdx_has_emulated_msr (index, true); + +return vmx_has_emulated_msr (kvm, index); +} + +static int vt_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr_info) +{ +if (unlikely (is_td_vcpu (vcpu))) +return tdx_get_msr (vcpu, msr_info); + +return vmx_get_msr (vcpu, msr_info); +} + +static void vt_msr_filter_changed (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_msr_filter_changed (vcpu); +} +static void vt_apicv_post_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -500,7 +536,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .hardware_enable = vmx_hardware_enable, .hardware_disable = vt_hardware_disable, - .has_emulated_msr = vmx_has_emulated_msr, + .has_emulated_msr = vt_has_emulated_msr, .is_vm_type_supported = vt_is_vm_type_supported, .max_vcpus = vt_max_vcpus,
@@ -522,8 +558,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .update_exception_bitmap = vmx_update_exception_bitmap, .get_msr_feature = vmx_get_msr_feature, - .get_msr = vmx_get_msr, - .set_msr = vmx_set_msr, + .get_msr = vt_get_msr, + .set_msr = vt_set_msr, .get_segment_base = vmx_get_segment_base, .get_segment = vmx_get_segment, .set_segment = vmx_set_segment,
@@ -632,7 +668,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .apic_init_signal_blocked = vmx_apic_init_signal_blocked, .migrate_timers = vmx_migrate_timers, - .msr_filter_changed = vmx_msr_filter_changed, + .msr_filter_changed = vt_msr_filter_changed, .complete_emulated_msr = kvm_complete_insn_gp, .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector,
arch/x86/kvm/vmx/tdx.c +74 -0
80 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +6 -0
@@ -171,6 +171,9 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,void tdx_inject_nmi (struct kvm_vcpu *vcpu);void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code); +bool tdx_has_emulated_msr (u32 index,bool write); +int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr); +int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);
@@ -212,6 +215,9 @@ static inline void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mostatic inline void tdx_inject_nmi (struct kvm_vcpu *vcpu) {}static inline void tdx_get_exit_info (struct kvm_vcpu *vcpu,u32 *reason,u64 *info1,u64 *info2,u32 *intr_info,u32 *error_code) {} +static inline bool tdx_has_emulated_msr (u32 index,bool write) {return false; } +static inline int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; } +static inline int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; }static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }
arch/x86/kvm/x86.c +0 -1
@@ -87,7 +87,6 @@#include "trace.h" #define MAX_IO_MSRS 256 -#define KVM_MAX_MCE_BANKS 32struct kvm_caps kvm_caps__read_mostly = { .supported_mce_cap =MCG_CTL_P |MCG_SER_P ,
arch/x86/kvm/x86.h +2 -0
@@ -8,6 +8,8 @@#include "kvm_cache_regs.h" #include "kvm_emulate.h" +#define KVM_MAX_MCE_BANKS 32 +bool __kvm_is_vm_type_supported (unsigned long type);struct kvm_caps {
Signature · 1 line
2.25.1
PATCH v12 89/106 KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV rdmsr/wrmsr hypercall to the KVM backend function.
--- arch/x86/kvm/vmx/tdx.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
arch/x86/kvm/vmx/tdx.c +39 -0
@@ -1190,6 +1190,41 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_emulate_rdmsr (struct kvm_vcpu *vcpu) +{ +u32 index =tdvmcall_a0_read (vcpu); +u64 data; + +if (!kvm_msr_allowed (vcpu, index,KVM_MSR_FILTER_READ ) || +kvm_get_msr (vcpu, index, &data)) { +trace_kvm_msr_read_ex (index); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } +trace_kvm_msr_read (index, data); + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +tdvmcall_set_return_val (vcpu, data); +return 1 ; +} + +static int tdx_emulate_wrmsr (struct kvm_vcpu *vcpu) +{ +u32 index =tdvmcall_a0_read (vcpu); +u64 data =tdvmcall_a1_read (vcpu); + +if (!kvm_msr_allowed (vcpu, index,KVM_MSR_FILTER_WRITE ) || +kvm_set_msr (vcpu, index, data)) { +trace_kvm_msr_write_ex (index, data); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } + +trace_kvm_msr_write (index, data); +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1204,6 +1239,10 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_io (vcpu);case EXIT_REASON_EPT_VIOLATION :return tdx_emulate_mmio (vcpu); +case EXIT_REASON_MSR_READ : +return tdx_emulate_rdmsr (vcpu); +case EXIT_REASON_MSR_WRITE : +return tdx_emulate_wrmsr (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v12 90/106 KVM: TDX: Handle TDX PV report fatal error hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV report fatal error hypercall to exit to device model so that it can gracefully handle it.
--- arch/x86/kvm/vmx/tdx.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/kvm/vmx/tdx.c +7 -0
@@ -1243,6 +1243,13 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_rdmsr (vcpu);case EXIT_REASON_MSR_WRITE :return tdx_emulate_wrmsr (vcpu); +case TDG_VP_VMCALL_REPORT_FATAL_ERROR : +/* +* Exit to userspace device model for tear down. +* Because guest TD is already panicking, returning an error to +* guest TD doesn't make sense. No argument check is done. +*/ +return tdx_vp_vmcall_to_user (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v12 91/106 KVM: TDX: Handle TDX PV map_gpa hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Wire up TDX PV map_gpa hypercall to the kvm/mmu backend.
--- arch/x86/kvm/vmx/tdx.c | 53 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+)
arch/x86/kvm/vmx/tdx.c +53 -0
@@ -1225,6 +1225,57 @@ static int tdx_emulate_wrmsr(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_map_gpa (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +gpa_t gpa =tdvmcall_a0_read (vcpu); +gpa_t size =tdvmcall_a1_read (vcpu); +gpa_t end = gpa + size; +gfn_t s =gpa_to_gfn (gpa) & ~kvm_gfn_shared_mask (kvm); +gfn_t e =gpa_to_gfn (end) & ~kvm_gfn_shared_mask (kvm); +int i; + +if (!IS_ALIGNED (gpa,4096 ) || !IS_ALIGNED (size,4096 ) || + end < gpa || + end >kvm_gfn_shared_mask (kvm) << (PAGE_SHIFT +1 ) || +kvm_is_private_gpa (kvm, gpa) !=kvm_is_private_gpa (kvm, end)) { +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; + } + +/* +* Check how the requested region overlaps with the KVM memory slots. +* For simplicity, require that it must be contained within a memslot or +* it must not overlap with any memslots (MMIO). +*/ +for (i =0 ; i <kvm_arch_nr_memslot_as_ids (kvm); i++) { +struct kvm_memslots *slots =__kvm_memslots (kvm, i); +struct kvm_memslot_iter iter; + +kvm_for_each_memslot_in_gfn_range (&iter, slots, s, e) { +struct kvm_memory_slot *slot = iter.slot; +gfn_t slot_s = slot->base_gfn; +gfn_t slot_e = slot->base_gfn + slot->npages; + +/* no overlap */ +if (e < slot_s || s >= slot_e) +continue ; + +/* contained in slot */ +if (slot_s <= s && e <= slot_e) { +if (kvm_slot_can_be_private (slot)) +return tdx_vp_vmcall_to_user (vcpu); +continue ; + } + +break ; + } + } + +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +return 1 ; +} +static int handle_tdvmcall (struct kvm_vcpu *vcpu) {if (tdvmcall_exit_type (vcpu))
@@ -1250,6 +1301,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)* guest TD doesn't make sense. No argument check is done. */ return tdx_vp_vmcall_to_user (vcpu); +case TDG_VP_VMCALL_MAP_GPA : +return tdx_map_gpa (vcpu);default :break ; }
Signature · 1 line
2.25.1
PATCH v12 92/106 KVM: TDX: Handle TDG.VP.VMCALL<GetTdVmCallInfo> hypercall
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Implement TDG.VP.VMCALL<GetTdVmCallInfo> hypercall. If the input value is zero, return success code and zero in output registers.
TDG.VP.VMCALL<GetTdVmCallInfo> hypercall is a subleaf of TDG.VP.VMCALL to enumerate which TDG.VP.VMCALL sub leaves are supported. This hypercall is for future enhancement of the Guest-Host-Communication Interface (GHCI) specification. The GHCI version of 344426-001US defines it to require input R12 to be zero and to return zero in output registers, R11, R12, R13, and R14 so that guest TD enumerates no enhancement.
--- arch/x86/kvm/vmx/tdx.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
arch/x86/kvm/vmx/tdx.c +16 -0
@@ -1225,6 +1225,20 @@ static int tdx_emulate_wrmsr(struct kvm_vcpu *vcpu)return 1 ; } +static int tdx_get_td_vm_call_info (struct kvm_vcpu *vcpu) +{ +if (tdvmcall_a0_read (vcpu)) +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_INVALID_OPERAND ); +else { +tdvmcall_set_return_code (vcpu,TDG_VP_VMCALL_SUCCESS ); +kvm_r11_write (vcpu,0 ); +tdvmcall_a0_write (vcpu,0 ); +tdvmcall_a1_write (vcpu,0 ); +tdvmcall_a2_write (vcpu,0 ); + } +return 1 ; +} +static int tdx_map_gpa (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -1294,6 +1308,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)return tdx_emulate_rdmsr (vcpu);case EXIT_REASON_MSR_WRITE :return tdx_emulate_wrmsr (vcpu); +case TDG_VP_VMCALL_GET_TD_VM_CALL_INFO : +return tdx_get_td_vm_call_info (vcpu);case TDG_VP_VMCALL_REPORT_FATAL_ERROR :/* * Exit to userspace device model for tear down.
Signature · 1 line
2.25.1
PATCH v12 93/106 KVM: TDX: Silently discard SMI request
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX doesn't support system-management mode (SMM) and system-management interrupt (SMI) in guest TDs. Because guest state (vcpu state, memory state) is protected, it must go through the TDX module APIs to change guest state, injecting SMI and changing vcpu mode into SMM. The TDX module doesn't provide a way for VMM to inject SMI into guest TD and a way for VMM to switch guest vcpu mode into SMM.
We have two options in KVM when handling SMM or SMI in the guest TD or the device model (e.g. QEMU): 1) silently ignore the request or 2) return a meaningful error.
For simplicity, we implemented the option 1).
--- arch/x86/kvm/smm.h | 7 +++++- arch/x86/kvm/vmx/main.c | 45 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 29 ++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 12 ++++++++++ 4 files changed, 88 insertions(+), 5 deletions(-)
arch/x86/kvm/smm.h +6 -1
@@ -142,7 +142,12 @@ union kvm_smram {static inline int kvm_inject_smi (struct kvm_vcpu *vcpu) { -kvm_make_request (KVM_REQ_SMI , vcpu); +/* +* If SMM isn't supported (e.g. TDX), silently discard SMI request. +* Assume that SMM supported = MSR_IA32_SMBASE supported. +*/ +if (static_call (kvm_x86_has_emulated_msr)(vcpu->kvm,MSR_IA32_SMBASE )) +kvm_make_request (KVM_REQ_SMI , vcpu);return 0 ; }
arch/x86/kvm/vmx/main.c +41 -4
57 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.c +29 -0
@@ -1862,6 +1862,35 @@ int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)return 1 ; } +#ifdef CONFIG_KVM_SMM +int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection) +{ +/* SMI isn't supported for TDX. */ +WARN_ON_ONCE (1 ); +return false; +} + +int tdx_enter_smm (struct kvm_vcpu *vcpu,union kvm_smram *smram) +{ +/* smi_allowed() is always false for TDX as above. */ +WARN_ON_ONCE (1 ); +return 0 ; +} + +int tdx_leave_smm (struct kvm_vcpu *vcpu,const union kvm_smram *smram) +{ +WARN_ON_ONCE (1 ); +return 0 ; +} + +void tdx_enable_smi_window (struct kvm_vcpu *vcpu) +{ +/* SMI isn't supported for TDX. Silently discard SMI request. */ +WARN_ON_ONCE (1 ); + vcpu->arch.smi_pending = false; +} +#endif +int tdx_dev_ioctl (void __user *argp) {struct kvm_tdx_capabilities __user *user_caps;
arch/x86/kvm/vmx/x86_ops.h +12 -0
@@ -226,4 +226,16 @@ static inline int tdx_sept_tlb_remote_flush(struct kvm *kvm) { return 0; }static inline void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level) {}#endif +#if defined(CONFIG_INTEL_TDX_HOST ) && defined(CONFIG_KVM_SMM ) +int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection); +int tdx_enter_smm (struct kvm_vcpu *vcpu,union kvm_smram *smram); +int tdx_leave_smm (struct kvm_vcpu *vcpu,const union kvm_smram *smram); +void tdx_enable_smi_window (struct kvm_vcpu *vcpu); +#else +static inline int tdx_smi_allowed (struct kvm_vcpu *vcpu,bool for_injection) {return false; } +static inline int tdx_enter_smm (struct kvm_vcpu *vcpu,union kvm_smram *smram) {return 0 ; } +static inline int tdx_leave_smm (struct kvm_vcpu *vcpu,const union kvm_smram *smram) {return 0 ; } +static inline void tdx_enable_smi_window (struct kvm_vcpu *vcpu) {} +#endif +#endif /* __KVM_X86_VMX_X86_OPS_H */
Signature · 1 line
2.25.1
PATCH v12 94/106 KVM: TDX: Silently ignore INIT/SIPI
In reply to [email protected]From: Isaku Yamahata <[email protected]>
The TDX module API doesn't provide API for VMM to inject INIT IPI and SIPI. Instead it defines the different protocols to boot application processors. Ignore INIT and SIPI events for the TDX guest.
There are two options. 1) (silently) ignore INIT/SIPI request or 2) return error to guest TDs somehow. Given that TDX guest is paravirtualized to boot AP, the option 1 is chosen for simplicity.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/lapic.c | 19 +++++++++++------- arch/x86/kvm/svm/svm.c | 1 + arch/x86/kvm/vmx/main.c | 32 ++++++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 4 ++-- 6 files changed, 48 insertions(+), 11 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -144,6 +144,7 @@ KVM_X86_OP_OPTIONAL(migrate_timers) KVM_X86_OP(msr_filter_changed) KVM_X86_OP(complete_emulated_msr) KVM_X86_OP(vcpu_deliver_sipi_vector) +KVM_X86_OP(vcpu_deliver_init) KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);#undef KVM_X86_OP
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1786,6 +1786,7 @@ struct kvm_x86_ops {int (*complete_emulated_msr)(struct kvm_vcpu *vcpu,int err);void (*vcpu_deliver_sipi_vector)(struct kvm_vcpu *vcpu,u8 vector); +void (*vcpu_deliver_init)(struct kvm_vcpu *vcpu);/* * Returns vCPU specific APICv inhibit reasons
@@ -1997,6 +1998,7 @@ void kvm_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg);void kvm_set_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var,int seg);int kvm_load_segment_descriptor (struct kvm_vcpu *vcpu,u16 selector,int seg);void kvm_vcpu_deliver_sipi_vector (struct kvm_vcpu *vcpu,u8 vector); +void kvm_vcpu_deliver_init (struct kvm_vcpu *vcpu);int kvm_task_switch (struct kvm_vcpu *vcpu,u16 tss_selector,int idt_index,int reason,bool has_error_code,u32 error_code);
arch/x86/kvm/lapic.c +12 -7
@@ -3193,6 +3193,16 @@ int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len)return 0 ; } +void kvm_vcpu_deliver_init (struct kvm_vcpu *vcpu) +{ +kvm_vcpu_reset (vcpu, true); +if (kvm_vcpu_is_bsp (vcpu)) + vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; +else + vcpu->arch.mp_state =KVM_MP_STATE_INIT_RECEIVED ; +} +EXPORT_SYMBOL_GPL (kvm_vcpu_deliver_init); +int kvm_apic_accept_events (struct kvm_vcpu *vcpu) {struct kvm_lapic *apic = vcpu->arch.apic;
@@ -3224,13 +3234,8 @@ int kvm_apic_accept_events(struct kvm_vcpu *vcpu)return 0 ; } -if (test_and_clear_bit (KVM_APIC_INIT , &apic->pending_events)) { -kvm_vcpu_reset (vcpu, true); -if (kvm_vcpu_is_bsp (apic->vcpu)) - vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; -else - vcpu->arch.mp_state =KVM_MP_STATE_INIT_RECEIVED ; - } +if (test_and_clear_bit (KVM_APIC_INIT , &apic->pending_events)) +static_call (kvm_x86_vcpu_deliver_init)(vcpu);if (test_and_clear_bit (KVM_APIC_SIPI , &apic->pending_events)) {if (vcpu->arch.mp_state ==KVM_MP_STATE_INIT_RECEIVED ) {/* evaluate pending_events before reading the vector */
arch/x86/kvm/svm/svm.c +1 -0
@@ -4830,6 +4830,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .complete_emulated_msr = svm_complete_emulated_msr, .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector, + .vcpu_deliver_init = kvm_vcpu_deliver_init, .vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons, };
arch/x86/kvm/vmx/main.c +30 -2
@@ -310,6 +310,14 @@ static void vt_enable_smi_window(struct kvm_vcpu *vcpu) }#endif +static bool vt_apic_init_signal_blocked (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return true; + +return vmx_apic_init_signal_blocked (vcpu); +} +static void vt_apicv_post_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -338,6 +346,25 @@ static void vt_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,vmx_deliver_interrupt (apic, delivery_mode, trig_mode, vector); } +static void vt_vcpu_deliver_sipi_vector (struct kvm_vcpu *vcpu,u8 vector) +{ +if (is_td_vcpu (vcpu)) +return ; + +kvm_vcpu_deliver_sipi_vector (vcpu, vector); +} + +static void vt_vcpu_deliver_init (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) { +/* TDX doesn't support INIT. Ignore INIT event */ + vcpu->arch.mp_state =KVM_MP_STATE_RUNNABLE ; +return ; + } + +kvm_vcpu_deliver_init (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -702,13 +729,14 @@ struct kvm_x86_ops vt_x86_ops __initdata = {#endif .can_emulate_instruction = vmx_can_emulate_instruction, - .apic_init_signal_blocked = vmx_apic_init_signal_blocked, + .apic_init_signal_blocked = vt_apic_init_signal_blocked, .migrate_timers = vmx_migrate_timers, .msr_filter_changed = vt_msr_filter_changed, .complete_emulated_msr = kvm_complete_insn_gp, - .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector, + .vcpu_deliver_sipi_vector = vt_vcpu_deliver_sipi_vector, + .vcpu_deliver_init = vt_vcpu_deliver_init, .dev_mem_enc_ioctl = tdx_dev_ioctl, .mem_enc_ioctl = vt_mem_enc_ioctl,
arch/x86/kvm/vmx/tdx.c +2 -2
@@ -729,8 +729,8 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {struct msr_data apic_base_msr; -/* Ignore INIT silently because TDX doesn't support INIT event. */ -if (init_event) +/* vcpu_deliver_init method silently discards INIT event. */ +if (KVM_BUG_ON (init_event, vcpu->kvm))return ;if (KVM_BUG_ON (is_td_vcpu_created (to_tdx (vcpu)), vcpu->kvm))return ;
Signature · 1 line
2.25.1
PATCH v12 95/106 KVM: TDX: Add methods to ignore accesses to CPU state
In reply to [email protected]From: Sean Christopherson <[email protected]>
TDX protects TDX guest state from VMM. Implement access methods for TDX guest state to ignore them or return zero. Because those methods can be called by kvm ioctls to set/get cpu registers, they don't have KVM_BUG_ON except one method.
--- arch/x86/kvm/vmx/main.c | 269 +++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 49 ++++++- arch/x86/kvm/vmx/x86_ops.h | 13 ++ 3 files changed, 304 insertions(+), 27 deletions(-)
arch/x86/kvm/vmx/main.c +244 -25
@@ -365,6 +365,184 @@ static void vt_vcpu_deliver_init(struct kvm_vcpu *vcpu)kvm_vcpu_deliver_init (vcpu); } +static void vt_vcpu_after_set_cpuid (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_vcpu_after_set_cpuid (vcpu); +} + +static void vt_update_exception_bitmap (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_update_exception_bitmap (vcpu); +} + +static u64 vt_get_segment_base (struct kvm_vcpu *vcpu,int seg) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_segment_base (vcpu, seg); + +return vmx_get_segment_base (vcpu, seg); +} + +static void vt_get_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var, +int seg) +{ +if (is_td_vcpu (vcpu)) { +tdx_get_segment (vcpu, var, seg); +return ; + } + +vmx_get_segment (vcpu, var, seg); +} + +static void vt_set_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var, +int seg) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_segment (vcpu, var, seg); +} + +static int vt_get_cpl (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_cpl (vcpu); + +return vmx_get_cpl (vcpu); +} + +static void vt_get_cs_db_l_bits (struct kvm_vcpu *vcpu,int *db,int *l) +{ +if (is_td_vcpu (vcpu)) { + *db =0 ; + *l =0 ; +return ; + } + +vmx_get_cs_db_l_bits (vcpu, db, l); +} + +static void vt_set_cr0 (struct kvm_vcpu *vcpu,unsigned long cr0) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_cr0 (vcpu, cr0); +} + +static void vt_set_cr4 (struct kvm_vcpu *vcpu,unsigned long cr4) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_cr4 (vcpu, cr4); +} + +static int vt_set_efer (struct kvm_vcpu *vcpu,u64 efer) +{ +if (is_td_vcpu (vcpu)) +return 0 ; + +return vmx_set_efer (vcpu, efer); +} + +static void vt_get_idt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (is_td_vcpu (vcpu)) { +memset (dt,0 ,sizeof (*dt)); +return ; + } + +vmx_get_idt (vcpu, dt); +} + +static void vt_set_idt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_idt (vcpu, dt); +} + +static void vt_get_gdt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (is_td_vcpu (vcpu)) { +memset (dt,0 ,sizeof (*dt)); +return ; + } + +vmx_get_gdt (vcpu, dt); +} + +static void vt_set_gdt (struct kvm_vcpu *vcpu,struct desc_ptr *dt) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_gdt (vcpu, dt); +} + +static void vt_set_dr7 (struct kvm_vcpu *vcpu,unsigned long val) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_dr7 (vcpu, val); +} + +static void vt_sync_dirty_debug_regs (struct kvm_vcpu *vcpu) +{ +/* +* MOV-DR exiting is always cleared for TD guest, even in debug mode. +* Thus KVM_DEBUGREG_WONT_EXIT can never be set and it should never +* reach here for TD vcpu. +*/ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_sync_dirty_debug_regs (vcpu); +} + +static void vt_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg) +{ +if (is_td_vcpu (vcpu)) { +tdx_cache_reg (vcpu, reg); +return ; + } + +vmx_cache_reg (vcpu, reg); +} + +static unsigned long vt_get_rflags (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_get_rflags (vcpu); + +return vmx_get_rflags (vcpu); +} + +static void vt_set_rflags (struct kvm_vcpu *vcpu,unsigned long rflags) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_rflags (vcpu, rflags); +} + +static bool vt_get_if_flag (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return false; + +return vmx_get_if_flag (vcpu); +} +static void vt_flush_tlb_all (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) {
@@ -509,6 +687,14 @@ static void vt_inject_irq(struct kvm_vcpu *vcpu, bool reinjected)vmx_inject_irq (vcpu, reinjected); } +static void vt_inject_exception (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_inject_exception (vcpu); +} +static void vt_cancel_injection (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -555,6 +741,39 @@ static void vt_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,vmx_get_exit_info (vcpu, reason, info1, info2, intr_info, error_code); } + +static void vt_update_cr8_intercept (struct kvm_vcpu *vcpu,int tpr,int irr) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_update_cr8_intercept (vcpu, tpr, irr); +} + +static void vt_load_eoi_exitmap (struct kvm_vcpu *vcpu,u64 *eoi_exit_bitmap) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_load_eoi_exitmap (vcpu, eoi_exit_bitmap); +} + +static int vt_set_tss_addr (struct kvm *kvm,unsigned int addr) +{ +if (is_td (kvm)) +return 0 ; + +return vmx_set_tss_addr (kvm, addr); +} + +static int vt_set_identity_map_addr (struct kvm *kvm,u64 ident_addr) +{ +if (is_td (kvm)) +return 0 ; + +return vmx_set_identity_map_addr (kvm, ident_addr); +} +static u8 vt_get_mt_mask (struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio) {if (is_td_vcpu (vcpu))
@@ -620,29 +839,29 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_load = vt_vcpu_load, .vcpu_put = vt_vcpu_put, - .update_exception_bitmap = vmx_update_exception_bitmap, + .update_exception_bitmap = vt_update_exception_bitmap, .get_msr_feature = vmx_get_msr_feature, .get_msr = vt_get_msr, .set_msr = vt_set_msr, - .get_segment_base = vmx_get_segment_base, - .get_segment = vmx_get_segment, - .set_segment = vmx_set_segment, - .get_cpl = vmx_get_cpl, - .get_cs_db_l_bits = vmx_get_cs_db_l_bits, - .set_cr0 = vmx_set_cr0, + .get_segment_base = vt_get_segment_base, + .get_segment = vt_get_segment, + .set_segment = vt_set_segment, + .get_cpl = vt_get_cpl, + .get_cs_db_l_bits = vt_get_cs_db_l_bits, + .set_cr0 = vt_set_cr0, .is_valid_cr4 = vmx_is_valid_cr4, - .set_cr4 = vmx_set_cr4, - .set_efer = vmx_set_efer, - .get_idt = vmx_get_idt, - .set_idt = vmx_set_idt, - .get_gdt = vmx_get_gdt, - .set_gdt = vmx_set_gdt, - .set_dr7 = vmx_set_dr7, - .sync_dirty_debug_regs = vmx_sync_dirty_debug_regs, - .cache_reg = vmx_cache_reg, - .get_rflags = vmx_get_rflags, - .set_rflags = vmx_set_rflags, - .get_if_flag = vmx_get_if_flag, + .set_cr4 = vt_set_cr4, + .set_efer = vt_set_efer, + .get_idt = vt_get_idt, + .set_idt = vt_set_idt, + .get_gdt = vt_get_gdt, + .set_gdt = vt_set_gdt, + .set_dr7 = vt_set_dr7, + .sync_dirty_debug_regs = vt_sync_dirty_debug_regs, + .cache_reg = vt_cache_reg, + .get_rflags = vt_get_rflags, + .set_rflags = vt_set_rflags, + .get_if_flag = vt_get_if_flag, .flush_tlb_all = vt_flush_tlb_all, .flush_tlb_current = vt_flush_tlb_current,
@@ -659,7 +878,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .patch_hypercall = vmx_patch_hypercall, .inject_irq = vt_inject_irq, .inject_nmi = vt_inject_nmi, - .inject_exception = vmx_inject_exception, + .inject_exception = vt_inject_exception, .cancel_injection = vt_cancel_injection, .interrupt_allowed = vt_interrupt_allowed, .nmi_allowed = vt_nmi_allowed,
@@ -667,11 +886,11 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_nmi_mask = vt_set_nmi_mask, .enable_nmi_window = vt_enable_nmi_window, .enable_irq_window = vt_enable_irq_window, - .update_cr8_intercept = vmx_update_cr8_intercept, + .update_cr8_intercept = vt_update_cr8_intercept, .set_virtual_apic_mode = vmx_set_virtual_apic_mode, .set_apic_access_page_addr = vmx_set_apic_access_page_addr, .refresh_apicv_exec_ctrl = vmx_refresh_apicv_exec_ctrl, - .load_eoi_exitmap = vmx_load_eoi_exitmap, + .load_eoi_exitmap = vt_load_eoi_exitmap, .apicv_post_state_restore = vt_apicv_post_state_restore, .required_apicv_inhibits = VMX_REQUIRED_APICV_INHIBITS, .hwapic_irr_update = vmx_hwapic_irr_update,
@@ -682,13 +901,13 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .dy_apicv_has_pending_interrupt = pi_has_pending_interrupt, .protected_apic_has_interrupt = vt_protected_apic_has_interrupt, - .set_tss_addr = vmx_set_tss_addr, - .set_identity_map_addr = vmx_set_identity_map_addr, + .set_tss_addr = vt_set_tss_addr, + .set_identity_map_addr = vt_set_identity_map_addr, .get_mt_mask = vt_get_mt_mask, .get_exit_info = vt_get_exit_info, - .vcpu_after_set_cpuid = vmx_vcpu_after_set_cpuid, + .vcpu_after_set_cpuid = vt_vcpu_after_set_cpuid, .has_wbinvd_exit = cpu_has_vmx_wbinvd_exit,
arch/x86/kvm/vmx/tdx.c +47 -2
67 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h +13 -0
@@ -175,6 +175,12 @@ bool tdx_has_emulated_msr(u32 index, bool write);int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr);int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr); +int tdx_get_cpl (struct kvm_vcpu *vcpu); +void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg); +unsigned long tdx_get_rflags (struct kvm_vcpu *vcpu); +u64 tdx_get_segment_base (struct kvm_vcpu *vcpu,int seg); +void tdx_get_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var,int seg); +int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp);void tdx_flush_tlb (struct kvm_vcpu *vcpu);
@@ -219,6 +225,13 @@ static inline bool tdx_has_emulated_msr(u32 index, bool write) { return false; }static inline int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; }static inline int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; } +static inline int tdx_get_cpl (struct kvm_vcpu *vcpu) {return 0 ; } +static inline void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg) {} +static inline unsigned long tdx_get_rflags (struct kvm_vcpu *vcpu) {return 0 ; } +static inline u64 tdx_get_segment_base (struct kvm_vcpu *vcpu,int seg) {return 0 ; } +static inline void tdx_get_segment (struct kvm_vcpu *vcpu,struct kvm_segment *var, +int seg) {} +static inline int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp) {return -EOPNOTSUPP ; }static inline void tdx_flush_tlb (struct kvm_vcpu *vcpu) {}
Signature · 1 line
2.25.1
PATCH v12 96/106 KVM: TDX: Add methods to ignore guest instruction emulation
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because TDX protects TDX guest state from VMM, instructions in guest memory cannot be emulated. Implement methods to ignore guest instruction emulator.
--- arch/x86/kvm/vmx/main.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +26 -2
@@ -310,6 +310,30 @@ static void vt_enable_smi_window(struct kvm_vcpu *vcpu) }#endif +static bool vt_can_emulate_instruction (struct kvm_vcpu *vcpu,int emul_type, +void *insn,int insn_len) +{ +if (is_td_vcpu (vcpu)) +return false; + +return vmx_can_emulate_instruction (vcpu, emul_type, insn, insn_len); +} + +static int vt_check_intercept (struct kvm_vcpu *vcpu, +struct x86_instruction_info *info, +enum x86_intercept_stage stage, +struct x86_exception *exception) +{ +/* +* This call back is triggered by the x86 instruction emulator. TDX +* doesn't allow guest memory inspection. +*/ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return X86EMUL_UNHANDLEABLE ; + +return vmx_check_intercept (vcpu, info, stage, exception); +} +static bool vt_apic_init_signal_blocked (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -918,7 +942,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .load_mmu_pgd = vt_load_mmu_pgd, - .check_intercept = vmx_check_intercept, + .check_intercept = vt_check_intercept, .handle_exit_irqoff = vt_handle_exit_irqoff, .request_immediate_exit = vt_request_immediate_exit,
@@ -947,7 +971,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .enable_smi_window = vt_enable_smi_window, #endif - .can_emulate_instruction = vmx_can_emulate_instruction, + .can_emulate_instruction = vt_can_emulate_instruction, .apic_init_signal_blocked = vt_apic_init_signal_blocked, .migrate_timers = vmx_migrate_timers,
Signature · 1 line
2.25.1
PATCH v12 97/106 KVM: TDX: Add a method to ignore dirty logging
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Currently TDX KVM doesn't support tracking dirty pages (yet). Implement a method to ignore it. Because the flag for kvm memory slot to enable dirty logging isn't accepted for TDX, warn on the method is called for TDX.
--- arch/x86/kvm/vmx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +9 -1
@@ -806,6 +806,14 @@ static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)return __vmx_get_mt_mask (vcpu, gfn, is_mmio, true); } +static void vt_update_cpu_dirty_logging (struct kvm_vcpu *vcpu) +{ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_update_cpu_dirty_logging (vcpu); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -950,7 +958,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .sched_in = vt_sched_in, .cpu_dirty_log_size = PML_ENTITY_NUM, - .update_cpu_dirty_logging = vmx_update_cpu_dirty_logging, + .update_cpu_dirty_logging = vt_update_cpu_dirty_logging, .nested_ops = &vmx_nested_ops,
Signature · 1 line
2.25.1
PATCH v12 98/106 KVM: TDX: Add methods to ignore VMX preemption timer
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX doesn't support VMX preemption timer. Implement access methods for VMM to ignore VMX preemption timer.
--- arch/x86/kvm/vmx/main.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +23 -2
@@ -814,6 +814,27 @@ static void vt_update_cpu_dirty_logging(struct kvm_vcpu *vcpu)vmx_update_cpu_dirty_logging (vcpu); } +#ifdef CONFIG_X86_64 +static int vt_set_hv_timer (struct kvm_vcpu *vcpu,u64 guest_deadline_tsc, +bool *expired) +{ +/* VMX-preemption timer isn't available for TDX. */ +if (is_td_vcpu (vcpu)) +return -EINVAL ; + +return vmx_set_hv_timer (vcpu, guest_deadline_tsc, expired); +} + +static void vt_cancel_hv_timer (struct kvm_vcpu *vcpu) +{ +/* VMX-preemption timer can't be set. See vt_set_hv_timer(). */ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_cancel_hv_timer (vcpu); +} +#endif +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -966,8 +987,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .pi_start_assignment = vmx_pi_start_assignment, #ifdef CONFIG_X86_64 - .set_hv_timer = vmx_set_hv_timer, - .cancel_hv_timer = vmx_cancel_hv_timer, + .set_hv_timer = vt_set_hv_timer, + .cancel_hv_timer = vt_cancel_hv_timer, #endif .setup_mce = vmx_setup_mce,
Signature · 1 line
2.25.1
PATCH v12 99/106 KVM: TDX: Add methods to ignore accesses to TSC
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX protects TDX guest TSC state from VMM. Implement access methods to ignore guest TSC.
--- arch/x86/kvm/vmx/main.c | 44 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 40 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/main.c +40 -4
@@ -806,6 +806,42 @@ static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)return __vmx_get_mt_mask (vcpu, gfn, is_mmio, true); } +static u64 vt_get_l2_tsc_offset (struct kvm_vcpu *vcpu) +{ +/* TDX doesn't support L2 guest at the moment. */ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return 0 ; + +return vmx_get_l2_tsc_offset (vcpu); +} + +static u64 vt_get_l2_tsc_multiplier (struct kvm_vcpu *vcpu) +{ +/* TDX doesn't support L2 guest at the moment. */ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return 0 ; + +return vmx_get_l2_tsc_multiplier (vcpu); +} + +static void vt_write_tsc_offset (struct kvm_vcpu *vcpu,u64 offset) +{ +/* In TDX, tsc offset can't be changed. */ +if (is_td_vcpu (vcpu)) +return ; + +vmx_write_tsc_offset (vcpu, offset); +} + +static void vt_write_tsc_multiplier (struct kvm_vcpu *vcpu,u64 multiplier) +{ +/* In TDX, tsc multiplier can't be changed. */ +if (is_td_vcpu (vcpu)) +return ; + +vmx_write_tsc_multiplier (vcpu, multiplier); +} +static void vt_update_cpu_dirty_logging (struct kvm_vcpu *vcpu) {if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm))
@@ -964,10 +1000,10 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .has_wbinvd_exit = cpu_has_vmx_wbinvd_exit, - .get_l2_tsc_offset = vmx_get_l2_tsc_offset, - .get_l2_tsc_multiplier = vmx_get_l2_tsc_multiplier, - .write_tsc_offset = vmx_write_tsc_offset, - .write_tsc_multiplier = vmx_write_tsc_multiplier, + .get_l2_tsc_offset = vt_get_l2_tsc_offset, + .get_l2_tsc_multiplier = vt_get_l2_tsc_multiplier, + .write_tsc_offset = vt_write_tsc_offset, + .write_tsc_multiplier = vt_write_tsc_multiplier, .load_mmu_pgd = vt_load_mmu_pgd,
Signature · 1 line
2.25.1
PATCH v12 100/106 KVM: TDX: Ignore setting up mce
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because vmx_set_mce function is VMX specific and it cannot be used for TDX. Add vt stub to ignore setting up mce for TDX.
--- arch/x86/kvm/vmx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +9 -1
@@ -871,6 +871,14 @@ static void vt_cancel_hv_timer(struct kvm_vcpu *vcpu) }#endif +static void vt_setup_mce (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_setup_mce (vcpu); +} +static int vt_mem_enc_ioctl (struct kvm *kvm,void __user *argp) {if (!is_td (kvm))
@@ -1027,7 +1035,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .cancel_hv_timer = vt_cancel_hv_timer, #endif - .setup_mce = vmx_setup_mce, + .setup_mce = vt_setup_mce, #ifdef CONFIG_KVM_SMM .smi_allowed = vt_smi_allowed,
Signature · 1 line
2.25.1
PATCH v12 101/106 KVM: TDX: Add a method to ignore for TDX to ignore hypercall patch
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because guest TD memory is protected, VMM patching guest binary for hypercall instruction isn't possible. Add a method to ignore hypercall patching with a warning. Note: guest TD kernel needs to be modified to use TDG.VP.VMCALL for hypercall.
--- arch/x86/kvm/vmx/main.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/main.c +14 -1
@@ -703,6 +703,19 @@ static u32 vt_get_interrupt_shadow(struct kvm_vcpu *vcpu)return vmx_get_interrupt_shadow (vcpu); } +static void vt_patch_hypercall (struct kvm_vcpu *vcpu, +unsigned char *hypercall) +{ +/* +* Because guest memory is protected, guest can't be patched. TD kernel +* is modified to use TDG.VP.VMCAL for hypercall. +*/ +if (KVM_BUG_ON (is_td_vcpu (vcpu), vcpu->kvm)) +return ; + +vmx_patch_hypercall (vcpu, hypercall); +} +static void vt_inject_irq (struct kvm_vcpu *vcpu,bool reinjected) {if (is_td_vcpu (vcpu))
@@ -972,7 +985,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .update_emulated_instruction = vmx_update_emulated_instruction, .set_interrupt_shadow = vt_set_interrupt_shadow, .get_interrupt_shadow = vt_get_interrupt_shadow, - .patch_hypercall = vmx_patch_hypercall, + .patch_hypercall = vt_patch_hypercall, .inject_irq = vt_inject_irq, .inject_nmi = vt_inject_nmi, .inject_exception = vt_inject_exception,
Signature · 1 line
2.25.1
PATCH v12 102/106 KVM: TDX: Add methods to ignore virtual apic related operation
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX protects TDX guest APIC state from VMM. Implement access methods of TDX guest vAPIC state to ignore them or return zero.
--- arch/x86/kvm/vmx/main.c | 61 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 6 ++++ arch/x86/kvm/vmx/x86_ops.h | 3 ++ 3 files changed, 64 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/main.c +55 -6
@@ -342,6 +342,14 @@ static bool vt_apic_init_signal_blocked(struct kvm_vcpu *vcpu)return vmx_apic_init_signal_blocked (vcpu); } +static void vt_set_virtual_apic_mode (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return tdx_set_virtual_apic_mode (vcpu); + +return vmx_set_virtual_apic_mode (vcpu); +} +static void vt_apicv_post_state_restore (struct kvm_vcpu *vcpu) {struct pi_desc *pi =vcpu_to_pi_desc (vcpu);
@@ -350,6 +358,31 @@ static void vt_apicv_post_state_restore(struct kvm_vcpu *vcpu)memset (pi->pir,0 ,sizeof (pi->pir)); } +static void vt_hwapic_irr_update (struct kvm_vcpu *vcpu,int max_irr) +{ +if (is_td_vcpu (vcpu)) +return ; + +return vmx_hwapic_irr_update (vcpu, max_irr); +} + +static void vt_hwapic_isr_update (int max_isr) +{ +if (is_td_vcpu (kvm_get_running_vcpu ())) +return ; + +return vmx_hwapic_isr_update (max_isr); +} + +static bool vt_guest_apic_has_interrupt (struct kvm_vcpu *vcpu) +{ +/* TDX doesn't support L2 at the moment. */ +if (WARN_ON_ONCE (is_td_vcpu (vcpu))) +return false; + +return vmx_guest_apic_has_interrupt (vcpu); +} +static int vt_sync_pir_to_irr (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu))
@@ -787,6 +820,22 @@ static void vt_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr)vmx_update_cr8_intercept (vcpu, tpr, irr); } +static void vt_set_apic_access_page_addr (struct kvm_vcpu *vcpu) +{ +if (is_td_vcpu (vcpu)) +return ; + +vmx_set_apic_access_page_addr (vcpu); +} + +static void vt_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu) +{ +if (WARN_ON_ONCE (is_td_vcpu (vcpu))) +return ; + +vmx_refresh_apicv_exec_ctrl (vcpu); +} +static void vt_load_eoi_exitmap (struct kvm_vcpu *vcpu,u64 *eoi_exit_bitmap) {if (is_td_vcpu (vcpu))
@@ -997,15 +1046,15 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .enable_nmi_window = vt_enable_nmi_window, .enable_irq_window = vt_enable_irq_window, .update_cr8_intercept = vt_update_cr8_intercept, - .set_virtual_apic_mode = vmx_set_virtual_apic_mode, - .set_apic_access_page_addr = vmx_set_apic_access_page_addr, - .refresh_apicv_exec_ctrl = vmx_refresh_apicv_exec_ctrl, + .set_virtual_apic_mode = vt_set_virtual_apic_mode, + .set_apic_access_page_addr = vt_set_apic_access_page_addr, + .refresh_apicv_exec_ctrl = vt_refresh_apicv_exec_ctrl, .load_eoi_exitmap = vt_load_eoi_exitmap, .apicv_post_state_restore = vt_apicv_post_state_restore, .required_apicv_inhibits = VMX_REQUIRED_APICV_INHIBITS, - .hwapic_irr_update = vmx_hwapic_irr_update, - .hwapic_isr_update = vmx_hwapic_isr_update, - .guest_apic_has_interrupt = vmx_guest_apic_has_interrupt, + .hwapic_irr_update = vt_hwapic_irr_update, + .hwapic_isr_update = vt_hwapic_isr_update, + .guest_apic_has_interrupt = vt_guest_apic_has_interrupt, .sync_pir_to_irr = vt_sync_pir_to_irr, .deliver_interrupt = vt_deliver_interrupt, .dy_apicv_has_pending_interrupt = pi_has_pending_interrupt,
arch/x86/kvm/vmx/tdx.c +6 -0
@@ -1899,6 +1899,12 @@ void tdx_enable_smi_window(struct kvm_vcpu *vcpu) }#endif +void tdx_set_virtual_apic_mode (struct kvm_vcpu *vcpu) +{ +/* Only x2APIC mode is supported for TD. */ +WARN_ON_ONCE (kvm_get_apic_mode (vcpu) !=LAPIC_MODE_X2APIC ); +} +int tdx_get_cpl (struct kvm_vcpu *vcpu) {return 0 ;
arch/x86/kvm/vmx/x86_ops.h +3 -0
@@ -174,6 +174,7 @@ void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,bool tdx_has_emulated_msr (u32 index,bool write);int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr);int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr); +void tdx_set_virtual_apic_mode (struct kvm_vcpu *vcpu);int tdx_get_cpl (struct kvm_vcpu *vcpu);void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg);
@@ -225,6 +226,8 @@ static inline bool tdx_has_emulated_msr(u32 index, bool write) { return false; }static inline int tdx_get_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; }static inline int tdx_set_msr (struct kvm_vcpu *vcpu,struct msr_data *msr) {return 1 ; } +static inline void tdx_set_virtual_apic_mode (struct kvm_vcpu *vcpu) {} +static inline int tdx_get_cpl (struct kvm_vcpu *vcpu) {return 0 ; }static inline void tdx_cache_reg (struct kvm_vcpu *vcpu,enum kvm_reg reg) {}static inline unsigned long tdx_get_rflags (struct kvm_vcpu *vcpu) {return 0 ; }
Signature · 1 line
2.25.1
PATCH v12 103/106 Documentation/virt/kvm: Document on Trust Domain Extensions(TDX)
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add documentation to Intel Trusted Domain Extensions(TDX) support.
--- Documentation/virt/kvm/api.rst | 9 +- Documentation/virt/kvm/index.rst | 2 + Documentation/virt/kvm/intel-tdx.rst | 351 +++++++++++++++++++++++++++ 3 files changed, 361 insertions(+), 1 deletion(-) create mode 100644 Documentation/virt/kvm/intel-tdx.rst
Documentation/virt/kvm/api.rst +8 -1
@@ -1375,6 +1375,9 @@ the memory region are automatically reflected into the guest. For example, an mmap() that affects the region will be made visible immediately. Another example is madvise(MADV_DROP). +For TDX guest, deleting/moving memory region loses guest memory contents. +Read only region isn't supported. Only as-id 0 is supported. + Note: On arm64, a write generated by the page-table walker (to update the Access and Dirty flags, for example) never results in a KVM_EXIT_MMIO exit when the slot has the KVM_MEM_READONLY flag. This
@@ -4664,7 +4667,7 @@ H_GET_CPU_CHARACTERISTICS hypercall. :Capability: basic :Architectures: x86 -:Type: vm +:Type: vm ioctl, vcpu ioctl :Parameters: an opaque platform specific structure (in/out) :Returns: 0 on success; -1 on error
@@ -4676,6 +4679,10 @@ Currently, this ioctl is used for issuing Secure Encrypted Virtualization (SEV) commands on AMD Processors. The SEV commands are defined in Documentation/virt/kvm/x86/amd-memory-encryption.rst. +Currently, this ioctl is used for issuing Trusted Domain Extensions +(TDX) commands on Intel Processors. The TDX commands are defined in +Documentation/virt/kvm/intel-tdx.rst. + 4.111 KVM_MEMORY_ENCRYPT_REG_REGION -----------------------------------
Documentation/virt/kvm/index.rst +2 -0
@@ -19,3 +19,5 @@ KVM vcpu-requests halt-polling review-checklist + + intel-tdx
Documentation/virt/kvm/intel-tdx.rst new file +351 -0
351 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 104/106 KVM: x86: design documentation on TDX support of x86 KVM TDP MMU
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add a high level design document on TDX changes to TDP MMU.
--- Documentation/virt/kvm/index.rst | 1 + Documentation/virt/kvm/tdx-tdp-mmu.rst | 417 +++++++++++++++++++++++++ 2 files changed, 418 insertions(+) create mode 100644 Documentation/virt/kvm/tdx-tdp-mmu.rst
Documentation/virt/kvm/index.rst +1 -0
@@ -21,3 +21,4 @@ KVM review-checklist intel-tdx + tdx-tdp-mmu
Documentation/virt/kvm/tdx-tdp-mmu.rst new file +417 -0
417 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v12 105/106 RFC: KVM: TDX: Make busy with S-EPT on entry bug
In reply to [email protected]From: Isaku Yamahata <[email protected]>
TDX module has mitigation against zero-step attacks or single-step attacks. When the TDX module finds repeated EPT violations on the same guest RIP, i.e. no advance in guest, it starts to suspect the attack. The mitigation logic on the next entry tries to take the lock of S-EPT. It may result in an error of TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT. As KVM shouldn't spuriously zap private S-EPT so that guest can make progress, KVM shouldn't cause the TDX module to trigger the mitigation. Make (TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT) on entry KVM bug.
---
- This patch is RFC because this is only lightly tested and stress test isn't done.
--- arch/x86/kvm/vmx/tdx.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/tdx.c +14 -2
@@ -1712,8 +1712,20 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) {union tdx_exit_reason exit_reason =to_tdx (vcpu)->exit_reason; -/* See the comment of tdh_sept_seamcall(). */ -if (unlikely (exit_reason.full == (TDX_OPERAND_BUSY |TDX_OPERAND_ID_SEPT ))) +/* +* See the comment of tdh_sept_seamcall(). +* TDX module has mitigation against zero-step attacks or single-step +* attacks. When the TDX module finds repeated EPT violations on the +* same guest RIP, i.e. no advance in guest, it starts to suspect the +* attack. The mitigation logic on the next entry tries to take the +* lock of S-EPT. It may result in an error of (TDX_OPERAND_BUSY | +* TDX_OPERAND_ID_SEPT). As KVM shouldn't spuriously zap private S-EPT +* so that guest can make progress, KVM shouldn't cause the TDX module +* to trigger the mitigation. Make (TDX_OPERAND_BUSY | +* TDX_OPERAND_ID_SEPT) on entry KVM bug. +*/ +if (KVM_BUG_ON (exit_reason.full == (TDX_OPERAND_BUSY |TDX_OPERAND_ID_SEPT ), + vcpu->kvm))return 1 ;if (unlikely (exit_reason.non_recoverable || exit_reason.error)) {
Signature · 1 line
2.25.1
PATCH v12 106/106 the end of (the first phase of) TDX KVM patch series
In reply to [email protected]From: Isaku Yamahata <[email protected]>
This empty commit is to mark the end of (the first phase of) patch series of TDX KVM support.
--- .../virt/kvm/intel-tdx-layer-status.rst | 32 ------------------- 1 file changed, 32 deletions(-) delete mode 100644 Documentation/virt/kvm/intel-tdx-layer-status.rst
Documentation/virt/kvm/intel-tdx-layer-status.rst deleted +0 -32
@@ -1,32 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -=================================== -Intel Trust Dodmain Extensions(TDX) -=================================== - -Layer status -============ -What qemu can do ----------------- -- TDX VM TYPE is exposed to Qemu. -- Qemu can create/destroy guest of TDX vm type. -- Qemu can create/destroy vcpu of TDX vm type. -- Qemu can populate initial guest memory image. -- Qemu can finalize guest TD. -- Qemu can start to run vcpu. But vcpu can not make progress yet. - -Patch Layer status ------------------- - Patch layer Status -* TDX, VMX coexistence: Applied -* TDX architectural definitions: Applied -* TD VM creation/destruction: Applied -* TD vcpu creation/destruction: Applied -* TDX EPT violation: Applied -* TD finalization: Applied -* TD vcpu enter/exit: Applied -* TD vcpu interrupts/exit/hypercall: Not yet - -* KVM MMU GPA shared bits: Applied -* KVM TDP refactoring for TDX: Applied -* KVM TDP MMU hooks: Applied
Signature · 1 line
2.25.1
Quoted 11 lines · On 2/27/2023 4:21 PM, [email protected] wrote:
KVM TDX basic feature support
Hello. This is v12 the patch series vof KVM TDX support. This is based on v6.2-rc7 + the following patch series.
Related patch series This patch is based on:
- fd-based approach for supporing KVM v10 and
https://lore.kernel.org/lkml/[email protected]/
[email protected]:sean-jc/linux.git x86/upm_base_support
- TDX host kernel support v9
https://lore.kernel.org/lkml/[email protected]/T/#m2562c7f18eabae171793ed7ea81b3544d7af09c4
9fdabaaf0346 KVM: TDX: Make TDX VM type supported fa6b8135913c KVM: x86: Introduce vm_type to differentiate default VMs
from confidential VMs
6dbc1fdfb61f KVM: TDX: Add placeholders for TDX VM/vcpu structure 64a9c956f265 KVM: TDX: Initialize the TDX module when loading the KVM
intel kernel module
bacb6554fcf5 KVM: x86/vmx: Refactor KVM VMX module init/exit functions b17eee6a7d53 KVM: VMX: Move out vmx_x86_ops to 'main.c' to wrap VMX
and TDX
5a8b8d63a2e8 x86/mtrr: Mask out keyid bits from variable mtrr mask
register
29017e2c04cc KVM: arm64: Mark kvm_arm_init() and its unique
descendants as __init
I find above 8 patches not included in this patch series after TDX host kernel support series. Some of them were included in v11 before, and I think they are important to make the patch series intact. What's the reason for not including them?
Quoted 27 lines · Xiaoyao Li
Quoted 11 lines · On 2/27/2023 4:21 PM, [email protected] wrote:
KVM TDX basic feature support
Hello. This is v12 the patch series vof KVM TDX support. This is based on v6.2-rc7 + the following patch series.
Related patch series This patch is based on:
- fd-based approach for supporing KVM v10 and
https://lore.kernel.org/lkml/[email protected]/
[email protected]:sean-jc/linux.git x86/upm_base_support
- TDX host kernel support v9
https://lore.kernel.org/lkml/[email protected]/T/#m2562c7f18eabae171793ed7ea81b3544d7af09c4
9fdabaaf0346KVM: TDX: Make TDX VM type supported
fa6b8135913cKVM: x86: Introducevm_typeto differentiate default VMs from
confidential VMs
6dbc1fdfb61fKVM: TDX: Add placeholders for TDX VM/vcpu structure64a9c956f265KVM: TDX: Initialize the TDX module when loading the KVM intel
kernel module
bacb6554fcf5KVM: x86/vmx: Refactor KVM VMX module init/exit functionsb17eee6a7d53KVM: VMX: Move outvmx_x86_opsto 'main.c' to wrap VMX and TDX5a8b8d63a2e8x86/mtrr: Mask out keyid bits from variable mtrr mask register29017e2c04ccKVM: arm64: Markkvm_arm_init()and its unique descendants as__init
Quoted 4 lines
I find above 8 patches not included in this patch series after TDX host kernel support series. Some of them were included in v11 before, and I think they are important to make the patch series intact. What's the reason for not including them?
Thanks for pointing it out. My bad. I wrongly specified the start of patch series. Those should be included.
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v12 1/6 KVM: VMX: Move out vmx_x86_ops to 'main.c' to wrap VMX and TDX
From: Sean Christopherson <[email protected]>
KVM accesses Virtual Machine Control Structure (VMCS) with VMX instructions to operate on VM. TDX doesn't allow VMM to operate VMCS directly. Instead, TDX has its own data structures, and TDX SEAMCALL APIs for VMM to indirectly operate those data structures. This means we must have a TDX version of kvm_x86_ops.
The existing global struct kvm_x86_ops already defines an interface which fits with TDX. But kvm_x86_ops is system-wide, not per-VM structure. To allow VMX to coexist with TDs, the kvm_x86_ops callbacks will have wrappers "if (tdx) tdx_op() else vmx_op()" to switch VMX or TDX at run time.
To split the runtime switch, the VMX implementation, and the TDX implementation, add main.c, and move out the vmx_x86_ops hooks in preparation for adding TDX, which can coexist with VMX, i.e. KVM can run both VMs and TDs. Use 'vt' for the naming scheme as a nod to VT-x and as a concatenation of VmxTdx.
The current code looks as follows.
In vmx.c
static vmx_op() { ... }
static struct kvm_x86_ops vmx_x86_ops = {
.op = vmx_op,initialization code
The eventually converted code will look like In vmx.c, keep the VMX operations.
vmx_op() { ... }
VMX initialization
In tdx.c, define the TDX operations.
tdx_op() { ... }
TDX initialization
In x86_ops.h, declare the VMX and TDX operations.
vmx_op();
tdx_op();
In main.c, define common wrappers for VMX and TDX.
static vt_ops() { if (tdx) tdx_ops() else vmx_ops() }
static struct kvm_x86_ops vt_x86_ops = {
.op = vt_op,initialization to call VMX and TDX initialization
Opportunistically, fix the name inconsistency from vmx_create_vcpu() and vmx_free_vcpu() to vmx_vcpu_create() and vxm_vcpu_free().
--- arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/vmx/main.c | 167 +++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 370 ++++++++++--------------------------- arch/x86/kvm/vmx/x86_ops.h | 125 +++++++++++++ 4 files changed, 394 insertions(+), 270 deletions(-) create mode 100644 arch/x86/kvm/vmx/main.c create mode 100644 arch/x86/kvm/vmx/x86_ops.h
arch/x86/kvm/Makefile +1 -1
@@ -23,7 +23,7 @@ kvm-$(CONFIG_KVM_XEN) += xen.o kvm-$(CONFIG_KVM_SMM) += smm.o kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \ - vmx/hyperv.o vmx/nested.o vmx/posted_intr.o + vmx/hyperv.o vmx/nested.o vmx/posted_intr.o vmx/main.o kvm-intel-$(CONFIG_X86_SGX_KVM) += vmx/sgx.o kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o \
arch/x86/kvm/vmx/main.c new file +167 -0
167 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +101 -269
913 lines not on this page · raw message
arch/x86/kvm/vmx/x86_ops.h new file +125 -0
125 lines not on this page · raw message
Signature · 1 line
2.25.1