KVM TDX basic feature support

---
 arch/x86/kvm/Makefile      |   2 +-
 arch/x86/kvm/vmx/main.c    | 155 ++++++++++++++++
 arch/x86/kvm/vmx/vmx.c     | 363 +++++++++++--------------------------
 arch/x86/kvm/vmx/x86_ops.h | 125 +++++++++++++
 4 files changed, 386 insertions(+), 259 deletions(-)
 create mode 100644 arch/x86/kvm/vmx/main.c
 create mode 100644 arch/x86/kvm/vmx/x86_ops.h

arch/x86/kvm/Makefile +1 -1

index 30f244b64523..ee4d0999f20f 100644
@@ -22,7 +22,7 @@ kvm-$(CONFIG_X86_64) += mmu/tdp_iter.o mmu/tdp_mmu.o
 kvm-$(CONFIG_KVM_XEN)	+= xen.o
 
 kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \
-			   vmx/evmcs.o vmx/nested.o vmx/posted_intr.o
+			   vmx/evmcs.o vmx/nested.o vmx/posted_intr.o vmx/main.o
 kvm-intel-$(CONFIG_X86_SGX_KVM)	+= vmx/sgx.o
 
 kvm-amd-y		+= svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o

arch/x86/kvm/vmx/main.c new file +155 -0

155 lines not on this page · raw message

arch/x86/kvm/vmx/vmx.c +105 ~~-258~~

914 lines not on this page · raw message

arch/x86/kvm/vmx/x86_ops.h new file +125 -0

125 lines not on this page · raw message

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #2 lore raw

PATCH v10 2/108 KVM: x86: Refactor KVM VMX module init/exit functions

From: Isaku Yamahata <[email protected]>

Currently, KVM VMX module initialization/exit functions are a single function each. Refactor KVM VMX module initialization functions into KVM common part and VMX part so that TDX specific part can be added cleanly. Opportunistically refactor module exit function as well.

The current module initialization flow is, 1.) calculate the sizes of VMX kvm structure and VMX vcpu structure, 2.) hyper-v specific initialization 3.) report those sizes to the KVM common layer and KVM common initialization, and 4.) VMX specific system-wide initialization.

Refactor the KVM VMX module initialization function into functions with a wrapper function to separate VMX logic in vmx.c from a file, main.c, common among VMX and TDX. We have a wrapper function, "vt_init() {vmx kvm/vcpu size calculation; hv_vp_assist_page_init(); kvm_init(); vmx_init(); }" in main.c, and hv_vp_assist_page_init() and vmx_init() in vmx.c. hv_vp_assist_page_init() initializes hyper-v specific assist pages, kvm_init() does system-wide initialization of the KVM common layer, and vmx_init() does system-wide VMX initialization.

The KVM architecture common layer allocates struct kvm with reported size for architecture-specific code. The KVM VMX module defines its structure as struct vmx_kvm { struct kvm; VMX specific members;} and uses it as struct vmx kvm. Similar for vcpu structure. TDX KVM patches will define TDX specific kvm and vcpu structures, add tdx_pre_kvm_init() to report the sizes of them to the KVM common layer.

The current module exit function is also a single function, a combination of VMX specific logic and common KVM logic. Refactor it into VMX specific logic and KVM common logic. This is just refactoring to keep the VMX specific logic in vmx.c from main.c.

---
 arch/x86/kvm/vmx/main.c    | 37 +++++++++++++++
 arch/x86/kvm/vmx/vmx.c     | 95 ++++++++++++++++++--------------------
 arch/x86/kvm/vmx/x86_ops.h |  5 ++
 3 files changed, 88 insertions(+), 49 deletions(-)

arch/x86/kvm/vmx/main.c +37 -0

index 381059631e4b..4c7e71ec3e1d 100644
@@ -153,3 +153,40 @@ struct kvm_x86_init_ops vt_init_ops __initdata = {
 	.runtime_ops = &vt_x86_ops,
 	.pmu_ops = &intel_pmu_ops,
 };
+
+static int __init vt_init(void)
+{
+	unsigned int vcpu_size, vcpu_align;
+	int r;
+
+	vt_x86_ops.vm_size = sizeof(struct kvm_vmx);
+	vcpu_size = sizeof(struct vcpu_vmx);
+	vcpu_align = __alignof__(struct vcpu_vmx);
+
+	hv_vp_assist_page_init();
+
+	r = kvm_init(&vt_init_ops, vcpu_size, vcpu_align, THIS_MODULE);
+	if (r)
+		goto err_vmx_post_exit;
+
+	r = vmx_init();
+	if (r)
+		goto err_kvm_exit;
+
+	return 0;
+
+err_kvm_exit:
+	kvm_exit();
+err_vmx_post_exit:
+	hv_vp_assist_page_exit();
+	return r;
+}
+module_init(vt_init);
+
+static void vt_exit(void)
+{
+	vmx_exit();
+	kvm_exit();
+	hv_vp_assist_page_exit();
+}
+module_exit(vt_exit);

arch/x86/kvm/vmx/vmx.c +46 ~~-49~~

125 lines not on this page · raw message

arch/x86/kvm/vmx/x86_ops.h +5 -0

index 8cc2182fc6d7..83efb7f92a58 100644
@@ -8,6 +8,11 @@
 
 #include "x86.h"
 
+void __init hv_vp_assist_page_init(void);
+void hv_vp_assist_page_exit(void);
+int __init vmx_init(void);
+void vmx_exit(void);
+
 __init int vmx_cpu_has_kvm_support(void);
 __init int vmx_disabled_by_bios(void);
 __init int vmx_hardware_setup(void);

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #3 lore raw

PATCH v10 3/108 KVM: TDX: Add placeholders for TDX VM/vcpu structure

From: Isaku Yamahata <[email protected]>

Add placeholders TDX VM/vcpu structure that overlays with VMX VM/vcpu structures. Initialize VM structure size and vcpu size/align so that x86 KVM common code knows those size irrespective of VMX or TDX. Those structures will be populated as guest creation logic develops.

Add helper functions to check if the VM is guest TD and add conversion functions between KVM VM/VCPU and TDX VM/VCPU.

---
 arch/x86/kvm/vmx/main.c |  8 +++---
 arch/x86/kvm/vmx/tdx.h  | 54 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+), 3 deletions(-)
 create mode 100644 arch/x86/kvm/vmx/tdx.h

arch/x86/kvm/vmx/main.c +5 -3

index 4c7e71ec3e1d..a7d4af73228e 100644
@@ -5,6 +5,7 @@
 #include "vmx.h"
 #include "nested.h"
 #include "pmu.h"
+#include "tdx.h"
 
 struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = "kvm_intel",

@@ -159,9 +160,10 @@ static int __init vt_init(void)
 	unsigned int vcpu_size, vcpu_align;
 	int r;
 
-	vt_x86_ops.vm_size = sizeof(struct kvm_vmx);
-	vcpu_size = sizeof(struct vcpu_vmx);
-	vcpu_align = __alignof__(struct vcpu_vmx);
+	vt_x86_ops.vm_size = max(sizeof(struct kvm_vmx), sizeof(struct kvm_tdx));
+	vcpu_size = max(sizeof(struct vcpu_vmx), sizeof(struct vcpu_tdx));
+	vcpu_align = max(__alignof__(struct vcpu_vmx),
+			__alignof__(struct vcpu_tdx));
 
 	hv_vp_assist_page_init();

arch/x86/kvm/vmx/tdx.h new file +54 -0

54 lines not on this page · raw message

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #4 lore raw

PATCH v10 4/108 x86/virt/tdx: Add a helper function to return system wide info about TDX module

From: Isaku Yamahata <[email protected]>

TDX KVM needs system-wide information about the TDX module, struct tdsysinfo_struct. Add a helper function tdx_get_sysinfo() to return it instead of KVM getting it with various error checks. Move out the struct definition about it to common place arch/x86/include/asm/tdx.h.

---
 arch/x86/include/asm/tdx.h  | 55 +++++++++++++++++++++++++++++++++++++
 arch/x86/virt/vmx/tdx/tdx.c | 16 +++++++++--
 arch/x86/virt/vmx/tdx/tdx.h | 52 -----------------------------------
 3 files changed, 69 insertions(+), 54 deletions(-)

arch/x86/include/asm/tdx.h +55 -0

index d568f17da742..5cff7ed5b11e 100644
@@ -131,9 +131,64 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,
 #endif /* CONFIG_INTEL_TDX_GUEST && CONFIG_KVM_GUEST */
 
 #ifdef CONFIG_INTEL_TDX_HOST
+struct tdx_cpuid_config {
+	u32	leaf;
+	u32	sub_leaf;
+	u32	eax;
+	u32	ebx;
+	u32	ecx;
+	u32	edx;
+} __packed;
+
+#define TDSYSINFO_STRUCT_SIZE		1024
+#define TDSYSINFO_STRUCT_ALIGNMENT	1024
+
+struct tdsysinfo_struct {
+	/* TDX-SEAM Module Info */
+	u32	attributes;
+	u32	vendor_id;
+	u32	build_date;
+	u16	build_num;
+	u16	minor_version;
+	u16	major_version;
+	u8	reserved0[14];
+	/* Memory Info */
+	u16	max_tdmrs;
+	u16	max_reserved_per_tdmr;
+	u16	pamt_entry_size;
+	u8	reserved1[10];
+	/* Control Struct Info */
+	u16	tdcs_base_size;
+	u8	reserved2[2];
+	u16	tdvps_base_size;
+	u8	tdvps_xfam_dependent_size;
+	u8	reserved3[9];
+	/* TD Capabilities */
+	u64	attributes_fixed0;
+	u64	attributes_fixed1;
+	u64	xfam_fixed0;
+	u64	xfam_fixed1;
+	u8	reserved4[32];
+	u32	num_cpuid_config;
+	/*
+	 * The actual number of CPUID_CONFIG depends on above
+	 * 'num_cpuid_config'.  The size of 'struct tdsysinfo_struct'
+	 * is 1024B defined by TDX architecture.  Use a union with
+	 * specific padding to make 'sizeof(struct tdsysinfo_struct)'
+	 * equal to 1024.
+	 */
+	union {
+		struct tdx_cpuid_config	cpuid_configs[0];
+		u8			reserved5[892];
+	};
+} __packed __aligned(TDSYSINFO_STRUCT_ALIGNMENT);
+
+const struct tdsysinfo_struct *tdx_get_sysinfo(void);
 bool platform_tdx_enabled(void);
 int tdx_enable(void);
 #else	/* !CONFIG_INTEL_TDX_HOST */
+struct tdsysinfo_struct;
+static inline const struct tdsysinfo_struct *tdx_get_sysinfo(void) { return NULL; }
 static inline bool platform_tdx_enabled(void) { return false; }
 static inline int tdx_enable(void)  { return -ENODEV; }
 #endif	/* CONFIG_INTEL_TDX_HOST */

arch/x86/virt/vmx/tdx/tdx.c +14 -2

index 68ec1ebecb49..6fb630fa7d09 100644
@@ -503,7 +503,7 @@ static int check_cmrs(struct cmr_info *cmr_array, int *actual_cmr_num)
 	return 0;
 }
 
-static int tdx_get_sysinfo(void)
+static int __tdx_get_sysinfo(void)
 {
 	struct tdx_module_output out;
 	int ret;

@@ -530,6 +530,18 @@ static int tdx_get_sysinfo(void)
 	return check_cmrs(tdx_cmr_array, &tdx_cmr_num);
 }
 
+const struct tdsysinfo_struct *tdx_get_sysinfo(void)
+{
+	const struct tdsysinfo_struct *r = NULL;
+
+	mutex_lock(&tdx_module_lock);
+	if (tdx_module_status == TDX_MODULE_INITIALIZED)
+		r = &tdx_sysinfo;
+	mutex_unlock(&tdx_module_lock);
+	return r;
+}
+EXPORT_SYMBOL_GPL(tdx_get_sysinfo);
+
 /* Check whether the first range is the subrange of the second */
 static bool is_subrange(u64 r1_start, u64 r1_end, u64 r2_start, u64 r2_end)
 {

@@ -1238,7 +1250,7 @@ static int init_tdx_module(void)
 	if (ret)
 		goto out;
 
-	ret = tdx_get_sysinfo();
+	ret = __tdx_get_sysinfo();
 	if (ret)
 		goto out;

arch/x86/virt/vmx/tdx/tdx.h +0 ~~-52~~

58 lines not on this page · raw message

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #5 lore raw

PATCH v10 5/108 KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

From: Isaku Yamahata <[email protected]>

TDX requires several initialization steps for KVM to create guest TDs. Detect CPU feature, enable VMX (TDX is based on VMX), detect the TDX module availability, and initialize it. This patch implements those steps.

There are several options on when to initialize the TDX module. A.) kernel module loading time, B.) the first guest TD creation time. A.) was chosen. With B.), a user may hit an error of the TDX initialization when trying to create the first guest TD. The machine that fails to initialize the TDX module can't boot any guest TD further. Such failure is undesirable and a surprise because the user expects that the machine can accommodate guest TD, but actually not. So A.) is better than B.).

Introduce a module parameter, enable_tdx, to explicitly enable TDX KVM support. It's off by default to keep same behavior for those who don't use TDX. Implement hardware_setup method to detect TDX feature of CPU. Because TDX requires all present CPUs to enable VMX (VMXON). The x86 specific kvm_arch_post_hardware_enable_setup overrides the existing weak symbol of kvm_arch_post_hardware_enable_setup which is called at the KVM module initialization.

In reply to Isaku Yamahata

---
 arch/x86/kvm/Makefile      |  1 +
 arch/x86/kvm/vmx/main.c    | 18 ++++++-
 arch/x86/kvm/vmx/tdx.c     | 99 ++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/vmx.c     | 39 +++++++++++++++
 arch/x86/kvm/vmx/x86_ops.h |  9 ++++
 arch/x86/kvm/x86.c         | 32 +++++++-----
 6 files changed, 186 insertions(+), 12 deletions(-)
 create mode 100644 arch/x86/kvm/vmx/tdx.c

arch/x86/kvm/Makefile +1 -0

index ee4d0999f20f..e2c05195cb95 100644
@@ -24,6 +24,7 @@ kvm-$(CONFIG_KVM_XEN)	+= xen.o
 kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \
 			   vmx/evmcs.o vmx/nested.o vmx/posted_intr.o vmx/main.o
 kvm-intel-$(CONFIG_X86_SGX_KVM)	+= vmx/sgx.o
+kvm-intel-$(CONFIG_INTEL_TDX_HOST)	+= vmx/tdx.o
 
 kvm-amd-y		+= svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o

arch/x86/kvm/vmx/main.c +17 -1

index a7d4af73228e..a5cbf3ca2055 100644
@@ -7,6 +7,22 @@
 #include "pmu.h"
 #include "tdx.h"
 
+static bool __read_mostly enable_tdx = IS_ENABLED(CONFIG_INTEL_TDX_HOST);
+module_param_named(tdx, enable_tdx, bool, 0444);
+
+static __init int vt_hardware_setup(void)
+{
+	int ret;
+
+	ret = vmx_hardware_setup();
+	if (ret)
+		return ret;
+
+	enable_tdx = enable_tdx && !tdx_hardware_setup(&vt_x86_ops);
+
+	return 0;
+}
+
 struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = "kvm_intel",

@@ -148,7 +164,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 struct kvm_x86_init_ops vt_init_ops __initdata = {
 	.cpu_has_kvm_support = vmx_cpu_has_kvm_support,
 	.disabled_by_bios = vmx_disabled_by_bios,
-	.hardware_setup = vmx_hardware_setup,
+	.hardware_setup = vt_hardware_setup,
 	.handle_intel_pt_intr = NULL,
 
 	.runtime_ops = &vt_x86_ops,

arch/x86/kvm/vmx/tdx.c new file +99 -0

99 lines not on this page · raw message

arch/x86/kvm/vmx/vmx.c +39 -0

index 1963b28a2ea5..68aef67c5eb7 100644
@@ -2479,6 +2479,35 @@ int vmx_hardware_enable(void)
 	return 0;
 }
 
+static void __init vmxon(void *arg)
+{
+	int cpu = raw_smp_processor_id();
+	u64 phys_addr = __pa(per_cpu(vmxarea, cpu));
+	atomic_t *failed = arg;
+	int r;
+
+	if (cr4_read_shadow() & X86_CR4_VMXE) {
+		r = -EBUSY;
+		goto out;
+	}
+
+	r = kvm_cpu_vmxon(phys_addr);
+out:
+	if (r)
+		atomic_inc(failed);
+}
+
+int __init vmxon_all(void)
+{
+	atomic_t failed = ATOMIC_INIT(0);
+
+	on_each_cpu(vmxon, &failed, 1);
+
+	if (atomic_read(&failed))
+		return -EBUSY;
+	return 0;
+}
+
 static void vmclear_local_loaded_vmcss(void)
 {
 	int cpu = raw_smp_processor_id();

@@ -2499,6 +2528,16 @@ void vmx_hardware_disable(void)
 	intel_pt_handle_vmx(0);
 }
 
+static void __init vmxoff(void *junk)
+{
+	cpu_vmxoff();
+}
+
+void __init vmxoff_all(void)
+{
+	on_each_cpu(vmxoff, NULL, 1);
+}
+
 /*
  * There is no X86_FEATURE for SGX yet, but anyway we need to query CPUID
  * directly instead of going through cpu_has(), to ensure KVM is trapping

arch/x86/kvm/vmx/x86_ops.h +9 -0

index 83efb7f92a58..6196d651a00a 100644
@@ -17,6 +17,9 @@ __init int vmx_cpu_has_kvm_support(void);
 __init int vmx_disabled_by_bios(void);
 __init int vmx_hardware_setup(void);
 
+int __init vmxon_all(void);
+void __init vmxoff_all(void);
+
 extern struct kvm_x86_ops vt_x86_ops __initdata;
 extern struct kvm_x86_init_ops vt_init_ops __initdata;

@@ -127,4 +130,10 @@ void vmx_cancel_hv_timer(struct kvm_vcpu *vcpu);
 #endif
 void vmx_setup_mce(struct kvm_vcpu *vcpu);
 
+#ifdef CONFIG_INTEL_TDX_HOST
+int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);
+#else
+static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }
+#endif
+
 #endif /* __KVM_X86_VMX_X86_OPS_H */

arch/x86/kvm/x86.c +21 ~~-11~~

index 769b3a9a3151..715a53d4fc3d 100644
@@ -12244,6 +12244,16 @@ static void hardware_enable(void *arg)
 		atomic_inc(failed);
 }
 
+static int kvm_hardware_enable_all(void)
+{
+	atomic_t failed = ATOMIC_INIT(0);
+
+	on_each_cpu(hardware_enable, &failed, 1);
+	if (atomic_read(&failed))
+		return -EBUSY;
+	return 0;
+}
+
 static void hardware_disable(void *junk)
 {
 	WARN_ON_ONCE(preemptible());

@@ -12251,29 +12261,29 @@ static void hardware_disable(void *junk)
 	drop_user_return_notifiers();
 }
 
+static void kvm_hardware_disable_all(void)
+{
+	on_each_cpu(hardware_disable, NULL, 1);
+}
+
 /*
  * Called after the VM is otherwise initialized, but just before adding it to
  * the vm_list.
  */
 int kvm_arch_add_vm(struct kvm *kvm, int usage_count)
 {
-	atomic_t failed = ATOMIC_INIT(0);
-	int r = 0;
+	int r;
 
 	if (usage_count != 1)
 		return kvm_mmu_post_init_vm(kvm);
 
-	on_each_cpu(hardware_enable, &failed, 1);
-
-	if (atomic_read(&failed)) {
-		r = -EBUSY;
-		goto err;
-	}
+	r = kvm_hardware_enable_all();
+	if (r)
+		return r;
 
 	r = kvm_mmu_post_init_vm(kvm);
-err:
 	if (r)
-		on_each_cpu(hardware_disable, NULL, 1);
+		kvm_hardware_disable_all();
 	return r;
 }

@@ -12288,7 +12298,7 @@ int kvm_arch_drop_vm(int usage_count)
 	if (usage_count)
 		return 0;
 
-	on_each_cpu(hardware_disable, NULL, 1);
+	kvm_hardware_disable_all();
 	return 0;
 }

Signature · 1 line

2.25.1

Huang, Kai

2022-11-08 01:33 UTC #6 lore raw

Re: [PATCH v10 005/108] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

Quoted 12 lines

arch/x86/kvm/vmx/tdx.c new file +11 -0

+
+#define TDX_MAX_NR_CPUID_CONFIGS					\
+	((sizeof(struct tdsysinfo_struct) -				\
+		offsetof(struct tdsysinfo_struct, cpuid_configs))	\
+		/ sizeof(struct tdx_cpuid_config))
+
+struct tdx_capabilities {
+	u8 tdcs_nr_pages;
+	u8 tdvpx_nr_pages;
+
+	u64 attrs_fixed0;

arch/x86/kvm/vmx/tdx.c line 33

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c new file +10 -0

+	u64 attrs_fixed1;
+	u64 xfam_fixed0;
+	u64 xfam_fixed1;
+
+	u32 nr_cpuid_configs;
+	struct tdx_cpuid_config cpuid_configs[TDX_MAX_NR_CPUID_CONFIGS];
+};
+
+/* Capabilities of KVM + the TDX module. */
+static struct tdx_capabilities tdx_caps;

I think you can introduce this tdx_capabilities in another patch.

As claimed this patch can just focus on initializing the TDX module. Whether you need this tdx_capabilities or tdx_sysinfo is enough can be done in the patch when they are really needed. It makes review easier otherwise people won't be able to tell why tdx_capabilities is needed here.

Quoted 29 lines

arch/x86/kvm/vmx/tdx.c new file +29 -0

+
+static int __init tdx_module_setup(void)
+{
+	const struct tdsysinfo_struct *tdsysinfo;
+	int ret = 0;
+
+	BUILD_BUG_ON(sizeof(*tdsysinfo) != 1024);
+	BUILD_BUG_ON(TDX_MAX_NR_CPUID_CONFIGS != 37);
+
+	ret = tdx_enable();
+	if (ret) {
+		pr_info("Failed to initialize TDX module.\n");
+		return ret;
+	}
+
+	tdsysinfo = tdx_get_sysinfo();
+	if (tdsysinfo->num_cpuid_config > TDX_MAX_NR_CPUID_CONFIGS)
+		return -EIO;
+
+	tdx_caps = (struct tdx_capabilities) {
+		.tdcs_nr_pages = tdsysinfo->tdcs_base_size / PAGE_SIZE,
+		/*
+		 * TDVPS = TDVPR(4K page) + TDVPX(multiple 4K pages).
+		 * -1 for TDVPR.
+		 */
+		.tdvpx_nr_pages = tdsysinfo->tdvps_base_size / PAGE_SIZE - 1,
+		.attrs_fixed0 = tdsysinfo->attributes_fixed0,
+		.attrs_fixed1 = tdsysinfo->attributes_fixed1,
+		.xfam_fixed0 =	tdsysinfo->xfam_fixed0,

arch/x86/kvm/vmx/tdx.c line 72

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c new file +10 -0

+		.xfam_fixed1 = tdsysinfo->xfam_fixed1,
+		.nr_cpuid_configs = tdsysinfo->num_cpuid_config,
+	};
+	if (!memcpy(tdx_caps.cpuid_configs, tdsysinfo->cpuid_configs,
+			tdsysinfo->num_cpuid_config *
+			sizeof(struct tdx_cpuid_config)))
+		return -EIO;
+
+	pr_info("kvm: TDX is supported. x86 phys bits %d\n",
+		boot_cpu_data.x86_phys_bits);

What''s the benefit of print out x86_phys_bits? Looks a little bit weird here.

TDX host code will print out TDX private KeyID range. I think that is useful enough?

Quoted 6 lines

arch/x86/kvm/vmx/tdx.c new file +6 -0

+
+	return 0;
+}
+
+int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
+{

arch/x86/kvm/vmx/tdx.c line 88

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c new file +10 -0

+	int r;
+
+	if (!enable_ept) {
+		pr_warn("Cannot enable TDX with EPT disabled\n");
+		return -EINVAL;
+	}
+
+	/* MOVDIR64B instruction is needed. */
+	if (!static_cpu_has(X86_FEATURE_MOVDIR64B)) {
+		pr_warn("Cannot enable TDX with MOVDIR64B supported ");

					   ^
					   without

Review

Quoted 2 lines

(quoted diff) +2 -0
+		return -ENODEV;
+	}

I think you should explain why MOVDIR64B is required, otherwise this just comes out of blue.

Btw, is this absolutely required? TDX also supports Li-mode, which doesn't have integrity check. So theoretically with Li-mode, normal zeroing is also OK but doesn't need to use MOVDIR64B.

That being said, do we have a way to tell whether TDX works in Ci or Li mode?

Sean Christopherson

maintainer

2022-11-08 18:50 UTC #7 lore raw

Re: [PATCH v10 005/108] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

Quoted 11 lines · Kai

arch/x86/kvm/vmx/tdx.c line 87

Quoted 11 lines

arch/x86/kvm/vmx/tdx.c new file +11 -0

+int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
+{
+	int r;
+
+	if (!enable_ept) {
+		pr_warn("Cannot enable TDX with EPT disabled\n");
+		return -EINVAL;
+	}
+
+	/* MOVDIR64B instruction is needed. */
+	if (!static_cpu_has(X86_FEATURE_MOVDIR64B)) {

Nit unrelated to Kai's comments: use boot_cpu_has(), not static_cpu_has(). This is run-once code that's not a hot path so there's zero reason to trigger patching.

Quoted 17 lines

arch/x86/kvm/vmx/tdx.c line 88
Quoted 1 line
arch/x86/kvm/vmx/tdx.c new file +1 -0
+		pr_warn("Cannot enable TDX with MOVDIR64B supported ");
					   ^
					   without
Review
Quoted 2 lines
(quoted diff) +2 -0
+		return -ENODEV;
+	}
I think you should explain why MOVDIR64B is required, otherwise this just comes out of blue.
Btw, is this absolutely required? TDX also supports Li-mode, which doesn't have integrity check. So theoretically with Li-mode, normal zeroing is also OK but doesn't need to use MOVDIR64B.
That being said, do we have a way to tell whether TDX works in Ci or Li mode?

Isaku Yamahata

2022-11-14 23:18 UTC #8 lore raw

Re: [PATCH v10 005/108] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

In reply to Huang, Kai

Quoted 25 lines · Kai

Quoted 12 lines

arch/x86/kvm/vmx/tdx.c new file +11 -0

+
+#define TDX_MAX_NR_CPUID_CONFIGS					\
+	((sizeof(struct tdsysinfo_struct) -				\
+		offsetof(struct tdsysinfo_struct, cpuid_configs))	\
+		/ sizeof(struct tdx_cpuid_config))
+
+struct tdx_capabilities {
+	u8 tdcs_nr_pages;
+	u8 tdvpx_nr_pages;
+
+	u64 attrs_fixed0;

arch/x86/kvm/vmx/tdx.c line 33

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c new file +10 -0

+	u64 attrs_fixed1;
+	u64 xfam_fixed0;
+	u64 xfam_fixed1;
+
+	u32 nr_cpuid_configs;
+	struct tdx_cpuid_config cpuid_configs[TDX_MAX_NR_CPUID_CONFIGS];
+};
+
+/* Capabilities of KVM + the TDX module. */
+static struct tdx_capabilities tdx_caps;

I think you can introduce this tdx_capabilities in another patch.

Quoted 4 lines

As claimed this patch can just focus on initializing the TDX module. Whether you need this tdx_capabilities or tdx_sysinfo is enough can be done in the patch when they are really needed. It makes review easier otherwise people won't be able to tell why tdx_capabilities is needed here.

Ok, the previous patch ("x86/virt/tdx: Add a helper function to return system wide info about TDX module ") and this part will be moved right before the first use of tdx_caps. "KVM: TDX: create/destroy VM structure"

Quoted 42 lines

Quoted 29 lines

arch/x86/kvm/vmx/tdx.c new file +29 -0

+
+static int __init tdx_module_setup(void)
+{
+	const struct tdsysinfo_struct *tdsysinfo;
+	int ret = 0;
+
+	BUILD_BUG_ON(sizeof(*tdsysinfo) != 1024);
+	BUILD_BUG_ON(TDX_MAX_NR_CPUID_CONFIGS != 37);
+
+	ret = tdx_enable();
+	if (ret) {
+		pr_info("Failed to initialize TDX module.\n");
+		return ret;
+	}
+
+	tdsysinfo = tdx_get_sysinfo();
+	if (tdsysinfo->num_cpuid_config > TDX_MAX_NR_CPUID_CONFIGS)
+		return -EIO;
+
+	tdx_caps = (struct tdx_capabilities) {
+		.tdcs_nr_pages = tdsysinfo->tdcs_base_size / PAGE_SIZE,
+		/*
+		 * TDVPS = TDVPR(4K page) + TDVPX(multiple 4K pages).
+		 * -1 for TDVPR.
+		 */
+		.tdvpx_nr_pages = tdsysinfo->tdvps_base_size / PAGE_SIZE - 1,
+		.attrs_fixed0 = tdsysinfo->attributes_fixed0,
+		.attrs_fixed1 = tdsysinfo->attributes_fixed1,
+		.xfam_fixed0 =	tdsysinfo->xfam_fixed0,

arch/x86/kvm/vmx/tdx.c line 72

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c new file +10 -0

+		.xfam_fixed1 = tdsysinfo->xfam_fixed1,
+		.nr_cpuid_configs = tdsysinfo->num_cpuid_config,
+	};
+	if (!memcpy(tdx_caps.cpuid_configs, tdsysinfo->cpuid_configs,
+			tdsysinfo->num_cpuid_config *
+			sizeof(struct tdx_cpuid_config)))
+		return -EIO;
+
+	pr_info("kvm: TDX is supported. x86 phys bits %d\n",
+		boot_cpu_data.x86_phys_bits);

What''s the benefit of print out x86_phys_bits? Looks a little bit weird here.

Quoted 2 lines

TDX host code will print out TDX private KeyID range. I think that is useful enough?

Ok, please make TDX host code print it. I will remove key id rane.

Quoted 24 lines

Quoted 6 lines
arch/x86/kvm/vmx/tdx.c new file +6 -0
+
+	return 0;
+}
+
+int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
+{
arch/x86/kvm/vmx/tdx.c line 88
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c new file +10 -0
+	int r;
+
+	if (!enable_ept) {
+		pr_warn("Cannot enable TDX with EPT disabled\n");
+		return -EINVAL;
+	}
+
+	/* MOVDIR64B instruction is needed. */
+	if (!static_cpu_has(X86_FEATURE_MOVDIR64B)) {
+		pr_warn("Cannot enable TDX with MOVDIR64B supported ");
					   ^
					   without
Review
Quoted 2 lines
(quoted diff) +2 -0
+		return -ENODEV;
+	}
I think you should explain why MOVDIR64B is required, otherwise this just comes out of blue.

Quoted 5 lines

Btw, is this absolutely required? TDX also supports Li-mode, which doesn't have integrity check. So theoretically with Li-mode, normal zeroing is also OK but doesn't need to use MOVDIR64B.
That being said, do we have a way to tell whether TDX works in Ci or Li mode?

As long as I don't know. When clearing page, we can use if (featuremovdir64b) movdir64b else memset(0).

Signature · 1 line

Isaku Yamahata <[email protected]>

Huang, Kai

2022-11-15 01:58 UTC #9 lore raw

Re: [PATCH v10 005/108] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

Quoted 26 lines · Isaku Yamahata

Quoted 18 lines
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c new file +5 -0
+struct tdx_capabilities {
+	u8 tdcs_nr_pages;
+	u8 tdvpx_nr_pages;
+
+	u64 attrs_fixed0;
arch/x86/kvm/vmx/tdx.c line 33
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c new file +10 -0
+	u64 attrs_fixed1;
+	u64 xfam_fixed0;
+	u64 xfam_fixed1;
+
+	u32 nr_cpuid_configs;
+	struct tdx_cpuid_config cpuid_configs[TDX_MAX_NR_CPUID_CONFIGS];
+};
+
+/* Capabilities of KVM + the TDX module. */
+static struct tdx_capabilities tdx_caps;
I think you can introduce this tdx_capabilities in another patch.
Quoted 7 lines
As claimed this patch can just focus on initializing the TDX module. Whether
you need this tdx_capabilities or tdx_sysinfo is enough can be done in the patch
when they are really needed. It makes review easier otherwise people won't be
able to tell why tdx_capabilities is needed here.

Quoted 4 lines

Ok, the previous patch ("x86/virt/tdx: Add a helper function to return system wide info about TDX module ") and this part will be moved right before the
first use of tdx_caps. "KVM: TDX: create/destroy VM structure"

I think it's OK to put all host dependency patches at the beginning of the series, but just move 'tdx_capabilities' to the patch where it is needed.

Erdem Aktas

2022-11-15 12:23 UTC #10 lore raw

Re: [PATCH v10 005/108] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

Quoted 11 lines · Isaku Yamahata

Quoted 8 lines
I think you should explain why MOVDIR64B is required, otherwise this just comes out of blue.
Btw, is this absolutely required? TDX also supports Li-mode, which doesn't have integrity check. So theoretically with Li-mode, normal zeroing is also OK but doesn't need to use MOVDIR64B.
That being said, do we have a way to tell whether TDX works in Ci or Li mode?
As long as I don't know. When clearing page, we can use if (featuremovdir64b) movdir64b else memset(0).

-- sorry for resending the same email, for some reason my previous email was not in plain text mode--
As far as I know, Li mode and Ci mode both are working very similarly and require movdir64b to clear any poison bit previously set. Why would movdir64b is not supported in a tdx capable cpu?

Quoted 2 lines

--
Isaku Yamahata <[email protected]>

Isaku Yamahata

2022-11-17 17:33 UTC #11 lore raw

Re: [PATCH v10 005/108] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

Quoted 10 lines · Erdem Aktas

Quoted 9 lines · Isaku Yamahata
Quoted 8 lines
I think you should explain why MOVDIR64B is required, otherwise this just comes out of blue.
Btw, is this absolutely required? TDX also supports Li-mode, which doesn't have integrity check. So theoretically with Li-mode, normal zeroing is also OK but doesn't need to use MOVDIR64B.
That being said, do we have a way to tell whether TDX works in Ci or Li mode?

Quoted 8 lines

Quoted 2 lines
As long as I don't know. When clearing page, we can use if (featuremovdir64b) movdir64b else memset(0).
-- sorry for resending the same email, for some reason my previous email was not in plain text mode--
As far as I know, Li mode and Ci mode both are working very similarly and require movdir64b to clear any poison bit previously set. Why would movdir64b is not supported in a tdx capable cpu?

In practice, movdir64b can be safely assumed to be supported, I think. Strictly it's not guaranteed by SDM.
There are two options.

1. Assume modir64b. If it's not supported, KVM refuses to enable TDX.
+ Don't care CI-mode or Li-mode
- not strictly conforming to SDM. In practice, it won't matter.
2. Don't assume movdir64b. If movdir64b not supported, use memset(0).
+ strictly conforming to SDM
- needs to care Ci-mode or Li-mode.

So I'll choose option 1.

Signature · 1 line

Isaku Yamahata <[email protected]>

Erdem Aktas

2023-01-11 22:03 UTC #12 lore raw

Re: [PATCH v10 005/108] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

Quoted 13 lines · On Sat, Oct 29, 2022 at 11:24 PM <[email protected]> wrote:

From: Isaku Yamahata <[email protected]>

arch/x86/kvm/vmx/tdx.c new file +10 -0

+int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
+{
+       int r;
+
+       if (!enable_ept) {
+               pr_warn("Cannot enable TDX with EPT disabled\n");
+               return -EINVAL;
+       }
+
+       /* MOVDIR64B instruction is needed. */

arch/x86/kvm/vmx/tdx.c line 96

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c new file +10 -0

+       if (!static_cpu_has(X86_FEATURE_MOVDIR64B)) {
+               pr_warn("Cannot enable TDX with MOVDIR64B supported ");
+               return -ENODEV;
+       }
+
+       /* TDX requires VMX. */
+       r = vmxon_all();
+       if (!r)
+               r = tdx_module_setup();
+       vmxoff_all();

if few CPUs have VMX enabled, this will disable VMX in all of them. Depending on what enabled VMX on those CPUs, would this not cause kernel crashes/problems?

Isaku Yamahata

2023-01-12 03:09 UTC #13 lore raw

Re: [PATCH v10 005/108] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

Quoted 20 lines · Erdem Aktas

Quoted 19 lines · On Sat, Oct 29, 2022 at 11:24 PM <[email protected]> wrote:

From: Isaku Yamahata <[email protected]>

arch/x86/kvm/vmx/tdx.c new file +16 -0

+int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
+{
+       int r;
+
+       if (!enable_ept) {
+               pr_warn("Cannot enable TDX with EPT disabled\n");
+               return -EINVAL;
+       }
+
+       /* MOVDIR64B instruction is needed. */
+       if (!static_cpu_has(X86_FEATURE_MOVDIR64B)) {
+               pr_warn("Cannot enable TDX with MOVDIR64B supported ");
+               return -ENODEV;
+       }
+
+       /* TDX requires VMX. */

Quoted 8 lines

arch/x86/kvm/vmx/tdx.c line 96
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c new file +4 -0
+       r = vmxon_all();
+       if (!r)
+               r = tdx_module_setup();
+       vmxoff_all();
if few CPUs have VMX enabled, this will disable VMX in all of them. Depending on what enabled VMX on those CPUs, would this not cause kernel crashes/problems?

Are you seeing any issues? or is this a question in theory?

In theory, you're right. In practice, unless we have other kernel components that uses VMX, we don't have to worry. In fact, only KVM in the kernel tree uses VMX.

The flow is as follows,
- VMX is off on all CPUs on start up.
NOTE: VMX is disabled on kexec or reboot.
- On loading kvm_intel.ko (or kernel boot if builtin), this function is called. vmxon/off_all()
- VMX is enabled on all online CPUs when creating the first guest.
- VMX is disabled on all online CPUs when destroying the last guest.
- VMX is disabled on offlining CPU

Thanks,

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:24 UTC #14 lore raw

PATCH v10 6/108 KVM: x86: Introduce vm_type to differentiate default VMs from confidential VMs

From: Sean Christopherson <[email protected]>

Unlike default VMs, confidential VMs (Intel TDX and AMD SEV-ES) don't allow some operations (e.g., memory read/write, register state access, etc).

Introduce vm_type to track the type of the VM to x86 KVM. Other arch KVMs already use vm_type, KVM_INIT_VM accepts vm_type, and x86 KVM callback vm_init accepts vm_type. So follow them. Further, a different policy can be made based on vm_type. Define KVM_X86_DEFAULT_VM for default VM as default and define KVM_X86_TDX_VM for Intel TDX VM. The wrapper function will be defined as "bool is_td(kvm) { return vm_type == VM_TYPE_TDX; }"

Add a capability KVM_CAP_VM_TYPES to effectively allow device model, e.g. qemu, to query what VM types are supported by KVM. This (introduce a new capability and add vm_type) is chosen to align with other arch KVMs that have VM types already. Other arch KVMs uses different name to query supported vm types and there is no common name for it, so new name was chosen.

Co-developed-by Xiaoyao Li <[email protected]>

Signed-off-by Xiaoyao Li <[email protected]>

Signed-off-by Sean Christopherson <[email protected]>

---
 Documentation/virt/kvm/api.rst        | 21 +++++++++++++++++++++
 arch/x86/include/asm/kvm-x86-ops.h    |  1 +
 arch/x86/include/asm/kvm_host.h       |  2 ++
 arch/x86/include/uapi/asm/kvm.h       |  3 +++
 arch/x86/kvm/svm/svm.c                |  6 ++++++
 arch/x86/kvm/vmx/main.c               |  1 +
 arch/x86/kvm/vmx/tdx.h                |  6 +-----
 arch/x86/kvm/vmx/vmx.c                |  5 +++++
 arch/x86/kvm/vmx/x86_ops.h            |  1 +
 arch/x86/kvm/x86.c                    |  9 ++++++++-
 include/uapi/linux/kvm.h              |  1 +
 tools/arch/x86/include/uapi/asm/kvm.h |  3 +++
 tools/include/uapi/linux/kvm.h        |  1 +
 13 files changed, 54 insertions(+), 6 deletions(-)

Documentation/virt/kvm/api.rst +21 -0

index 08253cf498d1..b6f08e8a8320 100644
@@ -147,10 +147,31 @@ described as 'basic' will be available.
 The new VM has no virtual cpus and no memory.
 You probably want to use 0 as machine type.
 
+X86:
+^^^^
+
+Supported vm type can be queried from KVM_CAP_VM_TYPES, which returns the
+bitmap of supported vm types. The 1-setting of bit @n means vm type with
+value @n is supported.
+
+S390:
+^^^^^
+
 In order to create user controlled virtual machines on S390, check
 KVM_CAP_S390_UCONTROL and use the flag KVM_VM_S390_UCONTROL as
 privileged user (CAP_SYS_ADMIN).
 
+MIPS:
+^^^^^
+
+To use hardware assisted virtualization on MIPS (VZ ASE) rather than
+the default trap & emulate implementation (which changes the virtual
+memory layout to fit in user mode), check KVM_CAP_MIPS_VZ and use the
+flag KVM_VM_MIPS_VZ.
+
+ARM64:
+^^^^^^
+
 On arm64, the physical address size for a VM (IPA Size limit) is limited
 to 40bits by default. The limit can be configured if the host supports the
 extension KVM_CAP_ARM_VM_IPA_SIZE. When supported, use

arch/x86/include/asm/kvm-x86-ops.h +1 -0

index 15c1515a0760..8a5c5ae70bc5 100644
@@ -19,6 +19,7 @@ KVM_X86_OP(hardware_disable)
 KVM_X86_OP(hardware_unsetup)
 KVM_X86_OP(has_emulated_msr)
 KVM_X86_OP(vcpu_after_set_cpuid)
+KVM_X86_OP(is_vm_type_supported)
 KVM_X86_OP(vm_init)
 KVM_X86_OP_OPTIONAL(vm_destroy)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate)

arch/x86/include/asm/kvm_host.h +2 -0

index 66b1ff1cec61..2a41a93a80f3 100644
@@ -1151,6 +1151,7 @@ enum kvm_apicv_inhibit {
 };
 
 struct kvm_arch {
+	unsigned long vm_type;
 	unsigned long n_used_mmu_pages;
 	unsigned long n_requested_mmu_pages;
 	unsigned long n_max_mmu_pages;

@@ -1468,6 +1469,7 @@ struct kvm_x86_ops {
 	bool (*has_emulated_msr)(struct kvm *kvm, u32 index);
 	void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);
 
+	bool (*is_vm_type_supported)(unsigned long vm_type);
 	unsigned int vm_size;
 	int (*vm_init)(struct kvm *kvm);
 	void (*vm_destroy)(struct kvm *kvm);

arch/x86/include/uapi/asm/kvm.h +3 -0

index 46de10a809ec..54b08789c402 100644
@@ -532,4 +532,7 @@ struct kvm_pmu_event_filter {
 #define KVM_VCPU_TSC_CTRL 0 /* control group for the timestamp counter (TSC) */
 #define   KVM_VCPU_TSC_OFFSET 0 /* attribute for the TSC offset */
 
+#define KVM_X86_DEFAULT_VM	0
+#define KVM_X86_TDX_VM		1
+
 #endif /* _ASM_X86_KVM_H */

arch/x86/kvm/svm/svm.c +6 -0

index aaf5f0623011..2bcf2e1a5271 100644
@@ -4713,6 +4713,11 @@ static void svm_vm_destroy(struct kvm *kvm)
 	sev_vm_destroy(kvm);
 }
 
+static bool svm_is_vm_type_supported(unsigned long type)
+{
+	return type == KVM_X86_DEFAULT_VM;
+}
+
 static int svm_vm_init(struct kvm *kvm)
 {
 	if (!pause_filter_count || !pause_filter_thresh)

@@ -4740,6 +4745,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.vcpu_free = svm_vcpu_free,
 	.vcpu_reset = svm_vcpu_reset,
 
+	.is_vm_type_supported = svm_is_vm_type_supported,
 	.vm_size = sizeof(struct kvm_svm),
 	.vm_init = svm_vm_init,
 	.vm_destroy = svm_vm_destroy,

arch/x86/kvm/vmx/main.c +1 -0

index a5cbf3ca2055..22bf49afc761 100644
@@ -33,6 +33,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.hardware_disable = vmx_hardware_disable,
 	.has_emulated_msr = vmx_has_emulated_msr,
 
+	.is_vm_type_supported = vmx_is_vm_type_supported,
 	.vm_size = sizeof(struct kvm_vmx),
 	.vm_init = vmx_vm_init,
 	.vm_destroy = vmx_vm_destroy,

arch/x86/kvm/vmx/tdx.h +1 -5

index 060bf48ec3d6..473013265bd8 100644
@@ -15,11 +15,7 @@ struct vcpu_tdx {
 
 static inline bool is_td(struct kvm *kvm)
 {
-	/*
-	 * TDX VM type isn't defined yet.
-	 * return kvm->arch.vm_type == KVM_X86_TDX_VM;
-	 */
-	return false;
+	return kvm->arch.vm_type == KVM_X86_TDX_VM;
 }
 
 static inline bool is_td_vcpu(struct kvm_vcpu *vcpu)

arch/x86/kvm/vmx/vmx.c +5 -0

index 68aef67c5eb7..dc05b78e0a1e 100644
@@ -7410,6 +7410,11 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu)
 	return err;
 }
 
+bool vmx_is_vm_type_supported(unsigned long type)
+{
+	return type == KVM_X86_DEFAULT_VM;
+}
+
 #define L1TF_MSG_SMT "L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.\n"
 #define L1TF_MSG_L1D "L1TF CPU bug present and virtualization mitigation disabled, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.\n"

arch/x86/kvm/vmx/x86_ops.h +1 -0

index 6196d651a00a..d4877f4f93de 100644
@@ -27,6 +27,7 @@ void vmx_hardware_unsetup(void);
 int vmx_check_processor_compatibility(void);
 int vmx_hardware_enable(void);
 void vmx_hardware_disable(void);
+bool vmx_is_vm_type_supported(unsigned long type);
 int vmx_vm_init(struct kvm *kvm);
 void vmx_vm_destroy(struct kvm *kvm);
 int vmx_vcpu_precreate(struct kvm *kvm);

arch/x86/kvm/x86.c +8 -1

index 715a53d4fc3d..91053fdc4512 100644
@@ -4520,6 +4520,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_X86_NOTIFY_VMEXIT:
 		r = kvm_caps.has_notify_vmexit;
 		break;
+	case KVM_CAP_VM_TYPES:
+		r = BIT(KVM_X86_DEFAULT_VM);
+		if (static_call(kvm_x86_is_vm_type_supported)(KVM_X86_TDX_VM))
+			r |= BIT(KVM_X86_TDX_VM);
+		break;
 	default:
 		break;
 	}

@@ -12507,9 +12512,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	int ret;
 	unsigned long flags;
 
-	if (type)
+	if (!static_call(kvm_x86_is_vm_type_supported)(type))
 		return -EINVAL;
 
+	kvm->arch.vm_type = type;
+
 	ret = kvm_page_track_init(kvm);
 	if (ret)
 		goto out;

include/uapi/linux/kvm.h +1 -0

index fa60b032a405..49386e4de8b8 100644
@@ -1216,6 +1216,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_CPU_TOPOLOGY 222
 #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
 #define KVM_CAP_PRIVATE_MEM 224
+#define KVM_CAP_VM_TYPES 225
 
 #ifdef KVM_CAP_IRQ_ROUTING

tools/arch/x86/include/uapi/asm/kvm.h +3 -0

index 46de10a809ec..54b08789c402 100644
@@ -532,4 +532,7 @@ struct kvm_pmu_event_filter {
 #define KVM_VCPU_TSC_CTRL 0 /* control group for the timestamp counter (TSC) */
 #define   KVM_VCPU_TSC_OFFSET 0 /* attribute for the TSC offset */
 
+#define KVM_X86_DEFAULT_VM	0
+#define KVM_X86_TDX_VM		1
+
 #endif /* _ASM_X86_KVM_H */

tools/include/uapi/linux/kvm.h +1 -0

index 0d5d4419139a..812b771d8702 100644
@@ -1178,6 +1178,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_ZPCI_OP 221
 #define KVM_CAP_S390_CPU_TOPOLOGY 222
 #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
+#define KVM_CAP_VM_TYPES 225
 
 #ifdef KVM_CAP_IRQ_ROUTING

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #15 lore raw

PATCH v10 7/108 KVM: TDX: Make TDX VM type supported

From: Isaku Yamahata <[email protected]>

NOTE: This patch is in position of the patch series for developers to be able to test codes during the middle of the patch series although this patch series doesn't provide functional features until the all the patches of this patch series. When merging this patch series, this patch can be moved to the end.

As first step TDX VM support, return that TDX VM type supported to device model, e.g. qemu. The callback to create guest TD is vm_init callback for KVM_CREATE_VM.

---
 arch/x86/kvm/vmx/main.c    | 18 ++++++++++++++++--
 arch/x86/kvm/vmx/tdx.c     |  6 ++++++
 arch/x86/kvm/vmx/vmx.c     |  5 -----
 arch/x86/kvm/vmx/x86_ops.h |  3 ++-
 4 files changed, 24 insertions(+), 8 deletions(-)

arch/x86/kvm/vmx/main.c +16 -2

index 22bf49afc761..0900ff2f2390 100644
@@ -10,6 +10,12 @@
 static bool __read_mostly enable_tdx = IS_ENABLED(CONFIG_INTEL_TDX_HOST);
 module_param_named(tdx, enable_tdx, bool, 0444);
 
+static bool vt_is_vm_type_supported(unsigned long type)
+{
+	return type == KVM_X86_DEFAULT_VM ||
+		(enable_tdx && tdx_is_vm_type_supported(type));
+}
+
 static __init int vt_hardware_setup(void)
 {
 	int ret;

@@ -23,6 +29,14 @@ static __init int vt_hardware_setup(void)
 	return 0;
 }
 
+static int vt_vm_init(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return -EOPNOTSUPP;	/* Not ready to create guest TD yet. */
+
+	return vmx_vm_init(kvm);
+}
+
 struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = "kvm_intel",

@@ -33,9 +47,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.hardware_disable = vmx_hardware_disable,
 	.has_emulated_msr = vmx_has_emulated_msr,
 
-	.is_vm_type_supported = vmx_is_vm_type_supported,
+	.is_vm_type_supported = vt_is_vm_type_supported,
 	.vm_size = sizeof(struct kvm_vmx),
-	.vm_init = vmx_vm_init,
+	.vm_init = vt_vm_init,
 	.vm_destroy = vmx_vm_destroy,
 
 	.vcpu_precreate = vmx_vcpu_precreate,

arch/x86/kvm/vmx/tdx.c +6 -0

index 6213a5c6b637..530e72f85762 100644
@@ -74,6 +74,12 @@ static int __init tdx_module_setup(void)
 	return 0;
 }
 
+bool tdx_is_vm_type_supported(unsigned long type)
+{
+	/* enable_tdx check is done by the caller. */
+	return type == KVM_X86_TDX_VM;
+}
+
 int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
 {
 	int r;

arch/x86/kvm/vmx/vmx.c +0 -5

index dc05b78e0a1e..68aef67c5eb7 100644
@@ -7410,11 +7410,6 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu)
 	return err;
 }
 
-bool vmx_is_vm_type_supported(unsigned long type)
-{
-	return type == KVM_X86_DEFAULT_VM;
-}
-
 #define L1TF_MSG_SMT "L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.\n"
 #define L1TF_MSG_L1D "L1TF CPU bug present and virtualization mitigation disabled, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.\n"

arch/x86/kvm/vmx/x86_ops.h +2 -1

index d4877f4f93de..ac1688b0b0e3 100644
@@ -27,7 +27,6 @@ void vmx_hardware_unsetup(void);
 int vmx_check_processor_compatibility(void);
 int vmx_hardware_enable(void);
 void vmx_hardware_disable(void);
-bool vmx_is_vm_type_supported(unsigned long type);
 int vmx_vm_init(struct kvm *kvm);
 void vmx_vm_destroy(struct kvm *kvm);
 int vmx_vcpu_precreate(struct kvm *kvm);

@@ -133,8 +132,10 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);
 
 #ifdef CONFIG_INTEL_TDX_HOST
 int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);
+bool tdx_is_vm_type_supported(unsigned long type);
 #else
 static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }
+static inline bool tdx_is_vm_type_supported(unsigned long type) { return false; }
 #endif
 
 #endif /* __KVM_X86_VMX_X86_OPS_H */

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #16 lore raw

PATCH v10 8/108 The start of TDX KVM patch series: TDX architectural definitions

From: Isaku Yamahata <[email protected]>

This empty commit is to mark the start of patch series of TDX architectural definitions.

---
 .../virt/kvm/intel-tdx-layer-status.rst       | 29 +++++++++++++++++++
 1 file changed, 29 insertions(+)
 create mode 100644 Documentation/virt/kvm/intel-tdx-layer-status.rst

Documentation/virt/kvm/intel-tdx-layer-status.rst new file +29 -0

new file mode 100644
index 000000000000..b7a14bc73853
@@ -0,0 +1,29 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===================================
+Intel Trust Dodmain Extensions(TDX)
+===================================
+
+Layer status
+============
+What qemu can do
+----------------
+- TDX VM TYPE is exposed to Qemu.
+- Qemu can try to create VM of TDX VM type and then fails.
+
+Patch Layer status
+------------------
+  Patch layer                          Status
+* TDX, VMX coexistence:                 Applied
+* TDX architectural definitions:        Applying
+* TD VM creation/destruction:           Not yet
+* TD vcpu creation/destruction:         Not yet
+* TDX EPT violation:                    Not yet
+* TD finalization:                      Not yet
+* TD vcpu enter/exit:                   Not yet
+* TD vcpu interrupts/exit/hypercall:    Not yet
+
+* KVM MMU GPA shared bits:              Not yet
+* KVM TDP refactoring for TDX:          Not yet
+* KVM TDP MMU hooks:                    Not yet
+* KVM TDP MMU MapGPA:                   Not yet

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #17 lore raw

PATCH v10 9/108 KVM: TDX: Define TDX architectural definitions

From: Isaku Yamahata <[email protected]>

Define architectural definitions for KVM to issue the TDX SEAMCALLs.

Structures and values that are architecturally defined in the TDX module specifications the chapter of ABI Reference.

Co-developed-by Sean Christopherson <[email protected]>

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/kvm/vmx/tdx_arch.h | 166 ++++++++++++++++++++++++++++++++++++
 1 file changed, 166 insertions(+)
 create mode 100644 arch/x86/kvm/vmx/tdx_arch.h

arch/x86/kvm/vmx/tdx_arch.h new file +166 -0

new file mode 100644
index 000000000000..18604734fb14
@@ -0,0 +1,166 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* architectural constants/data definitions for TDX SEAMCALLs */
+
+#ifndef __KVM_X86_TDX_ARCH_H
+#define __KVM_X86_TDX_ARCH_H
+
+#include <linux/types.h>
+
+/*
+ * TDX SEAMCALL API function leaves
+ */
+#define TDH_VP_ENTER			0
+#define TDH_MNG_ADDCX			1
+#define TDH_MEM_PAGE_ADD		2
+#define TDH_MEM_SEPT_ADD		3
+#define TDH_VP_ADDCX			4
+#define TDH_MEM_PAGE_RELOCATE		5
+#define TDH_MEM_PAGE_AUG		6
+#define TDH_MEM_RANGE_BLOCK		7
+#define TDH_MNG_KEY_CONFIG		8
+#define TDH_MNG_CREATE			9
+#define TDH_VP_CREATE			10
+#define TDH_MNG_RD			11
+#define TDH_MR_EXTEND			16
+#define TDH_MR_FINALIZE			17
+#define TDH_VP_FLUSH			18
+#define TDH_MNG_VPFLUSHDONE		19
+#define TDH_MNG_KEY_FREEID		20
+#define TDH_MNG_INIT			21
+#define TDH_VP_INIT			22
+#define TDH_VP_RD			26
+#define TDH_MNG_KEY_RECLAIMID		27
+#define TDH_PHYMEM_PAGE_RECLAIM		28
+#define TDH_MEM_PAGE_REMOVE		29
+#define TDH_MEM_SEPT_REMOVE		30
+#define TDH_MEM_TRACK			38
+#define TDH_MEM_RANGE_UNBLOCK		39
+#define TDH_PHYMEM_CACHE_WB		40
+#define TDH_PHYMEM_PAGE_WBINVD		41
+#define TDH_VP_WR			43
+#define TDH_SYS_LP_SHUTDOWN		44
+
+#define TDG_VP_VMCALL_GET_TD_VM_CALL_INFO		0x10000
+#define TDG_VP_VMCALL_MAP_GPA				0x10001
+#define TDG_VP_VMCALL_GET_QUOTE				0x10002
+#define TDG_VP_VMCALL_REPORT_FATAL_ERROR		0x10003
+#define TDG_VP_VMCALL_SETUP_EVENT_NOTIFY_INTERRUPT	0x10004
+
+/* TDX control structure (TDR/TDCS/TDVPS) field access codes */
+#define TDX_NON_ARCH			BIT_ULL(63)
+#define TDX_CLASS_SHIFT			56
+#define TDX_FIELD_MASK			GENMASK_ULL(31, 0)
+
+#define __BUILD_TDX_FIELD(non_arch, class, field)	\
+	(((non_arch) ? TDX_NON_ARCH : 0) |		\
+	 ((u64)(class) << TDX_CLASS_SHIFT) |		\
+	 ((u64)(field) & TDX_FIELD_MASK))
+
+#define BUILD_TDX_FIELD(class, field)			\
+	__BUILD_TDX_FIELD(false, (class), (field))
+
+#define BUILD_TDX_FIELD_NON_ARCH(class, field)		\
+	__BUILD_TDX_FIELD(true, (class), (field))
+
+
+/* Class code for TD */
+#define TD_CLASS_EXECUTION_CONTROLS	17ULL
+
+/* Class code for TDVPS */
+#define TDVPS_CLASS_VMCS		0ULL
+#define TDVPS_CLASS_GUEST_GPR		16ULL
+#define TDVPS_CLASS_OTHER_GUEST		17ULL
+#define TDVPS_CLASS_MANAGEMENT		32ULL
+
+enum tdx_tdcs_execution_control {
+	TD_TDCS_EXEC_TSC_OFFSET = 10,
+};
+
+/* @field is any of enum tdx_tdcs_execution_control */
+#define TDCS_EXEC(field)		BUILD_TDX_FIELD(TD_CLASS_EXECUTION_CONTROLS, (field))
+
+/* @field is the VMCS field encoding */
+#define TDVPS_VMCS(field)		BUILD_TDX_FIELD(TDVPS_CLASS_VMCS, (field))
+
+enum tdx_vcpu_guest_other_state {
+	TD_VCPU_STATE_DETAILS_NON_ARCH = 0x100,
+};
+
+union tdx_vcpu_state_details {
+	struct {
+		u64 vmxip	: 1;
+		u64 reserved	: 63;
+	};
+	u64 full;
+};
+
+/* @field is any of enum tdx_guest_other_state */
+#define TDVPS_STATE(field)		BUILD_TDX_FIELD(TDVPS_CLASS_OTHER_GUEST, (field))
+#define TDVPS_STATE_NON_ARCH(field)	BUILD_TDX_FIELD_NON_ARCH(TDVPS_CLASS_OTHER_GUEST, (field))
+
+/* Management class fields */
+enum tdx_vcpu_guest_management {
+	TD_VCPU_PEND_NMI = 11,
+};
+
+/* @field is any of enum tdx_vcpu_guest_management */
+#define TDVPS_MANAGEMENT(field)		BUILD_TDX_FIELD(TDVPS_CLASS_MANAGEMENT, (field))
+
+#define TDX_EXTENDMR_CHUNKSIZE		256
+
+struct tdx_cpuid_value {
+	u32 eax;
+	u32 ebx;
+	u32 ecx;
+	u32 edx;
+} __packed;
+
+#define TDX_TD_ATTRIBUTE_DEBUG		BIT_ULL(0)
+#define TDX_TD_ATTRIBUTE_PKS		BIT_ULL(30)
+#define TDX_TD_ATTRIBUTE_KL		BIT_ULL(31)
+#define TDX_TD_ATTRIBUTE_PERFMON	BIT_ULL(63)
+
+/*
+ * TD_PARAMS is provided as an input to TDH_MNG_INIT, the size of which is 1024B.
+ */
+struct td_params {
+	u64 attributes;
+	u64 xfam;
+	u32 max_vcpus;
+	u32 reserved0;
+
+	u64 eptp_controls;
+	u64 exec_controls;
+	u16 tsc_frequency;
+	u8  reserved1[38];
+
+	u64 mrconfigid[6];
+	u64 mrowner[6];
+	u64 mrownerconfig[6];
+	u64 reserved2[4];
+
+	union {
+		struct tdx_cpuid_value cpuid_values[0];
+		u8 reserved3[768];
+	};
+} __packed __aligned(1024);
+
+/*
+ * Guest uses MAX_PA for GPAW when set.
+ * 0: GPA.SHARED bit is GPA[47]
+ * 1: GPA.SHARED bit is GPA[51]
+ */
+#define TDX_EXEC_CONTROL_MAX_GPAW      BIT_ULL(0)
+
+/*
+ * TDX requires the frequency to be defined in units of 25MHz, which is the
+ * frequency of the core crystal clock on TDX-capable platforms, i.e. the TDX
+ * module can only program frequencies that are multiples of 25MHz.  The
+ * frequency must be between 100mhz and 10ghz (inclusive).
+ */
+#define TDX_TSC_KHZ_TO_25MHZ(tsc_in_khz)	((tsc_in_khz) / (25 * 1000))
+#define TDX_TSC_25MHZ_TO_KHZ(tsc_in_25mhz)	((tsc_in_25mhz) * (25 * 1000))
+#define TDX_MIN_TSC_FREQUENCY_KHZ		(100 * 1000)
+#define TDX_MAX_TSC_FREQUENCY_KHZ		(10 * 1000 * 1000)
+
+#endif /* __KVM_X86_TDX_ARCH_H */

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #18 lore raw

PATCH v10 10/108 KVM: TDX: Add TDX "architectural" error codes

From: Sean Christopherson <[email protected]>

Add error codes for the TDX SEAMCALLs both for TDX VMM side for TDH SEAMCALL and TDX guest side for TDG.VP.VMCALL. KVM issues the TDX SEAMCALLs and checks its error code. KVM handles hypercall from the TDX guest and may return an error. So error code for the TDX guest is also needed.

TDX SEAMCALL uses bits 31:0 to return more information, so these error codes will only exactly match RAX[63:32]. Error codes for TDG.VP.VMCALL is defined by TDX Guest-Host-Communication interface spec.

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/kvm/vmx/tdx_errno.h | 38 ++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 arch/x86/kvm/vmx/tdx_errno.h

arch/x86/kvm/vmx/tdx_errno.h new file +38 -0

new file mode 100644
index 000000000000..ce246ba62454
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* architectural status code for SEAMCALL */
+
+#ifndef __KVM_X86_TDX_ERRNO_H
+#define __KVM_X86_TDX_ERRNO_H
+
+#define TDX_SEAMCALL_STATUS_MASK		0xFFFFFFFF00000000ULL
+
+/*
+ * TDX SEAMCALL Status Codes (returned in RAX)
+ */
+#define TDX_SUCCESS				0x0000000000000000ULL
+#define TDX_NON_RECOVERABLE_VCPU		0x4000000100000000ULL
+#define TDX_INTERRUPTED_RESUMABLE		0x8000000300000000ULL
+#define TDX_OPERAND_BUSY                        0x8000020000000000ULL
+#define TDX_VCPU_NOT_ASSOCIATED			0x8000070200000000ULL
+#define TDX_KEY_GENERATION_FAILED		0x8000080000000000ULL
+#define TDX_KEY_STATE_INCORRECT			0xC000081100000000ULL
+#define TDX_KEY_CONFIGURED			0x0000081500000000ULL
+#define TDX_NO_HKID_READY_TO_WBCACHE		0x0000082100000000ULL
+#define TDX_EPT_WALK_FAILED			0xC0000B0000000000ULL
+
+/*
+ * TDG.VP.VMCALL Status Codes (returned in R10)
+ */
+#define TDG_VP_VMCALL_SUCCESS			0x0000000000000000ULL
+#define TDG_VP_VMCALL_RETRY			0x0000000000000001ULL
+#define TDG_VP_VMCALL_INVALID_OPERAND		0x8000000000000000ULL
+#define TDG_VP_VMCALL_TDREPORT_FAILED		0x8000000000000001ULL
+
+/*
+ * TDX module operand ID, appears in 31:0 part of error code as
+ * detail information
+ */
+#define TDX_OPERAND_ID_RCX			0x01
+#define TDX_OPERAND_ID_SEPT			0x92
+
+#endif /* __KVM_X86_TDX_ERRNO_H */

Signature · 1 line

2.25.1

Binbin Wu

2022-10-31 09:24 UTC #19 lore raw

Re: [PATCH v10 010/108] KVM: TDX: Add TDX "architectural" error codes

Quoted 31 lines · On 2022/10/30 14:22, [email protected] wrote:

From: Sean Christopherson <[email protected]>
Add error codes for the TDX SEAMCALLs both for TDX VMM side for TDH SEAMCALL and TDX guest side for TDG.VP.VMCALL. KVM issues the TDX SEAMCALLs and checks its error code. KVM handles hypercall from the TDX guest and may return an error. So error code for the TDX guest is also needed.
TDX SEAMCALL uses bits 31:0 to return more information, so these error codes will only exactly match RAX[63:32]. Error codes for TDG.VP.VMCALL is defined by TDX Guest-Host-Communication interface spec.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
---
  arch/x86/kvm/vmx/tdx_errno.h | 38 ++++++++++++++++++++++++++++++++++++
  1 file changed, 38 insertions(+)
create mode 100644 arch/x86/kvm/vmx/tdx_errno.h
arch/x86/kvm/vmx/tdx_errno.h new file +5 -0
new file mode 100644
index 000000000000..ce246ba62454
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* architectural status code for SEAMCALL */
+
+#ifndef __KVM_X86_TDX_ERRNO_H
+#define __KVM_X86_TDX_ERRNO_H

arch/x86/kvm/vmx/tdx_errno.h line 15

Quoted 10 lines

arch/x86/kvm/vmx/tdx_errno.h new file +10 -0

+
+#define TDX_SEAMCALL_STATUS_MASK		0xFFFFFFFF00000000ULL
+
+/*
+ * TDX SEAMCALL Status Codes (returned in RAX)
+ */
+#define TDX_SUCCESS				0x0000000000000000ULL
+#define TDX_NON_RECOVERABLE_VCPU		0x4000000100000000ULL
+#define TDX_INTERRUPTED_RESUMABLE		0x8000000300000000ULL
+#define TDX_OPERAND_BUSY                        0x8000020000000000ULL

one minor format issue, spaces are used instread of tabs this line.

Quoted 23 lines

arch/x86/kvm/vmx/tdx_errno.h new file +23 -0

+#define TDX_VCPU_NOT_ASSOCIATED			0x8000070200000000ULL
+#define TDX_KEY_GENERATION_FAILED		0x8000080000000000ULL
+#define TDX_KEY_STATE_INCORRECT			0xC000081100000000ULL
+#define TDX_KEY_CONFIGURED			0x0000081500000000ULL
+#define TDX_NO_HKID_READY_TO_WBCACHE		0x0000082100000000ULL
+#define TDX_EPT_WALK_FAILED			0xC0000B0000000000ULL
+
+/*
+ * TDG.VP.VMCALL Status Codes (returned in R10)
+ */
+#define TDG_VP_VMCALL_SUCCESS			0x0000000000000000ULL
+#define TDG_VP_VMCALL_RETRY			0x0000000000000001ULL
+#define TDG_VP_VMCALL_INVALID_OPERAND		0x8000000000000000ULL
+#define TDG_VP_VMCALL_TDREPORT_FAILED		0x8000000000000001ULL
+
+/*
+ * TDX module operand ID, appears in 31:0 part of error code as
+ * detail information
+ */
+#define TDX_OPERAND_ID_RCX			0x01
+#define TDX_OPERAND_ID_SEPT			0x92
+
+#endif /* __KVM_X86_TDX_ERRNO_H */

Isaku Yamahata

2022-11-03 00:05 UTC #20 lore raw

Re: [PATCH v10 010/108] KVM: TDX: Add TDX "architectural" error codes

Quoted 37 lines · Binbin Wu

Quoted 35 lines · On 2022/10/30 14:22, [email protected] wrote:
From: Sean Christopherson <[email protected]>
Add error codes for the TDX SEAMCALLs both for TDX VMM side for TDH SEAMCALL and TDX guest side for TDG.VP.VMCALL. KVM issues the TDX SEAMCALLs and checks its error code. KVM handles hypercall from the TDX guest and may return an error. So error code for the TDX guest is also needed.
TDX SEAMCALL uses bits 31:0 to return more information, so these error codes will only exactly match RAX[63:32]. Error codes for TDG.VP.VMCALL is defined by TDX Guest-Host-Communication interface spec.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
Reviewed-by Paolo Bonzini <[email protected]>
---
  arch/x86/kvm/vmx/tdx_errno.h | 38 ++++++++++++++++++++++++++++++++++++
  1 file changed, 38 insertions(+)
create mode 100644 arch/x86/kvm/vmx/tdx_errno.h
arch/x86/kvm/vmx/tdx_errno.h new file +9 -0
new file mode 100644
index 000000000000..ce246ba62454
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* architectural status code for SEAMCALL */
+
+#ifndef __KVM_X86_TDX_ERRNO_H
+#define __KVM_X86_TDX_ERRNO_H
+
+#define TDX_SEAMCALL_STATUS_MASK		0xFFFFFFFF00000000ULL
+
+/*

Quoted 8 lines

arch/x86/kvm/vmx/tdx_errno.h line 15

Quoted 6 lines

arch/x86/kvm/vmx/tdx_errno.h new file +6 -0

+ * TDX SEAMCALL Status Codes (returned in RAX)
+ */
+#define TDX_SUCCESS				0x0000000000000000ULL
+#define TDX_NON_RECOVERABLE_VCPU		0x4000000100000000ULL
+#define TDX_INTERRUPTED_RESUMABLE		0x8000000300000000ULL
+#define TDX_OPERAND_BUSY                        0x8000020000000000ULL

one minor format issue, spaces are used instread of tabs this line.

Thanks, fixed it locally. It will be include in the next respin.

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:24 UTC #21 lore raw

PATCH v10 11/108 KVM: TDX: Add C wrapper functions for SEAMCALLs to the TDX module

From: Isaku Yamahata <[email protected]>

A VMM interacts with the TDX module using a new instruction (SEAMCALL). A TDX VMM uses SEAMCALLs where a VMX VMM would have directly interacted with VMX instructions. For instance, a TDX VMM does not have full access to the VM control structure corresponding to VMX VMCS. Instead, a VMM induces the TDX module to act on behalf via SEAMCALLs.

Export __seamcall and define C wrapper functions for SEAMCALLs for readability. Some SEAMCALL APIs donates pages to TDX module or guest TD. The pages are encrypted with TDX private host key id set in high bits of physical address. If any modified cache lines may exit for these pages, flush them to memory by clflush_cache_range().

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/tdx.h       |   2 +
 arch/x86/kvm/vmx/tdx_ops.h       | 185 +++++++++++++++++++++++++++++++
 arch/x86/virt/vmx/tdx/seamcall.S |   2 +
 3 files changed, 189 insertions(+)
 create mode 100644 arch/x86/kvm/vmx/tdx_ops.h

arch/x86/include/asm/tdx.h +2 -0

index 5cff7ed5b11e..ba2e4c69fb9f 100644
@@ -186,6 +186,8 @@ struct tdsysinfo_struct {
 const struct tdsysinfo_struct *tdx_get_sysinfo(void);
 bool platform_tdx_enabled(void);
 int tdx_enable(void);
+u64 __seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9,
+	       struct tdx_module_output *out);
 #else	/* !CONFIG_INTEL_TDX_HOST */
 struct tdsysinfo_struct;
 static inline const struct tdsysinfo_struct *tdx_get_sysinfo(void) { return NULL; }

arch/x86/kvm/vmx/tdx_ops.h new file +185 -0

185 lines not on this page · raw message

arch/x86/virt/vmx/tdx/seamcall.S +2 -0

index f81be6b9c133..b90a7fe05494 100644
@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 #include <linux/linkage.h>
+#include <asm/export.h>
 #include <asm/frame.h>
 
 #include "tdxcall.S"

@@ -50,3 +51,4 @@ SYM_FUNC_START(__seamcall)
 	FRAME_END
 	RET
 SYM_FUNC_END(__seamcall)
+EXPORT_SYMBOL_GPL(__seamcall)

Signature · 1 line

2.25.1

Zhi Wang

2022-11-23 10:12 UTC #22 lore raw

Re: [PATCH v10 011/108] KVM: TDX: Add C wrapper functions for SEAMCALLs to the TDX module

Quoted 9 lines · On Sat, 29 Oct 2022 23:22:12 -0700 [email protected] wrote:

From: Isaku Yamahata <[email protected]>
A VMM interacts with the TDX module using a new instruction (SEAMCALL). A TDX VMM uses SEAMCALLs where a VMX VMM would have directly interacted with VMX instructions. For instance, a TDX VMM does not have full access to the VM control structure corresponding to VMX VMCS. Instead, a VMM induces the TDX module to act on behalf via SEAMCALLs.

Quoted 3 lines

Export __seamcall and define C wrapper functions for SEAMCALLs for readability. Some SEAMCALL APIs donates pages to TDX module or guest TD. The pages are encrypted with TDX private host key id set in high

It seems a little bit confusing here.

I guess you are talking about the host pages donated to TDX module will be encrypted with TDX module global key id, which is used to encrypted the pages only used by TDX module. For pages donated to the TD guest, the pages are encrypted with TD guest prviate key id?

It might be better that you can talk about the actual reaason instead of the key id stuff. For example Some SEMCALL APIs will zap or write the pages donated by VMM via MOVDIR64B, which requires the VMM to flush the cachelines.

Quoted 1 line

bits of physical address. If any modified cache lines may exit for

typo exit->exist

Quoted 239 lines

these pages, flush them to memory by clflush_cache_range().

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/tdx.h       |   2 +
 arch/x86/kvm/vmx/tdx_ops.h       | 185
+++++++++++++++++++++++++++++++ arch/x86/virt/vmx/tdx/seamcall.S |
2 + 3 files changed, 189 insertions(+)
 create mode 100644 arch/x86/kvm/vmx/tdx_ops.h

arch/x86/include/asm/tdx.h +2 -0

index 5cff7ed5b11e..ba2e4c69fb9f 100644
@@ -186,6 +186,8 @@ struct tdsysinfo_struct {
 const struct tdsysinfo_struct *tdx_get_sysinfo(void);
 bool platform_tdx_enabled(void);
 int tdx_enable(void);
+u64 __seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9,
+	       struct tdx_module_output *out);
 #else	/* !CONFIG_INTEL_TDX_HOST */
 struct tdsysinfo_struct;
 static inline const struct tdsysinfo_struct *tdx_get_sysinfo(void) {

return NULL; } diff --git a/arch/x86/kvm/vmx/tdx_ops.h b/arch/x86/kvm/vmx/tdx_ops.h new file mode 100644 index 000000000000..85adbf49c277

arch/x86/kvm/vmx/tdx_ops.h new file +24 -0

@@ -0,0 +1,185 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* constants/data definitions for TDX SEAMCALLs */
+
+#ifndef __KVM_X86_TDX_OPS_H
+#define __KVM_X86_TDX_OPS_H
+
+#include <linux/compiler.h>
+
+#include <asm/cacheflush.h>
+#include <asm/asm.h>
+#include <asm/kvm_host.h>
+
+#include "tdx_errno.h"
+#include "tdx_arch.h"
+
+#ifdef CONFIG_INTEL_TDX_HOST
+
+static inline u64 tdh_mng_addcx(hpa_t tdr, hpa_t addr)
+{
+	clflush_cache_range(__va(addr), PAGE_SIZE);
+	return __seamcall(TDH_MNG_ADDCX, addr, tdr, 0, 0, NULL);
+}
+
+static inline u64 tdh_mem_page_add(hpa_t tdr, gpa_t gpa, hpa_t hpa,

hpa_t source,

arch/x86/kvm/vmx/tdx_ops.h new file +4 -0

+				   struct tdx_module_output *out)
+{
+	clflush_cache_range(__va(hpa), PAGE_SIZE);
+	return __seamcall(TDH_MEM_PAGE_ADD, gpa, tdr, hpa, source,

out); +}

arch/x86/kvm/vmx/tdx_ops.h new file +2 -0

+
+static inline u64 tdh_mem_sept_add(hpa_t tdr, gpa_t gpa, int level,

hpa_t page,

arch/x86/kvm/vmx/tdx_ops.h new file +4 -0

+				   struct tdx_module_output *out)
+{
+	clflush_cache_range(__va(page), PAGE_SIZE);
+	return __seamcall(TDH_MEM_SEPT_ADD, gpa | level, tdr, page,

0, out); +}

arch/x86/kvm/vmx/tdx_ops.h new file +2 -0

+
+static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int

level,

arch/x86/kvm/vmx/tdx_ops.h new file +3 -0

+				      struct tdx_module_output *out)
+{
+	return __seamcall(TDH_MEM_SEPT_REMOVE, gpa | level, tdr, 0,

0, out); +}

arch/x86/kvm/vmx/tdx_ops.h new file +8 -0

+
+static inline u64 tdh_vp_addcx(hpa_t tdvpr, hpa_t addr)
+{
+	clflush_cache_range(__va(addr), PAGE_SIZE);
+	return __seamcall(TDH_VP_ADDCX, addr, tdvpr, 0, 0, NULL);
+}
+
+static inline u64 tdh_mem_page_relocate(hpa_t tdr, gpa_t gpa, hpa_t

hpa,

arch/x86/kvm/vmx/tdx_ops.h new file +1 -0

+					struct tdx_module_output

*out) +{

arch/x86/kvm/vmx/tdx_ops.h new file +2 -0

+	clflush_cache_range(__va(hpa), PAGE_SIZE);
+	return __seamcall(TDH_MEM_PAGE_RELOCATE, gpa, tdr, hpa, 0,

out); +}

arch/x86/kvm/vmx/tdx_ops.h new file +9 -0

+
+static inline u64 tdh_mem_page_aug(hpa_t tdr, gpa_t gpa, hpa_t hpa,
+				   struct tdx_module_output *out)
+{
+	clflush_cache_range(__va(hpa), PAGE_SIZE);
+	return __seamcall(TDH_MEM_PAGE_AUG, gpa, tdr, hpa, 0, out);
+}
+
+static inline u64 tdh_mem_range_block(hpa_t tdr, gpa_t gpa, int

level,

arch/x86/kvm/vmx/tdx_ops.h new file +3 -0

+				      struct tdx_module_output *out)
+{
+	return __seamcall(TDH_MEM_RANGE_BLOCK, gpa | level, tdr, 0,

0, out); +}

arch/x86/kvm/vmx/tdx_ops.h new file +19 -0

+
+static inline u64 tdh_mng_key_config(hpa_t tdr)
+{
+	return __seamcall(TDH_MNG_KEY_CONFIG, tdr, 0, 0, 0, NULL);
+}
+
+static inline u64 tdh_mng_create(hpa_t tdr, int hkid)
+{
+	clflush_cache_range(__va(tdr), PAGE_SIZE);
+	return __seamcall(TDH_MNG_CREATE, tdr, hkid, 0, 0, NULL);
+}
+
+static inline u64 tdh_vp_create(hpa_t tdr, hpa_t tdvpr)
+{
+	clflush_cache_range(__va(tdvpr), PAGE_SIZE);
+	return __seamcall(TDH_VP_CREATE, tdvpr, tdr, 0, 0, NULL);
+}
+
+static inline u64 tdh_mng_rd(hpa_t tdr, u64 field, struct

tdx_module_output *out) +{

arch/x86/kvm/vmx/tdx_ops.h new file +53 -0

+	return __seamcall(TDH_MNG_RD, tdr, field, 0, 0, out);
+}
+
+static inline u64 tdh_mr_extend(hpa_t tdr, gpa_t gpa,
+				struct tdx_module_output *out)
+{
+	return __seamcall(TDH_MR_EXTEND, gpa, tdr, 0, 0, out);
+}
+
+static inline u64 tdh_mr_finalize(hpa_t tdr)
+{
+	return __seamcall(TDH_MR_FINALIZE, tdr, 0, 0, 0, NULL);
+}
+
+static inline u64 tdh_vp_flush(hpa_t tdvpr)
+{
+	return __seamcall(TDH_VP_FLUSH, tdvpr, 0, 0, 0, NULL);
+}
+
+static inline u64 tdh_mng_vpflushdone(hpa_t tdr)
+{
+	return __seamcall(TDH_MNG_VPFLUSHDONE, tdr, 0, 0, 0, NULL);
+}
+
+static inline u64 tdh_mng_key_freeid(hpa_t tdr)
+{
+	return __seamcall(TDH_MNG_KEY_FREEID, tdr, 0, 0, 0, NULL);
+}
+
+static inline u64 tdh_mng_init(hpa_t tdr, hpa_t td_params,
+			       struct tdx_module_output *out)
+{
+	return __seamcall(TDH_MNG_INIT, tdr, td_params, 0, 0, out);
+}
+
+static inline u64 tdh_vp_init(hpa_t tdvpr, u64 rcx)
+{
+	return __seamcall(TDH_VP_INIT, tdvpr, rcx, 0, 0, NULL);
+}
+
+static inline u64 tdh_vp_rd(hpa_t tdvpr, u64 field,
+			    struct tdx_module_output *out)
+{
+	return __seamcall(TDH_VP_RD, tdvpr, field, 0, 0, out);
+}
+
+static inline u64 tdh_mng_key_reclaimid(hpa_t tdr)
+{
+	return __seamcall(TDH_MNG_KEY_RECLAIMID, tdr, 0, 0, 0, NULL);
+}
+
+static inline u64 tdh_phymem_page_reclaim(hpa_t page,
+					  struct tdx_module_output

*out) +{

arch/x86/kvm/vmx/tdx_ops.h new file +1 -0

+	return __seamcall(TDH_PHYMEM_PAGE_RECLAIM, page, 0, 0, 0,

out); +}

arch/x86/kvm/vmx/tdx_ops.h new file +2 -0

+
+static inline u64 tdh_mem_page_remove(hpa_t tdr, gpa_t gpa, int

level,

arch/x86/kvm/vmx/tdx_ops.h new file +3 -0

+				      struct tdx_module_output *out)
+{
+	return __seamcall(TDH_MEM_PAGE_REMOVE, gpa | level, tdr, 0,

0, out); +}

arch/x86/kvm/vmx/tdx_ops.h new file +12 -0

+
+static inline u64 tdh_sys_lp_shutdown(void)
+{
+	return __seamcall(TDH_SYS_LP_SHUTDOWN, 0, 0, 0, 0, NULL);
+}
+
+static inline u64 tdh_mem_track(hpa_t tdr)
+{
+	return __seamcall(TDH_MEM_TRACK, tdr, 0, 0, 0, NULL);
+}
+
+static inline u64 tdh_mem_range_unblock(hpa_t tdr, gpa_t gpa, int

level,

arch/x86/kvm/vmx/tdx_ops.h new file +1 -0

+					struct tdx_module_output

*out) +{

arch/x86/kvm/vmx/tdx_ops.h new file +1 -0

+	return __seamcall(TDH_MEM_RANGE_UNBLOCK, gpa | level, tdr,

0, 0, out); +}

arch/x86/kvm/vmx/tdx_ops.h new file +4 -0

+
+static inline u64 tdh_phymem_cache_wb(bool resume)
+{
+	return __seamcall(TDH_PHYMEM_CACHE_WB, resume ? 1 : 0, 0, 0,

0, NULL); +}

arch/x86/kvm/vmx/tdx_ops.h new file +4 -0

+
+static inline u64 tdh_phymem_page_wbinvd(hpa_t page)
+{
+	return __seamcall(TDH_PHYMEM_PAGE_WBINVD, page, 0, 0, 0,

NULL); +}

arch/x86/kvm/vmx/tdx_ops.h new file +2 -0

+
+static inline u64 tdh_vp_wr(hpa_t tdvpr, u64 field, u64 val, u64

mask,

arch/x86/kvm/vmx/tdx_ops.h new file +7 -0

+			    struct tdx_module_output *out)
+{
+	return __seamcall(TDH_VP_WR, tdvpr, field, val, mask, out);
+}
+#endif /* CONFIG_INTEL_TDX_HOST */
+
+#endif /* __KVM_X86_TDX_OPS_H */

(quoted diff) +0 -0

b/arch/x86/virt/vmx/tdx/seamcall.S index f81be6b9c133..b90a7fe05494 100644 --- a/arch/x86/virt/vmx/tdx/seamcall.S
+++ b/arch/x86/virt/vmx/tdx/seamcall.S

arch/x86/virt/vmx/tdx/seamcall.S +2 -0

@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 #include <linux/linkage.h>
+#include <asm/export.h>
 #include <asm/frame.h>
 
 #include "tdxcall.S"

@@ -50,3 +51,4 @@ SYM_FUNC_START(__seamcall)
 	FRAME_END
 	RET
 SYM_FUNC_END(__seamcall)
+EXPORT_SYMBOL_GPL(__seamcall)

Isaku Yamahata

2022-12-15 19:45 UTC #23 lore raw

Re: [PATCH v10 011/108] KVM: TDX: Add C wrapper functions for SEAMCALLs to the TDX module

Quoted 16 lines · Zhi Wang

Quoted 9 lines · On Sat, 29 Oct 2022 23:22:12 -0700 [email protected] wrote:
From: Isaku Yamahata <[email protected]>
A VMM interacts with the TDX module using a new instruction (SEAMCALL). A TDX VMM uses SEAMCALLs where a VMX VMM would have directly interacted with VMX instructions. For instance, a TDX VMM does not have full access to the VM control structure corresponding to VMX VMCS. Instead, a VMM induces the TDX module to act on behalf via SEAMCALLs.
Quoted 3 lines
Export __seamcall and define C wrapper functions for SEAMCALLs for readability. Some SEAMCALL APIs donates pages to TDX module or guest TD. The pages are encrypted with TDX private host key id set in high

Quoted 6 lines

It seems a little bit confusing here.
I guess you are talking about the host pages donated to TDX module will be encrypted with TDX module global key id, which is used to encrypted the pages only used by TDX module. For pages donated to the TD guest, the pages are encrypted with TD guest prviate key id?

Basically yes. Only TDR is encrypted with the global key id.

Quoted 4 lines

It might be better that you can talk about the actual reaason instead of the key id stuff. For example Some SEMCALL APIs will zap or write the pages donated by VMM via MOVDIR64B, which requires the VMM to flush the cachelines.

How about the followings?

Some SEAMCALL APIs donates host pages to TDX module or guest TD and the donated pages are encrypted. Some of such SEAMCALLs flush cache lines (typically by movdir64b instruction), some don't. Those that doesn't clear cache lines require the VMM to flush the cache lines to avoid cache line alias.

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:24 UTC #24 lore raw

PATCH v10 12/108 KVM: TDX: Add helper functions to print TDX SEAMCALL error

From: Isaku Yamahata <[email protected]>

Add helper functions to print out errors from the TDX module in a uniform manner.

---
 arch/x86/kvm/Makefile        |  2 +-
 arch/x86/kvm/vmx/tdx_error.c | 21 +++++++++++++++++++++
 arch/x86/kvm/vmx/tdx_ops.h   |  3 +++
 3 files changed, 25 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/kvm/vmx/tdx_error.c

arch/x86/kvm/Makefile +1 -1

index e2c05195cb95..f1ad445df505 100644
@@ -24,7 +24,7 @@ kvm-$(CONFIG_KVM_XEN)	+= xen.o
 kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \
 			   vmx/evmcs.o vmx/nested.o vmx/posted_intr.o vmx/main.o
 kvm-intel-$(CONFIG_X86_SGX_KVM)	+= vmx/sgx.o
-kvm-intel-$(CONFIG_INTEL_TDX_HOST)	+= vmx/tdx.o
+kvm-intel-$(CONFIG_INTEL_TDX_HOST)	+= vmx/tdx.o vmx/tdx_error.o
 
 kvm-amd-y		+= svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o

arch/x86/kvm/vmx/tdx_error.c new file +21 -0

new file mode 100644
index 000000000000..574b72d34e1e
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0
+/* functions to record TDX SEAMCALL error */
+
+#include <linux/kernel.h>
+#include <linux/bug.h>
+
+#include "tdx_ops.h"
+
+void pr_tdx_error(u64 op, u64 error_code, const struct tdx_module_output *out)
+{
+	if (!out) {
+		pr_err_ratelimited("SEAMCALL[%lld] failed: 0x%llx\n",
+				   op, error_code);
+		return;
+	}
+
+	pr_err_ratelimited("SEAMCALL[%lld] failed: 0x%llx RCX 0x%llx, RDX 0x%llx,"
+			   " R8 0x%llx, R9 0x%llx, R10 0x%llx, R11 0x%llx\n",
+			   op, error_code,
+			   out->rcx, out->rdx, out->r8, out->r9, out->r10, out->r11);
+}

arch/x86/kvm/vmx/tdx_ops.h +3 -0

index 85adbf49c277..8cc2f01c509b 100644
@@ -9,12 +9,15 @@
 #include <asm/cacheflush.h>
 #include <asm/asm.h>
 #include <asm/kvm_host.h>
+#include <asm/tdx.h>
 
 #include "tdx_errno.h"
 #include "tdx_arch.h"
 
 #ifdef CONFIG_INTEL_TDX_HOST
 
+void pr_tdx_error(u64 op, u64 error_code, const struct tdx_module_output *out);
+
 static inline u64 tdh_mng_addcx(hpa_t tdr, hpa_t addr)
 {
 	clflush_cache_range(__va(addr), PAGE_SIZE);

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #25 lore raw

PATCH v10 13/108 The start of TDX KVM patch series: TD VM creation/destruction

From: Isaku Yamahata <[email protected]>

This empty commit is to mark the start of patch series of TD VM creation/destruction.

---
 Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2

index b7a14bc73853..5e0deaebf843 100644
@@ -15,8 +15,8 @@ Patch Layer status
 ------------------
   Patch layer                          Status
 * TDX, VMX coexistence:                 Applied
-* TDX architectural definitions:        Applying
-* TD VM creation/destruction:           Not yet
+* TDX architectural definitions:        Applied
+* TD VM creation/destruction:           Applying
 * TD vcpu creation/destruction:         Not yet
 * TDX EPT violation:                    Not yet
 * TD finalization:                      Not yet

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #26 lore raw

PATCH v10 14/108 KVM: TDX: Stub in tdx.h with structs, accessors, and VMCS helpers

From: Sean Christopherson <[email protected]>

Stub in kvm_tdx, vcpu_tdx, and their various accessors. TDX defines SEAMCALL APIs to access TDX control structures corresponding to the VMX VMCS. Introduce helper accessors to hide its SEAMCALL ABI details.

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/kvm/vmx/tdx.h | 118 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 116 insertions(+), 2 deletions(-)

arch/x86/kvm/vmx/tdx.h +116 -2

index 473013265bd8..98999bf3f188 100644
@@ -3,14 +3,27 @@
 #define __KVM_X86_TDX_H
 
 #ifdef CONFIG_INTEL_TDX_HOST
+
+#include "tdx_ops.h"
+
+struct tdx_td_page {
+	unsigned long va;
+	hpa_t pa;
+	bool added;
+};
+
 struct kvm_tdx {
 	struct kvm kvm;
-	/* TDX specific members follow. */
+
+	struct tdx_td_page tdr;
+	struct tdx_td_page *tdcs;
 };
 
 struct vcpu_tdx {
 	struct kvm_vcpu	vcpu;
-	/* TDX specific members follow. */
+
+	struct tdx_td_page tdvpr;
+	struct tdx_td_page *tdvpx;
 };
 
 static inline bool is_td(struct kvm *kvm)

@@ -32,6 +45,107 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu)
 {
 	return container_of(vcpu, struct vcpu_tdx, vcpu);
 }
+
+static __always_inline void tdvps_vmcs_check(u32 field, u8 bits)
+{
+#define VMCS_ENC_ACCESS_TYPE_MASK	0x1UL
+#define VMCS_ENC_ACCESS_TYPE_FULL	0x0UL
+#define VMCS_ENC_ACCESS_TYPE_HIGH	0x1UL
+#define VMCS_ENC_ACCESS_TYPE(field)	((field) & VMCS_ENC_ACCESS_TYPE_MASK)
+
+	/* TDX is 64bit only.  HIGH field isn't supported. */
+	BUILD_BUG_ON_MSG(__builtin_constant_p(field) &&
+			 VMCS_ENC_ACCESS_TYPE(field) == VMCS_ENC_ACCESS_TYPE_HIGH,
+			 "Read/Write to TD VMCS *_HIGH fields not supported");
+
+	BUILD_BUG_ON(bits != 16 && bits != 32 && bits != 64);
+
+#define VMCS_ENC_WIDTH_MASK	GENMASK(14, 13)
+#define VMCS_ENC_WIDTH_16BIT	(0UL << 13)
+#define VMCS_ENC_WIDTH_64BIT	(1UL << 13)
+#define VMCS_ENC_WIDTH_32BIT	(2UL << 13)
+#define VMCS_ENC_WIDTH_NATURAL	(3UL << 13)
+#define VMCS_ENC_WIDTH(field)	((field) & VMCS_ENC_WIDTH_MASK)
+
+	/* TDX is 64bit only.  i.e. natural width = 64bit. */
+	BUILD_BUG_ON_MSG(bits != 64 && __builtin_constant_p(field) &&
+			 (VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_64BIT ||
+			  VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_NATURAL),
+			 "Invalid TD VMCS access for 64-bit field");
+	BUILD_BUG_ON_MSG(bits != 32 && __builtin_constant_p(field) &&
+			 VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_32BIT,
+			 "Invalid TD VMCS access for 32-bit field");
+	BUILD_BUG_ON_MSG(bits != 16 && __builtin_constant_p(field) &&
+			 VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_16BIT,
+			 "Invalid TD VMCS access for 16-bit field");
+}
+
+static __always_inline void tdvps_state_non_arch_check(u64 field, u8 bits) {}
+static __always_inline void tdvps_management_check(u64 field, u8 bits) {}
+
+#define TDX_BUILD_TDVPS_ACCESSORS(bits, uclass, lclass)				\
+static __always_inline u##bits td_##lclass##_read##bits(struct vcpu_tdx *tdx,	\
+							u32 field)		\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_rd(tdx->tdvpr.pa, TDVPS_##uclass(field), &out);		\
+	if (unlikely(err)) {							\
+		pr_err("TDH_VP_RD["#uclass".0x%x] failed: 0x%llx\n",		\
+		       field, err);						\
+		return 0;							\
+	}									\
+	return (u##bits)out.r8;							\
+}										\
+static __always_inline void td_##lclass##_write##bits(struct vcpu_tdx *tdx,	\
+						      u32 field, u##bits val)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), val,		\
+		      GENMASK_ULL(bits - 1, 0), &out);				\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] = 0x%llx failed: 0x%llx\n",	\
+		       field, (u64)val, err);					\
+}										\
+static __always_inline void td_##lclass##_setbit##bits(struct vcpu_tdx *tdx,	\
+						       u32 field, u64 bit)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), bit, bit,		\
+			&out);							\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] |= 0x%llx failed: 0x%llx\n",	\
+		       field, bit, err);					\
+}										\
+static __always_inline void td_##lclass##_clearbit##bits(struct vcpu_tdx *tdx,	\
+							 u32 field, u64 bit)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), 0, bit,		\
+			&out);							\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] &= ~0x%llx failed: 0x%llx\n",	\
+		       field, bit,  err);					\
+}
+
+TDX_BUILD_TDVPS_ACCESSORS(16, VMCS, vmcs);
+TDX_BUILD_TDVPS_ACCESSORS(32, VMCS, vmcs);
+TDX_BUILD_TDVPS_ACCESSORS(64, VMCS, vmcs);
+
+TDX_BUILD_TDVPS_ACCESSORS(64, STATE_NON_ARCH, state_non_arch);
+TDX_BUILD_TDVPS_ACCESSORS(8, MANAGEMENT, management);
+
 #else
 struct kvm_tdx {
 	struct kvm kvm;

Signature · 1 line

2.25.1

Binbin Wu

2022-10-31 11:39 UTC #27 lore raw

Re: [PATCH v10 014/108] KVM: TDX: Stub in tdx.h with structs, accessors, and VMCS helpers

Quoted 133 lines · On 2022/10/30 14:22, [email protected] wrote:

From: Sean Christopherson <[email protected]>

Signed-off-by Sean Christopherson <[email protected]>

---
  arch/x86/kvm/vmx/tdx.h | 118 ++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 116 insertions(+), 2 deletions(-)

arch/x86/kvm/vmx/tdx.h +98 -2

index 473013265bd8..98999bf3f188 100644
@@ -3,14 +3,27 @@
  #define __KVM_X86_TDX_H
 
  #ifdef CONFIG_INTEL_TDX_HOST
+
+#include "tdx_ops.h"
+
+struct tdx_td_page {
+	unsigned long va;
+	hpa_t pa;
+	bool added;
+};
+
  struct kvm_tdx {
  	struct kvm kvm;
-	/* TDX specific members follow. */
+
+	struct tdx_td_page tdr;
+	struct tdx_td_page *tdcs;
  };
 
  struct vcpu_tdx {
  	struct kvm_vcpu	vcpu;
-	/* TDX specific members follow. */
+
+	struct tdx_td_page tdvpr;
+	struct tdx_td_page *tdvpx;
  };
 
  static inline bool is_td(struct kvm *kvm)

@@ -32,6 +45,107 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu)
  {
  	return container_of(vcpu, struct vcpu_tdx, vcpu);
  }
+
+static __always_inline void tdvps_vmcs_check(u32 field, u8 bits)
+{
+#define VMCS_ENC_ACCESS_TYPE_MASK	0x1UL
+#define VMCS_ENC_ACCESS_TYPE_FULL	0x0UL
+#define VMCS_ENC_ACCESS_TYPE_HIGH	0x1UL
+#define VMCS_ENC_ACCESS_TYPE(field)	((field) & VMCS_ENC_ACCESS_TYPE_MASK)
+
+	/* TDX is 64bit only.  HIGH field isn't supported. */
+	BUILD_BUG_ON_MSG(__builtin_constant_p(field) &&
+			 VMCS_ENC_ACCESS_TYPE(field) == VMCS_ENC_ACCESS_TYPE_HIGH,
+			 "Read/Write to TD VMCS *_HIGH fields not supported");
+
+	BUILD_BUG_ON(bits != 16 && bits != 32 && bits != 64);
+
+#define VMCS_ENC_WIDTH_MASK	GENMASK(14, 13)
+#define VMCS_ENC_WIDTH_16BIT	(0UL << 13)
+#define VMCS_ENC_WIDTH_64BIT	(1UL << 13)
+#define VMCS_ENC_WIDTH_32BIT	(2UL << 13)
+#define VMCS_ENC_WIDTH_NATURAL	(3UL << 13)
+#define VMCS_ENC_WIDTH(field)	((field) & VMCS_ENC_WIDTH_MASK)
+
+	/* TDX is 64bit only.  i.e. natural width = 64bit. */
+	BUILD_BUG_ON_MSG(bits != 64 && __builtin_constant_p(field) &&
+			 (VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_64BIT ||
+			  VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_NATURAL),
+			 "Invalid TD VMCS access for 64-bit field");
+	BUILD_BUG_ON_MSG(bits != 32 && __builtin_constant_p(field) &&
+			 VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_32BIT,
+			 "Invalid TD VMCS access for 32-bit field");
+	BUILD_BUG_ON_MSG(bits != 16 && __builtin_constant_p(field) &&
+			 VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_16BIT,
+			 "Invalid TD VMCS access for 16-bit field");
+}
+
+static __always_inline void tdvps_state_non_arch_check(u64 field, u8 bits) {}
+static __always_inline void tdvps_management_check(u64 field, u8 bits) {}
+
+#define TDX_BUILD_TDVPS_ACCESSORS(bits, uclass, lclass)				\
+static __always_inline u##bits td_##lclass##_read##bits(struct vcpu_tdx *tdx,	\
+							u32 field)		\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_rd(tdx->tdvpr.pa, TDVPS_##uclass(field), &out);		\
+	if (unlikely(err)) {							\
+		pr_err("TDH_VP_RD["#uclass".0x%x] failed: 0x%llx\n",		\
+		       field, err);						\
+		return 0;							\
+	}									\
+	return (u##bits)out.r8;							\
+}										\
+static __always_inline void td_##lclass##_write##bits(struct vcpu_tdx *tdx,	\
+						      u32 field, u##bits val)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), val,		\
+		      GENMASK_ULL(bits - 1, 0), &out);				\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] = 0x%llx failed: 0x%llx\n",	\
+		       field, (u64)val, err);					\
+}										\
+static __always_inline void td_##lclass##_setbit##bits(struct vcpu_tdx *tdx,	\
+						       u32 field, u64 bit)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), bit, bit,		\
+			&out);							\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] |= 0x%llx failed: 0x%llx\n",	\
+		       field, bit, err);					\
+}										\
+static __always_inline void td_##lclass##_clearbit##bits(struct vcpu_tdx *tdx,	\
+							 u32 field, u64 bit)	\
+{										\

arch/x86/kvm/vmx/tdx.h static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) · line 140

Quoted 10 lines

arch/x86/kvm/vmx/tdx.h +10 -0

+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), 0, bit,		\
+			&out);							\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] &= ~0x%llx failed: 0x%llx\n",	\
+		       field, bit,  err);					\
+}

For the set of accessors, although there will be kernel errer message when tdh_vp_{rd,wr} fails, the caller doesn't know these function calls succeed or not. Won't this cause any unexpected behavior?

Quoted 11 lines

arch/x86/kvm/vmx/tdx.h +8 -0

+
+TDX_BUILD_TDVPS_ACCESSORS(16, VMCS, vmcs);
+TDX_BUILD_TDVPS_ACCESSORS(32, VMCS, vmcs);
+TDX_BUILD_TDVPS_ACCESSORS(64, VMCS, vmcs);
+
+TDX_BUILD_TDVPS_ACCESSORS(64, STATE_NON_ARCH, state_non_arch);
+TDX_BUILD_TDVPS_ACCESSORS(8, MANAGEMENT, management);
+
  #else
  struct kvm_tdx {
  	struct kvm kvm;

Isaku Yamahata

2022-11-03 00:57 UTC #28 lore raw

Re: [PATCH v10 014/108] KVM: TDX: Stub in tdx.h with structs, accessors, and VMCS helpers

Quoted 140 lines · Binbin Wu

Quoted 138 lines · On 2022/10/30 14:22, [email protected] wrote:

From: Sean Christopherson <[email protected]>

Signed-off-by Sean Christopherson <[email protected]>

---
  arch/x86/kvm/vmx/tdx.h | 118 ++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 116 insertions(+), 2 deletions(-)

arch/x86/kvm/vmx/tdx.h +106 -2

index 473013265bd8..98999bf3f188 100644
@@ -3,14 +3,27 @@
  #define __KVM_X86_TDX_H
  #ifdef CONFIG_INTEL_TDX_HOST
+
+#include "tdx_ops.h"
+
+struct tdx_td_page {
+	unsigned long va;
+	hpa_t pa;
+	bool added;
+};
+
  struct kvm_tdx {
  	struct kvm kvm;
-	/* TDX specific members follow. */
+
+	struct tdx_td_page tdr;
+	struct tdx_td_page *tdcs;
  };
  struct vcpu_tdx {
  	struct kvm_vcpu	vcpu;
-	/* TDX specific members follow. */
+
+	struct tdx_td_page tdvpr;
+	struct tdx_td_page *tdvpx;
  };
  static inline bool is_td(struct kvm *kvm)

@@ -32,6 +45,107 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu)
  {
  	return container_of(vcpu, struct vcpu_tdx, vcpu);
  }
+
+static __always_inline void tdvps_vmcs_check(u32 field, u8 bits)
+{
+#define VMCS_ENC_ACCESS_TYPE_MASK	0x1UL
+#define VMCS_ENC_ACCESS_TYPE_FULL	0x0UL
+#define VMCS_ENC_ACCESS_TYPE_HIGH	0x1UL
+#define VMCS_ENC_ACCESS_TYPE(field)	((field) & VMCS_ENC_ACCESS_TYPE_MASK)
+
+	/* TDX is 64bit only.  HIGH field isn't supported. */
+	BUILD_BUG_ON_MSG(__builtin_constant_p(field) &&
+			 VMCS_ENC_ACCESS_TYPE(field) == VMCS_ENC_ACCESS_TYPE_HIGH,
+			 "Read/Write to TD VMCS *_HIGH fields not supported");
+
+	BUILD_BUG_ON(bits != 16 && bits != 32 && bits != 64);
+
+#define VMCS_ENC_WIDTH_MASK	GENMASK(14, 13)
+#define VMCS_ENC_WIDTH_16BIT	(0UL << 13)
+#define VMCS_ENC_WIDTH_64BIT	(1UL << 13)
+#define VMCS_ENC_WIDTH_32BIT	(2UL << 13)
+#define VMCS_ENC_WIDTH_NATURAL	(3UL << 13)
+#define VMCS_ENC_WIDTH(field)	((field) & VMCS_ENC_WIDTH_MASK)
+
+	/* TDX is 64bit only.  i.e. natural width = 64bit. */
+	BUILD_BUG_ON_MSG(bits != 64 && __builtin_constant_p(field) &&
+			 (VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_64BIT ||
+			  VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_NATURAL),
+			 "Invalid TD VMCS access for 64-bit field");
+	BUILD_BUG_ON_MSG(bits != 32 && __builtin_constant_p(field) &&
+			 VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_32BIT,
+			 "Invalid TD VMCS access for 32-bit field");
+	BUILD_BUG_ON_MSG(bits != 16 && __builtin_constant_p(field) &&
+			 VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_16BIT,
+			 "Invalid TD VMCS access for 16-bit field");
+}
+
+static __always_inline void tdvps_state_non_arch_check(u64 field, u8 bits) {}
+static __always_inline void tdvps_management_check(u64 field, u8 bits) {}
+
+#define TDX_BUILD_TDVPS_ACCESSORS(bits, uclass, lclass)				\
+static __always_inline u##bits td_##lclass##_read##bits(struct vcpu_tdx *tdx,	\
+							u32 field)		\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_rd(tdx->tdvpr.pa, TDVPS_##uclass(field), &out);		\
+	if (unlikely(err)) {							\
+		pr_err("TDH_VP_RD["#uclass".0x%x] failed: 0x%llx\n",		\
+		       field, err);						\
+		return 0;							\
+	}									\
+	return (u##bits)out.r8;							\
+}										\
+static __always_inline void td_##lclass##_write##bits(struct vcpu_tdx *tdx,	\
+						      u32 field, u##bits val)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), val,		\
+		      GENMASK_ULL(bits - 1, 0), &out);				\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] = 0x%llx failed: 0x%llx\n",	\
+		       field, (u64)val, err);					\
+}										\
+static __always_inline void td_##lclass##_setbit##bits(struct vcpu_tdx *tdx,	\
+						       u32 field, u64 bit)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), bit, bit,		\
+			&out);							\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] |= 0x%llx failed: 0x%llx\n",	\
+		       field, bit, err);					\
+}										\
+static __always_inline void td_##lclass##_clearbit##bits(struct vcpu_tdx *tdx,	\
+							 u32 field, u64 bit)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), 0, bit,		\
+			&out);							\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] &= ~0x%llx failed: 0x%llx\n",	\

Quoted 8 lines

arch/x86/kvm/vmx/tdx.h static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) · line 140
Quoted 2 lines
arch/x86/kvm/vmx/tdx.h +2 -0
+		       field, bit,  err);					\
+}
For the set of accessors, although there will be kernel errer message when tdh_vp_{rd,wr} fails,
the caller doesn't know these function calls succeed or not. Won't this cause any unexpected
behavior?

Oh right. Basically tdh_vp_{rd, wr} should fail as long as parameters are correct. So I'll add KVM_BUG_ON(). TDX module takes shared lock on TDVPR, TDR and TDCS. It should fails with TDX_OPERAND_BUSY on run time. (At least with TDX module 1.0)

Thanks,

Signature · 1 line

Isaku Yamahata <[email protected]>

Huang, Kai

2022-11-10 11:11 UTC #29 lore raw

Re: [PATCH v10 014/108] KVM: TDX: Stub in tdx.h with structs, accessors, and VMCS helpers

Quoted 34 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Sean Christopherson <[email protected]>
Stub in kvm_tdx, vcpu_tdx, and their various accessors. TDX defines SEAMCALL APIs to access TDX control structures corresponding to the VMX VMCS. Introduce helper accessors to hide its SEAMCALL ABI details.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
---
 arch/x86/kvm/vmx/tdx.h | 118 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 116 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/tdx.h +11 -1
index 473013265bd8..98999bf3f188 100644
@@ -3,14 +3,27 @@
 #define __KVM_X86_TDX_H
 
 #ifdef CONFIG_INTEL_TDX_HOST
+
+#include "tdx_ops.h"
+
+struct tdx_td_page {
+	unsigned long va;
+	hpa_t pa;
+	bool added;
+};
+
 struct kvm_tdx {
 	struct kvm kvm;
-	/* TDX specific members follow. */
+
+	struct tdx_td_page tdr;

arch/x86/kvm/vmx/tdx.h line 27

Quoted 10 lines

arch/x86/kvm/vmx/tdx.h +4 -1

+	struct tdx_td_page *tdcs;
 };
 
 struct vcpu_tdx {
 	struct kvm_vcpu	vcpu;
-	/* TDX specific members follow. */
+
+	struct tdx_td_page tdvpr;
+	struct tdx_td_page *tdvpx;
 };

As replied to the patch "KVM: TDX: create/destroy VM structure", I think it's better to introduce this part together with that patch to make reviewer easier.

If I am not seeing mistakenly, the "accessors" below are not used in that patch, so they can be done in a later patch when needed.

Quoted 108 lines

arch/x86/kvm/vmx/tdx.h +101 -0

 static inline bool is_td(struct kvm *kvm)

@@ -32,6 +45,107 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu)
 {
 	return container_of(vcpu, struct vcpu_tdx, vcpu);
 }
+
+static __always_inline void tdvps_vmcs_check(u32 field, u8 bits)
+{
+#define VMCS_ENC_ACCESS_TYPE_MASK	0x1UL
+#define VMCS_ENC_ACCESS_TYPE_FULL	0x0UL
+#define VMCS_ENC_ACCESS_TYPE_HIGH	0x1UL
+#define VMCS_ENC_ACCESS_TYPE(field)	((field) & VMCS_ENC_ACCESS_TYPE_MASK)
+
+	/* TDX is 64bit only.  HIGH field isn't supported. */
+	BUILD_BUG_ON_MSG(__builtin_constant_p(field) &&
+			 VMCS_ENC_ACCESS_TYPE(field) == VMCS_ENC_ACCESS_TYPE_HIGH,
+			 "Read/Write to TD VMCS *_HIGH fields not supported");
+
+	BUILD_BUG_ON(bits != 16 && bits != 32 && bits != 64);
+
+#define VMCS_ENC_WIDTH_MASK	GENMASK(14, 13)
+#define VMCS_ENC_WIDTH_16BIT	(0UL << 13)
+#define VMCS_ENC_WIDTH_64BIT	(1UL << 13)
+#define VMCS_ENC_WIDTH_32BIT	(2UL << 13)
+#define VMCS_ENC_WIDTH_NATURAL	(3UL << 13)
+#define VMCS_ENC_WIDTH(field)	((field) & VMCS_ENC_WIDTH_MASK)
+
+	/* TDX is 64bit only.  i.e. natural width = 64bit. */
+	BUILD_BUG_ON_MSG(bits != 64 && __builtin_constant_p(field) &&
+			 (VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_64BIT ||
+			  VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_NATURAL),
+			 "Invalid TD VMCS access for 64-bit field");
+	BUILD_BUG_ON_MSG(bits != 32 && __builtin_constant_p(field) &&
+			 VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_32BIT,
+			 "Invalid TD VMCS access for 32-bit field");
+	BUILD_BUG_ON_MSG(bits != 16 && __builtin_constant_p(field) &&
+			 VMCS_ENC_WIDTH(field) == VMCS_ENC_WIDTH_16BIT,
+			 "Invalid TD VMCS access for 16-bit field");
+}
+
+static __always_inline void tdvps_state_non_arch_check(u64 field, u8 bits) {}
+static __always_inline void tdvps_management_check(u64 field, u8 bits) {}
+
+#define TDX_BUILD_TDVPS_ACCESSORS(bits, uclass, lclass)				\
+static __always_inline u##bits td_##lclass##_read##bits(struct vcpu_tdx *tdx,	\
+							u32 field)		\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_rd(tdx->tdvpr.pa, TDVPS_##uclass(field), &out);		\
+	if (unlikely(err)) {							\
+		pr_err("TDH_VP_RD["#uclass".0x%x] failed: 0x%llx\n",		\
+		       field, err);						\
+		return 0;							\
+	}									\
+	return (u##bits)out.r8;							\
+}										\
+static __always_inline void td_##lclass##_write##bits(struct vcpu_tdx *tdx,	\
+						      u32 field, u##bits val)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), val,		\
+		      GENMASK_ULL(bits - 1, 0), &out);				\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] = 0x%llx failed: 0x%llx\n",	\
+		       field, (u64)val, err);					\
+}										\
+static __always_inline void td_##lclass##_setbit##bits(struct vcpu_tdx *tdx,	\
+						       u32 field, u64 bit)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), bit, bit,		\
+			&out);							\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] |= 0x%llx failed: 0x%llx\n",	\
+		       field, bit, err);					\
+}										\
+static __always_inline void td_##lclass##_clearbit##bits(struct vcpu_tdx *tdx,	\
+							 u32 field, u64 bit)	\
+{										\
+	struct tdx_module_output out;						\
+	u64 err;								\
+										\
+	tdvps_##lclass##_check(field, bits);					\
+	err = tdh_vp_wr(tdx->tdvpr.pa, TDVPS_##uclass(field), 0, bit,		\
+			&out);							\
+	if (unlikely(err))							\
+		pr_err("TDH_VP_WR["#uclass".0x%x] &= ~0x%llx failed: 0x%llx\n",	\
+		       field, bit,  err);					\
+}
+
+TDX_BUILD_TDVPS_ACCESSORS(16, VMCS, vmcs);
+TDX_BUILD_TDVPS_ACCESSORS(32, VMCS, vmcs);
+TDX_BUILD_TDVPS_ACCESSORS(64, VMCS, vmcs);
+
+TDX_BUILD_TDVPS_ACCESSORS(64, STATE_NON_ARCH, state_non_arch);
+TDX_BUILD_TDVPS_ACCESSORS(8, MANAGEMENT, management);
+

[email protected]

author

2022-10-30 06:25 UTC #30 lore raw

PATCH v10 15/108 x86/cpu: Add helper functions to allocate/free TDX private host key id

From: Isaku Yamahata <[email protected]>

TDX private host key id is assigned to guest TD. The memory controller encrypts guest TD memory with the assigned TDX private host key id (HIKD). Add helper functions to allocate/free TDX private host key id so that TDX KVM manage it.

Also export the global TDX private host key id that is used to encrypt TDX module, its memory and some dynamic data (TDR). When VMM releasing encrypted page to reuse it, the page needs to be flushed with the used host key id. VMM needs the global TDX private host key id to flush such pages TDX module accesses with the global TDX private host key id.

---
 arch/x86/include/asm/tdx.h  | 13 +++++++++++++
 arch/x86/virt/vmx/tdx/tdx.c | 28 +++++++++++++++++++++++++++-
 2 files changed, 40 insertions(+), 1 deletion(-)

arch/x86/include/asm/tdx.h +13 -0

index ba2e4c69fb9f..cd304d323d33 100644
@@ -186,6 +186,17 @@ struct tdsysinfo_struct {
 const struct tdsysinfo_struct *tdx_get_sysinfo(void);
 bool platform_tdx_enabled(void);
 int tdx_enable(void);
+
+/*
+ * Key id globally used by TDX module: TDX module maps TDR with this TDX global
+ * key id.  TDR includes key id assigned to the TD.  Then TDX module maps other
+ * TD-related pages with the assigned key id.  TDR requires this TDX global key
+ * id for cache flush unlike other TD-related pages.
+ */
+extern u32 tdx_global_keyid __read_mostly;
+int tdx_keyid_alloc(void);
+void tdx_keyid_free(int keyid);
+
 u64 __seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9,
 	       struct tdx_module_output *out);
 #else	/* !CONFIG_INTEL_TDX_HOST */

@@ -193,6 +204,8 @@ struct tdsysinfo_struct;
 static inline const struct tdsysinfo_struct *tdx_get_sysinfo(void) { return NULL; }
 static inline bool platform_tdx_enabled(void) { return false; }
 static inline int tdx_enable(void)  { return -ENODEV; }
+static inline int tdx_keyid_alloc(void) { return -EOPNOTSUPP; }
+static inline void tdx_keyid_free(int keyid) { }
 #endif	/* CONFIG_INTEL_TDX_HOST */
 
 #endif /* !__ASSEMBLY__ */

arch/x86/virt/vmx/tdx/tdx.c +27 -1

index 6fb630fa7d09..0625ced219d7 100644
@@ -64,7 +64,8 @@ static struct cmr_info tdx_cmr_array[MAX_CMRS] __aligned(CMR_INFO_ARRAY_ALIGNMEN
 static int tdx_cmr_num;
 
 /* TDX module global KeyID.  Used in TDH.SYS.CONFIG ABI. */
-static u32 tdx_global_keyid;
+u32 tdx_global_keyid __read_mostly;
+EXPORT_SYMBOL_GPL(tdx_global_keyid);
 
 /*
  * Detect TDX private KeyIDs to see whether TDX has been enabled by the

@@ -113,6 +114,31 @@ static void __init clear_tdx(void)
 	tdx_keyid_start = tdx_keyid_num = 0;
 }
 
+/* TDX KeyID pool */
+static DEFINE_IDA(tdx_keyid_pool);
+
+int tdx_keyid_alloc(void)
+{
+	if (WARN_ON_ONCE(!tdx_keyid_start || !tdx_keyid_num))
+		return -EINVAL;
+
+	/* The first keyID is reserved for the global key. */
+	return ida_alloc_range(&tdx_keyid_pool, tdx_keyid_start + 1,
+			       tdx_keyid_start + tdx_keyid_num - 1,
+			       GFP_KERNEL);
+}
+EXPORT_SYMBOL_GPL(tdx_keyid_alloc);
+
+void tdx_keyid_free(int keyid)
+{
+	/* keyid = 0 is reserved. */
+	if (!keyid || keyid <= 0)
+		return;
+
+	ida_free(&tdx_keyid_pool, keyid);
+}
+EXPORT_SYMBOL_GPL(tdx_keyid_free);
+
 static void __init tdx_memory_destroy(void)
 {
 	while (!list_empty(&tdx_memlist)) {

Signature · 1 line

2.25.1

Huang, Kai

2022-11-08 09:17 UTC #31 lore raw

Re: [PATCH v10 015/108] x86/cpu: Add helper functions to allocate/free TDX private host key id

Quoted 4 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>
TDX private host key id is assigned to guest TD. The memory controller encrypts guest TD memory with the assigned TDX private host key id (HIKD).

								     ^
								     HKID.

And since you already mentioned in the first sentence, you can put (HKID) part there. And I think you can just use HKID in the rest places to save some typing as this is the purpose of using (HKID) I suppose.

[...]

Quoted 11 lines

arch/x86/virt/vmx/tdx/tdx.c +10 -0

+/* TDX KeyID pool */
+static DEFINE_IDA(tdx_keyid_pool);
+
+int tdx_keyid_alloc(void)
+{
+	if (WARN_ON_ONCE(!tdx_keyid_start || !tdx_keyid_num))
+		return -EINVAL;
+
+	/* The first keyID is reserved for the global key. */
+	return ida_alloc_range(&tdx_keyid_pool, tdx_keyid_start + 1,

arch/x86/virt/vmx/tdx/tdx.c static void __init clear_tdx(void) · line 136

Quoted 10 lines

arch/x86/virt/vmx/tdx/tdx.c +10 -0

+			       tdx_keyid_start + tdx_keyid_num - 1,
+			       GFP_KERNEL);
+}
+EXPORT_SYMBOL_GPL(tdx_keyid_alloc);
+
+void tdx_keyid_free(int keyid)
+{
+	/* keyid = 0 is reserved. */
+	if (!keyid || keyid <= 0)
+		return;

Double check of keyid == 0.

I think you can just use:

	if (keyid <= tdx_keyid_start)
		return;

And/or add a WARN() as it's a bug if above happens.

Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c +5 -0

+
+	ida_free(&tdx_keyid_pool, keyid);
+}
+EXPORT_SYMBOL_GPL(tdx_keyid_free);
+
 static void __init tdx_memory_destroy(void)
 {
 	while (!list_empty(&tdx_memlist)) {

Isaku Yamahata

2022-11-17 17:35 UTC #32 lore raw

Re: [PATCH v10 015/108] x86/cpu: Add helper functions to allocate/free TDX private host key id

Quoted 38 lines · Kai

Quoted 4 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
TDX private host key id is assigned to guest TD. The memory controller encrypts guest TD memory with the assigned TDX private host key id (HIKD).
								     ^
								     HKID.
And since you already mentioned in the first sentence, you can put (HKID) part there. And I think you can just use HKID in the rest places to save some typing as this is the purpose of using (HKID) I suppose.
[...]
Quoted 11 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+/* TDX KeyID pool */
+static DEFINE_IDA(tdx_keyid_pool);
+
+int tdx_keyid_alloc(void)
+{
+	if (WARN_ON_ONCE(!tdx_keyid_start || !tdx_keyid_num))
+		return -EINVAL;
+
+	/* The first keyID is reserved for the global key. */
+	return ida_alloc_range(&tdx_keyid_pool, tdx_keyid_start + 1,
arch/x86/virt/vmx/tdx/tdx.c static void __init clear_tdx(void) · line 136
Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+			       tdx_keyid_start + tdx_keyid_num - 1,
+			       GFP_KERNEL);
+}
+EXPORT_SYMBOL_GPL(tdx_keyid_alloc);
+
+void tdx_keyid_free(int keyid)
+{
+	/* keyid = 0 is reserved. */
+	if (!keyid || keyid <= 0)
+		return;
Double check of keyid == 0.

Quoted 6 lines

I think you can just use:
	if (keyid <= tdx_keyid_start)
		return;
And/or add a WARN() as it's a bug if above happens.

Ok. Will fix it with WARN().

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:24 UTC #33 lore raw

PATCH v10 16/108 KVM: TDX: create/destroy VM structure

From: Sean Christopherson <[email protected]>

As the first step to create TDX guest, create/destroy VM struct. Assign TDX private Host Key ID (HKID) to the TDX guest for memory encryption and allocate extra pages for the TDX guest. On destruction, free allocated pages, and HKID.

Before tearing down private page tables, TDX requires some resources of the guest TD to be destroyed (i.e. keyID must have been reclaimed, etc). Add flush_shadow_all_private callback before tearing down private page tables for it.

Add a second kvm_x86_ops hook in kvm_arch_destroy_vm() to support TDX's destruction path, which needs to first put the VM into a teardown state, then free per-vCPU resources, and finally free per-VM resources.

Co-developed-by Kai Huang <[email protected]>

Signed-off-by Kai Huang <[email protected]>

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/kvm-x86-ops.h |   2 +
 arch/x86/include/asm/kvm_host.h    |   2 +
 arch/x86/kvm/vmx/main.c            |  34 ++-
 arch/x86/kvm/vmx/tdx.c             | 411 +++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx.h             |   2 +
 arch/x86/kvm/vmx/x86_ops.h         |  11 +
 arch/x86/kvm/x86.c                 |   8 +
 7 files changed, 467 insertions(+), 3 deletions(-)

arch/x86/include/asm/kvm-x86-ops.h +2 -0

index 8a5c5ae70bc5..3a29a6b31ee8 100644
@@ -21,7 +21,9 @@ KVM_X86_OP(has_emulated_msr)
 KVM_X86_OP(vcpu_after_set_cpuid)
 KVM_X86_OP(is_vm_type_supported)
 KVM_X86_OP(vm_init)
+KVM_X86_OP_OPTIONAL(flush_shadow_all_private)
 KVM_X86_OP_OPTIONAL(vm_destroy)
+KVM_X86_OP_OPTIONAL(vm_free)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate)
 KVM_X86_OP(vcpu_create)
 KVM_X86_OP(vcpu_free)

arch/x86/include/asm/kvm_host.h +2 -0

index 2a41a93a80f3..2870155ce6fb 100644
@@ -1472,7 +1472,9 @@ struct kvm_x86_ops {
 	bool (*is_vm_type_supported)(unsigned long vm_type);
 	unsigned int vm_size;
 	int (*vm_init)(struct kvm *kvm);
+	void (*flush_shadow_all_private)(struct kvm *kvm);
 	void (*vm_destroy)(struct kvm *kvm);
+	void (*vm_free)(struct kvm *kvm);
 
 	/* Create, but do not attach this VCPU */
 	int (*vcpu_precreate)(struct kvm *kvm);

arch/x86/kvm/vmx/main.c +31 -3

index 0900ff2f2390..d01a946a18cf 100644
@@ -29,18 +29,44 @@ static __init int vt_hardware_setup(void)
 	return 0;
 }
 
+static void vt_hardware_unsetup(void)
+{
+	tdx_hardware_unsetup();
+	vmx_hardware_unsetup();
+}
+
 static int vt_vm_init(struct kvm *kvm)
 {
 	if (is_td(kvm))
-		return -EOPNOTSUPP;	/* Not ready to create guest TD yet. */
+		return tdx_vm_init(kvm);
 
 	return vmx_vm_init(kvm);
 }
 
+static void vt_flush_shadow_all_private(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return tdx_mmu_release_hkid(kvm);
+}
+
+static void vt_vm_destroy(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return;
+
+	vmx_vm_destroy(kvm);
+}
+
+static void vt_vm_free(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return tdx_vm_free(kvm);
+}
+
 struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = "kvm_intel",
 
-	.hardware_unsetup = vmx_hardware_unsetup,
+	.hardware_unsetup = vt_hardware_unsetup,
 	.check_processor_compatibility = vmx_check_processor_compatibility,
 
 	.hardware_enable = vmx_hardware_enable,

@@ -50,7 +76,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.is_vm_type_supported = vt_is_vm_type_supported,
 	.vm_size = sizeof(struct kvm_vmx),
 	.vm_init = vt_vm_init,
-	.vm_destroy = vmx_vm_destroy,
+	.flush_shadow_all_private = vt_flush_shadow_all_private,
+	.vm_destroy = vt_vm_destroy,
+	.vm_free = vt_vm_free,
 
 	.vcpu_precreate = vmx_vcpu_precreate,
 	.vcpu_create = vmx_vcpu_create,

arch/x86/kvm/vmx/tdx.c +411 -0

432 lines not on this page · raw message

arch/x86/kvm/vmx/tdx.h +2 -0

index 98999bf3f188..938314635b47 100644
@@ -17,6 +17,8 @@ struct kvm_tdx {
 
 	struct tdx_td_page tdr;
 	struct tdx_td_page *tdcs;
+
+	int hkid;
 };
 
 struct vcpu_tdx {

arch/x86/kvm/vmx/x86_ops.h +11 -0

index ac1688b0b0e3..95da978c9aa9 100644
@@ -133,9 +133,20 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_INTEL_TDX_HOST
 int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);
 bool tdx_is_vm_type_supported(unsigned long type);
+void tdx_hardware_unsetup(void);
+
+int tdx_vm_init(struct kvm *kvm);
+void tdx_mmu_release_hkid(struct kvm *kvm);
+void tdx_vm_free(struct kvm *kvm);
 #else
 static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }
 static inline bool tdx_is_vm_type_supported(unsigned long type) { return false; }
+static inline void tdx_hardware_unsetup(void) {}
+
+static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; }
+static inline void tdx_mmu_release_hkid(struct kvm *kvm) {}
+static inline void tdx_flush_shadow_all_private(struct kvm *kvm) {}
+static inline void tdx_vm_free(struct kvm *kvm) {}
 #endif
 
 #endif /* __KVM_X86_VMX_X86_OPS_H */

arch/x86/kvm/x86.c +8 -0

index 91053fdc4512..4b22196cb12c 100644
@@ -12702,6 +12702,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 	kvm_page_track_cleanup(kvm);
 	kvm_xen_destroy_vm(kvm);
 	kvm_hv_destroy_vm(kvm);
+	static_call_cond(kvm_x86_vm_free)(kvm);
 }
 
 static void memslot_rmap_free(struct kvm_memory_slot *slot)

@@ -13012,6 +13013,13 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 
 void kvm_arch_flush_shadow_all(struct kvm *kvm)
 {
+	/*
+	 * kvm_mmu_zap_all() zaps both private and shared page tables.  Before
+	 * tearing down private page tables, TDX requires some TD resources to
+	 * be destroyed (i.e. keyID must have been reclaimed, etc).  Invoke
+	 * kvm_x86_flush_shadow_all_private() for this.
+	 */
+	static_call_cond(kvm_x86_flush_shadow_all_private)(kvm);
 	kvm_mmu_zap_all(kvm);
 }

Signature · 1 line

2.25.1

Huang, Kai

2022-11-10 11:04 UTC #34 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

Quoted 9 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

Quoted 9 lines
From: Sean Christopherson <[email protected]>
As the first step to create TDX guest, create/destroy VM struct. Assign TDX private Host Key ID (HKID) to the TDX guest for memory encryption and allocate extra pages for the TDX guest. On destruction, free allocated pages, and HKID.
Before tearing down private page tables, TDX requires some resources of the guest TD to be destroyed (i.e. keyID must have been reclaimed, etc). Add

				 ^
				 HKID

It's better to use consistent words throughout patch/series I guess.

Quoted 6 lines

Quoted 6 lines
flush_shadow_all_private callback before tearing down private page tables for it.
Add a second kvm_x86_ops hook in kvm_arch_destroy_vm() to support TDX's destruction path, which needs to first put the VM into a teardown state, then free per-vCPU resources, and finally free per-VM resources.

Perhaps explicitly call out the hook is vm_free() and why the existing vm_destroy() hook cannot meet TDX's purpose, so that people can understand easily why you need vm_free().

[...]

Quoted 4 lines

arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 59
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4 -0
+static inline void tdx_hkid_free(struct kvm_tdx *kvm_tdx)
+{
+	tdx_keyid_free(kvm_tdx->hkid);
+	kvm_tdx->hkid = -1;

Why -1? Can it be set to 0, which is the initial value when kvm_tdx is allocated anyway?

Quoted 12 lines

arch/x86/kvm/vmx/tdx.c +12 -0

+}
+
+static inline bool is_hkid_assigned(struct kvm_tdx *kvm_tdx)
+{
+	return kvm_tdx->hkid > 0;
+}
+
+static void tdx_clear_page(unsigned long page)
+{
+	const void *zero_page = (const void *) __va(page_to_phys(ZERO_PAGE(0)));
+	unsigned long i;
+

Quoted 8 lines

arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 79

Quoted 8 lines

arch/x86/kvm/vmx/tdx.c +8 -0

+	/*
+	 * Zeroing the page is only necessary for systems with MKTME-i:
+	 * when re-assign one page from old keyid to a new keyid, MOVDIR64B is
+	 * required to clear/write the page with new keyid to prevent integrity
+	 * error when read on the page with new keyid.
+	 *
+	 * The cache line could be poisoned (even without MKTME-i), clear the
+	 * poison bit.

Does this happen only when there's potential kernel bug?

Quoted 73 lines

arch/x86/kvm/vmx/tdx.c +73 -0

+	 */
+	for (i = 0; i < PAGE_SIZE; i += 64)
+		movdir64b((void *)(page + i), zero_page);
+	/*
+	 * MOVDIR64B store uses WC buffer.  Prevent following memory reads
+	 * from seeing potentially poisoned cache.
+	 */
+	__mb();
+}
+
+static int tdx_reclaim_page(unsigned long va, hpa_t pa, bool do_wb, u16 hkid)
+{
+	struct tdx_module_output out;
+	u64 err;
+
+	do {
+		err = tdh_phymem_page_reclaim(pa, &out);
+		/*
+		 * TDH.PHYMEM.PAGE.RECLAIM is allowed only when TD is shutdown.
+		 * state.  i.e. destructing TD.
+		 * TDH.PHYMEM.PAGE.RECLAIM  requires TDR and target page.
+		 * Because we're destructing TD, it's rare to contend with TDR.
+		 */
+	} while (err == (TDX_OPERAND_BUSY | TDX_OPERAND_ID_RCX));
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_PHYMEM_PAGE_RECLAIM, err, &out);
+		return -EIO;
+	}
+
+	if (do_wb) {
+		/*
+		 * Only TDR page gets into this path.  No contention is expected
+		 * because the last page of TD.
+		 */
+		err = tdh_phymem_page_wbinvd(set_hkid_to_hpa(pa, hkid));
+		if (WARN_ON_ONCE(err)) {
+			pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err, NULL);
+			return -EIO;
+		}
+	}
+
+	tdx_clear_page(va);
+	return 0;
+}
+
+static int tdx_alloc_td_page(struct tdx_td_page *page)
+{
+	page->va = __get_free_page(GFP_KERNEL_ACCOUNT);
+	if (!page->va)
+		return -ENOMEM;
+
+	page->pa = __pa(page->va);
+	return 0;
+}
+
+static inline void tdx_mark_td_page_added(struct tdx_td_page *page)
+{
+	WARN_ON_ONCE(page->added);
+	page->added = true;
+}
+
+static void tdx_reclaim_td_page(struct tdx_td_page *page)
+{
+	if (page->added) {
+		/*
+		 * TDCX are being reclaimed.  TDX module maps TDCX with HKID
+		 * assigned to the TD.  Here the cache associated to the TD
+		 * was already flushed by TDH.PHYMEM.CACHE.WB before here, So
+		 * cache doesn't need to be flushed again.
+		 */
+		if (tdx_reclaim_page(page->va, page->pa, false, 0))
+			return;
+

Quoted 8 lines

arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 160
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8 -0
+		page->added = false;
+	}
+	if (page->va) {
+		free_page(page->va);
+		page->va = 0;
+	}
+}
+

I am wondering why this 'struct tdx_td_page' is needed?

It appears the page->pa is used by SEAMCALLs and page->va is used by tdx_clear_page() as MOVDIR64B needs a virtual address.

But since GFP_KERNEL_ACCOUNT is used in memory allocation, so you can actually just get the pa and va from the page easily (using page_to_phys() and __va()). Also it's 64-bit kernel so you don't even need to consider HIGHMEM.

Also, it seems page->added can be replaced with simply checking whether page is NULL, correct?

Btw, I think the introduce of 'struct tdx_td_page' and the new 'struct tdx_td_page tdr' to 'struct kvm_tdx' should come together with this patch, but not in the previous patch "KVM: TDX: Stub in tdx.h with structs, accessors, and VMCS helpers". This makes the code review easier.

The "accessors" can be introduced in later patch when they are needed -- it doesn't seem any of them is used in this patch?

Isaku Yamahata

2022-11-17 17:45 UTC #35 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

Quoted 10 lines · Kai

Quoted 6 lines
Quoted 6 lines
flush_shadow_all_private callback before tearing down private page tables for it.
Add a second kvm_x86_ops hook in kvm_arch_destroy_vm() to support TDX's destruction path, which needs to first put the VM into a teardown state, then free per-vCPU resources, and finally free per-VM resources.
Perhaps explicitly call out the hook is vm_free() and why the existing vm_destroy() hook cannot meet TDX's purpose, so that people can understand easily why you need vm_free().

Sure, will update the commit message.

Quoted 7 lines

Quoted 4 lines
arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 59
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4 -0
+static inline void tdx_hkid_free(struct kvm_tdx *kvm_tdx)
+{
+	tdx_keyid_free(kvm_tdx->hkid);
+	kvm_tdx->hkid = -1;
Why -1? Can it be set to 0, which is the initial value when kvm_tdx is allocated anyway?

0 works. I'll replace it with 0.

Quoted 14 lines

arch/x86/kvm/vmx/tdx.c +14 -0

+}
+
+static inline bool is_hkid_assigned(struct kvm_tdx *kvm_tdx)
+{
+	return kvm_tdx->hkid > 0;
+}
+
+static void tdx_clear_page(unsigned long page)
+{
+	const void *zero_page = (const void *) __va(page_to_phys(ZERO_PAGE(0)));
+	unsigned long i;
+
+	/*
+	 * Zeroing the page is only necessary for systems with MKTME-i:

Quoted 8 lines

Quoted 6 lines
arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 79
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +6 -0
+	 * when re-assign one page from old keyid to a new keyid, MOVDIR64B is
+	 * required to clear/write the page with new keyid to prevent integrity
+	 * error when read on the page with new keyid.
+	 *
+	 * The cache line could be poisoned (even without MKTME-i), clear the
+	 * poison bit.
Does this happen only when there's potential kernel bug?

That's right.

Quoted 40 lines

Quoted 28 lines

arch/x86/kvm/vmx/tdx.c +28 -0

+static int tdx_alloc_td_page(struct tdx_td_page *page)
+{
+	page->va = __get_free_page(GFP_KERNEL_ACCOUNT);
+	if (!page->va)
+		return -ENOMEM;
+
+	page->pa = __pa(page->va);
+	return 0;
+}
+
+static inline void tdx_mark_td_page_added(struct tdx_td_page *page)
+{
+	WARN_ON_ONCE(page->added);
+	page->added = true;
+}
+
+static void tdx_reclaim_td_page(struct tdx_td_page *page)
+{
+	if (page->added) {
+		/*
+		 * TDCX are being reclaimed.  TDX module maps TDCX with HKID
+		 * assigned to the TD.  Here the cache associated to the TD
+		 * was already flushed by TDH.PHYMEM.CACHE.WB before here, So
+		 * cache doesn't need to be flushed again.
+		 */
+		if (tdx_reclaim_page(page->va, page->pa, false, 0))
+			return;
+

Quoted 8 lines

arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 160
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8 -0
+		page->added = false;
+	}
+	if (page->va) {
+		free_page(page->va);
+		page->va = 0;
+	}
+}
+

I am wondering why this 'struct tdx_td_page' is needed?

Quoted 6 lines

It appears the page->pa is used by SEAMCALLs and page->va is used by tdx_clear_page() as MOVDIR64B needs a virtual address.
But since GFP_KERNEL_ACCOUNT is used in memory allocation, so you can actually just get the pa and va from the page easily (using page_to_phys() and __va()). Also it's 64-bit kernel so you don't even need to consider HIGHMEM.

Yes, page->va member can be dropped. Will drop it.

Quoted 2 lines

Also, it seems page->added can be replaced with simply checking whether page is NULL, correct?

No. It's subtle. Anyway let me check it.

Quoted 7 lines

Btw, I think the introduce of 'struct tdx_td_page' and the new 'struct tdx_td_page tdr' to 'struct kvm_tdx' should come together with this patch, but not in the previous patch "KVM: TDX: Stub in tdx.h with structs, accessors, and VMCS helpers". This makes the code review easier.
The "accessors" can be introduced in later patch when they are needed -- it doesn't seem any of them is used in this patch?

Ok, will move it to that patch.

Signature · 1 line

Isaku Yamahata <[email protected]>

Sagi Shahar

2022-11-15 00:06 UTC #36 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

Quoted 374 lines · On Sat, Oct 29, 2022 at 11:24 PM <[email protected]> wrote:

From: Sean Christopherson <[email protected]>

Co-developed-by Kai Huang <[email protected]>

Signed-off-by Kai Huang <[email protected]>

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/kvm-x86-ops.h |   2 +
 arch/x86/include/asm/kvm_host.h    |   2 +
 arch/x86/kvm/vmx/main.c            |  34 ++-
 arch/x86/kvm/vmx/tdx.c             | 411 +++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx.h             |   2 +
 arch/x86/kvm/vmx/x86_ops.h         |  11 +
 arch/x86/kvm/x86.c                 |   8 +
 7 files changed, 467 insertions(+), 3 deletions(-)

arch/x86/include/asm/kvm-x86-ops.h +2 -0

index 8a5c5ae70bc5..3a29a6b31ee8 100644
@@ -21,7 +21,9 @@ KVM_X86_OP(has_emulated_msr)
 KVM_X86_OP(vcpu_after_set_cpuid)
 KVM_X86_OP(is_vm_type_supported)
 KVM_X86_OP(vm_init)
+KVM_X86_OP_OPTIONAL(flush_shadow_all_private)
 KVM_X86_OP_OPTIONAL(vm_destroy)
+KVM_X86_OP_OPTIONAL(vm_free)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate)
 KVM_X86_OP(vcpu_create)
 KVM_X86_OP(vcpu_free)

arch/x86/include/asm/kvm_host.h +2 -0

index 2a41a93a80f3..2870155ce6fb 100644
@@ -1472,7 +1472,9 @@ struct kvm_x86_ops {
        bool (*is_vm_type_supported)(unsigned long vm_type);
        unsigned int vm_size;
        int (*vm_init)(struct kvm *kvm);
+       void (*flush_shadow_all_private)(struct kvm *kvm);
        void (*vm_destroy)(struct kvm *kvm);
+       void (*vm_free)(struct kvm *kvm);
 
        /* Create, but do not attach this VCPU */
        int (*vcpu_precreate)(struct kvm *kvm);

arch/x86/kvm/vmx/main.c +31 -3

index 0900ff2f2390..d01a946a18cf 100644
@@ -29,18 +29,44 @@ static __init int vt_hardware_setup(void)
        return 0;
 }
 
+static void vt_hardware_unsetup(void)
+{
+       tdx_hardware_unsetup();
+       vmx_hardware_unsetup();
+}
+
 static int vt_vm_init(struct kvm *kvm)
 {
        if (is_td(kvm))
-               return -EOPNOTSUPP;     /* Not ready to create guest TD yet. */
+               return tdx_vm_init(kvm);
 
        return vmx_vm_init(kvm);
 }
 
+static void vt_flush_shadow_all_private(struct kvm *kvm)
+{
+       if (is_td(kvm))
+               return tdx_mmu_release_hkid(kvm);
+}
+
+static void vt_vm_destroy(struct kvm *kvm)
+{
+       if (is_td(kvm))
+               return;
+
+       vmx_vm_destroy(kvm);
+}
+
+static void vt_vm_free(struct kvm *kvm)
+{
+       if (is_td(kvm))
+               return tdx_vm_free(kvm);
+}
+
 struct kvm_x86_ops vt_x86_ops __initdata = {
        .name = "kvm_intel",
 
-       .hardware_unsetup = vmx_hardware_unsetup,
+       .hardware_unsetup = vt_hardware_unsetup,
        .check_processor_compatibility = vmx_check_processor_compatibility,
 
        .hardware_enable = vmx_hardware_enable,

@@ -50,7 +76,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
        .is_vm_type_supported = vt_is_vm_type_supported,
        .vm_size = sizeof(struct kvm_vmx),
        .vm_init = vt_vm_init,
-       .vm_destroy = vmx_vm_destroy,
+       .flush_shadow_all_private = vt_flush_shadow_all_private,
+       .vm_destroy = vt_vm_destroy,
+       .vm_free = vt_vm_free,
 
        .vcpu_precreate = vmx_vcpu_precreate,
        .vcpu_create = vmx_vcpu_create,

arch/x86/kvm/vmx/tdx.c +245 -0

index 530e72f85762..ec88dde0d300 100644
@@ -32,6 +32,401 @@ struct tdx_capabilities {
 /* Capabilities of KVM + the TDX module. */
 static struct tdx_capabilities tdx_caps;
 
+/*
+ * Some TDX SEAMCALLs (TDH.MNG.CREATE, TDH.PHYMEM.CACHE.WB,
+ * TDH.MNG.KEY.RECLAIMID, TDH.MNG.KEY.FREEID etc) tries to acquire a global lock
+ * internally in TDX module.  If failed, TDX_OPERAND_BUSY is returned without
+ * spinning or waiting due to a constraint on execution time.  It's caller's
+ * responsibility to avoid race (or retry on TDX_OPERAND_BUSY).  Use this mutex
+ * to avoid race in TDX module because the kernel knows better about scheduling.
+ */
+static DEFINE_MUTEX(tdx_lock);
+static struct mutex *tdx_mng_key_config_lock;
+
+static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid)
+{
+       return pa | ((hpa_t)hkid << boot_cpu_data.x86_phys_bits);
+}
+
+static inline bool is_td_created(struct kvm_tdx *kvm_tdx)
+{
+       return kvm_tdx->tdr.added;
+}
+
+static inline void tdx_hkid_free(struct kvm_tdx *kvm_tdx)
+{
+       tdx_keyid_free(kvm_tdx->hkid);
+       kvm_tdx->hkid = -1;
+}
+
+static inline bool is_hkid_assigned(struct kvm_tdx *kvm_tdx)
+{
+       return kvm_tdx->hkid > 0;
+}
+
+static void tdx_clear_page(unsigned long page)
+{
+       const void *zero_page = (const void *) __va(page_to_phys(ZERO_PAGE(0)));
+       unsigned long i;
+
+       /*
+        * Zeroing the page is only necessary for systems with MKTME-i:
+        * when re-assign one page from old keyid to a new keyid, MOVDIR64B is
+        * required to clear/write the page with new keyid to prevent integrity
+        * error when read on the page with new keyid.
+        *
+        * The cache line could be poisoned (even without MKTME-i), clear the
+        * poison bit.
+        */
+       for (i = 0; i < PAGE_SIZE; i += 64)
+               movdir64b((void *)(page + i), zero_page);
+       /*
+        * MOVDIR64B store uses WC buffer.  Prevent following memory reads
+        * from seeing potentially poisoned cache.
+        */
+       __mb();
+}
+
+static int tdx_reclaim_page(unsigned long va, hpa_t pa, bool do_wb, u16 hkid)
+{
+       struct tdx_module_output out;
+       u64 err;
+
+       do {
+               err = tdh_phymem_page_reclaim(pa, &out);
+               /*
+                * TDH.PHYMEM.PAGE.RECLAIM is allowed only when TD is shutdown.
+                * state.  i.e. destructing TD.
+                * TDH.PHYMEM.PAGE.RECLAIM  requires TDR and target page.
+                * Because we're destructing TD, it's rare to contend with TDR.
+                */
+       } while (err == (TDX_OPERAND_BUSY | TDX_OPERAND_ID_RCX));
+       if (WARN_ON_ONCE(err)) {
+               pr_tdx_error(TDH_PHYMEM_PAGE_RECLAIM, err, &out);
+               return -EIO;
+       }
+
+       if (do_wb) {
+               /*
+                * Only TDR page gets into this path.  No contention is expected
+                * because the last page of TD.
+                */
+               err = tdh_phymem_page_wbinvd(set_hkid_to_hpa(pa, hkid));
+               if (WARN_ON_ONCE(err)) {
+                       pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err, NULL);
+                       return -EIO;
+               }
+       }
+
+       tdx_clear_page(va);
+       return 0;
+}
+
+static int tdx_alloc_td_page(struct tdx_td_page *page)
+{
+       page->va = __get_free_page(GFP_KERNEL_ACCOUNT);
+       if (!page->va)
+               return -ENOMEM;
+
+       page->pa = __pa(page->va);
+       return 0;
+}
+
+static inline void tdx_mark_td_page_added(struct tdx_td_page *page)
+{
+       WARN_ON_ONCE(page->added);
+       page->added = true;
+}
+
+static void tdx_reclaim_td_page(struct tdx_td_page *page)
+{
+       if (page->added) {
+               /*
+                * TDCX are being reclaimed.  TDX module maps TDCX with HKID
+                * assigned to the TD.  Here the cache associated to the TD
+                * was already flushed by TDH.PHYMEM.CACHE.WB before here, So
+                * cache doesn't need to be flushed again.
+                */
+               if (tdx_reclaim_page(page->va, page->pa, false, 0))
+                       return;
+
+               page->added = false;
+       }
+       if (page->va) {
+               free_page(page->va);
+               page->va = 0;
+       }
+}
+
+static int tdx_do_tdh_phymem_cache_wb(void *param)
+{
+       u64 err = 0;
+
+       do {
+               err = tdh_phymem_cache_wb(!!err);
+       } while (err == TDX_INTERRUPTED_RESUMABLE);
+
+       /* Other thread may have done for us. */
+       if (err == TDX_NO_HKID_READY_TO_WBCACHE)
+               err = TDX_SUCCESS;
+       if (WARN_ON_ONCE(err)) {
+               pr_tdx_error(TDH_PHYMEM_CACHE_WB, err, NULL);
+               return -EIO;
+       }
+
+       return 0;
+}
+
+void tdx_mmu_release_hkid(struct kvm *kvm)
+{
+       struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       cpumask_var_t packages;
+       bool cpumask_allocated;
+       u64 err;
+       int ret;
+       int i;
+
+       if (!is_hkid_assigned(kvm_tdx))
+               return;
+
+       if (!is_td_created(kvm_tdx))
+               goto free_hkid;
+
+       cpumask_allocated = zalloc_cpumask_var(&packages, GFP_KERNEL);
+       cpus_read_lock();
+       for_each_online_cpu(i) {
+               if (cpumask_allocated &&
+                       cpumask_test_and_set_cpu(topology_physical_package_id(i),
+                                               packages))
+                       continue;
+
+               /*
+                * We can destroy multiple the guest TDs simultaneously.
+                * Prevent tdh_phymem_cache_wb from returning TDX_BUSY by
+                * serialization.
+                */
+               mutex_lock(&tdx_lock);
+               ret = smp_call_on_cpu(i, tdx_do_tdh_phymem_cache_wb, NULL, 1);
+               mutex_unlock(&tdx_lock);
+               if (ret)
+                       break;
+       }
+       cpus_read_unlock();
+       free_cpumask_var(packages);
+
+       mutex_lock(&tdx_lock);
+       err = tdh_mng_key_freeid(kvm_tdx->tdr.pa);
+       mutex_unlock(&tdx_lock);
+       if (WARN_ON_ONCE(err)) {
+               pr_tdx_error(TDH_MNG_KEY_FREEID, err, NULL);
+               pr_err("tdh_mng_key_freeid failed. HKID %d is leaked.\n",
+                       kvm_tdx->hkid);
+               return;
+       }
+
+free_hkid:
+       tdx_hkid_free(kvm_tdx);
+}
+
+void tdx_vm_free(struct kvm *kvm)
+{
+       struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       int i;
+
+       /* Can't reclaim or free TD pages if teardown failed. */
+       if (is_hkid_assigned(kvm_tdx))
+               return;
+
+       if (kvm_tdx->tdcs) {
+               for (i = 0; i < tdx_caps.tdcs_nr_pages; i++)
+                       tdx_reclaim_td_page(&kvm_tdx->tdcs[i]);
+               kfree(kvm_tdx->tdcs);
+       }
+
+       /*
+        * TDX module maps TDR with TDX global HKID.  TDX module may access TDR
+        * while operating on TD (Especially reclaiming TDCS).  Cache flush with
+        * TDX global HKID is needed.
+        */
+       if (kvm_tdx->tdr.added &&
+               tdx_reclaim_page(kvm_tdx->tdr.va, kvm_tdx->tdr.pa, true,
+                               tdx_global_keyid))
+               return;
+
+       free_page(kvm_tdx->tdr.va);
+}
+
+static int tdx_do_tdh_mng_key_config(void *param)
+{
+       hpa_t *tdr_p = param;
+       u64 err;
+
+       do {
+               err = tdh_mng_key_config(*tdr_p);
+
+               /*
+                * If it failed to generate a random key, retry it because this
+                * is typically caused by an entropy error of the CPU's random
+                * number generator.
+                */
+       } while (err == TDX_KEY_GENERATION_FAILED);
+
+       if (WARN_ON_ONCE(err)) {
+               pr_tdx_error(TDH_MNG_KEY_CONFIG, err, NULL);
+               return -EIO;
+       }
+
+       return 0;

arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 289

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c +10 -0

+}
+
+int tdx_vm_init(struct kvm *kvm)
+{
+       struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       cpumask_var_t packages;
+       int ret, i;
+       u64 err;
+
+       ret = tdx_keyid_alloc();

Can we skip the hkid allocation at such an early stage? This makes intra-host migration more complicated as the hkid of the destination VM is already allocated before we have a chance to migrate the state from the source VM.

I remember you had an internal version that already did that in
https://github.com/intel/tdx/blob/552dd80c48f67ca01bcdd10667e0c11efd375177/arch/x86/kvm/vmx/tdx.c#L508

Quoted 244 lines

arch/x86/kvm/vmx/tdx.c +156 -0

+       if (ret < 0)
+               return ret;
+       kvm_tdx->hkid = ret;
+
+       ret = tdx_alloc_td_page(&kvm_tdx->tdr);
+       if (ret)
+               goto free_hkid;
+
+       kvm_tdx->tdcs = kcalloc(tdx_caps.tdcs_nr_pages, sizeof(*kvm_tdx->tdcs),
+                               GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+       if (!kvm_tdx->tdcs)
+               goto free_tdr;
+       for (i = 0; i < tdx_caps.tdcs_nr_pages; i++) {
+               ret = tdx_alloc_td_page(&kvm_tdx->tdcs[i]);
+               if (ret)
+                       goto free_tdcs;
+       }
+
+       if (!zalloc_cpumask_var(&packages, GFP_KERNEL)) {
+               ret = -ENOMEM;
+               goto free_tdcs;
+       }
+       cpus_read_lock();
+       /*
+        * Need at least one CPU of the package to be online in order to
+        * program all packages for host key id.  Check it.
+        */
+       for_each_present_cpu(i)
+               cpumask_set_cpu(topology_physical_package_id(i), packages);
+       for_each_online_cpu(i)
+               cpumask_clear_cpu(topology_physical_package_id(i), packages);
+       if (!cpumask_empty(packages)) {
+               ret = -EIO;
+               /*
+                * Because it's hard for human operator to figure out the
+                * reason, warn it.
+                */
+               pr_warn("All packages need to have online CPU to create TD. Online CPU and retry.\n");
+               goto free_packages;
+       }
+
+       /*
+        * Acquire global lock to avoid TDX_OPERAND_BUSY:
+        * TDH.MNG.CREATE and other APIs try to lock the global Key Owner
+        * Table (KOT) to track the assigned TDX private HKID.  It doesn't spin
+        * to acquire the lock, returns TDX_OPERAND_BUSY instead, and let the
+        * caller to handle the contention.  This is because of time limitation
+        * usable inside the TDX module and OS/VMM knows better about process
+        * scheduling.
+        *
+        * APIs to acquire the lock of KOT:
+        * TDH.MNG.CREATE, TDH.MNG.KEY.FREEID, TDH.MNG.VPFLUSHDONE, and
+        * TDH.PHYMEM.CACHE.WB.
+        */
+       mutex_lock(&tdx_lock);
+       err = tdh_mng_create(kvm_tdx->tdr.pa, kvm_tdx->hkid);
+       mutex_unlock(&tdx_lock);
+       if (WARN_ON_ONCE(err)) {
+               pr_tdx_error(TDH_MNG_CREATE, err, NULL);
+               ret = -EIO;
+               goto free_packages;
+       }
+       tdx_mark_td_page_added(&kvm_tdx->tdr);
+
+       for_each_online_cpu(i) {
+               int pkg = topology_physical_package_id(i);
+
+               if (cpumask_test_and_set_cpu(pkg, packages))
+                       continue;
+
+               /*
+                * Program the memory controller in the package with an
+                * encryption key associated to a TDX private host key id
+                * assigned to this TDR.  Concurrent operations on same memory
+                * controller results in TDX_OPERAND_BUSY.  Avoid this race by
+                * mutex.
+                */
+               mutex_lock(&tdx_mng_key_config_lock[pkg]);
+               ret = smp_call_on_cpu(i, tdx_do_tdh_mng_key_config,
+                                     &kvm_tdx->tdr.pa, true);
+               mutex_unlock(&tdx_mng_key_config_lock[pkg]);
+               if (ret)
+                       break;
+       }
+       cpus_read_unlock();
+       free_cpumask_var(packages);
+       if (ret)
+               goto teardown;
+
+       for (i = 0; i < tdx_caps.tdcs_nr_pages; i++) {
+               err = tdh_mng_addcx(kvm_tdx->tdr.pa, kvm_tdx->tdcs[i].pa);
+               if (WARN_ON_ONCE(err)) {
+                       pr_tdx_error(TDH_MNG_ADDCX, err, NULL);
+                       ret = -EIO;
+                       goto teardown;
+               }
+               tdx_mark_td_page_added(&kvm_tdx->tdcs[i]);
+       }
+
+       /*
+        * Note, TDH_MNG_INIT cannot be invoked here.  TDH_MNG_INIT requires a dedicated
+        * ioctl() to define the configure CPUID values for the TD.
+        */
+       return 0;
+
+       /*
+        * The sequence for freeing resources from a partially initialized TD
+        * varies based on where in the initialization flow failure occurred.
+        * Simply use the full teardown and destroy, which naturally play nice
+        * with partial initialization.
+        */
+teardown:
+       tdx_mmu_release_hkid(kvm);
+       tdx_vm_free(kvm);
+       return ret;
+
+free_packages:
+       cpus_read_unlock();
+       free_cpumask_var(packages);
+free_tdcs:
+       for (i = 0; i < tdx_caps.tdcs_nr_pages; i++) {
+               if (!kvm_tdx->tdcs[i].va)
+                       continue;
+               free_page(kvm_tdx->tdcs[i].va);
+       }
+       kfree(kvm_tdx->tdcs);
+       kvm_tdx->tdcs = NULL;
+free_tdr:
+       if (kvm_tdx->tdr.va) {
+               free_page(kvm_tdx->tdr.va);
+               kvm_tdx->tdr.added = false;
+               kvm_tdx->tdr.va = 0;
+               kvm_tdx->tdr.pa = 0;
+       }
+free_hkid:
+       if (kvm_tdx->hkid != -1)
+               tdx_hkid_free(kvm_tdx);
+       return ret;
+}
+
 static int __init tdx_module_setup(void)
 {
        const struct tdsysinfo_struct *tdsysinfo;

@@ -82,6 +477,8 @@ bool tdx_is_vm_type_supported(unsigned long type)
 
 int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
 {
+       int max_pkgs;
+       int i;
        int r;
 
        if (!enable_ept) {

@@ -95,6 +492,14 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
                return -ENODEV;
        }
 
+       max_pkgs = topology_max_packages();
+       tdx_mng_key_config_lock = kcalloc(max_pkgs, sizeof(*tdx_mng_key_config_lock),
+                                  GFP_KERNEL);
+       if (!tdx_mng_key_config_lock)
+               return -ENOMEM;
+       for (i = 0; i < max_pkgs; i++)
+               mutex_init(&tdx_mng_key_config_lock[i]);
+
        /* TDX requires VMX. */
        r = vmxon_all();
        if (!r)

@@ -103,3 +508,9 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
 
        return r;
 }
+
+void tdx_hardware_unsetup(void)
+{
+       /* kfree accepts NULL. */
+       kfree(tdx_mng_key_config_lock);
+}

arch/x86/kvm/vmx/tdx.h +2 -0

index 98999bf3f188..938314635b47 100644
@@ -17,6 +17,8 @@ struct kvm_tdx {
 
        struct tdx_td_page tdr;
        struct tdx_td_page *tdcs;
+
+       int hkid;
 };
 
 struct vcpu_tdx {

arch/x86/kvm/vmx/x86_ops.h +11 -0

index ac1688b0b0e3..95da978c9aa9 100644
@@ -133,9 +133,20 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_INTEL_TDX_HOST
 int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);
 bool tdx_is_vm_type_supported(unsigned long type);
+void tdx_hardware_unsetup(void);
+
+int tdx_vm_init(struct kvm *kvm);
+void tdx_mmu_release_hkid(struct kvm *kvm);
+void tdx_vm_free(struct kvm *kvm);
 #else
 static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }
 static inline bool tdx_is_vm_type_supported(unsigned long type) { return false; }
+static inline void tdx_hardware_unsetup(void) {}
+
+static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; }
+static inline void tdx_mmu_release_hkid(struct kvm *kvm) {}
+static inline void tdx_flush_shadow_all_private(struct kvm *kvm) {}
+static inline void tdx_vm_free(struct kvm *kvm) {}
 #endif
 
 #endif /* __KVM_X86_VMX_X86_OPS_H */

arch/x86/kvm/x86.c +8 -1

index 91053fdc4512..4b22196cb12c 100644
@@ -12702,6 +12702,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
        kvm_page_track_cleanup(kvm);
        kvm_xen_destroy_vm(kvm);
        kvm_hv_destroy_vm(kvm);
+       static_call_cond(kvm_x86_vm_free)(kvm);
 }
 
 static void memslot_rmap_free(struct kvm_memory_slot *slot)

@@ -13012,6 +13013,13 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 
 void kvm_arch_flush_shadow_all(struct kvm *kvm)
 {
+       /*
+        * kvm_mmu_zap_all() zaps both private and shared page tables.  Before
+        * tearing down private page tables, TDX requires some TD resources to
+        * be destroyed (i.e. keyID must have been reclaimed, etc).  Invoke
+        * kvm_x86_flush_shadow_all_private() for this.
+        */
+       static_call_cond(kvm_x86_flush_shadow_all_private)(kvm);
        kvm_mmu_zap_all(kvm);
 }
 
--

2.25.1

Isaku Yamahata

2022-11-17 17:48 UTC #37 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

Quoted 14 lines · Sagi Shahar

arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 289
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8 -0
+int tdx_vm_init(struct kvm *kvm)
+{
+       struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       cpumask_var_t packages;
+       int ret, i;
+       u64 err;
+
+       ret = tdx_keyid_alloc();
Can we skip the hkid allocation at such an early stage? This makes intra-host migration more complicated as the hkid of the destination VM is already allocated before we have a chance to migrate the state from the source VM.

Quoted 2 lines

I remember you had an internal version that already did that in
https://github.com/intel/tdx/blob/552dd80c48f67ca01bcdd10667e0c11efd375177/arch/x86/kvm/vmx/tdx.c#L508

With the patch of "KVM: TDX: initialize VM with TDX specific parameters", the allocation code is split out into its own code with KVM_TDX_INIT_VM. I think that's what you want.

Signature · 1 line

Isaku Yamahata <[email protected]>

Sagi Shahar

2022-11-17 23:46 UTC #38 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

Quoted 18 lines · Isaku Yamahata

Quoted 14 lines · Sagi Shahar
arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 289
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8 -0
+int tdx_vm_init(struct kvm *kvm)
+{
+       struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       cpumask_var_t packages;
+       int ret, i;
+       u64 err;
+
+       ret = tdx_keyid_alloc();
Can we skip the hkid allocation at such an early stage? This makes intra-host migration more complicated as the hkid of the destination VM is already allocated before we have a chance to migrate the state from the source VM.

Quoted 6 lines

Quoted 2 lines
I remember you had an internal version that already did that in
https://github.com/intel/tdx/blob/552dd80c48f67ca01bcdd10667e0c11efd375177/arch/x86/kvm/vmx/tdx.c#L508
With the patch of "KVM: TDX: initialize VM with TDX specific parameters", the allocation code is split out into its own code with KVM_TDX_INIT_VM. I think that's what you want.

Right, I missed that. Thanks.

Quoted 2 lines

--
Isaku Yamahata <[email protected]>

Zhi Wang

2022-11-23 10:52 UTC #39 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

Quoted 184 lines · On Sat, 29 Oct 2022 23:22:17 -0700 [email protected] wrote:

From: Sean Christopherson <[email protected]>
As the first step to create TDX guest, create/destroy VM struct. Assign TDX private Host Key ID (HKID) to the TDX guest for memory encryption and allocate extra pages for the TDX guest. On destruction, free allocated pages, and HKID.
Before tearing down private page tables, TDX requires some resources of the guest TD to be destroyed (i.e. keyID must have been reclaimed, etc). Add flush_shadow_all_private callback before tearing down private page tables for it.
Add a second kvm_x86_ops hook in kvm_arch_destroy_vm() to support TDX's destruction path, which needs to first put the VM into a teardown state, then free per-vCPU resources, and finally free per-VM resources.
Co-developed-by Kai Huang <[email protected]>
Signed-off-by Kai Huang <[email protected]>
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
---
 arch/x86/include/asm/kvm-x86-ops.h |   2 +
 arch/x86/include/asm/kvm_host.h    |   2 +
 arch/x86/kvm/vmx/main.c            |  34 ++-
 arch/x86/kvm/vmx/tdx.c             | 411
+++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h             |
2 + arch/x86/kvm/vmx/x86_ops.h         |  11 +
 arch/x86/kvm/x86.c                 |   8 +
 7 files changed, 467 insertions(+), 3 deletions(-)
(quoted diff) +0 -0
b/arch/x86/include/asm/kvm-x86-ops.h index 8a5c5ae70bc5..3a29a6b31ee8 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
arch/x86/include/asm/kvm-x86-ops.h +2 -0
@@ -21,7 +21,9 @@ KVM_X86_OP(has_emulated_msr)
 KVM_X86_OP(vcpu_after_set_cpuid)
 KVM_X86_OP(is_vm_type_supported)
 KVM_X86_OP(vm_init)
+KVM_X86_OP_OPTIONAL(flush_shadow_all_private)
 KVM_X86_OP_OPTIONAL(vm_destroy)
+KVM_X86_OP_OPTIONAL(vm_free)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate)
 KVM_X86_OP(vcpu_create)
 KVM_X86_OP(vcpu_free)
(quoted diff) +0 -0
b/arch/x86/include/asm/kvm_host.h index 2a41a93a80f3..2870155ce6fb 100644 --- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1472,7 +1472,9 @@ struct kvm_x86_ops {
 	bool (*is_vm_type_supported)(unsigned long vm_type);
 	unsigned int vm_size;
 	int (*vm_init)(struct kvm *kvm);
+	void (*flush_shadow_all_private)(struct kvm *kvm);
 	void (*vm_destroy)(struct kvm *kvm);
+	void (*vm_free)(struct kvm *kvm);
 
 	/* Create, but do not attach this VCPU */
 	int (*vcpu_precreate)(struct kvm *kvm);
arch/x86/kvm/vmx/main.c +6 -1
index 0900ff2f2390..d01a946a18cf 100644
@@ -29,18 +29,44 @@ static __init int vt_hardware_setup(void)
 	return 0;
 }
 
+static void vt_hardware_unsetup(void)
+{
+	tdx_hardware_unsetup();
+	vmx_hardware_unsetup();
+}
+
 static int vt_vm_init(struct kvm *kvm)
 {
 	if (is_td(kvm))
-		return -EOPNOTSUPP;	/* Not ready to create
guest TD yet. */
arch/x86/kvm/vmx/main.c +22 -1
+		return tdx_vm_init(kvm);
 
 	return vmx_vm_init(kvm);
 }
 
+static void vt_flush_shadow_all_private(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return tdx_mmu_release_hkid(kvm);
+}
+
+static void vt_vm_destroy(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return;
+
+	vmx_vm_destroy(kvm);
+}
+
+static void vt_vm_free(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return tdx_vm_free(kvm);
+}
+
 struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = "kvm_intel",
 
-	.hardware_unsetup = vmx_hardware_unsetup,
+	.hardware_unsetup = vt_hardware_unsetup,
 	.check_processor_compatibility =
vmx_check_processor_compatibility,
arch/x86/kvm/vmx/main.c +3 -1
 	.hardware_enable = vmx_hardware_enable,
@@ -50,7 +76,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.is_vm_type_supported = vt_is_vm_type_supported,
 	.vm_size = sizeof(struct kvm_vmx),
 	.vm_init = vt_vm_init,
-	.vm_destroy = vmx_vm_destroy,
+	.flush_shadow_all_private = vt_flush_shadow_all_private,
+	.vm_destroy = vt_vm_destroy,
+	.vm_free = vt_vm_free,
 
 	.vcpu_precreate = vmx_vcpu_precreate,
 	.vcpu_create = vmx_vcpu_create,
arch/x86/kvm/vmx/tdx.c +3 -0
index 530e72f85762..ec88dde0d300 100644
@@ -32,6 +32,401 @@ struct tdx_capabilities {
 /* Capabilities of KVM + the TDX module. */
 static struct tdx_capabilities tdx_caps;
 
+/*
+ * Some TDX SEAMCALLs (TDH.MNG.CREATE, TDH.PHYMEM.CACHE.WB,
+ * TDH.MNG.KEY.RECLAIMID, TDH.MNG.KEY.FREEID etc) tries to acquire a
global lock
+ * internally in TDX module. If failed, TDX_OPERAND_BUSY is returned without
+ * spinning or waiting due to a constraint on execution time. It's caller's
+ * responsibility to avoid race (or retry on TDX_OPERAND_BUSY). Use this mutex
+ * to avoid race in TDX module because the kernel knows better about scheduling.
+ */
arch/x86/kvm/vmx/tdx.c +27 -0
+static DEFINE_MUTEX(tdx_lock);
+static struct mutex *tdx_mng_key_config_lock;
+
+static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid)
+{
+	return pa | ((hpa_t)hkid << boot_cpu_data.x86_phys_bits);
+}
+
+static inline bool is_td_created(struct kvm_tdx *kvm_tdx)
+{
+	return kvm_tdx->tdr.added;
+}
+
+static inline void tdx_hkid_free(struct kvm_tdx *kvm_tdx)
+{
+	tdx_keyid_free(kvm_tdx->hkid);
+	kvm_tdx->hkid = -1;
+}
+
+static inline bool is_hkid_assigned(struct kvm_tdx *kvm_tdx)
+{
+	return kvm_tdx->hkid > 0;
+}
+
+static void tdx_clear_page(unsigned long page)
+{
+	const void *zero_page = (const void *)
__va(page_to_phys(ZERO_PAGE(0)));
arch/x86/kvm/vmx/tdx.c +4 -0
+	unsigned long i;
+
+	/*
+	 * Zeroing the page is only necessary for systems with
MKTME-i:
arch/x86/kvm/vmx/tdx.c +1 -0
+	 * when re-assign one page from old keyid to a new keyid,
MOVDIR64B is
arch/x86/kvm/vmx/tdx.c +1 -0
+	 * required to clear/write the page with new keyid to
prevent integrity
arch/x86/kvm/vmx/tdx.c +3 -0
+	 * error when read on the page with new keyid.
+	 *
+	 * The cache line could be poisoned (even without MKTME-i),
clear the

arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 82

Quoted 4 lines

arch/x86/kvm/vmx/tdx.c +4 -0

+	 * poison bit.
+	 */
+	for (i = 0; i < PAGE_SIZE; i += 64)
+		movdir64b((void *)(page + i), zero_page);

In patch 11, the clflush is used on the movdir64b target. Should we also use the clflush here?

I suppose the clflush is to prevent the unexpected cacheline writeback after the movdir64b.

Quoted 34 lines

arch/x86/kvm/vmx/tdx.c +2 -0

+	/*
+	 * MOVDIR64B store uses WC buffer.  Prevent following memory

reads

arch/x86/kvm/vmx/tdx.c +6 -0

+	 * from seeing potentially poisoned cache.
+	 */
+	__mb();
+}
+
+static int tdx_reclaim_page(unsigned long va, hpa_t pa, bool do_wb,

u16 hkid) +{

arch/x86/kvm/vmx/tdx.c +7 -0

+	struct tdx_module_output out;
+	u64 err;
+
+	do {
+		err = tdh_phymem_page_reclaim(pa, &out);
+		/*
+		 * TDH.PHYMEM.PAGE.RECLAIM is allowed only when TD

is shutdown.

arch/x86/kvm/vmx/tdx.c +2 -0

+		 * state.  i.e. destructing TD.
+		 * TDH.PHYMEM.PAGE.RECLAIM  requires TDR and target

page.

arch/x86/kvm/vmx/tdx.c +1 -0

+		 * Because we're destructing TD, it's rare to

contend with TDR.

arch/x86/kvm/vmx/tdx.c +10 -0

+		 */
+	} while (err == (TDX_OPERAND_BUSY | TDX_OPERAND_ID_RCX));
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_PHYMEM_PAGE_RECLAIM, err, &out);
+		return -EIO;
+	}
+
+	if (do_wb) {
+		/*
+		 * Only TDR page gets into this path.  No contention

is expected

arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 112

Quoted 1 line

arch/x86/kvm/vmx/tdx.c +1 -0
+		 * because the last page of TD.

because of

Quoted 459 lines

arch/x86/kvm/vmx/tdx.c +2 -0

+		 */
+		err = tdh_phymem_page_wbinvd(set_hkid_to_hpa(pa,

hkid));

arch/x86/kvm/vmx/tdx.c +2 -0

+		if (WARN_ON_ONCE(err)) {
+			pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err,

NULL);

arch/x86/kvm/vmx/tdx.c +29 -0

+			return -EIO;
+		}
+	}
+
+	tdx_clear_page(va);
+	return 0;
+}
+
+static int tdx_alloc_td_page(struct tdx_td_page *page)
+{
+	page->va = __get_free_page(GFP_KERNEL_ACCOUNT);
+	if (!page->va)
+		return -ENOMEM;
+
+	page->pa = __pa(page->va);
+	return 0;
+}
+
+static inline void tdx_mark_td_page_added(struct tdx_td_page *page)
+{
+	WARN_ON_ONCE(page->added);
+	page->added = true;
+}
+
+static void tdx_reclaim_td_page(struct tdx_td_page *page)
+{
+	if (page->added) {
+		/*
+		 * TDCX are being reclaimed.  TDX module maps TDCX

with HKID

arch/x86/kvm/vmx/tdx.c +1 -0

+		 * assigned to the TD.  Here the cache associated to

the TD

arch/x86/kvm/vmx/tdx.c +1 -0

+		 * was already flushed by TDH.PHYMEM.CACHE.WB before

here, So

arch/x86/kvm/vmx/tdx.c +48 -0

+		 * cache doesn't need to be flushed again.
+		 */
+		if (tdx_reclaim_page(page->va, page->pa, false, 0))
+			return;
+
+		page->added = false;
+	}
+	if (page->va) {
+		free_page(page->va);
+		page->va = 0;
+	}
+}
+
+static int tdx_do_tdh_phymem_cache_wb(void *param)
+{
+	u64 err = 0;
+
+	do {
+		err = tdh_phymem_cache_wb(!!err);
+	} while (err == TDX_INTERRUPTED_RESUMABLE);
+
+	/* Other thread may have done for us. */
+	if (err == TDX_NO_HKID_READY_TO_WBCACHE)
+		err = TDX_SUCCESS;
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_PHYMEM_CACHE_WB, err, NULL);
+		return -EIO;
+	}
+
+	return 0;
+}
+
+void tdx_mmu_release_hkid(struct kvm *kvm)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	cpumask_var_t packages;
+	bool cpumask_allocated;
+	u64 err;
+	int ret;
+	int i;
+
+	if (!is_hkid_assigned(kvm_tdx))
+		return;
+
+	if (!is_td_created(kvm_tdx))
+		goto free_hkid;
+
+	cpumask_allocated = zalloc_cpumask_var(&packages,

GFP_KERNEL);

arch/x86/kvm/vmx/tdx.c +4 -0

+	cpus_read_lock();
+	for_each_online_cpu(i) {
+		if (cpumask_allocated &&
+

cpumask_test_and_set_cpu(topology_physical_package_id(i),

arch/x86/kvm/vmx/tdx.c +5 -0

+						packages))
+			continue;
+
+		/*
+		 * We can destroy multiple the guest TDs

simultaneously.

arch/x86/kvm/vmx/tdx.c +1 -0

+		 * Prevent tdh_phymem_cache_wb from returning

TDX_BUSY by

arch/x86/kvm/vmx/tdx.c +4 -0

+		 * serialization.
+		 */
+		mutex_lock(&tdx_lock);
+		ret = smp_call_on_cpu(i, tdx_do_tdh_phymem_cache_wb,

NULL, 1);

arch/x86/kvm/vmx/tdx.c +13 -0

+		mutex_unlock(&tdx_lock);
+		if (ret)
+			break;
+	}
+	cpus_read_unlock();
+	free_cpumask_var(packages);
+
+	mutex_lock(&tdx_lock);
+	err = tdh_mng_key_freeid(kvm_tdx->tdr.pa);
+	mutex_unlock(&tdx_lock);
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_MNG_KEY_FREEID, err, NULL);
+		pr_err("tdh_mng_key_freeid failed. HKID %d is

leaked.\n",

arch/x86/kvm/vmx/tdx.c +25 -0

+			kvm_tdx->hkid);
+		return;
+	}
+
+free_hkid:
+	tdx_hkid_free(kvm_tdx);
+}
+
+void tdx_vm_free(struct kvm *kvm)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	int i;
+
+	/* Can't reclaim or free TD pages if teardown failed. */
+	if (is_hkid_assigned(kvm_tdx))
+		return;
+
+	if (kvm_tdx->tdcs) {
+		for (i = 0; i < tdx_caps.tdcs_nr_pages; i++)
+			tdx_reclaim_td_page(&kvm_tdx->tdcs[i]);
+		kfree(kvm_tdx->tdcs);
+	}
+
+	/*
+	 * TDX module maps TDR with TDX global HKID.  TDX module may

access TDR

arch/x86/kvm/vmx/tdx.c +1 -0

+	 * while operating on TD (Especially reclaiming TDCS).

Cache flush with

arch/x86/kvm/vmx/tdx.c +4 -0

+	 * TDX global HKID is needed.
+	 */
+	if (kvm_tdx->tdr.added &&
+		tdx_reclaim_page(kvm_tdx->tdr.va, kvm_tdx->tdr.pa,

true,

arch/x86/kvm/vmx/tdx.c +16 -0

+				tdx_global_keyid))
+		return;
+
+	free_page(kvm_tdx->tdr.va);
+}
+
+static int tdx_do_tdh_mng_key_config(void *param)
+{
+	hpa_t *tdr_p = param;
+	u64 err;
+
+	do {
+		err = tdh_mng_key_config(*tdr_p);
+
+		/*
+		 * If it failed to generate a random key, retry it

because this

arch/x86/kvm/vmx/tdx.c +1 -0

+		 * is typically caused by an entropy error of the

CPU's random

arch/x86/kvm/vmx/tdx.c +29 -0

+		 * number generator.
+		 */
+	} while (err == TDX_KEY_GENERATION_FAILED);
+
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_MNG_KEY_CONFIG, err, NULL);
+		return -EIO;
+	}
+
+	return 0;
+}
+
+int tdx_vm_init(struct kvm *kvm)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	cpumask_var_t packages;
+	int ret, i;
+	u64 err;
+
+	ret = tdx_keyid_alloc();
+	if (ret < 0)
+		return ret;
+	kvm_tdx->hkid = ret;
+
+	ret = tdx_alloc_td_page(&kvm_tdx->tdr);
+	if (ret)
+		goto free_hkid;
+
+	kvm_tdx->tdcs = kcalloc(tdx_caps.tdcs_nr_pages,

sizeof(*kvm_tdx->tdcs),

arch/x86/kvm/vmx/tdx.c +16 -0

+				GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+	if (!kvm_tdx->tdcs)
+		goto free_tdr;
+	for (i = 0; i < tdx_caps.tdcs_nr_pages; i++) {
+		ret = tdx_alloc_td_page(&kvm_tdx->tdcs[i]);
+		if (ret)
+			goto free_tdcs;
+	}
+
+	if (!zalloc_cpumask_var(&packages, GFP_KERNEL)) {
+		ret = -ENOMEM;
+		goto free_tdcs;
+	}
+	cpus_read_lock();
+	/*
+	 * Need at least one CPU of the package to be online in

order to

arch/x86/kvm/vmx/tdx.c +4 -0

+	 * program all packages for host key id.  Check it.
+	 */
+	for_each_present_cpu(i)
+		cpumask_set_cpu(topology_physical_package_id(i),

packages);

arch/x86/kvm/vmx/tdx.c +2 -0

+	for_each_online_cpu(i)
+		cpumask_clear_cpu(topology_physical_package_id(i),

packages);

arch/x86/kvm/vmx/tdx.c +4 -0

+	if (!cpumask_empty(packages)) {
+		ret = -EIO;
+		/*
+		 * Because it's hard for human operator to figure

out the

arch/x86/kvm/vmx/tdx.c +3 -0

+		 * reason, warn it.
+		 */
+		pr_warn("All packages need to have online CPU to

create TD. Online CPU and retry.\n");

arch/x86/kvm/vmx/tdx.c +6 -0

+		goto free_packages;
+	}
+
+	/*
+	 * Acquire global lock to avoid TDX_OPERAND_BUSY:
+	 * TDH.MNG.CREATE and other APIs try to lock the global Key

Owner

arch/x86/kvm/vmx/tdx.c +1 -0

+	 * Table (KOT) to track the assigned TDX private HKID.  It

doesn't spin

arch/x86/kvm/vmx/tdx.c +1 -0

+	 * to acquire the lock, returns TDX_OPERAND_BUSY instead,

and let the

arch/x86/kvm/vmx/tdx.c +1 -0

+	 * caller to handle the contention.  This is because of time

limitation

arch/x86/kvm/vmx/tdx.c +1 -0

+	 * usable inside the TDX module and OS/VMM knows better

about process

arch/x86/kvm/vmx/tdx.c +4 -0

+	 * scheduling.
+	 *
+	 * APIs to acquire the lock of KOT:
+	 * TDH.MNG.CREATE, TDH.MNG.KEY.FREEID, TDH.MNG.VPFLUSHDONE,

and

arch/x86/kvm/vmx/tdx.c +20 -0

+	 * TDH.PHYMEM.CACHE.WB.
+	 */
+	mutex_lock(&tdx_lock);
+	err = tdh_mng_create(kvm_tdx->tdr.pa, kvm_tdx->hkid);
+	mutex_unlock(&tdx_lock);
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_MNG_CREATE, err, NULL);
+		ret = -EIO;
+		goto free_packages;
+	}
+	tdx_mark_td_page_added(&kvm_tdx->tdr);
+
+	for_each_online_cpu(i) {
+		int pkg = topology_physical_package_id(i);
+
+		if (cpumask_test_and_set_cpu(pkg, packages))
+			continue;
+
+		/*
+		 * Program the memory controller in the package with

arch/x86/kvm/vmx/tdx.c +1 -0

+		 * encryption key associated to a TDX private host

key id

arch/x86/kvm/vmx/tdx.c +1 -0

+		 * assigned to this TDR.  Concurrent operations on

same memory

arch/x86/kvm/vmx/tdx.c +1 -0

+		 * controller results in TDX_OPERAND_BUSY.  Avoid

this race by

arch/x86/kvm/vmx/tdx.c +16 -0

+		 * mutex.
+		 */
+		mutex_lock(&tdx_mng_key_config_lock[pkg]);
+		ret = smp_call_on_cpu(i, tdx_do_tdh_mng_key_config,
+				      &kvm_tdx->tdr.pa, true);
+		mutex_unlock(&tdx_mng_key_config_lock[pkg]);
+		if (ret)
+			break;
+	}
+	cpus_read_unlock();
+	free_cpumask_var(packages);
+	if (ret)
+		goto teardown;
+
+	for (i = 0; i < tdx_caps.tdcs_nr_pages; i++) {
+		err = tdh_mng_addcx(kvm_tdx->tdr.pa,

kvm_tdx->tdcs[i].pa);

arch/x86/kvm/vmx/tdx.c +10 -0

+		if (WARN_ON_ONCE(err)) {
+			pr_tdx_error(TDH_MNG_ADDCX, err, NULL);
+			ret = -EIO;
+			goto teardown;
+		}
+		tdx_mark_td_page_added(&kvm_tdx->tdcs[i]);
+	}
+
+	/*
+	 * Note, TDH_MNG_INIT cannot be invoked here.  TDH_MNG_INIT

requires a dedicated

arch/x86/kvm/vmx/tdx.c +6 -0

+	 * ioctl() to define the configure CPUID values for the TD.
+	 */
+	return 0;
+
+	/*
+	 * The sequence for freeing resources from a partially

initialized TD

arch/x86/kvm/vmx/tdx.c +1 -0

+	 * varies based on where in the initialization flow failure

occurred.

arch/x86/kvm/vmx/tdx.c +1 -0

+	 * Simply use the full teardown and destroy, which naturally

play nice

arch/x86/kvm/vmx/tdx.c +41 -0

+	 * with partial initialization.
+	 */
+teardown:
+	tdx_mmu_release_hkid(kvm);
+	tdx_vm_free(kvm);
+	return ret;
+
+free_packages:
+	cpus_read_unlock();
+	free_cpumask_var(packages);
+free_tdcs:
+	for (i = 0; i < tdx_caps.tdcs_nr_pages; i++) {
+		if (!kvm_tdx->tdcs[i].va)
+			continue;
+		free_page(kvm_tdx->tdcs[i].va);
+	}
+	kfree(kvm_tdx->tdcs);
+	kvm_tdx->tdcs = NULL;
+free_tdr:
+	if (kvm_tdx->tdr.va) {
+		free_page(kvm_tdx->tdr.va);
+		kvm_tdx->tdr.added = false;
+		kvm_tdx->tdr.va = 0;
+		kvm_tdx->tdr.pa = 0;
+	}
+free_hkid:
+	if (kvm_tdx->hkid != -1)
+		tdx_hkid_free(kvm_tdx);
+	return ret;
+}
+
 static int __init tdx_module_setup(void)
 {
 	const struct tdsysinfo_struct *tdsysinfo;

@@ -82,6 +477,8 @@ bool tdx_is_vm_type_supported(unsigned long type)
 
 int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
 {
+	int max_pkgs;
+	int i;
 	int r;
 
 	if (!enable_ept) {

@@ -95,6 +492,14 @@ int __init tdx_hardware_setup(struct kvm_x86_ops
 *x86_ops) return -ENODEV;
 	}
 
+	max_pkgs = topology_max_packages();
+	tdx_mng_key_config_lock = kcalloc(max_pkgs,
 sizeof(*tdx_mng_key_config_lock),
+				   GFP_KERNEL);
+	if (!tdx_mng_key_config_lock)
+		return -ENOMEM;
+	for (i = 0; i < max_pkgs; i++)
+		mutex_init(&tdx_mng_key_config_lock[i]);
+
 	/* TDX requires VMX. */
 	r = vmxon_all();

arch/x86/kvm/vmx/tdx.c +6 -0

 	if (!r)

@@ -103,3 +508,9 @@ int __init tdx_hardware_setup(struct kvm_x86_ops
 *x86_ops) 
 	return r;
 }
+
+void tdx_hardware_unsetup(void)
+{
+	/* kfree accepts NULL. */
+	kfree(tdx_mng_key_config_lock);
+}

arch/x86/kvm/vmx/tdx.h +2 -0

index 98999bf3f188..938314635b47 100644
@@ -17,6 +17,8 @@ struct kvm_tdx {
 
 	struct tdx_td_page tdr;
 	struct tdx_td_page *tdcs;
+
+	int hkid;
 };
 
 struct vcpu_tdx {

arch/x86/kvm/vmx/x86_ops.h +5 -0

index ac1688b0b0e3..95da978c9aa9 100644
@@ -133,9 +133,20 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_INTEL_TDX_HOST
 int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);
 bool tdx_is_vm_type_supported(unsigned long type);
+void tdx_hardware_unsetup(void);
+
+int tdx_vm_init(struct kvm *kvm);
+void tdx_mmu_release_hkid(struct kvm *kvm);
+void tdx_vm_free(struct kvm *kvm);
 #else
 static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) {

return 0; } static inline bool tdx_is_vm_type_supported(unsigned long type) { return false; } +static inline void tdx_hardware_unsetup(void) {} +
+static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; } +static inline void tdx_mmu_release_hkid(struct kvm *kvm) {}

arch/x86/kvm/vmx/x86_ops.h +2 -0

+static inline void tdx_flush_shadow_all_private(struct kvm *kvm) {}
+static inline void tdx_vm_free(struct kvm *kvm) {}
 #endif
 
 #endif /* __KVM_X86_VMX_X86_OPS_H */

arch/x86/kvm/x86.c +3 -0

index 91053fdc4512..4b22196cb12c 100644
@@ -12702,6 +12702,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 	kvm_page_track_cleanup(kvm);
 	kvm_xen_destroy_vm(kvm);
 	kvm_hv_destroy_vm(kvm);
+	static_call_cond(kvm_x86_vm_free)(kvm);
 }
 
 static void memslot_rmap_free(struct kvm_memory_slot *slot)

@@ -13012,6 +13013,13 @@ void kvm_arch_commit_memory_region(struct
 kvm *kvm, 
 void kvm_arch_flush_shadow_all(struct kvm *kvm)
 {
+	/*
+	 * kvm_mmu_zap_all() zaps both private and shared page

tables. Before

arch/x86/kvm/x86.c +1 -0

+	 * tearing down private page tables, TDX requires some TD

resources to

arch/x86/kvm/x86.c +1 -0

+	 * be destroyed (i.e. keyID must have been reclaimed, etc).

Invoke

arch/x86/kvm/x86.c +3 -0

+	 * kvm_x86_flush_shadow_all_private() for this.
+	 */
+	static_call_cond(kvm_x86_flush_shadow_all_private)(kvm);
 	kvm_mmu_zap_all(kvm);
 }

Isaku Yamahata

2022-12-15 19:59 UTC #40 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

Quoted 195 lines · Zhi Wang

Quoted 184 lines · On Sat, 29 Oct 2022 23:22:17 -0700 [email protected] wrote:
From: Sean Christopherson <[email protected]>
As the first step to create TDX guest, create/destroy VM struct. Assign TDX private Host Key ID (HKID) to the TDX guest for memory encryption and allocate extra pages for the TDX guest. On destruction, free allocated pages, and HKID.
Before tearing down private page tables, TDX requires some resources of the guest TD to be destroyed (i.e. keyID must have been reclaimed, etc). Add flush_shadow_all_private callback before tearing down private page tables for it.
Add a second kvm_x86_ops hook in kvm_arch_destroy_vm() to support TDX's destruction path, which needs to first put the VM into a teardown state, then free per-vCPU resources, and finally free per-VM resources.
Co-developed-by Kai Huang <[email protected]>
Signed-off-by Kai Huang <[email protected]>
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
---
 arch/x86/include/asm/kvm-x86-ops.h |   2 +
 arch/x86/include/asm/kvm_host.h    |   2 +
 arch/x86/kvm/vmx/main.c            |  34 ++-
 arch/x86/kvm/vmx/tdx.c             | 411
+++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h             |
2 + arch/x86/kvm/vmx/x86_ops.h         |  11 +
 arch/x86/kvm/x86.c                 |   8 +
 7 files changed, 467 insertions(+), 3 deletions(-)
(quoted diff) +0 -0
b/arch/x86/include/asm/kvm-x86-ops.h index 8a5c5ae70bc5..3a29a6b31ee8 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
arch/x86/include/asm/kvm-x86-ops.h +2 -0
@@ -21,7 +21,9 @@ KVM_X86_OP(has_emulated_msr)
 KVM_X86_OP(vcpu_after_set_cpuid)
 KVM_X86_OP(is_vm_type_supported)
 KVM_X86_OP(vm_init)
+KVM_X86_OP_OPTIONAL(flush_shadow_all_private)
 KVM_X86_OP_OPTIONAL(vm_destroy)
+KVM_X86_OP_OPTIONAL(vm_free)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate)
 KVM_X86_OP(vcpu_create)
 KVM_X86_OP(vcpu_free)
(quoted diff) +0 -0
b/arch/x86/include/asm/kvm_host.h index 2a41a93a80f3..2870155ce6fb 100644 --- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1472,7 +1472,9 @@ struct kvm_x86_ops {
 	bool (*is_vm_type_supported)(unsigned long vm_type);
 	unsigned int vm_size;
 	int (*vm_init)(struct kvm *kvm);
+	void (*flush_shadow_all_private)(struct kvm *kvm);
 	void (*vm_destroy)(struct kvm *kvm);
+	void (*vm_free)(struct kvm *kvm);
 
 	/* Create, but do not attach this VCPU */
 	int (*vcpu_precreate)(struct kvm *kvm);
arch/x86/kvm/vmx/main.c +6 -1
index 0900ff2f2390..d01a946a18cf 100644
@@ -29,18 +29,44 @@ static __init int vt_hardware_setup(void)
 	return 0;
 }
 
+static void vt_hardware_unsetup(void)
+{
+	tdx_hardware_unsetup();
+	vmx_hardware_unsetup();
+}
+
 static int vt_vm_init(struct kvm *kvm)
 {
 	if (is_td(kvm))
-		return -EOPNOTSUPP;	/* Not ready to create
guest TD yet. */
arch/x86/kvm/vmx/main.c +22 -1
+		return tdx_vm_init(kvm);
 
 	return vmx_vm_init(kvm);
 }
 
+static void vt_flush_shadow_all_private(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return tdx_mmu_release_hkid(kvm);
+}
+
+static void vt_vm_destroy(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return;
+
+	vmx_vm_destroy(kvm);
+}
+
+static void vt_vm_free(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return tdx_vm_free(kvm);
+}
+
 struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = "kvm_intel",
 
-	.hardware_unsetup = vmx_hardware_unsetup,
+	.hardware_unsetup = vt_hardware_unsetup,
 	.check_processor_compatibility =
vmx_check_processor_compatibility,
arch/x86/kvm/vmx/main.c +3 -1
 	.hardware_enable = vmx_hardware_enable,
@@ -50,7 +76,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.is_vm_type_supported = vt_is_vm_type_supported,
 	.vm_size = sizeof(struct kvm_vmx),
 	.vm_init = vt_vm_init,
-	.vm_destroy = vmx_vm_destroy,
+	.flush_shadow_all_private = vt_flush_shadow_all_private,
+	.vm_destroy = vt_vm_destroy,
+	.vm_free = vt_vm_free,
 
 	.vcpu_precreate = vmx_vcpu_precreate,
 	.vcpu_create = vmx_vcpu_create,
arch/x86/kvm/vmx/tdx.c +3 -0
index 530e72f85762..ec88dde0d300 100644
@@ -32,6 +32,401 @@ struct tdx_capabilities {
 /* Capabilities of KVM + the TDX module. */
 static struct tdx_capabilities tdx_caps;
 
+/*
+ * Some TDX SEAMCALLs (TDH.MNG.CREATE, TDH.PHYMEM.CACHE.WB,
+ * TDH.MNG.KEY.RECLAIMID, TDH.MNG.KEY.FREEID etc) tries to acquire a
global lock
+ * internally in TDX module. If failed, TDX_OPERAND_BUSY is returned without
+ * spinning or waiting due to a constraint on execution time. It's caller's
+ * responsibility to avoid race (or retry on TDX_OPERAND_BUSY). Use this mutex
+ * to avoid race in TDX module because the kernel knows better about scheduling.
+ */
arch/x86/kvm/vmx/tdx.c +27 -0
+static DEFINE_MUTEX(tdx_lock);
+static struct mutex *tdx_mng_key_config_lock;
+
+static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid)
+{
+	return pa | ((hpa_t)hkid << boot_cpu_data.x86_phys_bits);
+}
+
+static inline bool is_td_created(struct kvm_tdx *kvm_tdx)
+{
+	return kvm_tdx->tdr.added;
+}
+
+static inline void tdx_hkid_free(struct kvm_tdx *kvm_tdx)
+{
+	tdx_keyid_free(kvm_tdx->hkid);
+	kvm_tdx->hkid = -1;
+}
+
+static inline bool is_hkid_assigned(struct kvm_tdx *kvm_tdx)
+{
+	return kvm_tdx->hkid > 0;
+}
+
+static void tdx_clear_page(unsigned long page)
+{
+	const void *zero_page = (const void *)
__va(page_to_phys(ZERO_PAGE(0)));
arch/x86/kvm/vmx/tdx.c +4 -0
+	unsigned long i;
+
+	/*
+	 * Zeroing the page is only necessary for systems with
MKTME-i:
arch/x86/kvm/vmx/tdx.c +1 -0
+	 * when re-assign one page from old keyid to a new keyid,
MOVDIR64B is
arch/x86/kvm/vmx/tdx.c +1 -0
+	 * required to clear/write the page with new keyid to
prevent integrity
arch/x86/kvm/vmx/tdx.c +3 -0
+	 * error when read on the page with new keyid.
+	 *
+	 * The cache line could be poisoned (even without MKTME-i),
clear the
arch/x86/kvm/vmx/tdx.c struct tdx_capabilities { · line 82
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4 -0
+	 * poison bit.
+	 */
+	for (i = 0; i < PAGE_SIZE; i += 64)
+		movdir64b((void *)(page + i), zero_page);
In patch 11, the clflush is used on the movdir64b target. Should we also use the clflush here?

Quoted 2 lines

I suppose the clflush is to prevent the unexpected cacheline writeback after the movdir64b.

No, clflush doesn't work. The cache with HKID set needs to be flushed and (potential) poison needs to be cleared. In host OS/VMM, clflush flushes on cache with HKID=0 (or HKID of MKTME). clflush doesn't clear poison.

Signature · 1 line

Isaku Yamahata <[email protected]>

Ackerley Tng

2022-12-09 19:15 UTC #41 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

In tdx_vm_init, it is possible to have a double-reclaim, which eventually causes a host crash. I have a selftest that reliably reproduces this, and I believe the problem is that withiin tdx_vm_free(), we don't reset kvm->tdcs = NULL and kvm->tdr.added to false.

Quoted 115 lines

(quoted diff) +46 -0

+int tdx_vm_init(struct kvm *kvm)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	cpumask_var_t packages;
+	int ret, i;
+	u64 err;
+
+	ret = tdx_keyid_alloc();
+	if (ret < 0)
+		return ret;
+	kvm_tdx->hkid = ret;
+
+	ret = tdx_alloc_td_page(&kvm_tdx->tdr);
+	if (ret)
+		goto free_hkid;
+
+	kvm_tdx->tdcs = kcalloc(tdx_caps.tdcs_nr_pages, sizeof(*kvm_tdx->tdcs),
+				GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+	if (!kvm_tdx->tdcs)
+		goto free_tdr;
+	for (i = 0; i < tdx_caps.tdcs_nr_pages; i++) {
+		ret = tdx_alloc_td_page(&kvm_tdx->tdcs[i]);
+		if (ret)
+			goto free_tdcs;
+	}
+
+	if (!zalloc_cpumask_var(&packages, GFP_KERNEL)) {
+		ret = -ENOMEM;
+		goto free_tdcs;
+	}
+	cpus_read_lock();
+	/*
+	 * Need at least one CPU of the package to be online in order to
+	 * program all packages for host key id.  Check it.
+	 */
+	for_each_present_cpu(i)
+		cpumask_set_cpu(topology_physical_package_id(i), packages);
+	for_each_online_cpu(i)
+		cpumask_clear_cpu(topology_physical_package_id(i), packages);
+	if (!cpumask_empty(packages)) {
+		ret = -EIO;
+		/*
+		 * Because it's hard for human operator to figure out the
+		 * reason, warn it.
+		 */
+		pr_warn("All packages need to have online CPU to create TD. Online CPU

and retry.\n");

(quoted diff) +63 -0

+		goto free_packages;
+	}
+
+	/*
+	 * Acquire global lock to avoid TDX_OPERAND_BUSY:
+	 * TDH.MNG.CREATE and other APIs try to lock the global Key Owner
+	 * Table (KOT) to track the assigned TDX private HKID.  It doesn't spin
+	 * to acquire the lock, returns TDX_OPERAND_BUSY instead, and let the
+	 * caller to handle the contention.  This is because of time limitation
+	 * usable inside the TDX module and OS/VMM knows better about process
+	 * scheduling.
+	 *
+	 * APIs to acquire the lock of KOT:
+	 * TDH.MNG.CREATE, TDH.MNG.KEY.FREEID, TDH.MNG.VPFLUSHDONE, and
+	 * TDH.PHYMEM.CACHE.WB.
+	 */
+	mutex_lock(&tdx_lock);
+	err = tdh_mng_create(kvm_tdx->tdr.pa, kvm_tdx->hkid);
+	mutex_unlock(&tdx_lock);
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_MNG_CREATE, err, NULL);
+		ret = -EIO;
+		goto free_packages;
+	}
+	tdx_mark_td_page_added(&kvm_tdx->tdr);
+
+	for_each_online_cpu(i) {
+		int pkg = topology_physical_package_id(i);
+
+		if (cpumask_test_and_set_cpu(pkg, packages))
+			continue;
+
+		/*
+		 * Program the memory controller in the package with an
+		 * encryption key associated to a TDX private host key id
+		 * assigned to this TDR.  Concurrent operations on same memory
+		 * controller results in TDX_OPERAND_BUSY.  Avoid this race by
+		 * mutex.
+		 */
+		mutex_lock(&tdx_mng_key_config_lock[pkg]);
+		ret = smp_call_on_cpu(i, tdx_do_tdh_mng_key_config,
+				      &kvm_tdx->tdr.pa, true);
+		mutex_unlock(&tdx_mng_key_config_lock[pkg]);
+		if (ret)
+			break;
+	}
+	cpus_read_unlock();
+	free_cpumask_var(packages);
+	if (ret)
+		goto teardown;
+
+	for (i = 0; i < tdx_caps.tdcs_nr_pages; i++) {
+		err = tdh_mng_addcx(kvm_tdx->tdr.pa, kvm_tdx->tdcs[i].pa);
+		if (WARN_ON_ONCE(err)) {
+			pr_tdx_error(TDH_MNG_ADDCX, err, NULL);
+			ret = -EIO;
+			goto teardown;
+		}
+		tdx_mark_td_page_added(&kvm_tdx->tdcs[i]);
+	}
+
+	/*
+	 * Note, TDH_MNG_INIT cannot be invoked here.  TDH_MNG_INIT requires a

dedicated

(quoted diff) +4 -0

+	 * ioctl() to define the configure CPUID values for the TD.
+	 */
+	return 0;
+

Review

Quoted 10 lines

(quoted diff) +10 -0

+	/*
+	 * The sequence for freeing resources from a partially initialized TD
+	 * varies based on where in the initialization flow failure occurred.
+	 * Simply use the full teardown and destroy, which naturally play nice
+	 * with partial initialization.
+	 */
+teardown:
+	tdx_mmu_release_hkid(kvm);
+	tdx_vm_free(kvm);
+	return ret;

If there is some error that causes an exit through teardown, tdx_vm_free() will be called, which causes the resources to be freed. However, tdx_vm_free() is called a second time when the selftest (or qemu) exits, which causes a second reclaim to be performed.

Quoted 14 lines

(quoted diff) +14 -0

+
+free_packages:
+	cpus_read_unlock();
+	free_cpumask_var(packages);
+free_tdcs:
+	for (i = 0; i < tdx_caps.tdcs_nr_pages; i++) {
+		if (!kvm_tdx->tdcs[i].va)
+			continue;
+		free_page(kvm_tdx->tdcs[i].va);
+	}
+	kfree(kvm_tdx->tdcs);
+	kvm_tdx->tdcs = NULL;
+free_tdr:
+	if (kvm_tdx->tdr.va) {

Review

Quoted 10 lines

(quoted diff) +10 -0

+		free_page(kvm_tdx->tdr.va);
+		kvm_tdx->tdr.added = false;
+		kvm_tdx->tdr.va = 0;
+		kvm_tdx->tdr.pa = 0;
+	}
+free_hkid:
+	if (kvm_tdx->hkid != -1)
+		tdx_hkid_free(kvm_tdx);
+	return ret;
+}

The second reclaim is performed because kvm_tdx->tdcs is still set, and kvm_tdx->tdr.added is still set, so the second two if blocks in tdx_vm_free() are executed.

Quoted 17 lines

(quoted diff) +17 -0

+void tdx_vm_free(struct kvm *kvm)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	int i;
+
+	/* Can't reclaim or free TD pages if teardown failed. */
+	if (is_hkid_assigned(kvm_tdx))
+		return;
+
+	if (kvm_tdx->tdcs) {
+		for (i = 0; i < tdx_caps.tdcs_nr_pages; i++)
+			tdx_reclaim_td_page(&kvm_tdx->tdcs[i]);
+		kfree(kvm_tdx->tdcs);
+	}
+
+	/*
+	 * TDX module maps TDR with TDX global HKID.  TDX module may access TDR

Review

Quoted 10 lines

(quoted diff) +10 -0

+	 * while operating on TD (Especially reclaiming TDCS).  Cache flush with
+	 * TDX global HKID is needed.
+	 */
+	if (kvm_tdx->tdr.added &&
+		tdx_reclaim_page(kvm_tdx->tdr.va, kvm_tdx->tdr.pa, true,
+				tdx_global_keyid))
+		return;
+
+	free_page(kvm_tdx->tdr.va);
+}

Here's the fix that stopped the crash I was observing

arch/x86/kvm/vmx/tdx.c +4 -0

index 2e7916fb72a7..41d1ff1510c3 100644
@@ -405,6 +405,7 @@ void tdx_vm_free(struct kvm *kvm)
                for (i = 0; i < tdx_caps.tdcs_nr_pages; i++)
                        tdx_reclaim_td_page(&kvm_tdx->tdcs[i]);
                kfree(kvm_tdx->tdcs);
+               kvm_tdx->tdcs = NULL;
        }
 
        /*

@@ -418,6 +419,9 @@ void tdx_vm_free(struct kvm *kvm)
                return;
 
        free_page(kvm_tdx->tdr.va);
+       kvm_tdx->tdr.added = false;
+       kvm_tdx->tdr.va = 0;
+       kvm_tdx->tdr.pa = 0;
 }
 
 static int tdx_do_tdh_mng_key_config(void *param)

Isaku Yamahata

2022-12-15 20:59 UTC #42 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

Quoted 6 lines · Ackerley Tng

In tdx_vm_init, it is possible to have a double-reclaim, which eventually causes a host crash. I have a selftest that reliably reproduces this, and I believe the problem is that withiin tdx_vm_free(), we don't reset kvm->tdcs = NULL and kvm->tdr.added to false.

Thanks for the fix. Did you use error injection to trigger the error path?

Thanks,

Signature · 1 line

Isaku Yamahata <[email protected]>

Ackerley Tng

2023-01-03 18:17 UTC #43 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

Happy to help!

Nope it wasn't any special technique, I was working on the selftests and made a mistake configuring the TD while initializing it.

While setting up struct kvm_tdx_init_vm, I did not copy the entire struct kvm_cpuid2.

This reliably reproduces the crash:

--------

// SPDX-License-Identifier: GPL-2.0-only

#include "kvm_util_base.h"
#include <processor.h>

static int tdx_ioctl(int fd, int ioctl_no, uint32_t flags, void *data)

{
	struct kvm_tdx_cmd tdx_cmd;

	memset(&tdx_cmd, 0x0, sizeof(tdx_cmd));
	tdx_cmd.id = ioctl_no;
	tdx_cmd.flags = flags;
	tdx_cmd.data = (uint64_t)data;
	return ioctl(fd, KVM_MEMORY_ENCRYPT_OP, &tdx_cmd);
}

#define XFEATURE_LBR 15
#define XFEATURE_XTILECFG 17
#define XFEATURE_XTILEDATA 18
#define XFEATURE_CET_U 11
#define XFEATURE_CET_S 12

#define XFEATURE_MASK_LBR (1 << XFEATURE_LBR)
#define XFEATURE_MASK_CET_U (1 << XFEATURE_CET_U)
#define XFEATURE_MASK_CET_S (1 << XFEATURE_CET_S)
#define XFEATURE_MASK_CET (XFEATURE_MASK_CET_U | XFEATURE_MASK_CET_S)
#define XFEATURE_MASK_XTILECFG (1 << XFEATURE_XTILECFG)
#define XFEATURE_MASK_XTILEDATA (1 << XFEATURE_XTILEDATA)
#define XFEATURE_MASK_XTILE (XFEATURE_MASK_XTILECFG | XFEATURE_MASK_XTILEDATA)

static void apply_tdx_restrictions(struct kvm_cpuid2 *cpuid_data)

{
	for (int i = 0; i < 100; i++) {
		struct kvm_cpuid_entry2 *e = &cpuid_data->entries[i];

		if (e->function == 0xd && e->index == 0) {
			/*
			 * TDX module requires both XTILE_{CFG, DATA} to be set.
			 * Both bits are required for AMX to be functional.
			 */
			if ((e->eax & XFEATURE_MASK_XTILE) != XFEATURE_MASK_XTILE) {
				e->eax &= ~XFEATURE_MASK_XTILE;
			}
		}

		if (e->function == 0xd && e->index == 1) {
			/*
			 * TDX doesn't support LBR yet.
			 * Disable bits from the XCR0 register.
			 */
			e->ecx &= ~XFEATURE_MASK_LBR;

			/*
			 * TDX modules requires both CET_{U, S} to be set even
			 * if only one is supported.
			 */
			if (e->ecx & XFEATURE_MASK_CET)
				e->ecx |= XFEATURE_MASK_CET;
		}
	}
}

void initialize_td_bad(struct kvm_vm *vm)

{
	const struct kvm_cpuid2 *cpuid;
	int rc;

	/* No guest VMM controlled cpuid information yet. */
	struct kvm_tdx_init_vm init_vm;

	rc = kvm_check_cap(KVM_CAP_X2APIC_API);
	TEST_ASSERT(rc, "TDX: KVM_CAP_X2APIC_API is not supported!");
	rc = kvm_check_cap(KVM_CAP_SPLIT_IRQCHIP);
	TEST_ASSERT(rc, "TDX: KVM_CAP_SPLIT_IRQCHIP is not supported!");

	vm_enable_cap(vm, KVM_CAP_X2APIC_API,
		      KVM_X2APIC_API_USE_32BIT_IDS |
			      KVM_X2APIC_API_DISABLE_BROADCAST_QUIRK);
	vm_enable_cap(vm, KVM_CAP_SPLIT_IRQCHIP, 24);

	/* Allocate and setup memory for the td guest. */
	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, 0, 0, 1, 0);

	cpuid = kvm_get_supported_cpuid();
	apply_tdx_restrictions((struct kvm_cpuid2 *) cpuid);

	memset(&init_vm, 0, sizeof(init_vm));
	init_vm.attributes = 0;

	/*
	 * Should have been kvm_cpuid2_size(cpuid->nent) instead of
	 * sizeof(*cpuid)
	 */
	memcpy(&init_vm.cpuid, cpuid, sizeof(*cpuid));

	rc = tdx_ioctl(vm->fd, KVM_TDX_INIT_VM, 0, &init_vm);
	TEST_ASSERT(rc == 0, "KVM_TDX_INIT_VM failed: %d %d", rc, errno);
}

static int crash(void)

{
	struct kvm_vm *vm;

	/* Create a TD VM with no memory.*/
	vm = ____vm_create(VM_MODE_DEFAULT, 0, KVM_X86_TDX_VM);

	/* Allocate TD guest memory and initialize the TD.*/
	initialize_td_bad(vm);

	return 0;
}

int main(int argc, char **argv)

{
	/* Disable stdout buffering */
	setbuf(stdout, NULL);

	return crash();
}

Wang, Lei

2023-01-04 03:58 UTC #44 lore raw

Re: [PATCH v10 016/108] KVM: TDX: create/destroy VM structure

On 10/30/2022 2:22 PM, [email protected] wrote:> +static void vt_hardware_unsetup(void)

Quoted 9 lines

arch/x86/kvm/vmx/main.c +5 -1

+{
+	tdx_hardware_unsetup();
+	vmx_hardware_unsetup();
+}
+
 static int vt_vm_init(struct kvm *kvm)
 {
 	if (is_td(kvm))
-		return -EOPNOTSUPP;	/* Not ready to create guest TD yet. */

arch/x86/kvm/vmx/main.c static __init int vt_hardware_setup(void) · line 50

Quoted 10 lines

arch/x86/kvm/vmx/main.c +6 -0

+		return tdx_vm_init(kvm);
 
 	return vmx_vm_init(kvm);
 }
 
+static void vt_flush_shadow_all_private(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return tdx_mmu_release_hkid(kvm);
+}

nit: there is no need to use the "return" keyword.

Quoted 4 lines

arch/x86/kvm/vmx/main.c +4 -0

+
+static void vt_vm_destroy(struct kvm *kvm)
+{
+	if (is_td(kvm))

arch/x86/kvm/vmx/main.c static __init int vt_hardware_setup(void) · line 64

Quoted 10 lines

arch/x86/kvm/vmx/main.c +10 -0

+		return;
+
+	vmx_vm_destroy(kvm);
+}
+
+static void vt_vm_free(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return tdx_vm_free(kvm);
+}

ditto.

[email protected]

author

2022-10-30 06:24 UTC #45 lore raw

PATCH v10 17/108 KVM: TDX: Refuse to unplug the last cpu on the package

From: Isaku Yamahata <[email protected]>

In order to reclaim TDX host key id, (i.e. when deleting guest TD), needs to call TDH.PHYMEM.PAGE.WBINVD on all packages. If we have used TDX host key id, refuse to offline the last online cpu.

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/vmx/main.c            |  1 +
 arch/x86/kvm/vmx/tdx.c             | 40 +++++++++++++++++++++++++++++-
 arch/x86/kvm/vmx/x86_ops.h         |  2 ++
 arch/x86/kvm/x86.c                 | 27 ++++++++++++--------
 6 files changed, 61 insertions(+), 11 deletions(-)

arch/x86/include/asm/kvm-x86-ops.h +1 -0

index 3a29a6b31ee8..0ceb8e58a6c0 100644
@@ -17,6 +17,7 @@ BUILD_BUG_ON(1)
 KVM_X86_OP(hardware_enable)
 KVM_X86_OP(hardware_disable)
 KVM_X86_OP(hardware_unsetup)
+KVM_X86_OP_OPTIONAL_RET0(offline_cpu)
 KVM_X86_OP(has_emulated_msr)
 KVM_X86_OP(vcpu_after_set_cpuid)
 KVM_X86_OP(is_vm_type_supported)

arch/x86/include/asm/kvm_host.h +1 -0

index 2870155ce6fb..50b39d0071ff 100644
@@ -1466,6 +1466,7 @@ struct kvm_x86_ops {
 	int (*hardware_enable)(void);
 	void (*hardware_disable)(void);
 	void (*hardware_unsetup)(void);
+	int (*offline_cpu)(void);
 	bool (*has_emulated_msr)(struct kvm *kvm, u32 index);
 	void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);

arch/x86/kvm/vmx/main.c +1 -0

index d01a946a18cf..0918d1e6d2f3 100644
@@ -67,6 +67,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = "kvm_intel",
 
 	.hardware_unsetup = vt_hardware_unsetup,
+	.offline_cpu = tdx_offline_cpu,
 	.check_processor_compatibility = vmx_check_processor_compatibility,
 
 	.hardware_enable = vmx_hardware_enable,

arch/x86/kvm/vmx/tdx.c +39 -1

index ec88dde0d300..64229c3b3c5a 100644
@@ -42,6 +42,7 @@ static struct tdx_capabilities tdx_caps;
  */
 static DEFINE_MUTEX(tdx_lock);
 static struct mutex *tdx_mng_key_config_lock;
+static atomic_t nr_configured_hkid;
 
 static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid)
 {

@@ -222,7 +223,8 @@ void tdx_mmu_release_hkid(struct kvm *kvm)
 		pr_err("tdh_mng_key_freeid failed. HKID %d is leaked.\n",
 			kvm_tdx->hkid);
 		return;
-	}
+	} else
+		atomic_dec(&nr_configured_hkid);
 
 free_hkid:
 	tdx_hkid_free(kvm_tdx);

@@ -371,6 +373,8 @@ int tdx_vm_init(struct kvm *kvm)
 		if (ret)
 			break;
 	}
+	if (!ret)
+		atomic_inc(&nr_configured_hkid);
 	cpus_read_unlock();
 	free_cpumask_var(packages);
 	if (ret)

@@ -514,3 +518,37 @@ void tdx_hardware_unsetup(void)
 	/* kfree accepts NULL. */
 	kfree(tdx_mng_key_config_lock);
 }
+
+int tdx_offline_cpu(void)
+{
+	int curr_cpu = smp_processor_id();
+	cpumask_var_t packages;
+	int ret = 0;
+	int i;
+
+	if (!atomic_read(&nr_configured_hkid))
+		return 0;
+
+	/*
+	 * To reclaim hkid, need to call TDH.PHYMEM.PAGE.WBINVD on all packages.
+	 * If this is the last online cpu on the package, refuse offline.
+	 */
+	if (!zalloc_cpumask_var(&packages, GFP_KERNEL))
+		return -ENOMEM;
+
+	for_each_online_cpu(i) {
+		if (i != curr_cpu)
+			cpumask_set_cpu(topology_physical_package_id(i), packages);
+	}
+	if (!cpumask_test_cpu(topology_physical_package_id(curr_cpu), packages))
+		ret = -EBUSY;
+	free_cpumask_var(packages);
+	if (ret)
+		/*
+		 * Because it's hard for human operator to understand the
+		 * reason, warn it.
+		 */
+		pr_warn("TDX requires all packages to have an online CPU.  "
+			"Delete all TDs in order to offline all CPUs of a package.\n");
+	return ret;
+}

arch/x86/kvm/vmx/x86_ops.h +2 -0

index 95da978c9aa9..b2cb5786830a 100644
@@ -134,6 +134,7 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);
 int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);
 bool tdx_is_vm_type_supported(unsigned long type);
 void tdx_hardware_unsetup(void);
+int tdx_offline_cpu(void);
 
 int tdx_vm_init(struct kvm *kvm);
 void tdx_mmu_release_hkid(struct kvm *kvm);

@@ -142,6 +143,7 @@ void tdx_vm_free(struct kvm *kvm);
 static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }
 static inline bool tdx_is_vm_type_supported(unsigned long type) { return false; }
 static inline void tdx_hardware_unsetup(void) {}
+static inline int tdx_offline_cpu(void) { return 0; }
 
 static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; }
 static inline void tdx_mmu_release_hkid(struct kvm *kvm) {}

arch/x86/kvm/x86.c +17 ~~-10~~

index 4b22196cb12c..25c30c8c2d9b 100644
@@ -12337,16 +12337,23 @@ int kvm_arch_online_cpu(unsigned int cpu, int usage_count)
 
 int kvm_arch_offline_cpu(unsigned int cpu, int usage_count)
 {
-	if (usage_count) {
-		/*
-		 * arch callback kvm_arch_hardware_disable() assumes that
-		 * preemption is disabled for historical reason.  Disable
-		 * preemption until all arch callbacks are fixed.
-		 */
-		preempt_disable();
-		hardware_disable(NULL);
-		preempt_enable();
-	}
+	int ret;
+
+	if (!usage_count)
+		return 0;
+
+	ret = static_call(kvm_x86_offline_cpu)();
+	if (ret)
+		return ret;
+
+	/*
+	 * arch callback kvm_arch_hardware_disable() assumes that preemption is
+	 * disabled for historical reason.  Disable preemption until all arch
+	 * callbacks are fixed.
+	 */
+	preempt_disable();
+	hardware_disable(NULL);
+	preempt_enable();
 	return 0;
 }

Signature · 1 line

2.25.1

Binbin Wu

2022-11-02 08:06 UTC #46 lore raw

Re: [PATCH v10 017/108] KVM: TDX: Refuse to unplug the last cpu on the package

Quoted 155 lines · On 2022/10/30 14:22, [email protected] wrote:

From: Isaku Yamahata <[email protected]>

In order to reclaim TDX host key id, (i.e. when deleting guest TD), needs to call TDH.PHYMEM.PAGE.WBINVD on all packages. If we have used TDX host key id, refuse to offline the last online cpu.

---
  arch/x86/include/asm/kvm-x86-ops.h |  1 +
  arch/x86/include/asm/kvm_host.h    |  1 +
  arch/x86/kvm/vmx/main.c            |  1 +
  arch/x86/kvm/vmx/tdx.c             | 40 +++++++++++++++++++++++++++++-
  arch/x86/kvm/vmx/x86_ops.h         |  2 ++
  arch/x86/kvm/x86.c                 | 27 ++++++++++++--------
  6 files changed, 61 insertions(+), 11 deletions(-)

arch/x86/include/asm/kvm-x86-ops.h +1 -0

index 3a29a6b31ee8..0ceb8e58a6c0 100644
@@ -17,6 +17,7 @@ BUILD_BUG_ON(1)
  KVM_X86_OP(hardware_enable)
  KVM_X86_OP(hardware_disable)
  KVM_X86_OP(hardware_unsetup)
+KVM_X86_OP_OPTIONAL_RET0(offline_cpu)
  KVM_X86_OP(has_emulated_msr)
  KVM_X86_OP(vcpu_after_set_cpuid)
  KVM_X86_OP(is_vm_type_supported)

arch/x86/include/asm/kvm_host.h +1 -0

index 2870155ce6fb..50b39d0071ff 100644
@@ -1466,6 +1466,7 @@ struct kvm_x86_ops {
  	int (*hardware_enable)(void);
  	void (*hardware_disable)(void);
  	void (*hardware_unsetup)(void);
+	int (*offline_cpu)(void);
  	bool (*has_emulated_msr)(struct kvm *kvm, u32 index);
  	void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);

arch/x86/kvm/vmx/main.c +1 -0

index d01a946a18cf..0918d1e6d2f3 100644
@@ -67,6 +67,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
  	.name = "kvm_intel",
 
  	.hardware_unsetup = vt_hardware_unsetup,
+	.offline_cpu = tdx_offline_cpu,
  	.check_processor_compatibility = vmx_check_processor_compatibility,
 
  	.hardware_enable = vmx_hardware_enable,

arch/x86/kvm/vmx/tdx.c +39 -1

index ec88dde0d300..64229c3b3c5a 100644
@@ -42,6 +42,7 @@ static struct tdx_capabilities tdx_caps;
   */
  static DEFINE_MUTEX(tdx_lock);
  static struct mutex *tdx_mng_key_config_lock;
+static atomic_t nr_configured_hkid;
 
  static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid)
  {

@@ -222,7 +223,8 @@ void tdx_mmu_release_hkid(struct kvm *kvm)
  		pr_err("tdh_mng_key_freeid failed. HKID %d is leaked.\n",
  			kvm_tdx->hkid);
  		return;
-	}
+	} else
+		atomic_dec(&nr_configured_hkid);
 
  free_hkid:
  	tdx_hkid_free(kvm_tdx);

@@ -371,6 +373,8 @@ int tdx_vm_init(struct kvm *kvm)
  		if (ret)
  			break;
  	}
+	if (!ret)
+		atomic_inc(&nr_configured_hkid);
  	cpus_read_unlock();
  	free_cpumask_var(packages);
  	if (ret)

@@ -514,3 +518,37 @@ void tdx_hardware_unsetup(void)
  	/* kfree accepts NULL. */
  	kfree(tdx_mng_key_config_lock);
  }
+
+int tdx_offline_cpu(void)
+{
+	int curr_cpu = smp_processor_id();
+	cpumask_var_t packages;
+	int ret = 0;
+	int i;
+
+	if (!atomic_read(&nr_configured_hkid))
+		return 0;
+
+	/*
+	 * To reclaim hkid, need to call TDH.PHYMEM.PAGE.WBINVD on all packages.
+	 * If this is the last online cpu on the package, refuse offline.
+	 */
+	if (!zalloc_cpumask_var(&packages, GFP_KERNEL))
+		return -ENOMEM;
+
+	for_each_online_cpu(i) {
+		if (i != curr_cpu)
+			cpumask_set_cpu(topology_physical_package_id(i), packages);
+	}
+	if (!cpumask_test_cpu(topology_physical_package_id(curr_cpu), packages))
+		ret = -EBUSY;
+	free_cpumask_var(packages);
+	if (ret)
+		/*
+		 * Because it's hard for human operator to understand the
+		 * reason, warn it.
+		 */
+		pr_warn("TDX requires all packages to have an online CPU.  "
+			"Delete all TDs in order to offline all CPUs of a package.\n");
+	return ret;
+}

arch/x86/kvm/vmx/x86_ops.h +2 -0

index 95da978c9aa9..b2cb5786830a 100644
@@ -134,6 +134,7 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu);
  int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);
  bool tdx_is_vm_type_supported(unsigned long type);
  void tdx_hardware_unsetup(void);
+int tdx_offline_cpu(void);
 
  int tdx_vm_init(struct kvm *kvm);
  void tdx_mmu_release_hkid(struct kvm *kvm);

@@ -142,6 +143,7 @@ void tdx_vm_free(struct kvm *kvm);
  static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }
  static inline bool tdx_is_vm_type_supported(unsigned long type) { return false; }
  static inline void tdx_hardware_unsetup(void) {}
+static inline int tdx_offline_cpu(void) { return 0; }
 
  static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; }
  static inline void tdx_mmu_release_hkid(struct kvm *kvm) {}

arch/x86/kvm/x86.c +0 -6

index 4b22196cb12c..25c30c8c2d9b 100644
@@ -12337,16 +12337,23 @@ int kvm_arch_online_cpu(unsigned int cpu, int usage_count)
 
  int kvm_arch_offline_cpu(unsigned int cpu, int usage_count)
  {
-	if (usage_count) {
-		/*
-		 * arch callback kvm_arch_hardware_disable() assumes that
-		 * preemption is disabled for historical reason.  Disable
-		 * preemption until all arch callbacks are fixed.
-		 */

arch/x86/kvm/x86.c int kvm_arch_online_cpu(unsigned int cpu, int usage_count) · line 12345

Quoted 10 lines

arch/x86/kvm/x86.c +6 -4

-		preempt_disable();
-		hardware_disable(NULL);
-		preempt_enable();
-	}
+	int ret;
+
+	if (!usage_count)
+		return 0;
+
+	ret = static_call(kvm_x86_offline_cpu)();

Use static_call_cond instead?
Seems the new interface for x86 is only implemented for Intel.

Quoted 14 lines

arch/x86/kvm/x86.c +11 -0

+	if (ret)
+		return ret;
+
+	/*
+	 * arch callback kvm_arch_hardware_disable() assumes that preemption is
+	 * disabled for historical reason.  Disable preemption until all arch
+	 * callbacks are fixed.
+	 */
+	preempt_disable();
+	hardware_disable(NULL);
+	preempt_enable();
  	return 0;
  }

Isaku Yamahata

2022-11-03 02:01 UTC #47 lore raw

Re: [PATCH v10 017/108] KVM: TDX: Refuse to unplug the last cpu on the package

Quoted 18 lines · Binbin Wu

Quoted 18 lines

arch/x86/kvm/x86.c +1 ~~-10~~

index 4b22196cb12c..25c30c8c2d9b 100644
@@ -12337,16 +12337,23 @@ int kvm_arch_online_cpu(unsigned int cpu, int usage_count)
  int kvm_arch_offline_cpu(unsigned int cpu, int usage_count)
  {
-	if (usage_count) {
-		/*
-		 * arch callback kvm_arch_hardware_disable() assumes that
-		 * preemption is disabled for historical reason.  Disable
-		 * preemption until all arch callbacks are fixed.
-		 */
-		preempt_disable();
-		hardware_disable(NULL);
-		preempt_enable();
-	}
+	int ret;

Quoted 8 lines

arch/x86/kvm/x86.c int kvm_arch_online_cpu(unsigned int cpu, int usage_count) · line 12345
Quoted 5 lines
arch/x86/kvm/x86.c +5 -0
+
+	if (!usage_count)
+		return 0;
+
+	ret = static_call(kvm_x86_offline_cpu)();
Use static_call_cond instead?
Seems the new interface for x86 is only implemented for Intel.

Not needed because KVM_X86_OP_OPTIONAL_RET0(offline_cpu) is used.

Please remember

#define KVM_X86_OP_OPTIONAL_RET0(func) \
        static_call_update(kvm_x86_##func, (void *)kvm_x86_ops.func ? : \
                                           (void *)__static_call_return0);

Thanks,

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:24 UTC #48 lore raw

PATCH v10 18/108 KVM: TDX: x86: Add ioctl to get TDX systemwide parameters

From: Sean Christopherson <[email protected]>

Implement a system-scoped ioctl to get system-wide parameters for TDX.

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/kvm-x86-ops.h    |  1 +
 arch/x86/include/asm/kvm_host.h       |  1 +
 arch/x86/include/uapi/asm/kvm.h       | 48 +++++++++++++++++++++++++++
 arch/x86/kvm/vmx/main.c               |  2 ++
 arch/x86/kvm/vmx/tdx.c                | 46 +++++++++++++++++++++++++
 arch/x86/kvm/vmx/x86_ops.h            |  2 ++
 arch/x86/kvm/x86.c                    |  6 ++++
 tools/arch/x86/include/uapi/asm/kvm.h | 48 +++++++++++++++++++++++++++
 8 files changed, 154 insertions(+)

arch/x86/include/asm/kvm-x86-ops.h +1 -0

index 0ceb8e58a6c0..4425564647cb 100644
@@ -118,6 +118,7 @@ KVM_X86_OP(smi_allowed)
 KVM_X86_OP(enter_smm)
 KVM_X86_OP(leave_smm)
 KVM_X86_OP(enable_smi_window)
+KVM_X86_OP_OPTIONAL(dev_mem_enc_ioctl)
 KVM_X86_OP_OPTIONAL(mem_enc_ioctl)
 KVM_X86_OP_OPTIONAL(mem_enc_register_region)
 KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)

arch/x86/include/asm/kvm_host.h +1 -0

index 50b39d0071ff..1fced310ec63 100644
@@ -1626,6 +1626,7 @@ struct kvm_x86_ops {
 	int (*leave_smm)(struct kvm_vcpu *vcpu, const char *smstate);
 	void (*enable_smi_window)(struct kvm_vcpu *vcpu);
 
+	int (*dev_mem_enc_ioctl)(void __user *argp);
 	int (*mem_enc_ioctl)(struct kvm *kvm, void __user *argp);
 	int (*mem_enc_register_region)(struct kvm *kvm, struct kvm_enc_region *argp);
 	int (*mem_enc_unregister_region)(struct kvm *kvm, struct kvm_enc_region *argp);

arch/x86/include/uapi/asm/kvm.h +48 -0

52 lines not on this page · raw message

arch/x86/kvm/vmx/main.c +2 -0

index 0918d1e6d2f3..aedba5acb8eb 100644
@@ -203,6 +203,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.complete_emulated_msr = kvm_complete_insn_gp,
 
 	.vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector,
+
+	.dev_mem_enc_ioctl = tdx_dev_ioctl,
 };
 
 struct kvm_x86_init_ops vt_init_ops __initdata = {

arch/x86/kvm/vmx/tdx.c +46 -0

52 lines not on this page · raw message

arch/x86/kvm/vmx/x86_ops.h +2 -0

index b2cb5786830a..057f2be3d818 100644
@@ -135,6 +135,7 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops);
 bool tdx_is_vm_type_supported(unsigned long type);
 void tdx_hardware_unsetup(void);
 int tdx_offline_cpu(void);
+int tdx_dev_ioctl(void __user *argp);
 
 int tdx_vm_init(struct kvm *kvm);
 void tdx_mmu_release_hkid(struct kvm *kvm);

@@ -144,6 +145,7 @@ static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }
 static inline bool tdx_is_vm_type_supported(unsigned long type) { return false; }
 static inline void tdx_hardware_unsetup(void) {}
 static inline int tdx_offline_cpu(void) { return 0; }
+static inline int tdx_dev_ioctl(void __user *argp) { return -EOPNOTSUPP; };
 
 static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; }
 static inline void tdx_mmu_release_hkid(struct kvm *kvm) {}

arch/x86/kvm/x86.c +6 -0

index 25c30c8c2d9b..ddcbbcf13a55 100644
@@ -4678,6 +4678,12 @@ long kvm_arch_dev_ioctl(struct file *filp,
 		r = kvm_x86_dev_has_attr(&attr);
 		break;
 	}
+	case KVM_MEMORY_ENCRYPT_OP:
+		r = -EINVAL;
+		if (!kvm_x86_ops.dev_mem_enc_ioctl)
+			goto out;
+		r = static_call(kvm_x86_dev_mem_enc_ioctl)(argp);
+		break;
 	default:
 		r = -EINVAL;
 		break;

tools/arch/x86/include/uapi/asm/kvm.h +48 -0

52 lines not on this page · raw message

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #49 lore raw

PATCH v10 19/108 KVM: TDX: Add place holder for TDX VM specific mem_enc_op ioctl

From: Isaku Yamahata <[email protected]>

Add a place holder function for TDX specific VM-scoped ioctl as mem_enc_op. TDX specific sub-commands will be added to retrieve/pass TDX specific parameters.

KVM_MEMORY_ENCRYPT_OP was introduced for VM-scoped operations specific for guest state-protected VM. It defined subcommands for technology-specific operations under KVM_MEMORY_ENCRYPT_OP. Despite its name, the subcommands are not limited to memory encryption, but various technology-specific operations are defined. It's natural to repurpose KVM_MEMORY_ENCRYPT_OP for TDX specific operations and define subcommands.

TDX requires VM-scoped TDX-specific operations for device model, for example, qemu. Getting system-wide parameters, TDX-specific VM initialization.

---
 arch/x86/kvm/vmx/main.c    |  9 +++++++++
 arch/x86/kvm/vmx/tdx.c     | 26 ++++++++++++++++++++++++++
 arch/x86/kvm/vmx/x86_ops.h |  4 ++++
 3 files changed, 39 insertions(+)

arch/x86/kvm/vmx/main.c +9 -0

index aedba5acb8eb..b4e4c6c677f6 100644
@@ -63,6 +63,14 @@ static void vt_vm_free(struct kvm *kvm)
 		return tdx_vm_free(kvm);
 }
 
+static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp)
+{
+	if (!is_td(kvm))
+		return -ENOTTY;
+
+	return tdx_vm_ioctl(kvm, argp);
+}
+
 struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = "kvm_intel",

@@ -205,6 +213,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector,
 
 	.dev_mem_enc_ioctl = tdx_dev_ioctl,
+	.mem_enc_ioctl = vt_mem_enc_ioctl,
 };
 
 struct kvm_x86_init_ops vt_init_ops __initdata = {

arch/x86/kvm/vmx/tdx.c +26 -0

index 5a3ed8217a54..d77709a6da51 100644
@@ -477,6 +477,32 @@ int tdx_dev_ioctl(void __user *argp)
 	return 0;
 }
 
+int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)
+{
+	struct kvm_tdx_cmd tdx_cmd;
+	int r;
+
+	if (copy_from_user(&tdx_cmd, argp, sizeof(struct kvm_tdx_cmd)))
+		return -EFAULT;
+	if (tdx_cmd.error || tdx_cmd.unused)
+		return -EINVAL;
+
+	mutex_lock(&kvm->lock);
+
+	switch (tdx_cmd.id) {
+	default:
+		r = -EINVAL;
+		goto out;
+	}
+
+	if (copy_to_user(argp, &tdx_cmd, sizeof(struct kvm_tdx_cmd)))
+		r = -EFAULT;
+
+out:
+	mutex_unlock(&kvm->lock);
+	return r;
+}
+
 static int __init tdx_module_setup(void)
 {
 	const struct tdsysinfo_struct *tdsysinfo;

arch/x86/kvm/vmx/x86_ops.h +4 -0

index 057f2be3d818..93ffe2deb8e8 100644
@@ -140,6 +140,8 @@ int tdx_dev_ioctl(void __user *argp);
 int tdx_vm_init(struct kvm *kvm);
 void tdx_mmu_release_hkid(struct kvm *kvm);
 void tdx_vm_free(struct kvm *kvm);
+
+int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);
 #else
 static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }
 static inline bool tdx_is_vm_type_supported(unsigned long type) { return false; }

@@ -151,6 +153,8 @@ static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; }
 static inline void tdx_mmu_release_hkid(struct kvm *kvm) {}
 static inline void tdx_flush_shadow_all_private(struct kvm *kvm) {}
 static inline void tdx_vm_free(struct kvm *kvm) {}
+
+static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPNOTSUPP; }
 #endif
 
 #endif /* __KVM_X86_VMX_X86_OPS_H */

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #50 lore raw

PATCH v10 20/108 KVM: Support KVM_CAP_MAX_VCPUS for KVM_ENABLE_CAP

From: Isaku Yamahata <[email protected]>

TDX attestation includes the maximum number of vcpu that the guest can accommodate. For that, the maximum number of vcpu needs to be specified instead of constant, KVM_MAX_VCPUS. Make KVM_ENABLE_CAP support KVM_CAP_MAX_VCPUS.

---
 virt/kvm/kvm_main.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

virt/kvm/kvm_main.c +21 -0

index b9c270f97c88..3b05a3396f89 100644
@@ -4967,6 +4967,27 @@ static int kvm_vm_ioctl_enable_cap_generic(struct kvm *kvm,
 	case KVM_CAP_DIRTY_LOG_RING:
 	case KVM_CAP_DIRTY_LOG_RING_ACQ_REL:
 		return kvm_vm_ioctl_enable_dirty_log_ring(kvm, cap->args[0]);
+	case KVM_CAP_MAX_VCPUS: {
+		int r;
+
+		if (cap->flags || cap->args[0] == 0)
+			return -EINVAL;
+		if (cap->args[0] >  kvm_vm_ioctl_check_extension(kvm, KVM_CAP_MAX_VCPUS))
+			return -E2BIG;
+
+		mutex_lock(&kvm->lock);
+		/* Only decreasing is allowed. */
+		if (cap->args[0] > kvm->max_vcpus)
+			r = -E2BIG;
+		else if (kvm->created_vcpus)
+			r = -EBUSY;
+		else {
+			kvm->max_vcpus = cap->args[0];
+			r = 0;
+		}
+		mutex_unlock(&kvm->lock);
+		return r;
+	}
 	default:
 		return kvm_vm_ioctl_enable_cap(kvm, cap);
 	}

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #51 lore raw

PATCH v10 21/108 KVM: TDX: initialize VM with TDX specific parameters

From: Xiaoyao Li <[email protected]>

TDX requires additional parameters for TDX VM for confidential execution to protect its confidentiality of its memory contents and its CPU state from any other software, including VMM. When creating guest TD VM before creating vcpu, the number of vcpu, TSC frequency (that is same among vcpus. and it can't be changed.) CPUIDs which is emulated by the TDX module. It means guest can trust those CPUIDs. and sha384 values for measurement.

Add new subcommand, KVM_TDX_INIT_VM, to pass parameters for TDX guest. It assigns encryption key to the TDX guest for memory encryption. TDX encrypts memory per-guest bases. It assigns device model passes per-VM parameters for the TDX guest. The maximum number of vcpus, tsc frequency (TDX guest has fised VM-wide TSC frequency. not per-vcpu. The TDX guest can not change it.), attributes (production or debug), available extended features (which is reflected into guest XCR0, IA32_XSS MSR), cpuids, sha384 measurements, and etc.

This subcommand is called before creating vcpu and KVM_SET_CPUID2, i.e. cpuids configurations aren't available yet. So CPUIDs configuration values needs to be passed in struct kvm_init_vm. It's device model responsibility to make this cpuid config for KVM_TDX_INIT_VM and KVM_SET_CPUID2.

Signed-off-by Xiaoyao Li <[email protected]>

---
 arch/x86/include/asm/tdx.h            |   3 +
 arch/x86/include/uapi/asm/kvm.h       |  31 +++
 arch/x86/kvm/vmx/tdx.c                | 296 ++++++++++++++++++++++----
 arch/x86/kvm/vmx/tdx.h                |  22 ++
 tools/arch/x86/include/uapi/asm/kvm.h |  33 +++
 5 files changed, 347 insertions(+), 38 deletions(-)

arch/x86/include/asm/tdx.h +3 -0

index cd304d323d33..05ac4bfc8f8a 100644
@@ -131,6 +131,9 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,
 #endif /* CONFIG_INTEL_TDX_GUEST && CONFIG_KVM_GUEST */
 
 #ifdef CONFIG_INTEL_TDX_HOST
+
+/* -1 indicates CPUID leaf with no sub-leaves. */
+#define TDX_CPUID_NO_SUBLEAF	((u32)-1)
 struct tdx_cpuid_config {
 	u32	leaf;
 	u32	sub_leaf;

arch/x86/include/uapi/asm/kvm.h +31 -0

index 2ad9666e02a5..26661879c031 100644
@@ -538,6 +538,7 @@ struct kvm_pmu_event_filter {
 /* Trust Domain eXtension sub-ioctl() commands. */
 enum kvm_tdx_cmd_id {
 	KVM_TDX_CAPABILITIES = 0,
+	KVM_TDX_INIT_VM,
 
 	KVM_TDX_CMD_NR_MAX,
 };

@@ -583,4 +584,34 @@ struct kvm_tdx_capabilities {
 	struct kvm_tdx_cpuid_config cpuid_configs[0];
 };
 
+struct kvm_tdx_init_vm {
+	__u64 attributes;
+	__u64 mrconfigid[6];	/* sha384 digest */
+	__u64 mrowner[6];	/* sha384 digest */
+	__u64 mrownerconfig[6];	/* sha348 digest */
+	union {
+		/*
+		 * KVM_TDX_INIT_VM is called before vcpu creation, thus before
+		 * KVM_SET_CPUID2.  CPUID configurations needs to be passed.
+		 *
+		 * This configuration supersedes KVM_SET_CPUID{,2}.
+		 * The user space VMM, e.g. qemu, should make them consistent
+		 * with this values.
+		 * sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(256)
+		 * = 8KB.
+		 */
+		struct {
+			struct kvm_cpuid2 cpuid;
+			/* 8KB with KVM_MAX_CPUID_ENTRIES. */
+			struct kvm_cpuid_entry2 entries[];
+		};
+		/*
+		 * For future extensibility.
+		 * The size(struct kvm_tdx_init_vm) = 16KB.
+		 * This should be enough given sizeof(TD_PARAMS) = 1024
+		 */
+		__u64 reserved[2029];
+	};
+};
+
 #endif /* _ASM_X86_KVM_H */

arch/x86/kvm/vmx/tdx.c +258 ~~-38~~

330 lines not on this page · raw message

arch/x86/kvm/vmx/tdx.h +22 -0

index 938314635b47..ff0ea9cad347 100644
@@ -18,7 +18,11 @@ struct kvm_tdx {
 	struct tdx_td_page tdr;
 	struct tdx_td_page *tdcs;
 
+	u64 attributes;
+	u64 xfam;
 	int hkid;
+
+	u64 tsc_offset;
 };
 
 struct vcpu_tdx {

@@ -48,6 +52,11 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu)
 	return container_of(vcpu, struct vcpu_tdx, vcpu);
 }
 
+static inline bool is_td_initialized(struct kvm *kvm)
+{
+	return to_kvm_tdx(kvm)->hkid > 0;
+}
+
 static __always_inline void tdvps_vmcs_check(u32 field, u8 bits)
 {
 #define VMCS_ENC_ACCESS_TYPE_MASK	0x1UL

@@ -148,6 +157,19 @@ TDX_BUILD_TDVPS_ACCESSORS(64, VMCS, vmcs);
 TDX_BUILD_TDVPS_ACCESSORS(64, STATE_NON_ARCH, state_non_arch);
 TDX_BUILD_TDVPS_ACCESSORS(8, MANAGEMENT, management);
 
+static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field)
+{
+	struct tdx_module_output out;
+	u64 err;
+
+	err = tdh_mng_rd(kvm_tdx->tdr.pa, TDCS_EXEC(field), &out);
+	if (unlikely(err)) {
+		pr_err("TDH_MNG_RD[EXEC.0x%x] failed: 0x%llx\n", field, err);
+		return 0;
+	}
+	return out.r8;
+}
+
 #else
 struct kvm_tdx {
 	struct kvm kvm;

tools/arch/x86/include/uapi/asm/kvm.h +33 -0

index 2ad9666e02a5..531a0033e530 100644
@@ -538,6 +538,7 @@ struct kvm_pmu_event_filter {
 /* Trust Domain eXtension sub-ioctl() commands. */
 enum kvm_tdx_cmd_id {
 	KVM_TDX_CAPABILITIES = 0,
+	KVM_TDX_INIT_VM,
 
 	KVM_TDX_CMD_NR_MAX,
 };

@@ -583,4 +584,36 @@ struct kvm_tdx_capabilities {
 	struct kvm_tdx_cpuid_config cpuid_configs[0];
 };
 
+struct kvm_tdx_init_vm {
+	__u64 attributes;
+	__u32 max_vcpus;
+	__u32 padding;
+	__u64 mrconfigid[6];    /* sha384 digest */
+	__u64 mrowner[6];       /* sha384 digest */
+	__u64 mrownerconfig[6]; /* sha348 digest */
+	union {
+		/*
+		 * KVM_TDX_INIT_VM is called before vcpu creation, thus before
+		 * KVM_SET_CPUID2.  CPUID configurations needs to be passed.
+		 *
+		 * This configuration supersedes KVM_SET_CPUID{,2}.
+		 * The user space VMM, e.g. qemu, should make them consistent
+		 * with this values.
+		 * sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(256)
+		 * = 8KB.
+		 */
+		struct {
+			struct kvm_cpuid2 cpuid;
+			/* 8KB with KVM_MAX_CPUID_ENTRIES. */
+			struct kvm_cpuid_entry2 entries[];
+		};
+		/*
+		 * For future extensibility.
+		 * The size(struct kvm_tdx_init_vm) = 16KB.
+		 * This should be enough given sizeof(TD_PARAMS) = 1024
+		 */
+		__u64 reserved[2028];
+	};
+};
+
 #endif /* _ASM_X86_KVM_H */

Signature · 1 line

2.25.1

Wang, Lei

2022-11-16 05:34 UTC #52 lore raw

Re: [PATCH v10 021/108] KVM: TDX: initialize VM with TDX specific parameters

Quoted 355 lines · On 10/30/2022 2:22 PM, [email protected] wrote:

From: Xiaoyao Li <[email protected]>

Signed-off-by Xiaoyao Li <[email protected]>

---
 arch/x86/include/asm/tdx.h            |   3 +
 arch/x86/include/uapi/asm/kvm.h       |  31 +++
 arch/x86/kvm/vmx/tdx.c                | 296 ++++++++++++++++++++++----
 arch/x86/kvm/vmx/tdx.h                |  22 ++
 tools/arch/x86/include/uapi/asm/kvm.h |  33 +++
 5 files changed, 347 insertions(+), 38 deletions(-)

arch/x86/include/asm/tdx.h +3 -0

index cd304d323d33..05ac4bfc8f8a 100644
@@ -131,6 +131,9 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,
 #endif /* CONFIG_INTEL_TDX_GUEST && CONFIG_KVM_GUEST */
 
 #ifdef CONFIG_INTEL_TDX_HOST
+
+/* -1 indicates CPUID leaf with no sub-leaves. */
+#define TDX_CPUID_NO_SUBLEAF	((u32)-1)
 struct tdx_cpuid_config {
 	u32	leaf;
 	u32	sub_leaf;

arch/x86/include/uapi/asm/kvm.h +31 -0

index 2ad9666e02a5..26661879c031 100644
@@ -538,6 +538,7 @@ struct kvm_pmu_event_filter {
 /* Trust Domain eXtension sub-ioctl() commands. */
 enum kvm_tdx_cmd_id {
 	KVM_TDX_CAPABILITIES = 0,
+	KVM_TDX_INIT_VM,
 
 	KVM_TDX_CMD_NR_MAX,
 };

@@ -583,4 +584,34 @@ struct kvm_tdx_capabilities {
 	struct kvm_tdx_cpuid_config cpuid_configs[0];
 };
 
+struct kvm_tdx_init_vm {
+	__u64 attributes;
+	__u64 mrconfigid[6];	/* sha384 digest */
+	__u64 mrowner[6];	/* sha384 digest */
+	__u64 mrownerconfig[6];	/* sha348 digest */
+	union {
+		/*
+		 * KVM_TDX_INIT_VM is called before vcpu creation, thus before
+		 * KVM_SET_CPUID2.  CPUID configurations needs to be passed.
+		 *
+		 * This configuration supersedes KVM_SET_CPUID{,2}.
+		 * The user space VMM, e.g. qemu, should make them consistent
+		 * with this values.
+		 * sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(256)
+		 * = 8KB.
+		 */
+		struct {
+			struct kvm_cpuid2 cpuid;
+			/* 8KB with KVM_MAX_CPUID_ENTRIES. */
+			struct kvm_cpuid_entry2 entries[];
+		};
+		/*
+		 * For future extensibility.
+		 * The size(struct kvm_tdx_init_vm) = 16KB.
+		 * This should be enough given sizeof(TD_PARAMS) = 1024
+		 */
+		__u64 reserved[2029];
+	};
+};
+
 #endif /* _ASM_X86_KVM_H */

arch/x86/kvm/vmx/tdx.c +217 ~~-17~~

index d77709a6da51..54045e0576e7 100644
@@ -284,6 +284,205 @@ static int tdx_do_tdh_mng_key_config(void *param)
 int tdx_vm_init(struct kvm *kvm)
 {
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+
+	kvm_tdx->hkid = -1;
+
+	/*
+	 * This function initializes only KVM software construct.  It doesn't
+	 * initialize TDX stuff, e.g. TDCS, TDR, TDCX, HKID etc.
+	 * It is handled by KVM_TDX_INIT_VM, __tdx_td_init().
+	 */
+
+	return 0;
+}
+
+int tdx_dev_ioctl(void __user *argp)
+{
+	struct kvm_tdx_capabilities __user *user_caps;
+	struct kvm_tdx_capabilities caps;
+	struct kvm_tdx_cmd cmd;
+
+	BUILD_BUG_ON(sizeof(struct kvm_tdx_cpuid_config) !=
+		     sizeof(struct tdx_cpuid_config));
+
+	if (copy_from_user(&cmd, argp, sizeof(cmd)))
+		return -EFAULT;
+	if (cmd.flags || cmd.error || cmd.unused)
+		return -EINVAL;
+	/*
+	 * Currently only KVM_TDX_CAPABILITIES is defined for system-scoped
+	 * mem_enc_ioctl().
+	 */
+	if (cmd.id != KVM_TDX_CAPABILITIES)
+		return -EINVAL;
+
+	user_caps = (void __user *)cmd.data;
+	if (copy_from_user(&caps, user_caps, sizeof(caps)))
+		return -EFAULT;
+
+	if (caps.nr_cpuid_configs < tdx_caps.nr_cpuid_configs)
+		return -E2BIG;
+
+	caps = (struct kvm_tdx_capabilities) {
+		.attrs_fixed0 = tdx_caps.attrs_fixed0,
+		.attrs_fixed1 = tdx_caps.attrs_fixed1,
+		.xfam_fixed0 = tdx_caps.xfam_fixed0,
+		.xfam_fixed1 = tdx_caps.xfam_fixed1,
+		.nr_cpuid_configs = tdx_caps.nr_cpuid_configs,
+		.padding = 0,
+	};
+
+	if (copy_to_user(user_caps, &caps, sizeof(caps)))
+		return -EFAULT;
+	if (copy_to_user(user_caps->cpuid_configs, &tdx_caps.cpuid_configs,
+			 tdx_caps.nr_cpuid_configs *
+			 sizeof(struct tdx_cpuid_config)))
+		return -EFAULT;
+
+	return 0;
+}
+
+/*
+ * cpuid entry lookup in TDX cpuid config way.
+ * The difference is how to specify index(subleaves).
+ * Specify index to TDX_CPUID_NO_SUBLEAF for CPUID leaf with no-subleaves.
+ */
+static const struct kvm_cpuid_entry2 *tdx_find_cpuid_entry(const struct kvm_cpuid2 *cpuid,
+							   u32 function, u32 index)
+{
+	int i;
+
+	/* In TDX CPU CONFIG, TDX_CPUID_NO_SUBLEAF means index = 0. */
+	if (index == TDX_CPUID_NO_SUBLEAF)
+		index = 0;
+
+	for (i = 0; i < cpuid->nent; i++) {
+		const struct kvm_cpuid_entry2 *e = &cpuid->entries[i];
+
+		if (e->function == function &&
+		    (e->index == index ||
+		     !(e->flags & KVM_CPUID_FLAG_SIGNIFCANT_INDEX)))
+			return e;
+	}
+	return NULL;
+}
+
+static int setup_tdparams(struct kvm *kvm, struct td_params *td_params,
+			struct kvm_tdx_init_vm *init_vm)
+{
+	const struct kvm_cpuid2 *cpuid = &init_vm->cpuid;
+	const struct kvm_cpuid_entry2 *entry;
+	u64 guest_supported_xcr0;
+	u64 guest_supported_xss;
+	int max_pa;
+	int i;
+
+	if (kvm->created_vcpus)
+		return -EBUSY;
+	td_params->max_vcpus = kvm->max_vcpus;
+	td_params->attributes = init_vm->attributes;
+	if (td_params->attributes & TDX_TD_ATTRIBUTE_PERFMON) {
+		/*
+		 * TODO: save/restore PMU related registers around TDENTER.
+		 * Once it's done, remove this guard.
+		 */
+		pr_warn("TD doesn't support perfmon yet. KVM needs to save/restore "
+			"host perf registers properly.\n");
+		return -EOPNOTSUPP;
+	}
+
+	for (i = 0; i < tdx_caps.nr_cpuid_configs; i++) {
+		const struct tdx_cpuid_config *config = &tdx_caps.cpuid_configs[i];
+		const struct kvm_cpuid_entry2 *entry =
+			tdx_find_cpuid_entry(cpuid, config->leaf, config->sub_leaf);
+		struct tdx_cpuid_value *value = &td_params->cpuid_values[i];
+
+		if (!entry)
+			continue;
+
+		value->eax = entry->eax & config->eax;
+		value->ebx = entry->ebx & config->ebx;
+		value->ecx = entry->ecx & config->ecx;
+		value->edx = entry->edx & config->edx;
+	}
+
+	max_pa = 36;
+	entry = tdx_find_cpuid_entry(cpuid, 0x80000008, 0);
+	if (entry)
+		max_pa = entry->eax & 0xff;
+
+	td_params->eptp_controls = VMX_EPTP_MT_WB;
+	/*
+	 * No CPU supports 4-level && max_pa > 48.
+	 * "5-level paging and 5-level EPT" section 4.1 4-level EPT
+	 * "4-level EPT is limited to translating 48-bit guest-physical
+	 *  addresses."
+	 * cpu_has_vmx_ept_5levels() check is just in case.
+	 */
+	if (cpu_has_vmx_ept_5levels() && max_pa > 48) {
+		td_params->eptp_controls |= VMX_EPTP_PWL_5;
+		td_params->exec_controls |= TDX_EXEC_CONTROL_MAX_GPAW;
+	} else {
+		td_params->eptp_controls |= VMX_EPTP_PWL_4;
+	}
+
+	/* Setup td_params.xfam */
+	entry = tdx_find_cpuid_entry(cpuid, 0xd, 0);
+	if (entry)
+		guest_supported_xcr0 = (entry->eax | ((u64)entry->edx << 32));
+	else
+		guest_supported_xcr0 = 0;
+	guest_supported_xcr0 &= kvm_caps.supported_xcr0;
+
+	entry = tdx_find_cpuid_entry(cpuid, 0xd, 1);
+	if (entry)
+		guest_supported_xss = (entry->ecx | ((u64)entry->edx << 32));
+	else
+		guest_supported_xss = 0;
+	/* PT can be exposed to TD guest regardless of KVM's XSS support */
+	guest_supported_xss &= (kvm_caps.supported_xss | XFEATURE_MASK_PT);
+
+	td_params->xfam = guest_supported_xcr0 | guest_supported_xss;
+	if (td_params->xfam & XFEATURE_MASK_LBR) {
+		/*
+		 * TODO: once KVM supports LBR(save/restore LBR related
+		 * registers around TDENTER), remove this guard.
+		 */
+		pr_warn("TD doesn't support LBR yet. KVM needs to save/restore "
+			"IA32_LBR_DEPTH properly.\n");
+		return -EOPNOTSUPP;
+	}
+
+	if (td_params->xfam & XFEATURE_MASK_XTILE) {
+		/*
+		 * TODO: once KVM supports AMX(save/restore AMX related
+		 * registers around TDENTER), remove this guard.
+		 */
+		pr_warn("TD doesn't support AMX yet. KVM needs to save/restore "
+			"IA32_XFD, IA32_XFD_ERR properly.\n");
+		return -EOPNOTSUPP;
+	}
+
+	td_params->tsc_frequency =
+		TDX_TSC_KHZ_TO_25MHZ(kvm->arch.default_tsc_khz);
+
+#define MEMCPY_SAME_SIZE(dst, src)				\
+	do {							\
+		BUILD_BUG_ON(sizeof(dst) != sizeof(src));	\
+		memcpy((dst), (src), sizeof(dst));		\
+	} while (0)
+
+	MEMCPY_SAME_SIZE(td_params->mrconfigid, init_vm->mrconfigid);
+	MEMCPY_SAME_SIZE(td_params->mrowner, init_vm->mrowner);
+	MEMCPY_SAME_SIZE(td_params->mrownerconfig, init_vm->mrownerconfig);
+
+	return 0;
+}
+
+static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	struct tdx_module_output out;
 	cpumask_var_t packages;
 	int ret, i;
 	u64 err;

@@ -390,10 +589,13 @@ int tdx_vm_init(struct kvm *kvm)
 		tdx_mark_td_page_added(&kvm_tdx->tdcs[i]);
 	}
 
-	/*
-	 * Note, TDH_MNG_INIT cannot be invoked here.  TDH_MNG_INIT requires a dedicated
-	 * ioctl() to define the configure CPUID values for the TD.
-	 */
+	err = tdh_mng_init(kvm_tdx->tdr.pa, __pa(td_params), &out);
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_MNG_INIT, err, &out);
+		ret = -EIO;
+		goto teardown;
+	}
+
 	return 0;
 
 	/*

@@ -431,50 +633,65 @@ int tdx_vm_init(struct kvm *kvm)
 	return ret;
 }
 
-int tdx_dev_ioctl(void __user *argp)
+static int tdx_td_init(struct kvm *kvm, struct kvm_tdx_cmd *cmd)
 {
-	struct kvm_tdx_capabilities __user *user_caps;
-	struct kvm_tdx_capabilities caps;
-	struct kvm_tdx_cmd cmd;
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	struct kvm_tdx_init_vm *init_vm = NULL;
+	struct td_params *td_params = NULL;
+	void *entries_end;
+	int ret;
 
-	BUILD_BUG_ON(sizeof(struct kvm_tdx_cpuid_config) !=
-		     sizeof(struct tdx_cpuid_config));
+	BUILD_BUG_ON(sizeof(*init_vm) != 16 * 1024);
+	BUILD_BUG_ON((sizeof(*init_vm) - offsetof(typeof(*init_vm), entries)) /
+		     sizeof(init_vm->entries[0]) < KVM_MAX_CPUID_ENTRIES);
+	BUILD_BUG_ON(sizeof(struct td_params) != 1024);
 
-	if (copy_from_user(&cmd, argp, sizeof(cmd)))
-		return -EFAULT;
-	if (cmd.flags || cmd.error || cmd.unused)
+	if (is_td_initialized(kvm))
 		return -EINVAL;
-	/*
-	 * Currently only KVM_TDX_CAPABILITIES is defined for system-scoped
-	 * mem_enc_ioctl().
-	 */

arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 656

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c +4 -4

-	if (cmd.id != KVM_TDX_CAPABILITIES)
+
+	if (cmd->flags)
 		return -EINVAL;
 
-	user_caps = (void __user *)cmd.data;
-	if (copy_from_user(&caps, user_caps, sizeof(caps)))
-		return -EFAULT;
+	init_vm = kzalloc(sizeof(*init_vm), GFP_KERNEL);
+	if (copy_from_user(init_vm, (void __user *)cmd->data, sizeof(*init_vm))) {

Pointer 'init_vm' returned from call to function 'kzalloc' may be NULL and will be dereferenced at this line.

Maybe a NULL-check here is needed?

Quoted 165 lines

arch/x86/kvm/vmx/tdx.c +37 ~~-17~~

+		ret = -EFAULT;
+		goto out;
+	}
 
-	if (caps.nr_cpuid_configs < tdx_caps.nr_cpuid_configs)
-		return -E2BIG;
+	ret = -EINVAL;
+	if (init_vm->cpuid.padding)
+		goto out;
+	/* init_vm->entries shouldn't overrun. */
+	entries_end = init_vm->entries + init_vm->cpuid.nent;
+	if (entries_end > (void *)(init_vm + 1))
+		goto out;
+	/* Unused part must be zero. */
+	if (memchr_inv(entries_end, 0, (void *)(init_vm + 1) - entries_end))
+		goto out;
 
-	caps = (struct kvm_tdx_capabilities) {
-		.attrs_fixed0 = tdx_caps.attrs_fixed0,
-		.attrs_fixed1 = tdx_caps.attrs_fixed1,
-		.xfam_fixed0 = tdx_caps.xfam_fixed0,
-		.xfam_fixed1 = tdx_caps.xfam_fixed1,
-		.nr_cpuid_configs = tdx_caps.nr_cpuid_configs,
-		.padding = 0,
-	};
+	td_params = kzalloc(sizeof(struct td_params), GFP_KERNEL);
+	if (!td_params) {
+		ret = -ENOMEM;
+		goto out;
+	}
 
-	if (copy_to_user(user_caps, &caps, sizeof(caps)))
-		return -EFAULT;
-	if (copy_to_user(user_caps->cpuid_configs, &tdx_caps.cpuid_configs,
-			 tdx_caps.nr_cpuid_configs *
-			 sizeof(struct tdx_cpuid_config)))
-		return -EFAULT;
+	ret = setup_tdparams(kvm, td_params, init_vm);
+	if (ret)
+		goto out;
 
-	return 0;
+	ret = __tdx_td_init(kvm, td_params);
+	if (ret)
+		goto out;
+
+	kvm_tdx->tsc_offset = td_tdcs_exec_read64(kvm_tdx, TD_TDCS_EXEC_TSC_OFFSET);
+	kvm_tdx->attributes = td_params->attributes;
+	kvm_tdx->xfam = td_params->xfam;
+
+out:
+	/* kfree() accepts NULL. */
+	kfree(init_vm);
+	kfree(td_params);
+	return ret;
 }
 
 int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)

@@ -490,6 +707,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)
 	mutex_lock(&kvm->lock);
 
 	switch (tdx_cmd.id) {
+	case KVM_TDX_INIT_VM:
+		r = tdx_td_init(kvm, &tdx_cmd);
+		break;
 	default:
 		r = -EINVAL;
 		goto out;

arch/x86/kvm/vmx/tdx.h +22 -0

index 938314635b47..ff0ea9cad347 100644
@@ -18,7 +18,11 @@ struct kvm_tdx {
 	struct tdx_td_page tdr;
 	struct tdx_td_page *tdcs;
 
+	u64 attributes;
+	u64 xfam;
 	int hkid;
+
+	u64 tsc_offset;
 };
 
 struct vcpu_tdx {

@@ -48,6 +52,11 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu)
 	return container_of(vcpu, struct vcpu_tdx, vcpu);
 }
 
+static inline bool is_td_initialized(struct kvm *kvm)
+{
+	return to_kvm_tdx(kvm)->hkid > 0;
+}
+
 static __always_inline void tdvps_vmcs_check(u32 field, u8 bits)
 {
 #define VMCS_ENC_ACCESS_TYPE_MASK	0x1UL

@@ -148,6 +157,19 @@ TDX_BUILD_TDVPS_ACCESSORS(64, VMCS, vmcs);
 TDX_BUILD_TDVPS_ACCESSORS(64, STATE_NON_ARCH, state_non_arch);
 TDX_BUILD_TDVPS_ACCESSORS(8, MANAGEMENT, management);
 
+static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field)
+{
+	struct tdx_module_output out;
+	u64 err;
+
+	err = tdh_mng_rd(kvm_tdx->tdr.pa, TDCS_EXEC(field), &out);
+	if (unlikely(err)) {
+		pr_err("TDH_MNG_RD[EXEC.0x%x] failed: 0x%llx\n", field, err);
+		return 0;
+	}
+	return out.r8;
+}
+
 #else
 struct kvm_tdx {
 	struct kvm kvm;

tools/arch/x86/include/uapi/asm/kvm.h +33 -0

index 2ad9666e02a5..531a0033e530 100644
@@ -538,6 +538,7 @@ struct kvm_pmu_event_filter {
 /* Trust Domain eXtension sub-ioctl() commands. */
 enum kvm_tdx_cmd_id {
 	KVM_TDX_CAPABILITIES = 0,
+	KVM_TDX_INIT_VM,
 
 	KVM_TDX_CMD_NR_MAX,
 };

@@ -583,4 +584,36 @@ struct kvm_tdx_capabilities {
 	struct kvm_tdx_cpuid_config cpuid_configs[0];
 };
 
+struct kvm_tdx_init_vm {
+	__u64 attributes;
+	__u32 max_vcpus;
+	__u32 padding;
+	__u64 mrconfigid[6];    /* sha384 digest */
+	__u64 mrowner[6];       /* sha384 digest */
+	__u64 mrownerconfig[6]; /* sha348 digest */
+	union {
+		/*
+		 * KVM_TDX_INIT_VM is called before vcpu creation, thus before
+		 * KVM_SET_CPUID2.  CPUID configurations needs to be passed.
+		 *
+		 * This configuration supersedes KVM_SET_CPUID{,2}.
+		 * The user space VMM, e.g. qemu, should make them consistent
+		 * with this values.
+		 * sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(256)
+		 * = 8KB.
+		 */
+		struct {
+			struct kvm_cpuid2 cpuid;
+			/* 8KB with KVM_MAX_CPUID_ENTRIES. */
+			struct kvm_cpuid_entry2 entries[];
+		};
+		/*
+		 * For future extensibility.
+		 * The size(struct kvm_tdx_init_vm) = 16KB.
+		 * This should be enough given sizeof(TD_PARAMS) = 1024
+		 */
+		__u64 reserved[2028];
+	};
+};
+
 #endif /* _ASM_X86_KVM_H */

Isaku Yamahata

2022-11-17 17:51 UTC #53 lore raw

Re: [PATCH v10 021/108] KVM: TDX: initialize VM with TDX specific parameters

Quoted 46 lines · Lei

Quoted 32 lines

arch/x86/kvm/vmx/tdx.c +11 ~~-13~~

@@ -431,50 +633,65 @@ int tdx_vm_init(struct kvm *kvm)
 	return ret;
 }
 
-int tdx_dev_ioctl(void __user *argp)
+static int tdx_td_init(struct kvm *kvm, struct kvm_tdx_cmd *cmd)
 {
-	struct kvm_tdx_capabilities __user *user_caps;
-	struct kvm_tdx_capabilities caps;
-	struct kvm_tdx_cmd cmd;
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	struct kvm_tdx_init_vm *init_vm = NULL;
+	struct td_params *td_params = NULL;
+	void *entries_end;
+	int ret;
 
-	BUILD_BUG_ON(sizeof(struct kvm_tdx_cpuid_config) !=
-		     sizeof(struct tdx_cpuid_config));
+	BUILD_BUG_ON(sizeof(*init_vm) != 16 * 1024);
+	BUILD_BUG_ON((sizeof(*init_vm) - offsetof(typeof(*init_vm), entries)) /
+		     sizeof(init_vm->entries[0]) < KVM_MAX_CPUID_ENTRIES);
+	BUILD_BUG_ON(sizeof(struct td_params) != 1024);
 
-	if (copy_from_user(&cmd, argp, sizeof(cmd)))
-		return -EFAULT;
-	if (cmd.flags || cmd.error || cmd.unused)
+	if (is_td_initialized(kvm))
 		return -EINVAL;
-	/*
-	 * Currently only KVM_TDX_CAPABILITIES is defined for system-scoped
-	 * mem_enc_ioctl().
-	 */

arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 656

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c +4 -4

-	if (cmd.id != KVM_TDX_CAPABILITIES)
+
+	if (cmd->flags)
 		return -EINVAL;
 
-	user_caps = (void __user *)cmd.data;
-	if (copy_from_user(&caps, user_caps, sizeof(caps)))
-		return -EFAULT;
+	init_vm = kzalloc(sizeof(*init_vm), GFP_KERNEL);
+	if (copy_from_user(init_vm, (void __user *)cmd->data, sizeof(*init_vm))) {

Pointer 'init_vm' returned from call to function 'kzalloc' may be NULL and will be dereferenced at this line.

Quoted 1 line

Maybe a NULL-check here is needed?

Yes, thank you for catching it.

Thanks,

Signature · 1 line

Isaku Yamahata <[email protected]>

Wang, Lei

2023-01-04 07:59 UTC #54 lore raw

Re: [PATCH v10 021/108] KVM: TDX: initialize VM with TDX specific parameters

Quoted 19 lines · On 10/30/2022 2:22 PM, [email protected] wrote:

From: Xiaoyao Li <[email protected]>
TDX requires additional parameters for TDX VM for confidential execution to protect its confidentiality of its memory contents and its CPU state from any other software, including VMM. When creating guest TD VM before creating vcpu, the number of vcpu, TSC frequency (that is same among vcpus. and it can't be changed.) CPUIDs which is emulated by the TDX module. It means guest can trust those CPUIDs. and sha384 values for measurement.
Add new subcommand, KVM_TDX_INIT_VM, to pass parameters for TDX guest. It assigns encryption key to the TDX guest for memory encryption. TDX encrypts memory per-guest bases. It assigns device model passes per-VM parameters for the TDX guest. The maximum number of vcpus, tsc frequency (TDX guest has fised VM-wide TSC frequency. not per-vcpu. The TDX guest can not change it.), attributes (production or debug), available extended features (which is reflected into guest XCR0, IA32_XSS MSR), cpuids, sha384 measurements, and etc.

Quoted 3 lines

This subcommand is called before creating vcpu and KVM_SET_CPUID2, i.e. cpuids configurations aren't available yet. So CPUIDs configuration values needs to be passed in struct kvm_init_vm. It's device model responsibility

I suppose this should be kvm_tdx_init_vm.

Quoted 128 lines

to make this cpuid config for KVM_TDX_INIT_VM and KVM_SET_CPUID2.

Signed-off-by Xiaoyao Li <[email protected]>

---
 arch/x86/include/asm/tdx.h            |   3 +
 arch/x86/include/uapi/asm/kvm.h       |  31 +++
 arch/x86/kvm/vmx/tdx.c                | 296 ++++++++++++++++++++++----
 arch/x86/kvm/vmx/tdx.h                |  22 ++
 tools/arch/x86/include/uapi/asm/kvm.h |  33 +++
 5 files changed, 347 insertions(+), 38 deletions(-)

arch/x86/include/asm/tdx.h +3 -0

index cd304d323d33..05ac4bfc8f8a 100644
@@ -131,6 +131,9 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,
 #endif /* CONFIG_INTEL_TDX_GUEST && CONFIG_KVM_GUEST */
 
 #ifdef CONFIG_INTEL_TDX_HOST
+
+/* -1 indicates CPUID leaf with no sub-leaves. */
+#define TDX_CPUID_NO_SUBLEAF	((u32)-1)
 struct tdx_cpuid_config {
 	u32	leaf;
 	u32	sub_leaf;

arch/x86/include/uapi/asm/kvm.h +31 -0

index 2ad9666e02a5..26661879c031 100644
@@ -538,6 +538,7 @@ struct kvm_pmu_event_filter {
 /* Trust Domain eXtension sub-ioctl() commands. */
 enum kvm_tdx_cmd_id {
 	KVM_TDX_CAPABILITIES = 0,
+	KVM_TDX_INIT_VM,
 
 	KVM_TDX_CMD_NR_MAX,
 };

@@ -583,4 +584,34 @@ struct kvm_tdx_capabilities {
 	struct kvm_tdx_cpuid_config cpuid_configs[0];
 };
 
+struct kvm_tdx_init_vm {
+	__u64 attributes;
+	__u64 mrconfigid[6];	/* sha384 digest */
+	__u64 mrowner[6];	/* sha384 digest */
+	__u64 mrownerconfig[6];	/* sha348 digest */
+	union {
+		/*
+		 * KVM_TDX_INIT_VM is called before vcpu creation, thus before
+		 * KVM_SET_CPUID2.  CPUID configurations needs to be passed.
+		 *
+		 * This configuration supersedes KVM_SET_CPUID{,2}.
+		 * The user space VMM, e.g. qemu, should make them consistent
+		 * with this values.
+		 * sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(256)
+		 * = 8KB.
+		 */
+		struct {
+			struct kvm_cpuid2 cpuid;
+			/* 8KB with KVM_MAX_CPUID_ENTRIES. */
+			struct kvm_cpuid_entry2 entries[];
+		};
+		/*
+		 * For future extensibility.
+		 * The size(struct kvm_tdx_init_vm) = 16KB.
+		 * This should be enough given sizeof(TD_PARAMS) = 1024
+		 */
+		__u64 reserved[2029];
+	};
+};
+
 #endif /* _ASM_X86_KVM_H */

arch/x86/kvm/vmx/tdx.c +47 -0

index d77709a6da51..54045e0576e7 100644
@@ -284,6 +284,205 @@ static int tdx_do_tdh_mng_key_config(void *param)
 int tdx_vm_init(struct kvm *kvm)
 {
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+
+	kvm_tdx->hkid = -1;
+
+	/*
+	 * This function initializes only KVM software construct.  It doesn't
+	 * initialize TDX stuff, e.g. TDCS, TDR, TDCX, HKID etc.
+	 * It is handled by KVM_TDX_INIT_VM, __tdx_td_init().
+	 */
+
+	return 0;
+}
+
+int tdx_dev_ioctl(void __user *argp)
+{
+	struct kvm_tdx_capabilities __user *user_caps;
+	struct kvm_tdx_capabilities caps;
+	struct kvm_tdx_cmd cmd;
+
+	BUILD_BUG_ON(sizeof(struct kvm_tdx_cpuid_config) !=
+		     sizeof(struct tdx_cpuid_config));
+
+	if (copy_from_user(&cmd, argp, sizeof(cmd)))
+		return -EFAULT;
+	if (cmd.flags || cmd.error || cmd.unused)
+		return -EINVAL;
+	/*
+	 * Currently only KVM_TDX_CAPABILITIES is defined for system-scoped
+	 * mem_enc_ioctl().
+	 */
+	if (cmd.id != KVM_TDX_CAPABILITIES)
+		return -EINVAL;
+
+	user_caps = (void __user *)cmd.data;
+	if (copy_from_user(&caps, user_caps, sizeof(caps)))
+		return -EFAULT;
+
+	if (caps.nr_cpuid_configs < tdx_caps.nr_cpuid_configs)
+		return -E2BIG;
+
+	caps = (struct kvm_tdx_capabilities) {
+		.attrs_fixed0 = tdx_caps.attrs_fixed0,
+		.attrs_fixed1 = tdx_caps.attrs_fixed1,
+		.xfam_fixed0 = tdx_caps.xfam_fixed0,
+		.xfam_fixed1 = tdx_caps.xfam_fixed1,
+		.nr_cpuid_configs = tdx_caps.nr_cpuid_configs,
+		.padding = 0,
+	};

arch/x86/kvm/vmx/tdx.c static int tdx_do_tdh_mng_key_config(void *param) · line 343

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c +10 -0

+
+	if (copy_to_user(user_caps, &caps, sizeof(caps)))
+		return -EFAULT;
+	if (copy_to_user(user_caps->cpuid_configs, &tdx_caps.cpuid_configs,
+			 tdx_caps.nr_cpuid_configs *
+			 sizeof(struct tdx_cpuid_config)))
+		return -EFAULT;
+
+	return 0;
+}

tdx_dev_ioctl() is introduced in previous patch with the same code added here, which means this is just a place change and it will confuse reviewers. Is it neccesary to do so?

Quoted 18 lines

arch/x86/kvm/vmx/tdx.c +18 -0

+
+/*
+ * cpuid entry lookup in TDX cpuid config way.
+ * The difference is how to specify index(subleaves).
+ * Specify index to TDX_CPUID_NO_SUBLEAF for CPUID leaf with no-subleaves.
+ */
+static const struct kvm_cpuid_entry2 *tdx_find_cpuid_entry(const struct kvm_cpuid2 *cpuid,
+							   u32 function, u32 index)
+{
+	int i;
+
+	/* In TDX CPU CONFIG, TDX_CPUID_NO_SUBLEAF means index = 0. */
+	if (index == TDX_CPUID_NO_SUBLEAF)
+		index = 0;
+
+	for (i = 0; i < cpuid->nent; i++) {
+		const struct kvm_cpuid_entry2 *e = &cpuid->entries[i];
+

arch/x86/kvm/vmx/tdx.c static int tdx_do_tdh_mng_key_config(void *param) · line 371

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c +10 -0

+		if (e->function == function &&
+		    (e->index == index ||
+		     !(e->flags & KVM_CPUID_FLAG_SIGNIFCANT_INDEX)))
+			return e;
+	}
+	return NULL;
+}
+
+static int setup_tdparams(struct kvm *kvm, struct td_params *td_params,
+			struct kvm_tdx_init_vm *init_vm)

Indentation is broken here, please fix it.

Quoted 262 lines

arch/x86/kvm/vmx/tdx.c +173 ~~-38~~

+{
+	const struct kvm_cpuid2 *cpuid = &init_vm->cpuid;
+	const struct kvm_cpuid_entry2 *entry;
+	u64 guest_supported_xcr0;
+	u64 guest_supported_xss;
+	int max_pa;
+	int i;
+
+	if (kvm->created_vcpus)
+		return -EBUSY;
+	td_params->max_vcpus = kvm->max_vcpus;
+	td_params->attributes = init_vm->attributes;
+	if (td_params->attributes & TDX_TD_ATTRIBUTE_PERFMON) {
+		/*
+		 * TODO: save/restore PMU related registers around TDENTER.
+		 * Once it's done, remove this guard.
+		 */
+		pr_warn("TD doesn't support perfmon yet. KVM needs to save/restore "
+			"host perf registers properly.\n");
+		return -EOPNOTSUPP;
+	}
+
+	for (i = 0; i < tdx_caps.nr_cpuid_configs; i++) {
+		const struct tdx_cpuid_config *config = &tdx_caps.cpuid_configs[i];
+		const struct kvm_cpuid_entry2 *entry =
+			tdx_find_cpuid_entry(cpuid, config->leaf, config->sub_leaf);
+		struct tdx_cpuid_value *value = &td_params->cpuid_values[i];
+
+		if (!entry)
+			continue;
+
+		value->eax = entry->eax & config->eax;
+		value->ebx = entry->ebx & config->ebx;
+		value->ecx = entry->ecx & config->ecx;
+		value->edx = entry->edx & config->edx;
+	}
+
+	max_pa = 36;
+	entry = tdx_find_cpuid_entry(cpuid, 0x80000008, 0);
+	if (entry)
+		max_pa = entry->eax & 0xff;
+
+	td_params->eptp_controls = VMX_EPTP_MT_WB;
+	/*
+	 * No CPU supports 4-level && max_pa > 48.
+	 * "5-level paging and 5-level EPT" section 4.1 4-level EPT
+	 * "4-level EPT is limited to translating 48-bit guest-physical
+	 *  addresses."
+	 * cpu_has_vmx_ept_5levels() check is just in case.
+	 */
+	if (cpu_has_vmx_ept_5levels() && max_pa > 48) {
+		td_params->eptp_controls |= VMX_EPTP_PWL_5;
+		td_params->exec_controls |= TDX_EXEC_CONTROL_MAX_GPAW;
+	} else {
+		td_params->eptp_controls |= VMX_EPTP_PWL_4;
+	}
+
+	/* Setup td_params.xfam */
+	entry = tdx_find_cpuid_entry(cpuid, 0xd, 0);
+	if (entry)
+		guest_supported_xcr0 = (entry->eax | ((u64)entry->edx << 32));
+	else
+		guest_supported_xcr0 = 0;
+	guest_supported_xcr0 &= kvm_caps.supported_xcr0;
+
+	entry = tdx_find_cpuid_entry(cpuid, 0xd, 1);
+	if (entry)
+		guest_supported_xss = (entry->ecx | ((u64)entry->edx << 32));
+	else
+		guest_supported_xss = 0;
+	/* PT can be exposed to TD guest regardless of KVM's XSS support */
+	guest_supported_xss &= (kvm_caps.supported_xss | XFEATURE_MASK_PT);
+
+	td_params->xfam = guest_supported_xcr0 | guest_supported_xss;
+	if (td_params->xfam & XFEATURE_MASK_LBR) {
+		/*
+		 * TODO: once KVM supports LBR(save/restore LBR related
+		 * registers around TDENTER), remove this guard.
+		 */
+		pr_warn("TD doesn't support LBR yet. KVM needs to save/restore "
+			"IA32_LBR_DEPTH properly.\n");
+		return -EOPNOTSUPP;
+	}
+
+	if (td_params->xfam & XFEATURE_MASK_XTILE) {
+		/*
+		 * TODO: once KVM supports AMX(save/restore AMX related
+		 * registers around TDENTER), remove this guard.
+		 */
+		pr_warn("TD doesn't support AMX yet. KVM needs to save/restore "
+			"IA32_XFD, IA32_XFD_ERR properly.\n");
+		return -EOPNOTSUPP;
+	}
+
+	td_params->tsc_frequency =
+		TDX_TSC_KHZ_TO_25MHZ(kvm->arch.default_tsc_khz);
+
+#define MEMCPY_SAME_SIZE(dst, src)				\
+	do {							\
+		BUILD_BUG_ON(sizeof(dst) != sizeof(src));	\
+		memcpy((dst), (src), sizeof(dst));		\
+	} while (0)
+
+	MEMCPY_SAME_SIZE(td_params->mrconfigid, init_vm->mrconfigid);
+	MEMCPY_SAME_SIZE(td_params->mrowner, init_vm->mrowner);
+	MEMCPY_SAME_SIZE(td_params->mrownerconfig, init_vm->mrownerconfig);
+
+	return 0;
+}
+
+static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	struct tdx_module_output out;
 	cpumask_var_t packages;
 	int ret, i;
 	u64 err;

@@ -390,10 +589,13 @@ int tdx_vm_init(struct kvm *kvm)
 		tdx_mark_td_page_added(&kvm_tdx->tdcs[i]);
 	}
 
-	/*
-	 * Note, TDH_MNG_INIT cannot be invoked here.  TDH_MNG_INIT requires a dedicated
-	 * ioctl() to define the configure CPUID values for the TD.
-	 */
+	err = tdh_mng_init(kvm_tdx->tdr.pa, __pa(td_params), &out);
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_MNG_INIT, err, &out);
+		ret = -EIO;
+		goto teardown;
+	}
+
 	return 0;
 
 	/*

@@ -431,50 +633,65 @@ int tdx_vm_init(struct kvm *kvm)
 	return ret;
 }
 
-int tdx_dev_ioctl(void __user *argp)
+static int tdx_td_init(struct kvm *kvm, struct kvm_tdx_cmd *cmd)
 {
-	struct kvm_tdx_capabilities __user *user_caps;
-	struct kvm_tdx_capabilities caps;
-	struct kvm_tdx_cmd cmd;
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	struct kvm_tdx_init_vm *init_vm = NULL;
+	struct td_params *td_params = NULL;
+	void *entries_end;
+	int ret;
 
-	BUILD_BUG_ON(sizeof(struct kvm_tdx_cpuid_config) !=
-		     sizeof(struct tdx_cpuid_config));
+	BUILD_BUG_ON(sizeof(*init_vm) != 16 * 1024);
+	BUILD_BUG_ON((sizeof(*init_vm) - offsetof(typeof(*init_vm), entries)) /
+		     sizeof(init_vm->entries[0]) < KVM_MAX_CPUID_ENTRIES);
+	BUILD_BUG_ON(sizeof(struct td_params) != 1024);
 
-	if (copy_from_user(&cmd, argp, sizeof(cmd)))
-		return -EFAULT;
-	if (cmd.flags || cmd.error || cmd.unused)
+	if (is_td_initialized(kvm))
 		return -EINVAL;
-	/*
-	 * Currently only KVM_TDX_CAPABILITIES is defined for system-scoped
-	 * mem_enc_ioctl().
-	 */
-	if (cmd.id != KVM_TDX_CAPABILITIES)
+
+	if (cmd->flags)
 		return -EINVAL;
 
-	user_caps = (void __user *)cmd.data;
-	if (copy_from_user(&caps, user_caps, sizeof(caps)))
-		return -EFAULT;
+	init_vm = kzalloc(sizeof(*init_vm), GFP_KERNEL);
+	if (copy_from_user(init_vm, (void __user *)cmd->data, sizeof(*init_vm))) {
+		ret = -EFAULT;
+		goto out;
+	}
 
-	if (caps.nr_cpuid_configs < tdx_caps.nr_cpuid_configs)
-		return -E2BIG;
+	ret = -EINVAL;
+	if (init_vm->cpuid.padding)
+		goto out;
+	/* init_vm->entries shouldn't overrun. */
+	entries_end = init_vm->entries + init_vm->cpuid.nent;
+	if (entries_end > (void *)(init_vm + 1))
+		goto out;
+	/* Unused part must be zero. */
+	if (memchr_inv(entries_end, 0, (void *)(init_vm + 1) - entries_end))
+		goto out;
 
-	caps = (struct kvm_tdx_capabilities) {
-		.attrs_fixed0 = tdx_caps.attrs_fixed0,
-		.attrs_fixed1 = tdx_caps.attrs_fixed1,
-		.xfam_fixed0 = tdx_caps.xfam_fixed0,
-		.xfam_fixed1 = tdx_caps.xfam_fixed1,
-		.nr_cpuid_configs = tdx_caps.nr_cpuid_configs,
-		.padding = 0,
-	};
+	td_params = kzalloc(sizeof(struct td_params), GFP_KERNEL);
+	if (!td_params) {
+		ret = -ENOMEM;
+		goto out;
+	}
 
-	if (copy_to_user(user_caps, &caps, sizeof(caps)))
-		return -EFAULT;
-	if (copy_to_user(user_caps->cpuid_configs, &tdx_caps.cpuid_configs,
-			 tdx_caps.nr_cpuid_configs *
-			 sizeof(struct tdx_cpuid_config)))
-		return -EFAULT;
+	ret = setup_tdparams(kvm, td_params, init_vm);
+	if (ret)
+		goto out;
 
-	return 0;
+	ret = __tdx_td_init(kvm, td_params);
+	if (ret)
+		goto out;
+
+	kvm_tdx->tsc_offset = td_tdcs_exec_read64(kvm_tdx, TD_TDCS_EXEC_TSC_OFFSET);
+	kvm_tdx->attributes = td_params->attributes;
+	kvm_tdx->xfam = td_params->xfam;
+
+out:
+	/* kfree() accepts NULL. */
+	kfree(init_vm);
+	kfree(td_params);
+	return ret;
 }
 
 int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)

@@ -490,6 +707,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)
 	mutex_lock(&kvm->lock);
 
 	switch (tdx_cmd.id) {
+	case KVM_TDX_INIT_VM:
+		r = tdx_td_init(kvm, &tdx_cmd);
+		break;
 	default:
 		r = -EINVAL;
 		goto out;

arch/x86/kvm/vmx/tdx.h +4 -0

index 938314635b47..ff0ea9cad347 100644
@@ -18,7 +18,11 @@ struct kvm_tdx {
 	struct tdx_td_page tdr;
 	struct tdx_td_page *tdcs;
 
+	u64 attributes;
+	u64 xfam;
 	int hkid;
+
+	u64 tsc_offset;
 };
 
 struct vcpu_tdx {

arch/x86/kvm/vmx/tdx.h static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) · line 58

Quoted 7 lines

arch/x86/kvm/vmx/tdx.h +4 -0

@@ -48,6 +52,11 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu)
 	return container_of(vcpu, struct vcpu_tdx, vcpu);
 }
 
+static inline bool is_td_initialized(struct kvm *kvm)
+{
+	return to_kvm_tdx(kvm)->hkid > 0;
+}

There is a similar function, is_hkid_assigned(), which is previously defined. Do you think redefining a new function here will bring code redundency?

Quoted 73 lines

arch/x86/kvm/vmx/tdx.h +14 -0

+
 static __always_inline void tdvps_vmcs_check(u32 field, u8 bits)
 {
 #define VMCS_ENC_ACCESS_TYPE_MASK	0x1UL

@@ -148,6 +157,19 @@ TDX_BUILD_TDVPS_ACCESSORS(64, VMCS, vmcs);
 TDX_BUILD_TDVPS_ACCESSORS(64, STATE_NON_ARCH, state_non_arch);
 TDX_BUILD_TDVPS_ACCESSORS(8, MANAGEMENT, management);
 
+static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field)
+{
+	struct tdx_module_output out;
+	u64 err;
+
+	err = tdh_mng_rd(kvm_tdx->tdr.pa, TDCS_EXEC(field), &out);
+	if (unlikely(err)) {
+		pr_err("TDH_MNG_RD[EXEC.0x%x] failed: 0x%llx\n", field, err);
+		return 0;
+	}
+	return out.r8;
+}
+
 #else
 struct kvm_tdx {
 	struct kvm kvm;

tools/arch/x86/include/uapi/asm/kvm.h +33 -0

index 2ad9666e02a5..531a0033e530 100644
@@ -538,6 +538,7 @@ struct kvm_pmu_event_filter {
 /* Trust Domain eXtension sub-ioctl() commands. */
 enum kvm_tdx_cmd_id {
 	KVM_TDX_CAPABILITIES = 0,
+	KVM_TDX_INIT_VM,
 
 	KVM_TDX_CMD_NR_MAX,
 };

@@ -583,4 +584,36 @@ struct kvm_tdx_capabilities {
 	struct kvm_tdx_cpuid_config cpuid_configs[0];
 };
 
+struct kvm_tdx_init_vm {
+	__u64 attributes;
+	__u32 max_vcpus;
+	__u32 padding;
+	__u64 mrconfigid[6];    /* sha384 digest */
+	__u64 mrowner[6];       /* sha384 digest */
+	__u64 mrownerconfig[6]; /* sha348 digest */
+	union {
+		/*
+		 * KVM_TDX_INIT_VM is called before vcpu creation, thus before
+		 * KVM_SET_CPUID2.  CPUID configurations needs to be passed.
+		 *
+		 * This configuration supersedes KVM_SET_CPUID{,2}.
+		 * The user space VMM, e.g. qemu, should make them consistent
+		 * with this values.
+		 * sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(256)
+		 * = 8KB.
+		 */
+		struct {
+			struct kvm_cpuid2 cpuid;
+			/* 8KB with KVM_MAX_CPUID_ENTRIES. */
+			struct kvm_cpuid_entry2 entries[];
+		};
+		/*
+		 * For future extensibility.
+		 * The size(struct kvm_tdx_init_vm) = 16KB.
+		 * This should be enough given sizeof(TD_PARAMS) = 1024
+		 */
+		__u64 reserved[2028];
+	};
+};
+
 #endif /* _ASM_X86_KVM_H */

Isaku Yamahata

2023-01-12 03:12 UTC #55 lore raw

Re: [PATCH v10 021/108] KVM: TDX: initialize VM with TDX specific parameters

Quoted 160 lines · Lei

Quoted 19 lines · On 10/30/2022 2:22 PM, [email protected] wrote:

From: Xiaoyao Li <[email protected]>
TDX requires additional parameters for TDX VM for confidential execution to protect its confidentiality of its memory contents and its CPU state from any other software, including VMM. When creating guest TD VM before creating vcpu, the number of vcpu, TSC frequency (that is same among vcpus. and it can't be changed.) CPUIDs which is emulated by the TDX module. It means guest can trust those CPUIDs. and sha384 values for measurement.
Add new subcommand, KVM_TDX_INIT_VM, to pass parameters for TDX guest. It assigns encryption key to the TDX guest for memory encryption. TDX encrypts memory per-guest bases. It assigns device model passes per-VM parameters for the TDX guest. The maximum number of vcpus, tsc frequency (TDX guest has fised VM-wide TSC frequency. not per-vcpu. The TDX guest can not change it.), attributes (production or debug), available extended features (which is reflected into guest XCR0, IA32_XSS MSR), cpuids, sha384 measurements, and etc.

Quoted 3 lines

This subcommand is called before creating vcpu and KVM_SET_CPUID2, i.e. cpuids configurations aren't available yet. So CPUIDs configuration values needs to be passed in struct kvm_init_vm. It's device model responsibility

I suppose this should be kvm_tdx_init_vm.

Quoted 134 lines

to make this cpuid config for KVM_TDX_INIT_VM and KVM_SET_CPUID2.

Signed-off-by Xiaoyao Li <[email protected]>

---
 arch/x86/include/asm/tdx.h            |   3 +
 arch/x86/include/uapi/asm/kvm.h       |  31 +++
 arch/x86/kvm/vmx/tdx.c                | 296 ++++++++++++++++++++++----
 arch/x86/kvm/vmx/tdx.h                |  22 ++
 tools/arch/x86/include/uapi/asm/kvm.h |  33 +++
 5 files changed, 347 insertions(+), 38 deletions(-)

arch/x86/include/asm/tdx.h +3 -0

index cd304d323d33..05ac4bfc8f8a 100644
@@ -131,6 +131,9 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,
 #endif /* CONFIG_INTEL_TDX_GUEST && CONFIG_KVM_GUEST */
 
 #ifdef CONFIG_INTEL_TDX_HOST
+
+/* -1 indicates CPUID leaf with no sub-leaves. */
+#define TDX_CPUID_NO_SUBLEAF	((u32)-1)
 struct tdx_cpuid_config {
 	u32	leaf;
 	u32	sub_leaf;

arch/x86/include/uapi/asm/kvm.h +31 -0

index 2ad9666e02a5..26661879c031 100644
@@ -538,6 +538,7 @@ struct kvm_pmu_event_filter {
 /* Trust Domain eXtension sub-ioctl() commands. */
 enum kvm_tdx_cmd_id {
 	KVM_TDX_CAPABILITIES = 0,
+	KVM_TDX_INIT_VM,
 
 	KVM_TDX_CMD_NR_MAX,
 };

@@ -583,4 +584,34 @@ struct kvm_tdx_capabilities {
 	struct kvm_tdx_cpuid_config cpuid_configs[0];
 };
 
+struct kvm_tdx_init_vm {
+	__u64 attributes;
+	__u64 mrconfigid[6];	/* sha384 digest */
+	__u64 mrowner[6];	/* sha384 digest */
+	__u64 mrownerconfig[6];	/* sha348 digest */
+	union {
+		/*
+		 * KVM_TDX_INIT_VM is called before vcpu creation, thus before
+		 * KVM_SET_CPUID2.  CPUID configurations needs to be passed.
+		 *
+		 * This configuration supersedes KVM_SET_CPUID{,2}.
+		 * The user space VMM, e.g. qemu, should make them consistent
+		 * with this values.
+		 * sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(256)
+		 * = 8KB.
+		 */
+		struct {
+			struct kvm_cpuid2 cpuid;
+			/* 8KB with KVM_MAX_CPUID_ENTRIES. */
+			struct kvm_cpuid_entry2 entries[];
+		};
+		/*
+		 * For future extensibility.
+		 * The size(struct kvm_tdx_init_vm) = 16KB.
+		 * This should be enough given sizeof(TD_PARAMS) = 1024
+		 */
+		__u64 reserved[2029];
+	};
+};
+
 #endif /* _ASM_X86_KVM_H */

arch/x86/kvm/vmx/tdx.c +53 -0

index d77709a6da51..54045e0576e7 100644
@@ -284,6 +284,205 @@ static int tdx_do_tdh_mng_key_config(void *param)
 int tdx_vm_init(struct kvm *kvm)
 {
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+
+	kvm_tdx->hkid = -1;
+
+	/*
+	 * This function initializes only KVM software construct.  It doesn't
+	 * initialize TDX stuff, e.g. TDCS, TDR, TDCX, HKID etc.
+	 * It is handled by KVM_TDX_INIT_VM, __tdx_td_init().
+	 */
+
+	return 0;
+}
+
+int tdx_dev_ioctl(void __user *argp)
+{
+	struct kvm_tdx_capabilities __user *user_caps;
+	struct kvm_tdx_capabilities caps;
+	struct kvm_tdx_cmd cmd;
+
+	BUILD_BUG_ON(sizeof(struct kvm_tdx_cpuid_config) !=
+		     sizeof(struct tdx_cpuid_config));
+
+	if (copy_from_user(&cmd, argp, sizeof(cmd)))
+		return -EFAULT;
+	if (cmd.flags || cmd.error || cmd.unused)
+		return -EINVAL;
+	/*
+	 * Currently only KVM_TDX_CAPABILITIES is defined for system-scoped
+	 * mem_enc_ioctl().
+	 */
+	if (cmd.id != KVM_TDX_CAPABILITIES)
+		return -EINVAL;
+
+	user_caps = (void __user *)cmd.data;
+	if (copy_from_user(&caps, user_caps, sizeof(caps)))
+		return -EFAULT;
+
+	if (caps.nr_cpuid_configs < tdx_caps.nr_cpuid_configs)
+		return -E2BIG;
+
+	caps = (struct kvm_tdx_capabilities) {
+		.attrs_fixed0 = tdx_caps.attrs_fixed0,
+		.attrs_fixed1 = tdx_caps.attrs_fixed1,
+		.xfam_fixed0 = tdx_caps.xfam_fixed0,
+		.xfam_fixed1 = tdx_caps.xfam_fixed1,
+		.nr_cpuid_configs = tdx_caps.nr_cpuid_configs,
+		.padding = 0,
+	};
+
+	if (copy_to_user(user_caps, &caps, sizeof(caps)))
+		return -EFAULT;
+	if (copy_to_user(user_caps->cpuid_configs, &tdx_caps.cpuid_configs,
+			 tdx_caps.nr_cpuid_configs *
+			 sizeof(struct tdx_cpuid_config)))

Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static int tdx_do_tdh_mng_key_config(void *param) · line 343
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4 -0
+		return -EFAULT;
+
+	return 0;
+}
tdx_dev_ioctl() is introduced in previous patch with the same code added here, which means this is just a place change and it will confuse reviewers. Is it neccesary to do so?

Right. Somehow diff was also confused. I'll try to twist the previous patch to introduce __tdx_td_init() with the previous patch.

...

Quoted 7 lines

arch/x86/kvm/vmx/tdx.c static int tdx_do_tdh_mng_key_config(void *param) · line 347
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4 -0
+static inline bool is_td_initialized(struct kvm *kvm)
+{
+	return to_kvm_tdx(kvm)->hkid > 0;
+}
There is a similar function, is_hkid_assigned(), which is previously defined. Do you think redefining a new function here will bring code redundency?

I'll drop this function and replace it with is_hkid_assigned().

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:25 UTC #56 lore raw

PATCH v10 22/108 KVM: TDX: Make pmu_intel.c ignore guest TD case

From: Isaku Yamahata <[email protected]>

Because TDX KVM doesn't support PMU yet (it's future work of TDX KVM support as another patch series) and pmu_intel.c touches vmx specific structure in vcpu initialization, as workaround add dummy structure to struct vcpu_tdx and pmu_intel.c can ignore TDX case.

---
 arch/x86/kvm/vmx/pmu_intel.c | 46 +++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/vmx/pmu_intel.h | 28 ++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx.h       |  7 ++++++
 arch/x86/kvm/vmx/vmx.c       |  2 +-
 arch/x86/kvm/vmx/vmx.h       | 32 +------------------------
 5 files changed, 82 insertions(+), 33 deletions(-)
 create mode 100644 arch/x86/kvm/vmx/pmu_intel.h

arch/x86/kvm/vmx/pmu_intel.c +45 -1

index 25b70a85bef5..41e97c21f0d7 100644
@@ -17,6 +17,7 @@
 #include "lapic.h"
 #include "nested.h"
 #include "pmu.h"
+#include "tdx.h"
 
 #define MSR_PMC_FULL_WIDTH_BIT      (MSR_IA32_PMC0 - MSR_IA32_PERFCTR0)

@@ -35,6 +36,26 @@ static struct kvm_event_hw_type_mapping intel_arch_events[] = {
 /* mapping between fixed pmc index and intel_arch_events array */
 static int fixed_pmc_events[] = {1, 0, 7};
 
+struct lbr_desc *vcpu_to_lbr_desc(struct kvm_vcpu *vcpu)
+{
+#ifdef CONFIG_INTEL_TDX_HOST
+	if (is_td_vcpu(vcpu))
+		return &to_tdx(vcpu)->lbr_desc;
+#endif
+
+	return &to_vmx(vcpu)->lbr_desc;
+}
+
+struct x86_pmu_lbr *vcpu_to_lbr_records(struct kvm_vcpu *vcpu)
+{
+#ifdef CONFIG_INTEL_TDX_HOST
+	if (is_td_vcpu(vcpu))
+		return &to_tdx(vcpu)->lbr_desc.records;
+#endif
+
+	return &to_vmx(vcpu)->lbr_desc.records;
+}
+
 static void reprogram_fixed_counters(struct kvm_pmu *pmu, u64 data)
 {
 	struct kvm_pmc *pmc;

@@ -167,6 +188,23 @@ static inline struct kvm_pmc *get_fw_gp_pmc(struct kvm_pmu *pmu, u32 msr)
 	return get_gp_pmc(pmu, msr, MSR_IA32_PMC0);
 }
 
+bool intel_pmu_lbr_is_compatible(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		return false;
+	return cpuid_model_is_consistent(vcpu);
+}
+
+bool intel_pmu_lbr_is_enabled(struct kvm_vcpu *vcpu)
+{
+	struct x86_pmu_lbr *lbr = vcpu_to_lbr_records(vcpu);
+
+	if (is_td_vcpu(vcpu))
+		return false;
+
+	return lbr->nr && (vcpu_get_perf_capabilities(vcpu) & PMU_CAP_LBR_FMT);
+}
+
 static bool intel_pmu_is_valid_lbr_msr(struct kvm_vcpu *vcpu, u32 index)
 {
 	struct x86_pmu_lbr *records = vcpu_to_lbr_records(vcpu);

@@ -277,6 +315,9 @@ int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu)
 					PERF_SAMPLE_BRANCH_USER,
 	};
 
+	if (WARN_ON_ONCE(is_td_vcpu(vcpu)))
+		return 0;
+
 	if (unlikely(lbr_desc->event)) {
 		__set_bit(INTEL_PMC_IDX_FIXED_VLBR, pmu->pmc_in_use);
 		return 0;

@@ -586,7 +627,7 @@ static void intel_pmu_refresh(struct kvm_vcpu *vcpu)
 		INTEL_PMC_MAX_GENERIC, pmu->nr_arch_fixed_counters);
 
 	perf_capabilities = vcpu_get_perf_capabilities(vcpu);
-	if (cpuid_model_is_consistent(vcpu) &&
+	if (intel_pmu_lbr_is_compatible(vcpu) &&
 	    (perf_capabilities & PMU_CAP_LBR_FMT))
 		x86_perf_get_lbr(&lbr_desc->records);
 	else

@@ -643,6 +684,9 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu)
 	struct kvm_pmc *pmc = NULL;
 	int i;
 
+	if (is_td_vcpu(vcpu))
+		return;
+
 	for (i = 0; i < INTEL_PMC_MAX_GENERIC; i++) {
 		pmc = &pmu->gp_counters[i];

arch/x86/kvm/vmx/pmu_intel.h new file +28 -0

new file mode 100644
index 000000000000..66bba47c1269
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __KVM_X86_VMX_PMU_INTEL_H
+#define  __KVM_X86_VMX_PMU_INTEL_H
+
+struct lbr_desc *vcpu_to_lbr_desc(struct kvm_vcpu *vcpu);
+struct x86_pmu_lbr *vcpu_to_lbr_records(struct kvm_vcpu *vcpu);
+
+bool intel_pmu_lbr_is_compatible(struct kvm_vcpu *vcpu);
+bool intel_pmu_lbr_is_enabled(struct kvm_vcpu *vcpu);
+int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu);
+
+struct lbr_desc {
+	/* Basic info about guest LBR records. */
+	struct x86_pmu_lbr records;
+
+	/*
+	 * Emulate LBR feature via passthrough LBR registers when the
+	 * per-vcpu guest LBR event is scheduled on the current pcpu.
+	 *
+	 * The records may be inaccurate if the host reclaims the LBR.
+	 */
+	struct perf_event *event;
+
+	/* True if LBRs are marked as not intercepted in the MSR bitmap */
+	bool msr_passthrough;
+};
+
+#endif /* __KVM_X86_VMX_PMU_INTEL_H */

arch/x86/kvm/vmx/tdx.h +7 -0

index ff0ea9cad347..5aea69716278 100644
@@ -4,6 +4,7 @@
 
 #ifdef CONFIG_INTEL_TDX_HOST
 
+#include "pmu_intel.h"
 #include "tdx_ops.h"
 
 struct tdx_td_page {

@@ -30,6 +31,12 @@ struct vcpu_tdx {
 
 	struct tdx_td_page tdvpr;
 	struct tdx_td_page *tdvpx;
+
+	/*
+	 * Dummy to make pmu_intel not corrupt memory.
+	 * TODO: Support PMU for TDX.  Future work.
+	 */
+	struct lbr_desc lbr_desc;
 };
 
 static inline bool is_td(struct kvm *kvm)

arch/x86/kvm/vmx/vmx.c +1 -1

index 68aef67c5eb7..f890191e8580 100644
@@ -2341,7 +2341,7 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 			if ((data & PMU_CAP_LBR_FMT) !=
 			    (vmx_get_perf_capabilities() & PMU_CAP_LBR_FMT))
 				return 1;
-			if (!cpuid_model_is_consistent(vcpu))
+			if (!intel_pmu_lbr_is_compatible(vcpu))
 				return 1;
 		}
 		if (data & PERF_CAP_PEBS_FORMAT) {

arch/x86/kvm/vmx/vmx.h +1 ~~-31~~

index a3da84f4ea45..d49d0ace9fb8 100644
@@ -11,6 +11,7 @@
 #include "capabilities.h"
 #include "../kvm_cache_regs.h"
 #include "posted_intr.h"
+#include "pmu_intel.h"
 #include "vmcs.h"
 #include "vmx_ops.h"
 #include "../cpuid.h"

@@ -105,22 +106,6 @@ static inline bool intel_pmu_has_perf_global_ctrl(struct kvm_pmu *pmu)
 	return pmu->version > 1;
 }
 
-struct lbr_desc {
-	/* Basic info about guest LBR records. */
-	struct x86_pmu_lbr records;
-
-	/*
-	 * Emulate LBR feature via passthrough LBR registers when the
-	 * per-vcpu guest LBR event is scheduled on the current pcpu.
-	 *
-	 * The records may be inaccurate if the host reclaims the LBR.
-	 */
-	struct perf_event *event;
-
-	/* True if LBRs are marked as not intercepted in the MSR bitmap */
-	bool msr_passthrough;
-};
-
 /*
  * The nested_vmx structure is part of vcpu_vmx, and holds information we need
  * for correct emulation of VMX (i.e., nested VMX) on this vcpu.

@@ -650,21 +635,6 @@ static inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu)
 	return container_of(vcpu, struct vcpu_vmx, vcpu);
 }
 
-static inline struct lbr_desc *vcpu_to_lbr_desc(struct kvm_vcpu *vcpu)
-{
-	return &to_vmx(vcpu)->lbr_desc;
-}
-
-static inline struct x86_pmu_lbr *vcpu_to_lbr_records(struct kvm_vcpu *vcpu)
-{
-	return &vcpu_to_lbr_desc(vcpu)->records;
-}
-
-static inline bool intel_pmu_lbr_is_enabled(struct kvm_vcpu *vcpu)
-{
-	return !!vcpu_to_lbr_records(vcpu)->nr;
-}
-
 void intel_pmu_cross_mapped_check(struct kvm_pmu *pmu);
 int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu);
 void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu);

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #57 lore raw

PATCH v10 23/108 The start of TDX KVM patch series: TD vcpu creation/destruction

From: Isaku Yamahata <[email protected]>

This empty commit is to mark the start of patch series of TD vcpu creation/destruction.

---
 Documentation/virt/kvm/intel-tdx-layer-status.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Documentation/virt/kvm/intel-tdx-layer-status.rst +3 -3

index 5e0deaebf843..3e8efde3e3f3 100644
@@ -9,15 +9,15 @@ Layer status
 What qemu can do
 ----------------
 - TDX VM TYPE is exposed to Qemu.
-- Qemu can try to create VM of TDX VM type and then fails.
+- Qemu can create/destroy guest of TDX vm type.
 
 Patch Layer status
 ------------------
   Patch layer                          Status
 * TDX, VMX coexistence:                 Applied
 * TDX architectural definitions:        Applied
-* TD VM creation/destruction:           Applying
-* TD vcpu creation/destruction:         Not yet
+* TD VM creation/destruction:           Applied
+* TD vcpu creation/destruction:         Applying
 * TDX EPT violation:                    Not yet
 * TD finalization:                      Not yet
 * TD vcpu enter/exit:                   Not yet

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #58 lore raw

PATCH v10 24/108 KVM: TDX: allocate/free TDX vcpu structure

From: Isaku Yamahata <[email protected]>

The next step of TDX guest creation is to create vcpu. Allocate TDX vcpu structures, initialize it. Allocate pages of TDX vcpu for the TDX module.

In the case of the conventional case, cpuid is empty at the initialization. and cpuid is configured after the vcpu initialization. Because TDX supports only X2APIC mode, cpuid is forcibly initialized to support X2APIC on the vcpu initialization.

---
 arch/x86/kvm/vmx/main.c    |  40 +++++++++--
 arch/x86/kvm/vmx/tdx.c     | 138 +++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/x86_ops.h |   8 +++
 3 files changed, 182 insertions(+), 4 deletions(-)

arch/x86/kvm/vmx/main.c +36 -4

index b4e4c6c677f6..c125b2e3e8b4 100644
@@ -63,6 +63,38 @@ static void vt_vm_free(struct kvm *kvm)
 		return tdx_vm_free(kvm);
 }
 
+static int vt_vcpu_precreate(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return 0;
+
+	return vmx_vcpu_precreate(kvm);
+}
+
+static int vt_vcpu_create(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_create(vcpu);
+
+	return vmx_vcpu_create(vcpu);
+}
+
+static void vt_vcpu_free(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_free(vcpu);
+
+	return vmx_vcpu_free(vcpu);
+}
+
+static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_reset(vcpu, init_event);
+
+	return vmx_vcpu_reset(vcpu, init_event);
+}
+
 static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp)
 {
 	if (!is_td(kvm))

@@ -89,10 +121,10 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.vm_destroy = vt_vm_destroy,
 	.vm_free = vt_vm_free,
 
-	.vcpu_precreate = vmx_vcpu_precreate,
-	.vcpu_create = vmx_vcpu_create,
-	.vcpu_free = vmx_vcpu_free,
-	.vcpu_reset = vmx_vcpu_reset,
+	.vcpu_precreate = vt_vcpu_precreate,
+	.vcpu_create = vt_vcpu_create,
+	.vcpu_free = vt_vcpu_free,
+	.vcpu_reset = vt_vcpu_reset,
 
 	.prepare_switch_to_guest = vmx_prepare_switch_to_guest,
 	.vcpu_load = vmx_vcpu_load,

arch/x86/kvm/vmx/tdx.c +138 -0

150 lines not on this page · raw message

arch/x86/kvm/vmx/x86_ops.h +8 -0

index 93ffe2deb8e8..f6841c3dd12d 100644
@@ -141,6 +141,10 @@ int tdx_vm_init(struct kvm *kvm);
 void tdx_mmu_release_hkid(struct kvm *kvm);
 void tdx_vm_free(struct kvm *kvm);
 
+int tdx_vcpu_create(struct kvm_vcpu *vcpu);
+void tdx_vcpu_free(struct kvm_vcpu *vcpu);
+void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);
+
 int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);
 #else
 static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }

@@ -154,6 +158,10 @@ static inline void tdx_mmu_release_hkid(struct kvm *kvm) {}
 static inline void tdx_flush_shadow_all_private(struct kvm *kvm) {}
 static inline void tdx_vm_free(struct kvm *kvm) {}
 
+static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTSUPP; }
+static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {}
+static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}
+
 static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPNOTSUPP; }
 #endif

Signature · 1 line

2.25.1

Yuan Yao

2022-11-14 06:46 UTC #59 lore raw

Re: [PATCH v10 024/108] KVM: TDX: allocate/free TDX vcpu structure

Quoted 192 lines · On Sat, Oct 29, 2022 at 11:22:25PM -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>

The next step of TDX guest creation is to create vcpu. Allocate TDX vcpu structures, initialize it. Allocate pages of TDX vcpu for the TDX module.

---
 arch/x86/kvm/vmx/main.c    |  40 +++++++++--
 arch/x86/kvm/vmx/tdx.c     | 138 +++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/x86_ops.h |   8 +++
 3 files changed, 182 insertions(+), 4 deletions(-)

arch/x86/kvm/vmx/main.c +36 -4

index b4e4c6c677f6..c125b2e3e8b4 100644
@@ -63,6 +63,38 @@ static void vt_vm_free(struct kvm *kvm)
 		return tdx_vm_free(kvm);
 }
 
+static int vt_vcpu_precreate(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return 0;
+
+	return vmx_vcpu_precreate(kvm);
+}
+
+static int vt_vcpu_create(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_create(vcpu);
+
+	return vmx_vcpu_create(vcpu);
+}
+
+static void vt_vcpu_free(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_free(vcpu);
+
+	return vmx_vcpu_free(vcpu);
+}
+
+static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_reset(vcpu, init_event);
+
+	return vmx_vcpu_reset(vcpu, init_event);
+}
+
 static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp)
 {
 	if (!is_td(kvm))

@@ -89,10 +121,10 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.vm_destroy = vt_vm_destroy,
 	.vm_free = vt_vm_free,
 
-	.vcpu_precreate = vmx_vcpu_precreate,
-	.vcpu_create = vmx_vcpu_create,
-	.vcpu_free = vmx_vcpu_free,
-	.vcpu_reset = vmx_vcpu_reset,
+	.vcpu_precreate = vt_vcpu_precreate,
+	.vcpu_create = vt_vcpu_create,
+	.vcpu_free = vt_vcpu_free,
+	.vcpu_reset = vt_vcpu_reset,
 
 	.prepare_switch_to_guest = vmx_prepare_switch_to_guest,
 	.vcpu_load = vmx_vcpu_load,

arch/x86/kvm/vmx/tdx.c +102 -0

index 54045e0576e7..0625c354b341 100644
@@ -49,6 +49,11 @@ static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid)
 	return pa | ((hpa_t)hkid << boot_cpu_data.x86_phys_bits);
 }
 
+static inline bool is_td_vcpu_created(struct vcpu_tdx *tdx)
+{
+	return tdx->tdvpr.added;
+}
+
 static inline bool is_td_created(struct kvm_tdx *kvm_tdx)
 {
 	return kvm_tdx->tdr.added;

@@ -296,6 +301,139 @@ int tdx_vm_init(struct kvm *kvm)
 	return 0;
 }
 
+int tdx_vcpu_create(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	int ret, i;
+
+	/* TDX only supports x2APIC, which requires an in-kernel local APIC. */
+	if (!vcpu->arch.apic)
+		return -EINVAL;
+
+	fpstate_set_confidential(&vcpu->arch.guest_fpu);
+
+	ret = tdx_alloc_td_page(&tdx->tdvpr);
+	if (ret)
+		return ret;
+
+	tdx->tdvpx = kcalloc(tdx_caps.tdvpx_nr_pages, sizeof(*tdx->tdvpx),
+			GFP_KERNEL_ACCOUNT);
+	if (!tdx->tdvpx) {
+		ret = -ENOMEM;
+		goto free_tdvpr;
+	}
+	for (i = 0; i < tdx_caps.tdvpx_nr_pages; i++) {
+		ret = tdx_alloc_td_page(&tdx->tdvpx[i]);
+		if (ret)
+			goto free_tdvpx;
+	}
+
+	vcpu->arch.efer = EFER_SCE | EFER_LME | EFER_LMA | EFER_NX;
+
+	vcpu->arch.cr0_guest_owned_bits = -1ul;
+	vcpu->arch.cr4_guest_owned_bits = -1ul;
+
+	vcpu->arch.tsc_offset = to_kvm_tdx(vcpu->kvm)->tsc_offset;
+	vcpu->arch.l1_tsc_offset = vcpu->arch.tsc_offset;
+	vcpu->arch.guest_state_protected =
+		!(to_kvm_tdx(vcpu->kvm)->attributes & TDX_TD_ATTRIBUTE_DEBUG);
+
+	return 0;
+
+free_tdvpx:
+	/* @i points at the TDVPX page that failed allocation. */
+	for (--i; i >= 0; i--)
+		free_page(tdx->tdvpx[i].va);
+	kfree(tdx->tdvpx);
+	tdx->tdvpx = NULL;
+free_tdvpr:
+	free_page(tdx->tdvpr.va);
+
+	return ret;
+}
+
+void tdx_vcpu_free(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	int i;
+
+	/* Can't reclaim or free pages if teardown failed. */
+	if (is_hkid_assigned(to_kvm_tdx(vcpu->kvm)))
+		return;
+
+	if (tdx->tdvpx) {
+		for (i = 0; i < tdx_caps.tdvpx_nr_pages; i++)
+			tdx_reclaim_td_page(&tdx->tdvpx[i]);
+		kfree(tdx->tdvpx);
+		tdx->tdvpx = NULL;
+	}
+	tdx_reclaim_td_page(&tdx->tdvpr);
+}
+
+void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct msr_data apic_base_msr;
+	u64 err;
+	int i;
+
+	/* TDX doesn't support INIT event. */
+	if (WARN_ON_ONCE(init_event))
+		goto td_bugged;
+	if (WARN_ON_ONCE(is_td_vcpu_created(tdx)))
+		goto td_bugged;
+
+	err = tdh_vp_create(kvm_tdx->tdr.pa, tdx->tdvpr.pa);
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_VP_CREATE, err, NULL);
+		goto td_bugged;
+	}
+	tdx_mark_td_page_added(&tdx->tdvpr);
+
+	for (i = 0; i < tdx_caps.tdvpx_nr_pages; i++) {
+		err = tdh_vp_addcx(tdx->tdvpr.pa, tdx->tdvpx[i].pa);
+		if (WARN_ON_ONCE(err)) {
+			pr_tdx_error(TDH_VP_ADDCX, err, NULL);
+			goto td_bugged;
+		}
+		tdx_mark_td_page_added(&tdx->tdvpx[i]);

arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 410

Quoted 10 lines

arch/x86/kvm/vmx/tdx.c +10 -0

+	}
+
+	if (!vcpu->arch.cpuid_entries) {
+		/*
+		 * On cpu creation, cpuid entry is blank.  Forcibly enable
+		 * X2APIC feature to allow X2APIC.
+		 */
+		struct kvm_cpuid_entry2 *e;
+
+		e = kvmalloc_array(1, sizeof(*e), GFP_KERNEL_ACCOUNT);

NULL checking is necessary for kvmalloc_array.

Quoted 58 lines

arch/x86/kvm/vmx/tdx.c +26 -0

+		*e  = (struct kvm_cpuid_entry2) {
+			.function = 1,	/* Features for X2APIC */
+			.index = 0,
+			.eax = 0,
+			.ebx = 0,
+			.ecx = 1ULL << 21,	/* X2APIC */
+			.edx = 0,
+		};
+		vcpu->arch.cpuid_entries = e;
+		vcpu->arch.cpuid_nent = 1;
+	}
+	apic_base_msr.data = APIC_DEFAULT_PHYS_BASE | LAPIC_MODE_X2APIC;
+	if (kvm_vcpu_is_reset_bsp(vcpu))
+		apic_base_msr.data |= MSR_IA32_APICBASE_BSP;
+	apic_base_msr.host_initiated = true;
+	if (WARN_ON_ONCE(kvm_set_apic_base(vcpu, &apic_base_msr)))
+		goto td_bugged;
+
+	vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
+
+	return;
+
+td_bugged:
+	vcpu->kvm->vm_bugged = true;
+}
+
 int tdx_dev_ioctl(void __user *argp)
 {
 	struct kvm_tdx_capabilities __user *user_caps;

arch/x86/kvm/vmx/x86_ops.h +8 -1

index 93ffe2deb8e8..f6841c3dd12d 100644
@@ -141,6 +141,10 @@ int tdx_vm_init(struct kvm *kvm);
 void tdx_mmu_release_hkid(struct kvm *kvm);
 void tdx_vm_free(struct kvm *kvm);
 
+int tdx_vcpu_create(struct kvm_vcpu *vcpu);
+void tdx_vcpu_free(struct kvm_vcpu *vcpu);
+void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);
+
 int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);
 #else
 static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }

@@ -154,6 +158,10 @@ static inline void tdx_mmu_release_hkid(struct kvm *kvm) {}
 static inline void tdx_flush_shadow_all_private(struct kvm *kvm) {}
 static inline void tdx_vm_free(struct kvm *kvm) {}
 
+static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTSUPP; }
+static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {}
+static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}
+
 static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPNOTSUPP; }
 #endif
 
--

2.25.1

Isaku Yamahata

2022-12-15 21:28 UTC #60 lore raw

Re: [PATCH v10 024/108] KVM: TDX: allocate/free TDX vcpu structure

Quoted 197 lines · Yuan Yao

Quoted 196 lines · On Sat, Oct 29, 2022 at 11:22:25PM -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>

The next step of TDX guest creation is to create vcpu. Allocate TDX vcpu structures, initialize it. Allocate pages of TDX vcpu for the TDX module.

---
 arch/x86/kvm/vmx/main.c    |  40 +++++++++--
 arch/x86/kvm/vmx/tdx.c     | 138 +++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/x86_ops.h |   8 +++
 3 files changed, 182 insertions(+), 4 deletions(-)

arch/x86/kvm/vmx/main.c +36 -4

index b4e4c6c677f6..c125b2e3e8b4 100644
@@ -63,6 +63,38 @@ static void vt_vm_free(struct kvm *kvm)
 		return tdx_vm_free(kvm);
 }
 
+static int vt_vcpu_precreate(struct kvm *kvm)
+{
+	if (is_td(kvm))
+		return 0;
+
+	return vmx_vcpu_precreate(kvm);
+}
+
+static int vt_vcpu_create(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_create(vcpu);
+
+	return vmx_vcpu_create(vcpu);
+}
+
+static void vt_vcpu_free(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_free(vcpu);
+
+	return vmx_vcpu_free(vcpu);
+}
+
+static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_reset(vcpu, init_event);
+
+	return vmx_vcpu_reset(vcpu, init_event);
+}
+
 static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp)
 {
 	if (!is_td(kvm))

@@ -89,10 +121,10 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.vm_destroy = vt_vm_destroy,
 	.vm_free = vt_vm_free,
 
-	.vcpu_precreate = vmx_vcpu_precreate,
-	.vcpu_create = vmx_vcpu_create,
-	.vcpu_free = vmx_vcpu_free,
-	.vcpu_reset = vmx_vcpu_reset,
+	.vcpu_precreate = vt_vcpu_precreate,
+	.vcpu_create = vt_vcpu_create,
+	.vcpu_free = vt_vcpu_free,
+	.vcpu_reset = vt_vcpu_reset,
 
 	.prepare_switch_to_guest = vmx_prepare_switch_to_guest,
 	.vcpu_load = vmx_vcpu_load,

arch/x86/kvm/vmx/tdx.c +106 -0

index 54045e0576e7..0625c354b341 100644
@@ -49,6 +49,11 @@ static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid)
 	return pa | ((hpa_t)hkid << boot_cpu_data.x86_phys_bits);
 }
 
+static inline bool is_td_vcpu_created(struct vcpu_tdx *tdx)
+{
+	return tdx->tdvpr.added;
+}
+
 static inline bool is_td_created(struct kvm_tdx *kvm_tdx)
 {
 	return kvm_tdx->tdr.added;

@@ -296,6 +301,139 @@ int tdx_vm_init(struct kvm *kvm)
 	return 0;
 }
 
+int tdx_vcpu_create(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	int ret, i;
+
+	/* TDX only supports x2APIC, which requires an in-kernel local APIC. */
+	if (!vcpu->arch.apic)
+		return -EINVAL;
+
+	fpstate_set_confidential(&vcpu->arch.guest_fpu);
+
+	ret = tdx_alloc_td_page(&tdx->tdvpr);
+	if (ret)
+		return ret;
+
+	tdx->tdvpx = kcalloc(tdx_caps.tdvpx_nr_pages, sizeof(*tdx->tdvpx),
+			GFP_KERNEL_ACCOUNT);
+	if (!tdx->tdvpx) {
+		ret = -ENOMEM;
+		goto free_tdvpr;
+	}
+	for (i = 0; i < tdx_caps.tdvpx_nr_pages; i++) {
+		ret = tdx_alloc_td_page(&tdx->tdvpx[i]);
+		if (ret)
+			goto free_tdvpx;
+	}
+
+	vcpu->arch.efer = EFER_SCE | EFER_LME | EFER_LMA | EFER_NX;
+
+	vcpu->arch.cr0_guest_owned_bits = -1ul;
+	vcpu->arch.cr4_guest_owned_bits = -1ul;
+
+	vcpu->arch.tsc_offset = to_kvm_tdx(vcpu->kvm)->tsc_offset;
+	vcpu->arch.l1_tsc_offset = vcpu->arch.tsc_offset;
+	vcpu->arch.guest_state_protected =
+		!(to_kvm_tdx(vcpu->kvm)->attributes & TDX_TD_ATTRIBUTE_DEBUG);
+
+	return 0;
+
+free_tdvpx:
+	/* @i points at the TDVPX page that failed allocation. */
+	for (--i; i >= 0; i--)
+		free_page(tdx->tdvpx[i].va);
+	kfree(tdx->tdvpx);
+	tdx->tdvpx = NULL;
+free_tdvpr:
+	free_page(tdx->tdvpr.va);
+
+	return ret;
+}
+
+void tdx_vcpu_free(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	int i;
+
+	/* Can't reclaim or free pages if teardown failed. */
+	if (is_hkid_assigned(to_kvm_tdx(vcpu->kvm)))
+		return;
+
+	if (tdx->tdvpx) {
+		for (i = 0; i < tdx_caps.tdvpx_nr_pages; i++)
+			tdx_reclaim_td_page(&tdx->tdvpx[i]);
+		kfree(tdx->tdvpx);
+		tdx->tdvpx = NULL;
+	}
+	tdx_reclaim_td_page(&tdx->tdvpr);
+}
+
+void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct msr_data apic_base_msr;
+	u64 err;
+	int i;
+
+	/* TDX doesn't support INIT event. */
+	if (WARN_ON_ONCE(init_event))
+		goto td_bugged;
+	if (WARN_ON_ONCE(is_td_vcpu_created(tdx)))
+		goto td_bugged;
+
+	err = tdh_vp_create(kvm_tdx->tdr.pa, tdx->tdvpr.pa);
+	if (WARN_ON_ONCE(err)) {
+		pr_tdx_error(TDH_VP_CREATE, err, NULL);
+		goto td_bugged;
+	}
+	tdx_mark_td_page_added(&tdx->tdvpr);
+
+	for (i = 0; i < tdx_caps.tdvpx_nr_pages; i++) {
+		err = tdh_vp_addcx(tdx->tdvpr.pa, tdx->tdvpx[i].pa);
+		if (WARN_ON_ONCE(err)) {
+			pr_tdx_error(TDH_VP_ADDCX, err, NULL);
+			goto td_bugged;
+		}
+		tdx_mark_td_page_added(&tdx->tdvpx[i]);
+	}
+
+	if (!vcpu->arch.cpuid_entries) {
+		/*

Quoted 8 lines

arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 410
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +6 -0
+		 * On cpu creation, cpuid entry is blank.  Forcibly enable
+		 * X2APIC feature to allow X2APIC.
+		 */
+		struct kvm_cpuid_entry2 *e;
+
+		e = kvmalloc_array(1, sizeof(*e), GFP_KERNEL_ACCOUNT);
NULL checking is necessary for kvmalloc_array.

Fixed. Because vcpu_reset() function doesn't return error, this logic is moved to tdx_vcpu_create().

Thanks,

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:25 UTC #61 lore raw

PATCH v10 25/108 KVM: TDX: Do TDX specific vcpu initialization

From: Sean Christopherson <[email protected]>

TD guest vcpu need to be configured before ready to run which requests addtional information from Device model (e.g. qemu), one 64bit value is passed to vcpu's RCX as an initial value. Repurpose KVM_MEMORY_ENCRYPT_OP to vcpu-scope and add new sub-commands KVM_TDX_INIT_VCPU under it for such additional vcpu configuration.

Add callback for kvm vCPU-scoped operations of KVM_MEMORY_ENCRYPT_OP and add a new subcommand, KVM_TDX_INIT_VCPU, for further vcpu initialization.

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/kvm-x86-ops.h    |   1 +
 arch/x86/include/asm/kvm_host.h       |   1 +
 arch/x86/include/uapi/asm/kvm.h       |   1 +
 arch/x86/kvm/vmx/main.c               |   9 ++
 arch/x86/kvm/vmx/tdx.c                | 166 ++++++++++++++++++--------
 arch/x86/kvm/vmx/tdx.h                |   4 +
 arch/x86/kvm/vmx/x86_ops.h            |   2 +
 arch/x86/kvm/x86.c                    |   6 +
 tools/arch/x86/include/uapi/asm/kvm.h |   1 +
 9 files changed, 139 insertions(+), 52 deletions(-)

arch/x86/include/asm/kvm-x86-ops.h +1 -0

index 4425564647cb..f28c9fd72ac4 100644
@@ -120,6 +120,7 @@ KVM_X86_OP(leave_smm)
 KVM_X86_OP(enable_smi_window)
 KVM_X86_OP_OPTIONAL(dev_mem_enc_ioctl)
 KVM_X86_OP_OPTIONAL(mem_enc_ioctl)
+KVM_X86_OP_OPTIONAL(vcpu_mem_enc_ioctl)
 KVM_X86_OP_OPTIONAL(mem_enc_register_region)
 KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)
 KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from)

arch/x86/include/asm/kvm_host.h +1 -0

index 1fced310ec63..829a07d23909 100644
@@ -1628,6 +1628,7 @@ struct kvm_x86_ops {
 
 	int (*dev_mem_enc_ioctl)(void __user *argp);
 	int (*mem_enc_ioctl)(struct kvm *kvm, void __user *argp);
+	int (*vcpu_mem_enc_ioctl)(struct kvm_vcpu *vcpu, void __user *argp);
 	int (*mem_enc_register_region)(struct kvm *kvm, struct kvm_enc_region *argp);
 	int (*mem_enc_unregister_region)(struct kvm *kvm, struct kvm_enc_region *argp);
 	int (*vm_copy_enc_context_from)(struct kvm *kvm, unsigned int source_fd);

arch/x86/include/uapi/asm/kvm.h +1 -0

index 26661879c031..80db152430e4 100644
@@ -539,6 +539,7 @@ struct kvm_pmu_event_filter {
 enum kvm_tdx_cmd_id {
 	KVM_TDX_CAPABILITIES = 0,
 	KVM_TDX_INIT_VM,
+	KVM_TDX_INIT_VCPU,
 
 	KVM_TDX_CMD_NR_MAX,
 };

arch/x86/kvm/vmx/main.c +9 -0

index c125b2e3e8b4..0d5ca65e9997 100644
@@ -103,6 +103,14 @@ static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp)
 	return tdx_vm_ioctl(kvm, argp);
 }
 
+static int vt_vcpu_mem_enc_ioctl(struct kvm_vcpu *vcpu, void __user *argp)
+{
+	if (!is_td_vcpu(vcpu))
+		return -EINVAL;
+
+	return tdx_vcpu_ioctl(vcpu, argp);
+}
+
 struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = "kvm_intel",

@@ -246,6 +254,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 
 	.dev_mem_enc_ioctl = tdx_dev_ioctl,
 	.mem_enc_ioctl = vt_mem_enc_ioctl,
+	.vcpu_mem_enc_ioctl = vt_vcpu_mem_enc_ioctl,
 };
 
 struct kvm_x86_init_ops vt_init_ops __initdata = {

arch/x86/kvm/vmx/tdx.c +114 ~~-52~~

221 lines not on this page · raw message

arch/x86/kvm/vmx/tdx.h +4 -0

index 5aea69716278..a95f25845f24 100644
@@ -23,6 +23,8 @@ struct kvm_tdx {
 	u64 xfam;
 	int hkid;
 
+	bool finalized;
+
 	u64 tsc_offset;
 };

@@ -32,6 +34,8 @@ struct vcpu_tdx {
 	struct tdx_td_page tdvpr;
 	struct tdx_td_page *tdvpx;
 
+	bool vcpu_initialized;
+
 	/*
 	 * Dummy to make pmu_intel not corrupt memory.
 	 * TODO: Support PMU for TDX.  Future work.

arch/x86/kvm/vmx/x86_ops.h +2 -0

index f6841c3dd12d..fda1b2eaebc6 100644
@@ -146,6 +146,7 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu);
 void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);
 
 int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);
+int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp);
 #else
 static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return 0; }
 static inline bool tdx_is_vm_type_supported(unsigned long type) { return false; }

@@ -163,6 +164,7 @@ static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {}
 static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}
 
 static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPNOTSUPP; }
+static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) { return -EOPNOTSUPP; }
 #endif
 
 #endif /* __KVM_X86_VMX_X86_OPS_H */

arch/x86/kvm/x86.c +6 -0

index ddcbbcf13a55..a811d643f71c 100644
@@ -5963,6 +5963,12 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 	case KVM_SET_DEVICE_ATTR:
 		r = kvm_vcpu_ioctl_device_attr(vcpu, ioctl, argp);
 		break;
+	case KVM_MEMORY_ENCRYPT_OP:
+		r = -ENOTTY;
+		if (!kvm_x86_ops.vcpu_mem_enc_ioctl)
+			goto out;
+		r = kvm_x86_ops.vcpu_mem_enc_ioctl(vcpu, argp);
+		break;
 	default:
 		r = -EINVAL;
 	}

tools/arch/x86/include/uapi/asm/kvm.h +1 -0

index 531a0033e530..35e3b4aa2e96 100644
@@ -539,6 +539,7 @@ struct kvm_pmu_event_filter {
 enum kvm_tdx_cmd_id {
 	KVM_TDX_CAPABILITIES = 0,
 	KVM_TDX_INIT_VM,
+	KVM_TDX_INIT_VCPU,
 
 	KVM_TDX_CMD_NR_MAX,
 };

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #62 lore raw

PATCH v10 26/108 KVM: TDX: Use private memory for TDX

From: Chao Peng <[email protected]>

Override kvm_arch_has_private_mem() to use fd-based private memory. Return true when a VM has a type of KVM_X86_TDX_VM.

Signed-off-by Chao Peng <[email protected]>

---
 arch/x86/kvm/x86.c | 5 +++++
 1 file changed, 5 insertions(+)

arch/x86/kvm/x86.c +5 -0

index a811d643f71c..ba4a9ce0ee80 100644
@@ -13862,6 +13862,11 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size,
 }
 EXPORT_SYMBOL_GPL(kvm_sev_es_string_io);
 
+bool kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	return kvm->arch.vm_type == KVM_X86_TDX_VM;
+}
+
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_entry);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_exit);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_fast_mmio);

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #63 lore raw

PATCH v10 27/108 The start of TDX KVM patch series: KVM MMU GPA shared bits

From: Isaku Yamahata <[email protected]>

This empty commit is to mark the start of patch series of KVM MMU GPA shared bits.

---
 Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Documentation/virt/kvm/intel-tdx-layer-status.rst +3 -2

index 3e8efde3e3f3..6e3f71ab6b59 100644
@@ -10,6 +10,7 @@ What qemu can do
 ----------------
 - TDX VM TYPE is exposed to Qemu.
 - Qemu can create/destroy guest of TDX vm type.
+- Qemu can create/destroy vcpu of TDX vm type.
 
 Patch Layer status
 ------------------

@@ -17,13 +18,13 @@ Patch Layer status
 * TDX, VMX coexistence:                 Applied
 * TDX architectural definitions:        Applied
 * TD VM creation/destruction:           Applied
-* TD vcpu creation/destruction:         Applying
+* TD vcpu creation/destruction:         Applied
 * TDX EPT violation:                    Not yet
 * TD finalization:                      Not yet
 * TD vcpu enter/exit:                   Not yet
 * TD vcpu interrupts/exit/hypercall:    Not yet
 
-* KVM MMU GPA shared bits:              Not yet
+* KVM MMU GPA shared bits:              Applying
 * KVM TDP refactoring for TDX:          Not yet
 * KVM TDP MMU hooks:                    Not yet
 * KVM TDP MMU MapGPA:                   Not yet

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #64 lore raw

PATCH v10 28/108 KVM: x86/mmu: introduce config for PRIVATE KVM MMU

From: Isaku Yamahata <[email protected]>

To keep the case of non TDX intact, introduce a new config option for private KVM MMU support. At the moment, this is synonym for CONFIG_INTEL_TDX_HOST && CONFIG_KVM_INTEL. The config makes it clear that the config is only for x86 KVM MMU.

---
 arch/x86/kvm/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)

arch/x86/kvm/Kconfig +4 -0

index 73fdfa429b20..6bafdb2ce284 100644
@@ -133,4 +133,8 @@ config KVM_XEN
 config KVM_EXTERNAL_WRITE_TRACKING
 	bool
 
+config KVM_MMU_PRIVATE
+	def_bool y
+	depends on INTEL_TDX_HOST && KVM_INTEL
+
 endif # VIRTUALIZATION

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:24 UTC #65 lore raw

PATCH v10 29/108 KVM: x86/mmu: Add address conversion functions for TDX shared bit of GPA

From: Isaku Yamahata <[email protected]>

TDX repurposes one GPA bit (51 bit or 47 bit based on configuration) to indicate the GPA is private(if cleared) or shared (if set) with VMM. If GPA.shared is set, GPA is covered by the existing conventional EPT pointed by EPTP. If GPA.shared bit is cleared, GPA is covered by TDX module. VMM has to issue SEAMCALLs to operate.

Add a member to remember GPA shared bit for each guest TDs, add address conversion functions between private GPA and shared GPA and test if GPA is private.

Because struct kvm_arch (or struct kvm which includes struct kvm_arch. See kvm_arch_alloc_vm() that passes __GPF_ZERO) is zero-cleared when allocated, the new member to remember GPA shared bit is guaranteed to be zero with this patch unless it's initialized explicitly.

Co-developed-by Rick Edgecombe <[email protected]>

Signed-off-by Rick Edgecombe <[email protected]>

---
 arch/x86/include/asm/kvm_host.h |  4 ++++
 arch/x86/kvm/mmu.h              | 32 ++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx.c          |  5 +++++
 3 files changed, 41 insertions(+)

arch/x86/include/asm/kvm_host.h +4 -0

index 829a07d23909..3374ec0d6d90 100644
@@ -1372,6 +1372,10 @@ struct kvm_arch {
 	 */
 #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
 	struct kvm_mmu_memory_cache split_desc_cache;
+
+#ifdef CONFIG_KVM_MMU_PRIVATE
+	gfn_t gfn_shared_mask;
+#endif
 };
 
 struct kvm_vm_stat {

arch/x86/kvm/mmu.h +32 -0

index 6bdaacb6faa0..a45f7a96b821 100644
@@ -278,4 +278,36 @@ static inline gpa_t kvm_translate_gpa(struct kvm_vcpu *vcpu,
 		return gpa;
 	return translate_nested_gpa(vcpu, gpa, access, exception);
 }
+
+static inline gfn_t kvm_gfn_shared_mask(const struct kvm *kvm)
+{
+#ifdef CONFIG_KVM_MMU_PRIVATE
+	return kvm->arch.gfn_shared_mask;
+#else
+	return 0;
+#endif
+}
+
+static inline gfn_t kvm_gfn_shared(const struct kvm *kvm, gfn_t gfn)
+{
+	return gfn | kvm_gfn_shared_mask(kvm);
+}
+
+static inline gfn_t kvm_gfn_private(const struct kvm *kvm, gfn_t gfn)
+{
+	return gfn & ~kvm_gfn_shared_mask(kvm);
+}
+
+static inline gpa_t kvm_gpa_private(const struct kvm *kvm, gpa_t gpa)
+{
+	return gpa & ~gfn_to_gpa(kvm_gfn_shared_mask(kvm));
+}
+
+static inline bool kvm_is_private_gpa(const struct kvm *kvm, gpa_t gpa)
+{
+	gfn_t mask = kvm_gfn_shared_mask(kvm);
+
+	return mask && !(gpa_to_gfn(gpa) & mask);
+}
+
 #endif

arch/x86/kvm/vmx/tdx.c +5 -0

index fd9210cb4f36..e80f9cf79b2e 100644
@@ -785,6 +785,11 @@ static int tdx_td_init(struct kvm *kvm, struct kvm_tdx_cmd *cmd)
 	kvm_tdx->attributes = td_params->attributes;
 	kvm_tdx->xfam = td_params->xfam;
 
+	if (td_params->exec_controls & TDX_EXEC_CONTROL_MAX_GPAW)
+		kvm->arch.gfn_shared_mask = gpa_to_gfn(BIT_ULL(51));
+	else
+		kvm->arch.gfn_shared_mask = gpa_to_gfn(BIT_ULL(47));
+
 out:
 	/* kfree() accepts NULL. */
 	kfree(init_vm);

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #66 lore raw

PATCH v10 30/108 The start of TDX KVM patch series: KVM TDP refactoring for TDX

From: Isaku Yamahata <[email protected]>

This empty commit is to mark the start of patch series of KVM TDP refactoring for TDX.

---
 Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2

index 6e3f71ab6b59..df003d2ed89e 100644
@@ -24,7 +24,7 @@ Patch Layer status
 * TD vcpu enter/exit:                   Not yet
 * TD vcpu interrupts/exit/hypercall:    Not yet
 
-* KVM MMU GPA shared bits:              Applying
-* KVM TDP refactoring for TDX:          Not yet
+* KVM MMU GPA shared bits:              Applied
+* KVM TDP refactoring for TDX:          Applying
 * KVM TDP MMU hooks:                    Not yet
 * KVM TDP MMU MapGPA:                   Not yet

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #67 lore raw

PATCH v10 31/108 KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE

From: Isaku Yamahata <[email protected]>

The TDX support will need the "suppress #VE" bit (bit 63) set as the initial value for SPTE. To reduce code change size, introduce a new macro SHADOW_NONPRESENT_VALUE for the initial value for the shadow page table entry (SPTE) and replace hard-coded value 0 for it. Initialize shadow page tables with their value.

The plan is to unconditionally set the "suppress #VE" bit for both AMD and Intel as: 1) AMD hardware doesn't use this bit; 2) for conventional VMX guests, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware.

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/kvm/mmu/mmu.c     | 50 +++++++++++++++++++++++++++++++++-----
 arch/x86/kvm/mmu/spte.h    |  2 ++
 arch/x86/kvm/mmu/tdp_mmu.c | 15 ++++++------
 3 files changed, 54 insertions(+), 13 deletions(-)

arch/x86/kvm/mmu/mmu.c +44 -6

index 10017a9f26ee..e7e11f51f8b4 100644
@@ -538,9 +538,9 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep)
 
 	if (!is_shadow_present_pte(old_spte) ||
 	    !spte_has_volatile_bits(old_spte))
-		__update_clear_spte_fast(sptep, 0ull);
+		__update_clear_spte_fast(sptep, SHADOW_NONPRESENT_VALUE);
 	else
-		old_spte = __update_clear_spte_slow(sptep, 0ull);
+		old_spte = __update_clear_spte_slow(sptep, SHADOW_NONPRESENT_VALUE);
 
 	if (!is_shadow_present_pte(old_spte))
 		return old_spte;

@@ -574,7 +574,7 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep)
  */
 static void mmu_spte_clear_no_track(u64 *sptep)
 {
-	__update_clear_spte_fast(sptep, 0ull);
+	__update_clear_spte_fast(sptep, SHADOW_NONPRESENT_VALUE);
 }
 
 static u64 mmu_spte_get_lockless(u64 *sptep)

@@ -642,6 +642,39 @@ static void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu)
 	}
 }
 
+#ifdef CONFIG_X86_64
+static inline void kvm_init_shadow_page(void *page)
+{
+	memset64(page, SHADOW_NONPRESENT_VALUE, 4096 / 8);
+}
+
+static int mmu_topup_shadow_page_cache(struct kvm_vcpu *vcpu)
+{
+	struct kvm_mmu_memory_cache *mc = &vcpu->arch.mmu_shadow_page_cache;
+	int start, end, i, r;
+
+	start = kvm_mmu_memory_cache_nr_free_objects(mc);
+	r = kvm_mmu_topup_memory_cache(mc, PT64_ROOT_MAX_LEVEL);
+
+	/*
+	 * Note, topup may have allocated objects even if it failed to allocate
+	 * the minimum number of objects required to make forward progress _at
+	 * this time_.  Initialize newly allocated objects even on failure, as
+	 * userspace can free memory and rerun the vCPU in response to -ENOMEM.
+	 */
+	end = kvm_mmu_memory_cache_nr_free_objects(mc);
+	for (i = start; i < end; i++)
+		kvm_init_shadow_page(mc->objects[i]);
+	return r;
+}
+#else
+static int mmu_topup_shadow_page_cache(struct kvm_vcpu *vcpu)
+{
+	return kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache,
+					  PT64_ROOT_MAX_LEVEL);
+}
+#endif /* CONFIG_X86_64 */
+
 static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect)
 {
 	int r;

@@ -651,8 +684,7 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect)
 				       1 + PT64_ROOT_MAX_LEVEL + PTE_PREFETCH_NUM);
 	if (r)
 		return r;
-	r = kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache,
-				       PT64_ROOT_MAX_LEVEL);
+	r = mmu_topup_shadow_page_cache(vcpu);
 	if (r)
 		return r;
 	if (maybe_indirect) {

@@ -5870,7 +5902,13 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)
 	vcpu->arch.mmu_page_header_cache.kmem_cache = mmu_page_header_cache;
 	vcpu->arch.mmu_page_header_cache.gfp_zero = __GFP_ZERO;
 
-	vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO;
+	/*
+	 * When X86_64, initial SEPT entries are initialized with
+	 * SHADOW_NONPRESENT_VALUE.  Otherwise zeroed.  See
+	 * mmu_topup_shadow_page_cache().
+	 */
+	if (!IS_ENABLED(CONFIG_X86_64))
+		vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO;
 
 	vcpu->arch.mmu = &vcpu->arch.root_mmu;
 	vcpu->arch.walk_mmu = &vcpu->arch.root_mmu;

arch/x86/kvm/mmu/spte.h +2 -0

index 7670c13ce251..42ecaa75da15 100644
@@ -148,6 +148,8 @@ static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11);
 
 #define MMIO_SPTE_GEN_MASK		GENMASK_ULL(MMIO_SPTE_GEN_LOW_BITS + MMIO_SPTE_GEN_HIGH_BITS - 1, 0)
 
+#define SHADOW_NONPRESENT_VALUE	0ULL
+
 extern u64 __read_mostly shadow_host_writable_mask;
 extern u64 __read_mostly shadow_mmu_writable_mask;
 extern u64 __read_mostly shadow_nx_mask;

arch/x86/kvm/mmu/tdp_mmu.c +8 -7

index eab765442d0b..38bc4c2f0f1f 100644
@@ -694,7 +694,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm,
 	 * here since the SPTE is going from non-present to non-present.  Use
 	 * the raw write helper to avoid an unnecessary check on volatile bits.
 	 */
-	__kvm_tdp_mmu_write_spte(iter->sptep, 0);
+	__kvm_tdp_mmu_write_spte(iter->sptep, SHADOW_NONPRESENT_VALUE);
 
 	return 0;
 }

@@ -871,8 +871,8 @@ static void __tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root,
 			continue;
 
 		if (!shared)
-			tdp_mmu_set_spte(kvm, &iter, 0);
-		else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0))
+			tdp_mmu_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE);
+		else if (tdp_mmu_set_spte_atomic(kvm, &iter, SHADOW_NONPRESENT_VALUE))
 			goto retry;
 	}
 }

@@ -928,8 +928,9 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
 	if (WARN_ON_ONCE(!is_shadow_present_pte(old_spte)))
 		return false;
 
-	__tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0,
-			   sp->gfn, sp->role.level + 1, true, true);
+	__tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte,
+			   SHADOW_NONPRESENT_VALUE, sp->gfn, sp->role.level + 1,
+			   true, true);
 
 	return true;
 }

@@ -963,7 +964,7 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,
 		    !is_last_spte(iter.old_spte, iter.level))
 			continue;
 
-		tdp_mmu_set_spte(kvm, &iter, 0);
+		tdp_mmu_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE);
 		flush = true;
 	}

@@ -1328,7 +1329,7 @@ static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter,
 	 * invariant that the PFN of a present * leaf SPTE can never change.
 	 * See __handle_changed_spte().
 	 */
-	tdp_mmu_set_spte(kvm, iter, 0);
+	tdp_mmu_set_spte(kvm, iter, SHADOW_NONPRESENT_VALUE);
 
 	if (!pte_write(range->pte)) {
 		new_spte = kvm_mmu_changed_pte_notifier_make_spte(iter->old_spte,

Signature · 1 line

2.25.1

Binbin Wu

2022-11-03 07:17 UTC #68 lore raw

Re: [PATCH v10 031/108] KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE

Quoted 47 lines · On 2022/10/30 14:22, [email protected] wrote:

From: Isaku Yamahata <[email protected]>
The TDX support will need the "suppress #VE" bit (bit 63) set as the initial value for SPTE. To reduce code change size, introduce a new macro SHADOW_NONPRESENT_VALUE for the initial value for the shadow page table entry (SPTE) and replace hard-coded value 0 for it. Initialize shadow page tables with their value.
The plan is to unconditionally set the "suppress #VE" bit for both AMD and Intel as: 1) AMD hardware doesn't use this bit; 2) for conventional VMX guests, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
---
  arch/x86/kvm/mmu/mmu.c     | 50 +++++++++++++++++++++++++++++++++-----
  arch/x86/kvm/mmu/spte.h    |  2 ++
  arch/x86/kvm/mmu/tdp_mmu.c | 15 ++++++------
  3 files changed, 54 insertions(+), 13 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -3
index 10017a9f26ee..e7e11f51f8b4 100644
@@ -538,9 +538,9 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep)
 
  	if (!is_shadow_present_pte(old_spte) ||
  	    !spte_has_volatile_bits(old_spte))
-		__update_clear_spte_fast(sptep, 0ull);
+		__update_clear_spte_fast(sptep, SHADOW_NONPRESENT_VALUE);
  	else
-		old_spte = __update_clear_spte_slow(sptep, 0ull);
+		old_spte = __update_clear_spte_slow(sptep, SHADOW_NONPRESENT_VALUE);
 
  	if (!is_shadow_present_pte(old_spte))
  		return old_spte;
@@ -574,7 +574,7 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep)
   */
  static void mmu_spte_clear_no_track(u64 *sptep)
  {
-	__update_clear_spte_fast(sptep, 0ull);
+	__update_clear_spte_fast(sptep, SHADOW_NONPRESENT_VALUE);
  }
 
  static u64 mmu_spte_get_lockless(u64 *sptep)

arch/x86/kvm/mmu/mmu.c static void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu) · line 644

Quoted 3 lines

arch/x86/kvm/mmu/mmu.c +0 -0

@@ -642,6 +642,39 @@ static void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu)
  	}
  }

Why the changes in mmu_spte_clear_track_bits and mmu_spte_clear_no_track are not handled differently for X86_64 and non X86_64 as the shadow page init below? It seems the two functions are the common code.

Quoted 128 lines

arch/x86/kvm/mmu/mmu.c +41 -3

+#ifdef CONFIG_X86_64
+static inline void kvm_init_shadow_page(void *page)
+{
+	memset64(page, SHADOW_NONPRESENT_VALUE, 4096 / 8);
+}
+
+static int mmu_topup_shadow_page_cache(struct kvm_vcpu *vcpu)
+{
+	struct kvm_mmu_memory_cache *mc = &vcpu->arch.mmu_shadow_page_cache;
+	int start, end, i, r;
+
+	start = kvm_mmu_memory_cache_nr_free_objects(mc);
+	r = kvm_mmu_topup_memory_cache(mc, PT64_ROOT_MAX_LEVEL);
+
+	/*
+	 * Note, topup may have allocated objects even if it failed to allocate
+	 * the minimum number of objects required to make forward progress _at
+	 * this time_.  Initialize newly allocated objects even on failure, as
+	 * userspace can free memory and rerun the vCPU in response to -ENOMEM.
+	 */
+	end = kvm_mmu_memory_cache_nr_free_objects(mc);
+	for (i = start; i < end; i++)
+		kvm_init_shadow_page(mc->objects[i]);
+	return r;
+}
+#else
+static int mmu_topup_shadow_page_cache(struct kvm_vcpu *vcpu)
+{
+	return kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache,
+					  PT64_ROOT_MAX_LEVEL);
+}
+#endif /* CONFIG_X86_64 */
+
  static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect)
  {
  	int r;

@@ -651,8 +684,7 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect)
  				       1 + PT64_ROOT_MAX_LEVEL + PTE_PREFETCH_NUM);
  	if (r)
  		return r;
-	r = kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache,
-				       PT64_ROOT_MAX_LEVEL);
+	r = mmu_topup_shadow_page_cache(vcpu);
  	if (r)
  		return r;
  	if (maybe_indirect) {

@@ -5870,7 +5902,13 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)
  	vcpu->arch.mmu_page_header_cache.kmem_cache = mmu_page_header_cache;
  	vcpu->arch.mmu_page_header_cache.gfp_zero = __GFP_ZERO;
 
-	vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO;
+	/*
+	 * When X86_64, initial SEPT entries are initialized with
+	 * SHADOW_NONPRESENT_VALUE.  Otherwise zeroed.  See
+	 * mmu_topup_shadow_page_cache().
+	 */
+	if (!IS_ENABLED(CONFIG_X86_64))
+		vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO;
 
  	vcpu->arch.mmu = &vcpu->arch.root_mmu;
  	vcpu->arch.walk_mmu = &vcpu->arch.root_mmu;

arch/x86/kvm/mmu/spte.h +2 -0

index 7670c13ce251..42ecaa75da15 100644
@@ -148,6 +148,8 @@ static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11);
 
  #define MMIO_SPTE_GEN_MASK		GENMASK_ULL(MMIO_SPTE_GEN_LOW_BITS + MMIO_SPTE_GEN_HIGH_BITS - 1, 0)
 
+#define SHADOW_NONPRESENT_VALUE	0ULL
+
  extern u64 __read_mostly shadow_host_writable_mask;
  extern u64 __read_mostly shadow_mmu_writable_mask;
  extern u64 __read_mostly shadow_nx_mask;

arch/x86/kvm/mmu/tdp_mmu.c +8 -7

index eab765442d0b..38bc4c2f0f1f 100644
@@ -694,7 +694,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm,
  	 * here since the SPTE is going from non-present to non-present.  Use
  	 * the raw write helper to avoid an unnecessary check on volatile bits.
  	 */
-	__kvm_tdp_mmu_write_spte(iter->sptep, 0);
+	__kvm_tdp_mmu_write_spte(iter->sptep, SHADOW_NONPRESENT_VALUE);
 
  	return 0;
  }

@@ -871,8 +871,8 @@ static void __tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root,
  			continue;
 
  		if (!shared)
-			tdp_mmu_set_spte(kvm, &iter, 0);
-		else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0))
+			tdp_mmu_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE);
+		else if (tdp_mmu_set_spte_atomic(kvm, &iter, SHADOW_NONPRESENT_VALUE))
  			goto retry;
  	}
  }

@@ -928,8 +928,9 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
  	if (WARN_ON_ONCE(!is_shadow_present_pte(old_spte)))
  		return false;
 
-	__tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0,
-			   sp->gfn, sp->role.level + 1, true, true);
+	__tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte,
+			   SHADOW_NONPRESENT_VALUE, sp->gfn, sp->role.level + 1,
+			   true, true);
 
  	return true;
  }

@@ -963,7 +964,7 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,
  		    !is_last_spte(iter.old_spte, iter.level))
  			continue;
 
-		tdp_mmu_set_spte(kvm, &iter, 0);
+		tdp_mmu_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE);
  		flush = true;
  	}

@@ -1328,7 +1329,7 @@ static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter,
  	 * invariant that the PFN of a present * leaf SPTE can never change.
  	 * See __handle_changed_spte().
  	 */
-	tdp_mmu_set_spte(kvm, iter, 0);
+	tdp_mmu_set_spte(kvm, iter, SHADOW_NONPRESENT_VALUE);
 
  	if (!pte_write(range->pte)) {
  		new_spte = kvm_mmu_changed_pte_notifier_make_spte(iter->old_spte,

Binbin Wu

2022-11-03 08:00 UTC #69 lore raw

Re: [PATCH v10 031/108] KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE

Quoted 47 lines · Binbin Wu

Quoted 45 lines · On 2022/10/30 14:22, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
The TDX support will need the "suppress #VE" bit (bit 63) set as the initial value for SPTE. To reduce code change size, introduce a new macro SHADOW_NONPRESENT_VALUE for the initial value for the shadow page table entry (SPTE) and replace hard-coded value 0 for it. Initialize shadow page tables with their value.
The plan is to unconditionally set the "suppress #VE" bit for both AMD and Intel as: 1) AMD hardware doesn't use this bit; 2) for conventional VMX guests, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
---
arch/x86/kvm/mmu/mmu.c | 50 +++++++++++++++++++++++++++++++++----- arch/x86/kvm/mmu/spte.h | 2 ++ arch/x86/kvm/mmu/tdp_mmu.c | 15 ++++++------
  3 files changed, 54 insertions(+), 13 deletions(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index 10017a9f26ee..e7e11f51f8b4 100644
@@ -538,9 +538,9 @@ static u64 mmu_spte_clear_track_bits(struct kvm 
 *kvm, u64 *sptep)
      if (!is_shadow_present_pte(old_spte) || !spte_has_volatile_bits(old_spte))
-        __update_clear_spte_fast(sptep, 0ull);
+        __update_clear_spte_fast(sptep, SHADOW_NONPRESENT_VALUE); else
-        old_spte = __update_clear_spte_slow(sptep, 0ull); +        old_spte = __update_clear_spte_slow(sptep, SHADOW_NONPRESENT_VALUE);
      if (!is_shadow_present_pte(old_spte))
          return old_spte;
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -574,7 +574,7 @@ static u64 mmu_spte_clear_track_bits(struct kvm 
 *kvm, u64 *sptep)
    */
   static void mmu_spte_clear_no_track(u64 *sptep)
   {
-    __update_clear_spte_fast(sptep, 0ull);
+    __update_clear_spte_fast(sptep, SHADOW_NONPRESENT_VALUE);
   }

Quoted 8 lines

arch/x86/kvm/mmu/mmu.c static void walk_shadow_page_lockless_end(struct · line 644
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +0 -0
     static u64 mmu_spte_get_lockless(u64 *sptep)
@@ -642,6 +642,39 @@ static void walk_shadow_page_lockless_end(struct 
 kvm_vcpu *vcpu)
       }
   }
Why the changes in mmu_spte_clear_track_bits and mmu_spte_clear_no_track are not handled differently for X86_64 and non X86_64 as the shadow page init below? It seems the two functions are the common code.

I see there are different definitions of SHADOW_NONPRESENT_VALUE for X86_64 and non X86_64 in the following patch. So got the answer now. : )

Quoted 132 lines

arch/x86/kvm/mmu/mmu.c +16 -0

+#ifdef CONFIG_X86_64
+static inline void kvm_init_shadow_page(void *page)
+{
+    memset64(page, SHADOW_NONPRESENT_VALUE, 4096 / 8);
+}
+
+static int mmu_topup_shadow_page_cache(struct kvm_vcpu *vcpu)
+{
+    struct kvm_mmu_memory_cache *mc = 
 &vcpu->arch.mmu_shadow_page_cache;
+    int start, end, i, r;
+
+    start = kvm_mmu_memory_cache_nr_free_objects(mc);
+    r = kvm_mmu_topup_memory_cache(mc, PT64_ROOT_MAX_LEVEL);
+
+    /*
+     * Note, topup may have allocated objects even if it failed to

allocate
+     * the minimum number of objects required to make forward progress _at
+     * this time_. Initialize newly allocated objects even on failure, as
+     * userspace can free memory and rerun the vCPU in response to

arch/x86/kvm/mmu/mmu.c +10 -1

-ENOMEM.
+     */
+    end = kvm_mmu_memory_cache_nr_free_objects(mc);
+    for (i = start; i < end; i++)
+        kvm_init_shadow_page(mc->objects[i]);
+    return r;
+}
+#else
+static int mmu_topup_shadow_page_cache(struct kvm_vcpu *vcpu)
+{
+    return

kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache, + PT64_ROOT_MAX_LEVEL);

arch/x86/kvm/mmu/mmu.c +3 -0

+}
+#endif /* CONFIG_X86_64 */
+

static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect) {
int r;

arch/x86/kvm/mmu/mmu.c +1 -2

@@ -651,8 +684,7 @@ static int mmu_topup_memory_caches(struct 
 kvm_vcpu *vcpu, bool maybe_indirect)
                          1 + PT64_ROOT_MAX_LEVEL + PTE_PREFETCH_NUM);
       if (r)
           return r;
-    r = kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache,
-                       PT64_ROOT_MAX_LEVEL);
+    r = mmu_topup_shadow_page_cache(vcpu);
       if (r)
           return r;

if (maybe_indirect) {

arch/x86/kvm/mmu/mmu.c +7 -0

@@ -5870,7 +5902,13 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)
       vcpu->arch.mmu_page_header_cache.kmem_cache = mmu_page_header_cache;
       vcpu->arch.mmu_page_header_cache.gfp_zero = __GFP_ZERO;
   -    vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO;
+    /*
+     * When X86_64, initial SEPT entries are initialized with
+     * SHADOW_NONPRESENT_VALUE.  Otherwise zeroed.  See
+     * mmu_topup_shadow_page_cache().
+     */
+    if (!IS_ENABLED(CONFIG_X86_64))
+        vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO;
         vcpu->arch.mmu = &vcpu->arch.root_mmu;
       vcpu->arch.walk_mmu = &vcpu->arch.root_mmu;

arch/x86/kvm/mmu/spte.h +0 -0

index 7670c13ce251..42ecaa75da15 100644
@@ -148,6 +148,8 @@ static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && 
 MMIO_SPTE_GEN_HIGH_BITS == 11);
     #define MMIO_SPTE_GEN_MASK GENMASK_ULL(MMIO_SPTE_GEN_LOW_BITS + MMIO_SPTE_GEN_HIGH_BITS - 1, 0)

+#define SHADOW_NONPRESENT_VALUE 0ULL
+
extern u64 __read_mostly shadow_host_writable_mask; extern u64 __read_mostly shadow_mmu_writable_mask; extern u64 __read_mostly shadow_nx_mask;

arch/x86/kvm/mmu/tdp_mmu.c +3 -3

index eab765442d0b..38bc4c2f0f1f 100644
@@ -694,7 +694,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct 
 kvm *kvm,
        * here since the SPTE is going from non-present to non-present.  Use
        * the raw write helper to avoid an unnecessary check on volatile bits.
        */
-    __kvm_tdp_mmu_write_spte(iter->sptep, 0);
+    __kvm_tdp_mmu_write_spte(iter->sptep, SHADOW_NONPRESENT_VALUE);
         return 0;
   }

@@ -871,8 +871,8 @@ static void __tdp_mmu_zap_root(struct kvm *kvm, 
 struct kvm_mmu_page *root,
               continue;
             if (!shared)
-            tdp_mmu_set_spte(kvm, &iter, 0);
-        else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0))
+            tdp_mmu_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE);
+        else if (tdp_mmu_set_spte_atomic(kvm, &iter, 
 SHADOW_NONPRESENT_VALUE))
               goto retry;
       }

}

arch/x86/kvm/mmu/tdp_mmu.c +4 -1

@@ -928,8 +928,9 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct 
 kvm_mmu_page *sp)
       if (WARN_ON_ONCE(!is_shadow_present_pte(old_spte)))
           return false;
   -    __tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0,
-               sp->gfn, sp->role.level + 1, true, true);
+    __tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte,
+               SHADOW_NONPRESENT_VALUE, sp->gfn, sp->role.level + 1,
+               true, true);
         return true;
   }

@@ -963,7 +964,7 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, 
 struct kvm_mmu_page *root,
               !is_last_spte(iter.old_spte, iter.level))
               continue;
   -        tdp_mmu_set_spte(kvm, &iter, 0);
+        tdp_mmu_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE);
           flush = true;
       }

@@ -1328,7 +1329,7 @@ static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter,
       * invariant that the PFN of a present * leaf SPTE can never change.
       * See __handle_changed_spte().
       */
-    tdp_mmu_set_spte(kvm, iter, 0);
+    tdp_mmu_set_spte(kvm, iter, SHADOW_NONPRESENT_VALUE);
      if (!pte_write(range->pte)) {
          new_spte = kvm_mmu_changed_pte_notifier_make_spte(iter->old_spte,

Huang, Kai

2022-11-08 11:33 UTC #70 lore raw

Re: [PATCH v10 031/108] KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE

https://lore.kernel.org/lkml/[email protected]/T/#m5ed59404da43677e99ac51dd1360dc22b964bcbb

Quoted 4 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

The plan is to unconditionally set the "suppress #VE" bit for both AMD and Intel as: 1) AMD hardware doesn't use this bit; 2) for conventional VMX guests, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware.

Hi Isaku,

"AMD hardware doesn't use this bit" is wrong:

Isaku Yamahata

2022-11-17 17:55 UTC #71 lore raw

Re: [PATCH v10 031/108] KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE

Quoted 11 lines · Kai

Quoted 4 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:
The plan is to unconditionally set the "suppress #VE" bit for both AMD and Intel as: 1) AMD hardware doesn't use this bit; 2) for conventional VMX guests, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware.
Hi Isaku,
"AMD hardware doesn't use this bit" is wrong:
https://lore.kernel.org/lkml/[email protected]/T/#m5ed59404da43677e99ac51dd1360dc22b964bcbb

Oops. I'll update the commit message.

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:25 UTC #72 lore raw

PATCH v10 32/108 KVM: x86/mmu: Make sync_page not use hard-coded 0 as the initial SPTE value

From: Isaku Yamahata <[email protected]>

FNAME(sync_page) in arch/x86/kvm/mmu/paging_tmpl.h assumes that the initial shadow page table entry (SPTE) is zero. Remove the assumption by using SHADOW_NONPRESENT_VALUE that will be updated from 0 to non-zero value.

---
 arch/x86/kvm/mmu/paging_tmpl.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

arch/x86/kvm/mmu/paging_tmpl.h +2 -1

index 5ab5f94dcb6f..6db3f2b5563a 100644
@@ -1036,7 +1036,8 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)
 		gpa_t pte_gpa;
 		gfn_t gfn;
 
-		if (!sp->spt[i])
+		/* spt[i] has initial value of shadow page table allocation */
+		if (sp->spt[i] == SHADOW_NONPRESENT_VALUE)
 			continue;
 
 		pte_gpa = first_pte_gpa + i * sizeof(pt_element_t);

Signature · 1 line

2.25.1

Huang, Kai

2022-11-09 11:24 UTC #73 lore raw

Re: [PATCH v10 032/108] KVM: x86/mmu: Make sync_page not use hard-coded 0 as the initial SPTE value

Quoted 16 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>
FNAME(sync_page) in arch/x86/kvm/mmu/paging_tmpl.h assumes that the initial shadow page table entry (SPTE) is zero. Remove the assumption by using SHADOW_NONPRESENT_VALUE that will be updated from 0 to non-zero value.
Signed-off-by Isaku Yamahata <[email protected]>
---
 arch/x86/kvm/mmu/paging_tmpl.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/paging_tmpl.h +0 -0
index 5ab5f94dcb6f..6db3f2b5563a 100644

arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) · line 1043

Quoted 9 lines

arch/x86/kvm/mmu/paging_tmpl.h +2 -1

@@ -1036,7 +1036,8 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)
 		gpa_t pte_gpa;
 		gfn_t gfn;
 
-		if (!sp->spt[i])
+		/* spt[i] has initial value of shadow page table allocation */
+		if (sp->spt[i] == SHADOW_NONPRESENT_VALUE)
 			continue;
 
 		pte_gpa = first_pte_gpa + i * sizeof(pt_element_t);

I think this patch can be merged to previous one? Looks there's no good reason to keep it as standalone.

Isaku Yamahata

2022-11-17 17:55 UTC #74 lore raw

Re: [PATCH v10 032/108] KVM: x86/mmu: Make sync_page not use hard-coded 0 as the initial SPTE value

Quoted 25 lines · Kai

Quoted 24 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
FNAME(sync_page) in arch/x86/kvm/mmu/paging_tmpl.h assumes that the initial shadow page table entry (SPTE) is zero. Remove the assumption by using SHADOW_NONPRESENT_VALUE that will be updated from 0 to non-zero value.
Signed-off-by Isaku Yamahata <[email protected]>
---
 arch/x86/kvm/mmu/paging_tmpl.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
index 5ab5f94dcb6f..6db3f2b5563a 100644
@@ -1036,7 +1036,8 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)
 		gpa_t pte_gpa;
 		gfn_t gfn;
 
-		if (!sp->spt[i])
+		/* spt[i] has initial value of shadow page table allocation */
+		if (sp->spt[i] == SHADOW_NONPRESENT_VALUE)
 			continue;
 

Quoted 4 lines

arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) · line 1043
Quoted 1 line
arch/x86/kvm/mmu/paging_tmpl.h +0 -0
 		pte_gpa = first_pte_gpa + i * sizeof(pt_element_t);
I think this patch can be merged to previous one? Looks there's no good reason to keep it as standalone.

Yes, will squash it.

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:25 UTC #75 lore raw

PATCH v10 33/108 KVM: x86/mmu: Allow non-zero value for non-present SPTE and removed SPTE

From: Sean Christopherson <[email protected]>

For TD guest, the current way to emulate MMIO doesn't work any more, as KVM is not able to access the private memory of TD guest and do the emulation. Instead, TD guest expects to receive #VE when it accesses the MMIO and then it can explicitly makes hypercall to KVM to get the expected information.

To achieve this, the TDX module always enables "EPT-violation #VE" in the VMCS control. And accordingly, KVM needs to configure the MMIO spte to trigger EPT violation (instead of misconfiguration) and at the same time, also clear the "suppress #VE" bit so the TD guest can get the #VE instead of causing actual EPT violation to KVM.

In order for KVM to be able to have chance to set up the correct SPTE for MMIO for TD guest, the default non-present SPTE must have the "suppress guest accesses the MMIO. Also, when TD guest accesses the actual shared memory, it should continue to trigger EPT violation to the KVM instead of receiving the #VE (the TDX module guarantees KVM will receive EPT violation for private memory access). This means for the shared memory, the SPTE also must have the "suppress #VE" bit set for the non-present SPTE.

Add "suppress VE" bit (bit 63) to SHADOW_NONPRESENT_VALUE and REMOVED_SPTE. Unconditionally set the "suppress #VE" bit (which is bit 63) for both AMD and Intel as: 1) AMD hardware doesn't use this bit when present bit is off;
2) for normal VMX guest, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware.

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/vmx.h |  1 +
 arch/x86/kvm/mmu/spte.c    |  4 +++-
 arch/x86/kvm/mmu/spte.h    | 22 +++++++++++++++++++++-
 arch/x86/kvm/mmu/tdp_mmu.c |  8 ++++++++
 4 files changed, 33 insertions(+), 2 deletions(-)

arch/x86/include/asm/vmx.h +1 -0

index 498dc600bd5c..cdbf12c1a83c 100644
@@ -511,6 +511,7 @@ enum vmcs_field {
 #define VMX_EPT_IPAT_BIT    			(1ull << 6)
 #define VMX_EPT_ACCESS_BIT			(1ull << 8)
 #define VMX_EPT_DIRTY_BIT			(1ull << 9)
+#define VMX_EPT_SUPPRESS_VE_BIT			(1ull << 63)
 #define VMX_EPT_RWX_MASK                        (VMX_EPT_READABLE_MASK |       \
 						 VMX_EPT_WRITABLE_MASK |       \
 						 VMX_EPT_EXECUTABLE_MASK)

arch/x86/kvm/mmu/spte.c +3 -1

index 2e08b2a45361..0b97a045c5f0 100644
@@ -419,7 +419,9 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only)
 	shadow_dirty_mask	= has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull;
 	shadow_nx_mask		= 0ull;
 	shadow_x_mask		= VMX_EPT_EXECUTABLE_MASK;
-	shadow_present_mask	= has_exec_only ? 0ull : VMX_EPT_READABLE_MASK;
+	/* VMX_EPT_SUPPRESS_VE_BIT is needed for W or X violation. */
+	shadow_present_mask	=
+		(has_exec_only ? 0ull : VMX_EPT_READABLE_MASK) | VMX_EPT_SUPPRESS_VE_BIT;
 	/*
 	 * EPT overrides the host MTRRs, and so KVM must program the desired
 	 * memtype directly into the SPTEs.  Note, this mask is just the mask

arch/x86/kvm/mmu/spte.h +21 -1

index 42ecaa75da15..7e0f79e8f45b 100644
@@ -148,7 +148,22 @@ static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11);
 
 #define MMIO_SPTE_GEN_MASK		GENMASK_ULL(MMIO_SPTE_GEN_LOW_BITS + MMIO_SPTE_GEN_HIGH_BITS - 1, 0)
 
+/*
+ * non-present SPTE value for both VMX and SVM for TDP MMU.
+ * For SVM NPT, for non-present spte (bit 0 = 0), other bits are ignored.
+ * For VMX EPT, bit 63 is ignored if #VE is disabled. (EPT_VIOLATION_VE=0)
+ *              bit 63 is #VE suppress if #VE is enabled. (EPT_VIOLATION_VE=1)
+ * For TDX:
+ *   Secure-EPT: TDX module sets EPT_VIOLATION_VE for Secure-EPT
+ *   private EPT: "suppress #VE" bit is ignored.  CPU doesn't walk it.
+ *   conventional EPT: "suppress #VE" bit must be set to get EPT violation
+ */
+#ifdef CONFIG_X86_64
+#define SHADOW_NONPRESENT_VALUE	BIT_ULL(63)
+static_assert(!(SHADOW_NONPRESENT_VALUE & SPTE_MMU_PRESENT_MASK));
+#else
 #define SHADOW_NONPRESENT_VALUE	0ULL
+#endif
 
 extern u64 __read_mostly shadow_host_writable_mask;
 extern u64 __read_mostly shadow_mmu_writable_mask;

@@ -189,13 +204,18 @@ extern u64 __read_mostly shadow_nonpresent_or_rsvd_mask;
  * non-present intermediate value. Other threads which encounter this value
  * should not modify the SPTE.
  *
+ * For X86_64 case, SHADOW_NONPRESENT_VALUE, "suppress #VE" bit, is set because
+ * "EPT violation #VE" in the secondary VM execution control may be enabled.
+ * Because TDX module sets "EPT violation #VE" for TD, "suppress #VE" bit for
+ * the conventional EPT needs to be set.
+ *
  * Use a semi-arbitrary value that doesn't set RWX bits, i.e. is not-present on
  * bot AMD and Intel CPUs, and doesn't set PFN bits, i.e. doesn't create a L1TF
  * vulnerability.  Use only low bits to avoid 64-bit immediates.
  *
  * Only used by the TDP MMU.
  */
-#define REMOVED_SPTE	0x5a0ULL
+#define REMOVED_SPTE	(SHADOW_NONPRESENT_VALUE | 0x5a0ULL)
 
 /* Removed SPTEs must not be misconstrued as shadow present PTEs. */
 static_assert(!(REMOVED_SPTE & SPTE_MMU_PRESENT_MASK));

arch/x86/kvm/mmu/tdp_mmu.c +8 -0

index 38bc4c2f0f1f..1eee9c159958 100644
@@ -693,6 +693,14 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm,
 	 * overwrite the special removed SPTE value. No bookkeeping is needed
 	 * here since the SPTE is going from non-present to non-present.  Use
 	 * the raw write helper to avoid an unnecessary check on volatile bits.
+	 *
+	 * Set non-present value to SHADOW_NONPRESENT_VALUE, rather than 0.
+	 * It is because when TDX is enabled, TDX module always
+	 * enables "EPT-violation #VE", so KVM needs to set
+	 * "suppress #VE" bit in EPT table entries, in order to get
+	 * real EPT violation, rather than TDVMCALL.  KVM sets
+	 * SHADOW_NONPRESENT_VALUE (which sets "suppress #VE" bit) so it
+	 * can be set when EPT table entries are zapped.
 	 */
 	__kvm_tdp_mmu_write_spte(iter->sptep, SHADOW_NONPRESENT_VALUE);

Signature · 1 line

2.25.1

Huang, Kai

2022-11-09 11:24 UTC #76 lore raw

Re: [PATCH v10 033/108] KVM: x86/mmu: Allow non-zero value for non-present SPTE and removed SPTE

Quoted 6 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Sean Christopherson <[email protected]>
For TD guest, the current way to emulate MMIO doesn't work any more, as KVM is not able to access the private memory of TD guest and do the emulation. Instead, TD guest expects to receive #VE when it accesses the MMIO and then it can explicitly makes hypercall to KVM to get the expected information.

		    ^
		    make

Quoted 11 lines

To achieve this, the TDX module always enables "EPT-violation #VE" in the VMCS control. And accordingly, KVM needs to configure the MMIO spte to trigger EPT violation (instead of misconfiguration) and at the same time, also clear the "suppress #VE" bit so the TD guest can get the #VE instead of causing actual EPT violation to KVM.
In order for KVM to be able to have chance to set up the correct SPTE for MMIO for TD guest, the default non-present SPTE must have the "suppress guest accesses the MMIO.

The above sentence is broken.

Quoted 38 lines

Also, when TD guest accesses the actual shared
memory, it should continue to trigger EPT violation to the KVM instead of receiving the #VE (the TDX module guarantees KVM will receive EPT violation for private memory access). This means for the shared memory, the SPTE also must have the "suppress #VE" bit set for the non-present SPTE.
Add "suppress VE" bit (bit 63) to SHADOW_NONPRESENT_VALUE and REMOVED_SPTE. Unconditionally set the "suppress #VE" bit (which is bit 63) for both AMD and Intel as: 1) AMD hardware doesn't use this bit when present bit is off;
2) for normal VMX guest, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware.
Signed-off-by Sean Christopherson <[email protected]>
Signed-off-by Isaku Yamahata <[email protected]>
---
 arch/x86/include/asm/vmx.h |  1 +
 arch/x86/kvm/mmu/spte.c    |  4 +++-
 arch/x86/kvm/mmu/spte.h    | 22 +++++++++++++++++++++-
 arch/x86/kvm/mmu/tdp_mmu.c |  8 ++++++++
 4 files changed, 33 insertions(+), 2 deletions(-)
arch/x86/include/asm/vmx.h +1 -0
index 498dc600bd5c..cdbf12c1a83c 100644
@@ -511,6 +511,7 @@ enum vmcs_field {
 #define VMX_EPT_IPAT_BIT    			(1ull << 6)
 #define VMX_EPT_ACCESS_BIT			(1ull << 8)
 #define VMX_EPT_DIRTY_BIT			(1ull << 9)
+#define VMX_EPT_SUPPRESS_VE_BIT			(1ull << 63)
 #define VMX_EPT_RWX_MASK                        (VMX_EPT_READABLE_MASK |       \
 						 VMX_EPT_WRITABLE_MASK |       \
 						 VMX_EPT_EXECUTABLE_MASK)
arch/x86/kvm/mmu/spte.c +0 -0
index 2e08b2a45361..0b97a045c5f0 100644

arch/x86/kvm/mmu/spte.c void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) · line 424

Quoted 7 lines

arch/x86/kvm/mmu/spte.c +3 -1

@@ -419,7 +419,9 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only)
 	shadow_dirty_mask	= has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull;
 	shadow_nx_mask		= 0ull;
 	shadow_x_mask		= VMX_EPT_EXECUTABLE_MASK;
-	shadow_present_mask	= has_exec_only ? 0ull : VMX_EPT_READABLE_MASK;
+	/* VMX_EPT_SUPPRESS_VE_BIT is needed for W or X violation. */
+	shadow_present_mask	=
+		(has_exec_only ? 0ull : VMX_EPT_READABLE_MASK) | VMX_EPT_SUPPRESS_VE_BIT;

I think this chunk can be in a separate patch since it doesn't handle fault from non-present to present, which is claimed by the patch title and changelog.

Or I think you need to describe handling W or X fault part in the changelog.

arch/x86/kvm/mmu/spte.h static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11); · line 152

Quoted 13 lines

arch/x86/kvm/mmu/spte.c +0 -0

 	/*
 	 * EPT overrides the host MTRRs, and so KVM must program the desired
 	 * memtype directly into the SPTEs.  Note, this mask is just the mask

arch/x86/kvm/mmu/spte.h +2 -0

index 42ecaa75da15..7e0f79e8f45b 100644
@@ -148,7 +148,22 @@ static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11);
 
 #define MMIO_SPTE_GEN_MASK		GENMASK_ULL(MMIO_SPTE_GEN_LOW_BITS + MMIO_SPTE_GEN_HIGH_BITS - 1, 0)
 
+/*
+ * non-present SPTE value for both VMX and SVM for TDP MMU.

      ^
      Non-present

arch/x86/kvm/mmu/spte.h static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11); · line 157

Quoted 5 lines

arch/x86/kvm/mmu/spte.h +5 -0

+ * For SVM NPT, for non-present spte (bit 0 = 0), other bits are ignored.
+ * For VMX EPT, bit 63 is ignored if #VE is disabled. (EPT_VIOLATION_VE=0)
+ *              bit 63 is #VE suppress if #VE is enabled. (EPT_VIOLATION_VE=1)
+ * For TDX:
+ *   Secure-EPT: TDX module sets EPT_VIOLATION_VE for Secure-EPT

"EPT-violation #VE" is a VMCS control, so it applies to both Secure-EPT and shared-EPT. Looks "for Secure-EPT" is a little bit confusing.

arch/x86/kvm/mmu/spte.h static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11); · line 158

Quoted 1 line

arch/x86/kvm/mmu/spte.h +1 -0

+ *   private EPT: "suppress #VE" bit is ignored.  CPU doesn't walk it.

Looks "private EPT" just comes out of blue. I am not sure whether you need to mention it here.

Quoted 12 lines

arch/x86/kvm/mmu/spte.h +7 -0

+ *   conventional EPT: "suppress #VE" bit must be set to get EPT violation
+ */
+#ifdef CONFIG_X86_64
+#define SHADOW_NONPRESENT_VALUE	BIT_ULL(63)
+static_assert(!(SHADOW_NONPRESENT_VALUE & SPTE_MMU_PRESENT_MASK));
+#else
 #define SHADOW_NONPRESENT_VALUE	0ULL
+#endif
 
 extern u64 __read_mostly shadow_host_writable_mask;
 extern u64 __read_mostly shadow_mmu_writable_mask;

arch/x86/kvm/mmu/spte.h extern u64 __read_mostly shadow_nonpresent_or_rsvd_mask; · line 211

Quoted 8 lines

arch/x86/kvm/mmu/spte.h +5 -0

@@ -189,13 +204,18 @@ extern u64 __read_mostly shadow_nonpresent_or_rsvd_mask;
  * non-present intermediate value. Other threads which encounter this value
  * should not modify the SPTE.
  *
+ * For X86_64 case, SHADOW_NONPRESENT_VALUE, "suppress #VE" bit, is set because
+ * "EPT violation #VE" in the secondary VM execution control may be enabled.
+ * Because TDX module sets "EPT violation #VE" for TD, "suppress #VE" bit for
+ * the conventional EPT needs to be set.
+ *

This comment looks duplicated with the above one. Not sure whether needed.

Quoted 20 lines

arch/x86/kvm/mmu/spte.h +1 -1

  * Use a semi-arbitrary value that doesn't set RWX bits, i.e. is not-present on
  * bot AMD and Intel CPUs, and doesn't set PFN bits, i.e. doesn't create a L1TF
  * vulnerability.  Use only low bits to avoid 64-bit immediates.
  *
  * Only used by the TDP MMU.
  */
-#define REMOVED_SPTE	0x5a0ULL
+#define REMOVED_SPTE	(SHADOW_NONPRESENT_VALUE | 0x5a0ULL)
 
 /* Removed SPTEs must not be misconstrued as shadow present PTEs. */
 static_assert(!(REMOVED_SPTE & SPTE_MMU_PRESENT_MASK));

arch/x86/kvm/mmu/tdp_mmu.c +1 -0

index 38bc4c2f0f1f..1eee9c159958 100644
@@ -693,6 +693,14 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm,
 	 * overwrite the special removed SPTE value. No bookkeeping is needed
 	 * here since the SPTE is going from non-present to non-present.  Use
 	 * the raw write helper to avoid an unnecessary check on volatile bits.
+	 *

arch/x86/kvm/mmu/tdp_mmu.c static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, · line 706

Quoted 10 lines

arch/x86/kvm/mmu/tdp_mmu.c +7 -0

+	 * Set non-present value to SHADOW_NONPRESENT_VALUE, rather than 0.
+	 * It is because when TDX is enabled, TDX module always
+	 * enables "EPT-violation #VE", so KVM needs to set
+	 * "suppress #VE" bit in EPT table entries, in order to get
+	 * real EPT violation, rather than TDVMCALL.  KVM sets
+	 * SHADOW_NONPRESENT_VALUE (which sets "suppress #VE" bit) so it
+	 * can be set when EPT table entries are zapped.
 	 */
 	__kvm_tdp_mmu_write_spte(iter->sptep, SHADOW_NONPRESENT_VALUE);

Ditto.

Isaku Yamahata

2022-11-17 17:58 UTC #77 lore raw

Re: [PATCH v10 033/108] KVM: x86/mmu: Allow non-zero value for non-present SPTE and removed SPTE

Quoted 16 lines · Kai

arch/x86/kvm/mmu/spte.c void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) · line 424
Quoted 12 lines
arch/x86/kvm/mmu/spte.c +3 -1
index 2e08b2a45361..0b97a045c5f0 100644
@@ -419,7 +419,9 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only)
 	shadow_dirty_mask	= has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull;
 	shadow_nx_mask		= 0ull;
 	shadow_x_mask		= VMX_EPT_EXECUTABLE_MASK;
-	shadow_present_mask	= has_exec_only ? 0ull : VMX_EPT_READABLE_MASK;
+	/* VMX_EPT_SUPPRESS_VE_BIT is needed for W or X violation. */
+	shadow_present_mask	=
+		(has_exec_only ? 0ull : VMX_EPT_READABLE_MASK) | VMX_EPT_SUPPRESS_VE_BIT;
I think this chunk can be in a separate patch since it doesn't handle fault from non-present to present, which is claimed by the patch title and changelog.

Quoted 1 line

Or I think you need to describe handling W or X fault part in the changelog.

Ok, let me try to split that hunk.

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:25 UTC #78 lore raw

PATCH v10 34/108 KVM: x86/mmu: Add Suppress VE bit to shadow_mmio_{value, mask}

From: Isaku Yamahata <[email protected]>

Because TDX will need shadow_mmio_mask to be VMX_SUPPRESS_VE | RWX and shadow_mmio_value to be 0, make VMX EPT case use same value for TDX shadow_mmio_mask. For VMX, VMX_SUPPRESS_VE doesn't matter, it doesn't affect VMX logic to add the bit to shadow_mmio_{value, mask}. Note that shadow_mmio_value will be per-VM value.

---
 arch/x86/kvm/mmu/spte.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

arch/x86/kvm/mmu/spte.c +2 -2

index 0b97a045c5f0..5d5c06d4fd89 100644
@@ -437,8 +437,8 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only)
 	 * EPT Misconfigurations are generated if the value of bits 2:0
 	 * of an EPT paging-structure entry is 110b (write/execute).
 	 */
-	kvm_mmu_set_mmio_spte_mask(VMX_EPT_MISCONFIG_WX_VALUE,
-				   VMX_EPT_RWX_MASK, 0);
+	kvm_mmu_set_mmio_spte_mask(VMX_EPT_MISCONFIG_WX_VALUE | VMX_EPT_SUPPRESS_VE_BIT,
+				   VMX_EPT_RWX_MASK | VMX_EPT_SUPPRESS_VE_BIT, 0);
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_set_ept_masks);

Signature · 1 line

2.25.1

Huang, Kai

2022-11-09 11:48 UTC #79 lore raw

Re: [PATCH v10 034/108] KVM: x86/mmu: Add Suppress VE bit to shadow_mmio_{value, mask}

Quoted 6 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>
Because TDX will need shadow_mmio_mask to be VMX_SUPPRESS_VE | RWX shadow_mmio_value to be 0, make VMX EPT case use same value for TDX shadow_mmio_mask.

TDX need to use different mmio_mask/value doesn't mean they need to be changed for VMX guest. I think the true purpose here is to still be able to use a global shadow_mmio_mask for both TDX and VMX guests. So please explicitly call out.

Quoted 3 lines

For VMX, VMX_SUPPRESS_VE doesn't matter, it doesn't
affect VMX logic to add the bit to shadow_mmio_{value, mask}. Note that shadow_mmio_value will be per-VM value.

Not sure why the last sentence matters.

Quoted 11 lines

Signed-off-by Isaku Yamahata <[email protected]>
---
 arch/x86/kvm/mmu/spte.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/spte.c +0 -0
index 0b97a045c5f0..5d5c06d4fd89 100644

arch/x86/kvm/mmu/spte.c void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) · line 440

Quoted 6 lines

arch/x86/kvm/mmu/spte.c +1 -2

@@ -437,8 +437,8 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only)
 	 * EPT Misconfigurations are generated if the value of bits 2:0
 	 * of an EPT paging-structure entry is 110b (write/execute).
 	 */
-	kvm_mmu_set_mmio_spte_mask(VMX_EPT_MISCONFIG_WX_VALUE,
-				   VMX_EPT_RWX_MASK, 0);
+	kvm_mmu_set_mmio_spte_mask(VMX_EPT_MISCONFIG_WX_VALUE | VMX_EPT_SUPPRESS_VE_BIT,

Is "suppress #VE" bit required for mmio_value for VMX guest? Why adding it?

Quoted 4 lines

arch/x86/kvm/mmu/spte.c +1 -0

+				   VMX_EPT_RWX_MASK | VMX_EPT_SUPPRESS_VE_BIT, 0);
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_set_ept_masks);

Isaku Yamahata

2022-11-17 18:02 UTC #80 lore raw

Re: [PATCH v10 034/108] KVM: x86/mmu: Add Suppress VE bit to shadow_mmio_{value, mask}

Quoted 8 lines · Kai

Quoted 6 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Because TDX will need shadow_mmio_mask to be VMX_SUPPRESS_VE | RWX shadow_mmio_value to be 0, make VMX EPT case use same value for TDX shadow_mmio_mask.

Quoted 4 lines

TDX need to use different mmio_mask/value doesn't mean they need to be changed for VMX guest. I think the true purpose here is to still be able to use a global shadow_mmio_mask for both TDX and VMX guests. So please explicitly call out.

That's right. With this change, per-VM shadow_mmio_{value, mask} can be avoided. The common value can be used for both VMX and TDX.

Signature · 1 line

Isaku Yamahata <[email protected]>

Sean Christopherson

maintainer

2022-11-28 23:51 UTC #81 lore raw

Re: [PATCH v10 034/108] KVM: x86/mmu: Add Suppress VE bit to shadow_mmio_{value, mask}

Quoted 11 lines · Isaku Yamahata

Quoted 8 lines · Kai
Quoted 6 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Because TDX will need shadow_mmio_mask to be VMX_SUPPRESS_VE | RWX shadow_mmio_value to be 0, make VMX EPT case use same value for TDX shadow_mmio_mask.

Quoted 7 lines

Quoted 4 lines
TDX need to use different mmio_mask/value doesn't mean they need to be changed for VMX guest. I think the true purpose here is to still be able to use a global shadow_mmio_mask for both TDX and VMX guests. So please explicitly call out.
That's right. With this change, per-VM shadow_mmio_{value, mask} can be avoided. The common value can be used for both VMX and TDX.

No, the mask can be global, but the value needs to be per-VM. VMX needs to generate an EPT misconfig, but TDX needs to generate an EPT violation to get the automagic #VE reflection.

[email protected]

author

2022-10-30 06:25 UTC #82 lore raw

PATCH v10 35/108 KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

From: Isaku Yamahata <[email protected]>

TDX will use a different shadow PTE entry value for MMIO from VMX. Add members to kvm_arch and track value for MMIO per-VM instead of global variables. By using the per-VM EPT entry value for MMIO, the existing VMX logic is kept working. To untangle the logic to initialize shadow_mmio_access_mask, introduce a separate setter function.

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/mmu.h              |  1 +
 arch/x86/kvm/mmu/mmu.c          |  7 ++++---
 arch/x86/kvm/mmu/spte.c         | 11 +++++++++--
 arch/x86/kvm/mmu/spte.h         |  4 ++--
 arch/x86/kvm/mmu/tdp_mmu.c      |  6 +++---
 6 files changed, 21 insertions(+), 10 deletions(-)

arch/x86/include/asm/kvm_host.h +2 -0

index 3374ec0d6d90..a1c801ca61d3 100644
@@ -1171,6 +1171,8 @@ struct kvm_arch {
 	 */
 	spinlock_t mmu_unsync_pages_lock;
 
+	u64 shadow_mmio_value;
+
 	struct list_head assigned_dev_head;
 	struct iommu_domain *iommu_domain;
 	bool iommu_noncoherent;

arch/x86/kvm/mmu.h +1 -0

index a45f7a96b821..50d240d52697 100644
@@ -101,6 +101,7 @@ static inline u8 kvm_get_shadow_phys_bits(void)
 }
 
 void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask);
+void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value);
 void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask);
 void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only);

arch/x86/kvm/mmu/mmu.c +4 -3

index e7e11f51f8b4..0d3fa29ccccc 100644
@@ -2421,7 +2421,7 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp,
 				return kvm_mmu_prepare_zap_page(kvm, child,
 								invalid_list);
 		}
-	} else if (is_mmio_spte(pte)) {
+	} else if (is_mmio_spte(kvm, pte)) {
 		mmu_spte_clear_no_track(spte);
 	}
 	return 0;

@@ -4081,7 +4081,7 @@ static int handle_mmio_page_fault(struct kvm_vcpu *vcpu, u64 addr, bool direct)
 	if (WARN_ON(reserved))
 		return -EINVAL;
 
-	if (is_mmio_spte(spte)) {
+	if (is_mmio_spte(vcpu->kvm, spte)) {
 		gfn_t gfn = get_mmio_spte_gfn(spte);
 		unsigned int access = get_mmio_spte_access(spte);

@@ -4578,7 +4578,7 @@ static unsigned long get_cr3(struct kvm_vcpu *vcpu)
 static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn,
 			   unsigned int access)
 {
-	if (unlikely(is_mmio_spte(*sptep))) {
+	if (unlikely(is_mmio_spte(vcpu->kvm, *sptep))) {
 		if (gfn != get_mmio_spte_gfn(*sptep)) {
 			mmu_spte_clear_no_track(sptep);
 			return true;

@@ -6061,6 +6061,7 @@ int kvm_mmu_init_vm(struct kvm *kvm)
 	struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;
 	int r;
 
+	kvm->arch.shadow_mmio_value = shadow_mmio_value;
 	INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
 	INIT_LIST_HEAD(&kvm->arch.zapped_obsolete_pages);
 	INIT_LIST_HEAD(&kvm->arch.lpage_disallowed_mmu_pages);

arch/x86/kvm/mmu/spte.c +9 -2

index 5d5c06d4fd89..8f468ee2b985 100644
@@ -74,10 +74,10 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)
 	u64 spte = generation_mmio_spte_mask(gen);
 	u64 gpa = gfn << PAGE_SHIFT;
 
-	WARN_ON_ONCE(!shadow_mmio_value);
+	WARN_ON_ONCE(!vcpu->kvm->arch.shadow_mmio_value);
 
 	access &= shadow_mmio_access_mask;
-	spte |= shadow_mmio_value | access;
+	spte |= vcpu->kvm->arch.shadow_mmio_value | access;
 	spte |= gpa | shadow_nonpresent_or_rsvd_mask;
 	spte |= (gpa & shadow_nonpresent_or_rsvd_mask)
 		<< SHADOW_NONPRESENT_OR_RSVD_MASK_LEN;

@@ -352,6 +352,7 @@ u64 mark_spte_for_access_track(u64 spte)
 void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask)
 {
 	BUG_ON((u64)(unsigned)access_mask != access_mask);
+
 	WARN_ON(mmio_value & shadow_nonpresent_or_rsvd_lower_gfn_mask);
 
 	/*

@@ -401,6 +402,12 @@ void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask)
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_set_mmio_spte_mask);
 
+void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value)
+{
+	kvm->arch.shadow_mmio_value = mmio_value;
+}
+EXPORT_SYMBOL_GPL(kvm_mmu_set_mmio_spte_value);
+
 void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask)
 {
 	/* shadow_me_value must be a subset of shadow_me_mask */

arch/x86/kvm/mmu/spte.h +2 -2

index 7e0f79e8f45b..82f0d5c08b77 100644
@@ -241,9 +241,9 @@ static inline int spte_index(u64 *sptep)
  */
 extern u64 __read_mostly shadow_nonpresent_or_rsvd_lower_gfn_mask;
 
-static inline bool is_mmio_spte(u64 spte)
+static inline bool is_mmio_spte(struct kvm *kvm, u64 spte)
 {
-	return (spte & shadow_mmio_mask) == shadow_mmio_value &&
+	return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&
 	       likely(enable_mmio_caching);
 }

arch/x86/kvm/mmu/tdp_mmu.c +3 -3

index 1eee9c159958..e07f14351d14 100644
@@ -580,8 +580,8 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,
 		 * impact the guest since both the former and current SPTEs
 		 * are nonpresent.
 		 */
-		if (WARN_ON(!is_mmio_spte(old_spte) &&
-			    !is_mmio_spte(new_spte) &&
+		if (WARN_ON(!is_mmio_spte(kvm, old_spte) &&
+			    !is_mmio_spte(kvm, new_spte) &&
 			    !is_removed_spte(new_spte)))
 			pr_err("Unexpected SPTE change! Nonpresent SPTEs\n"
 			       "should not be replaced with another,\n"

@@ -1105,7 +1105,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu,
 	}
 
 	/* If a MMIO SPTE is installed, the MMIO will need to be emulated. */
-	if (unlikely(is_mmio_spte(new_spte))) {
+	if (unlikely(is_mmio_spte(vcpu->kvm, new_spte))) {
 		vcpu->stat.pf_mmio_spte_created++;
 		trace_mark_mmio_spte(rcu_dereference(iter->sptep), iter->gfn,
 				     new_spte);

Signature · 1 line

2.25.1

Huang, Kai

2022-11-09 12:27 UTC #83 lore raw

Re: [PATCH v10 035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

Quoted 8 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>
TDX will use a different shadow PTE entry value for MMIO from VMX. Add members to kvm_arch and track value for MMIO per-VM instead of global variables. By using the per-VM EPT entry value for MMIO, the existing VMX logic is kept working. To untangle the logic to initialize shadow_mmio_access_mask, introduce a separate setter function.

It's weird to mention "shadow_mmio_access_mask" here (and this doesn't stand anymore anyway as we have changed to only make mmio_value per-vm). Just say something like "introduce a setter function to set mmio_value for the given guest so that TDX guest can override later".

Also, seems it's worth to mention the same mmio_mask is used for both TDX guest and VMX guest so we can still use global shadow_mmio_mask.

[...]

arch/x86/kvm/mmu/spte.c u64 mark_spte_for_access_track(u64 spte) · line 356

Quoted 5 lines

void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask) {
arch/x86/kvm/mmu/spte.c +1 -0
 	BUG_ON((u64)(unsigned)access_mask != access_mask);
+
 	WARN_ON(mmio_value & shadow_nonpresent_or_rsvd_lower_gfn_mask);

Looks unnecessary change.

Yan Zhao

2022-11-22 02:33 UTC #84 lore raw

Re: [PATCH v10 035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

Quoted 109 lines · On Sat, Oct 29, 2022 at 11:22:36PM -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/mmu.h              |  1 +
 arch/x86/kvm/mmu/mmu.c          |  7 ++++---
 arch/x86/kvm/mmu/spte.c         | 11 +++++++++--
 arch/x86/kvm/mmu/spte.h         |  4 ++--
 arch/x86/kvm/mmu/tdp_mmu.c      |  6 +++---
 6 files changed, 21 insertions(+), 10 deletions(-)

arch/x86/include/asm/kvm_host.h +2 -0

index 3374ec0d6d90..a1c801ca61d3 100644
@@ -1171,6 +1171,8 @@ struct kvm_arch {
 	 */
 	spinlock_t mmu_unsync_pages_lock;
 
+	u64 shadow_mmio_value;
+
 	struct list_head assigned_dev_head;
 	struct iommu_domain *iommu_domain;
 	bool iommu_noncoherent;

arch/x86/kvm/mmu.h +1 -0

index a45f7a96b821..50d240d52697 100644
@@ -101,6 +101,7 @@ static inline u8 kvm_get_shadow_phys_bits(void)
 }
 
 void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask);
+void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value);
 void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask);
 void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only);

arch/x86/kvm/mmu/mmu.c +4 -3

index e7e11f51f8b4..0d3fa29ccccc 100644
@@ -2421,7 +2421,7 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp,
 				return kvm_mmu_prepare_zap_page(kvm, child,
 								invalid_list);
 		}
-	} else if (is_mmio_spte(pte)) {
+	} else if (is_mmio_spte(kvm, pte)) {
 		mmu_spte_clear_no_track(spte);
 	}
 	return 0;

@@ -4081,7 +4081,7 @@ static int handle_mmio_page_fault(struct kvm_vcpu *vcpu, u64 addr, bool direct)
 	if (WARN_ON(reserved))
 		return -EINVAL;
 
-	if (is_mmio_spte(spte)) {
+	if (is_mmio_spte(vcpu->kvm, spte)) {
 		gfn_t gfn = get_mmio_spte_gfn(spte);
 		unsigned int access = get_mmio_spte_access(spte);

@@ -4578,7 +4578,7 @@ static unsigned long get_cr3(struct kvm_vcpu *vcpu)
 static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn,
 			   unsigned int access)
 {
-	if (unlikely(is_mmio_spte(*sptep))) {
+	if (unlikely(is_mmio_spte(vcpu->kvm, *sptep))) {
 		if (gfn != get_mmio_spte_gfn(*sptep)) {
 			mmu_spte_clear_no_track(sptep);
 			return true;

@@ -6061,6 +6061,7 @@ int kvm_mmu_init_vm(struct kvm *kvm)
 	struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;
 	int r;
 
+	kvm->arch.shadow_mmio_value = shadow_mmio_value;
 	INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
 	INIT_LIST_HEAD(&kvm->arch.zapped_obsolete_pages);
 	INIT_LIST_HEAD(&kvm->arch.lpage_disallowed_mmu_pages);

arch/x86/kvm/mmu/spte.c +3 -2

index 5d5c06d4fd89..8f468ee2b985 100644
@@ -74,10 +74,10 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)
 	u64 spte = generation_mmio_spte_mask(gen);
 	u64 gpa = gfn << PAGE_SHIFT;
 
-	WARN_ON_ONCE(!shadow_mmio_value);
+	WARN_ON_ONCE(!vcpu->kvm->arch.shadow_mmio_value);
 
 	access &= shadow_mmio_access_mask;
-	spte |= shadow_mmio_value | access;
+	spte |= vcpu->kvm->arch.shadow_mmio_value | access;
 	spte |= gpa | shadow_nonpresent_or_rsvd_mask;
 	spte |= (gpa & shadow_nonpresent_or_rsvd_mask)
 		<< SHADOW_NONPRESENT_OR_RSVD_MASK_LEN;

@@ -352,6 +352,7 @@ u64 mark_spte_for_access_track(u64 spte)
 void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask)
 {
 	BUG_ON((u64)(unsigned)access_mask != access_mask);
+
 	WARN_ON(mmio_value & shadow_nonpresent_or_rsvd_lower_gfn_mask);
 
 	/*

arch/x86/kvm/mmu/spte.c void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask) · line 408

Quoted 7 lines

arch/x86/kvm/mmu/spte.c +4 -0

@@ -401,6 +402,12 @@ void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask)
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_set_mmio_spte_mask);
 
+void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value)
+{
+	kvm->arch.shadow_mmio_value = mmio_value;
+}

Also make enable_mmio_caching to be a per-VM value?
As if the shadow_mmio_value is 0, mmio_caching needs to be disabled.

Quoted 10 lines

arch/x86/kvm/mmu/spte.c +2 -0

+EXPORT_SYMBOL_GPL(kvm_mmu_set_mmio_spte_value);
+
 void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask)
 {
 	/* shadow_me_value must be a subset of shadow_me_mask */

arch/x86/kvm/mmu/spte.h +0 -0

index 7e0f79e8f45b..82f0d5c08b77 100644

arch/x86/kvm/mmu/spte.h static inline int spte_index(u64 *sptep) · line 247

Quoted 9 lines

arch/x86/kvm/mmu/spte.h +2 -2

@@ -241,9 +241,9 @@ static inline int spte_index(u64 *sptep)
  */
 extern u64 __read_mostly shadow_nonpresent_or_rsvd_lower_gfn_mask;
 
-static inline bool is_mmio_spte(u64 spte)
+static inline bool is_mmio_spte(struct kvm *kvm, u64 spte)
 {
-	return (spte & shadow_mmio_mask) == shadow_mmio_value &&
+	return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&
 	       likely(enable_mmio_caching);

As above, also turn enable_mmio_caching to be per-vm ?

Quoted 29 lines

arch/x86/kvm/mmu/spte.h +0 -0

arch/x86/kvm/mmu/tdp_mmu.c +3 -3

index 1eee9c159958..e07f14351d14 100644
@@ -580,8 +580,8 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,
 		 * impact the guest since both the former and current SPTEs
 		 * are nonpresent.
 		 */
-		if (WARN_ON(!is_mmio_spte(old_spte) &&
-			    !is_mmio_spte(new_spte) &&
+		if (WARN_ON(!is_mmio_spte(kvm, old_spte) &&
+			    !is_mmio_spte(kvm, new_spte) &&
 			    !is_removed_spte(new_spte)))
 			pr_err("Unexpected SPTE change! Nonpresent SPTEs\n"
 			       "should not be replaced with another,\n"

@@ -1105,7 +1105,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu,
 	}
 
 	/* If a MMIO SPTE is installed, the MMIO will need to be emulated. */
-	if (unlikely(is_mmio_spte(new_spte))) {
+	if (unlikely(is_mmio_spte(vcpu->kvm, new_spte))) {
 		vcpu->stat.pf_mmio_spte_created++;
 		trace_mark_mmio_spte(rcu_dereference(iter->sptep), iter->gfn,
 				     new_spte);

Signature · 1 line

2.25.1

Huang, Kai

2022-11-25 00:13 UTC #85 lore raw

Re: [PATCH v10 035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

Quoted 2 lines · Yan Zhao

Also make enable_mmio_caching to be a per-VM value?
As if the shadow_mmio_value is 0, mmio_caching needs to be disabled.

If I recall correctly, Sean said we can disable TDX guests if mmio_caching is disabled (we also will need to change to allow enable_mmio_caching to still be true when mmio_value is 0).

SEV_ES has similar logic:

void __init sev_hardware_setup(void)
{

	...

        /*
         * SEV-ES requires MMIO caching as KVM doesn't have access to the guest
         * instruction stream, i.e. can't emulate in response to a #NPF and
         * instead relies on #NPF(RSVD) being reflected into the guest as #VC
         * (the guest can then do a #VMGEXIT to request MMIO emulation).
         */
        if (!enable_mmio_caching)
                goto out;

Yan Zhao

2022-11-25 00:35 UTC #86 lore raw

Re: [PATCH v10 035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

Quoted 13 lines · Kai

Quoted 2 lines · Yan Zhao
Also make enable_mmio_caching to be a per-VM value?
As if the shadow_mmio_value is 0, mmio_caching needs to be disabled.
If I recall correctly, Sean said we can disable TDX guests if mmio_caching is disabled (we also will need to change to allow enable_mmio_caching to still be true when mmio_value is 0).
SEV_ES has similar logic:
void __init sev_hardware_setup(void)
{

Quoted 11 lines

	...

        /*
         * SEV-ES requires MMIO caching as KVM doesn't have access to the guest
         * instruction stream, i.e. can't emulate in response to a #NPF and
         * instead relies on #NPF(RSVD) being reflected into the guest as #VC
         * (the guest can then do a #VMGEXIT to request MMIO emulation).
         */
        if (!enable_mmio_caching)
                goto out;

Would enabling mmio caching in per-VM basis be better?

Huang, Kai

2022-11-25 00:45 UTC #87 lore raw

Re: [PATCH v10 035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

Quoted 26 lines · Yan Zhao

Quoted 13 lines · Kai
Quoted 2 lines · Yan Zhao
Also make enable_mmio_caching to be a per-VM value?
As if the shadow_mmio_value is 0, mmio_caching needs to be disabled.
If I recall correctly, Sean said we can disable TDX guests if mmio_caching is disabled (we also will need to change to allow enable_mmio_caching to still be true when mmio_value is 0).
SEV_ES has similar logic:
void __init sev_hardware_setup(void)
{
Quoted 11 lines
	...

        /*
         * SEV-ES requires MMIO caching as KVM doesn't have access to the guest
         * instruction stream, i.e. can't emulate in response to a #NPF and
         * instead relies on #NPF(RSVD) being reflected into the guest as #VC
         * (the guest can then do a #VMGEXIT to request MMIO emulation).
         */
        if (!enable_mmio_caching)
                goto out;

Quoted 2 lines

Would enabling mmio caching in per-VM basis be better?

We need Paolo/Sean to decide.

The thing is TDX guests always require mmio_caching being enabled. For VMX guests, normally we will always enable mmio_caching too. So I think per-VM basis mmio_caching is not that useful.

Yan Zhao

2022-11-25 01:00 UTC #88 lore raw

Re: [PATCH v10 035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

Quoted 27 lines · Kai

Quoted 26 lines · Yan Zhao
Quoted 24 lines · Kai
Quoted 2 lines · Yan Zhao
Also make enable_mmio_caching to be a per-VM value?
As if the shadow_mmio_value is 0, mmio_caching needs to be disabled.
If I recall correctly, Sean said we can disable TDX guests if mmio_caching is disabled (we also will need to change to allow enable_mmio_caching to still be true when mmio_value is 0).
SEV_ES has similar logic:
void __init sev_hardware_setup(void)
{
	...

        /*
         * SEV-ES requires MMIO caching as KVM doesn't have access to the guest
         * instruction stream, i.e. can't emulate in response to a #NPF and
         * instead relies on #NPF(RSVD) being reflected into the guest as #VC
         * (the guest can then do a #VMGEXIT to request MMIO emulation).
         */
        if (!enable_mmio_caching)
                goto out;

Quoted 8 lines

Quoted 2 lines
Would enabling mmio caching in per-VM basis be better?
We need Paolo/Sean to decide.
The thing is TDX guests always require mmio_caching being enabled. For VMX guests, normally we will always enable mmio_caching too. So I think per-VM basis mmio_caching is not that useful.

With per-VM basis enabling, I think we can get rid of the kvm_gfn_shared_mask(kvm) in below code and also in handle_abnormal_pfn()

static inline bool is_mmio_spte(struct kvm *kvm, u64 spte)

{
        return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&
               likely(enable_mmio_caching || kvm_gfn_shared_mask(kvm));
}

Thanks
Yan

Huang, Kai

2022-11-25 01:07 UTC #89 lore raw

Re: [PATCH v10 035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

Quoted 36 lines · Yan Zhao

Quoted 27 lines · Kai
Quoted 26 lines · Yan Zhao
Quoted 24 lines · Kai
Quoted 2 lines · Yan Zhao
Also make enable_mmio_caching to be a per-VM value?
As if the shadow_mmio_value is 0, mmio_caching needs to be disabled.
If I recall correctly, Sean said we can disable TDX guests if mmio_caching is disabled (we also will need to change to allow enable_mmio_caching to still be true when mmio_value is 0).
SEV_ES has similar logic:
void __init sev_hardware_setup(void)
{
	...

        /*
         * SEV-ES requires MMIO caching as KVM doesn't have access to the guest
         * instruction stream, i.e. can't emulate in response to a #NPF and
         * instead relies on #NPF(RSVD) being reflected into the guest as #VC
         * (the guest can then do a #VMGEXIT to request MMIO emulation).
         */
        if (!enable_mmio_caching)
                goto out;
Quoted 8 lines
Quoted 2 lines
Would enabling mmio caching in per-VM basis be better?
We need Paolo/Sean to decide.
The thing is TDX guests always require mmio_caching being enabled. For VMX guests, normally we will always enable mmio_caching too. So I think per-VM basis mmio_caching is not that useful.

Quoted 9 lines

With per-VM basis enabling, I think we can get rid of the kvm_gfn_shared_mask(kvm) in below code and also in handle_abnormal_pfn()
static inline bool is_mmio_spte(struct kvm *kvm, u64 spte)
{
        return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&
               likely(enable_mmio_caching || kvm_gfn_shared_mask(kvm));
}

It needs to go anyway regardless per-VM mmio_caching or not, as explained we need to change to allow enable_mmio_caching to be true even mmio_value is 0.

Yan Zhao

2022-11-25 01:27 UTC #90 lore raw

Re: [PATCH v10 035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

Quoted 47 lines · Kai

Quoted 36 lines · Yan Zhao
Quoted 27 lines · Kai
Quoted 26 lines · Yan Zhao
Quoted 24 lines · Kai
Quoted 2 lines · Yan Zhao
Also make enable_mmio_caching to be a per-VM value?
As if the shadow_mmio_value is 0, mmio_caching needs to be disabled.
If I recall correctly, Sean said we can disable TDX guests if mmio_caching is disabled (we also will need to change to allow enable_mmio_caching to still be true when mmio_value is 0).
SEV_ES has similar logic:
void __init sev_hardware_setup(void)
{
	...

        /*
         * SEV-ES requires MMIO caching as KVM doesn't have access to the guest
         * instruction stream, i.e. can't emulate in response to a #NPF and
         * instead relies on #NPF(RSVD) being reflected into the guest as #VC
         * (the guest can then do a #VMGEXIT to request MMIO emulation).
         */
        if (!enable_mmio_caching)
                goto out;
Quoted 8 lines
Quoted 2 lines
Would enabling mmio caching in per-VM basis be better?
We need Paolo/Sean to decide.
The thing is TDX guests always require mmio_caching being enabled. For VMX guests, normally we will always enable mmio_caching too. So I think per-VM basis mmio_caching is not that useful.
Quoted 9 lines
With per-VM basis enabling, I think we can get rid of the kvm_gfn_shared_mask(kvm) in below code and also in handle_abnormal_pfn()
static inline bool is_mmio_spte(struct kvm *kvm, u64 spte)
{
        return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&
               likely(enable_mmio_caching || kvm_gfn_shared_mask(kvm));
}

Quoted 2 lines

It needs to go anyway regardless per-VM mmio_caching or not, as explained we need to change to allow enable_mmio_caching to be true even mmio_value is 0.

Or it's better to check enable_mmio_caching is true in kvm_mmu_set_mmio_spte_value() as below.

void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value)

{
        WARN_ON(!enable_mmio_caching);
        kvm->arch.shadow_mmio_value = mmio_value;
}

Sean Christopherson

maintainer

2022-11-28 23:49 UTC #91 lore raw

Re: [PATCH v10 035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis

Quoted 48 lines · Yan Zhao

Quoted 47 lines · Kai
Quoted 45 lines · Yan Zhao
Quoted 27 lines · Kai
Quoted 26 lines · Yan Zhao
Quoted 24 lines · Kai
Quoted 2 lines · Yan Zhao
Also make enable_mmio_caching to be a per-VM value?
As if the shadow_mmio_value is 0, mmio_caching needs to be disabled.
If I recall correctly, Sean said we can disable TDX guests if mmio_caching is disabled (we also will need to change to allow enable_mmio_caching to still be true when mmio_value is 0).
SEV_ES has similar logic:
void __init sev_hardware_setup(void)
{
	...

        /*
         * SEV-ES requires MMIO caching as KVM doesn't have access to the guest
         * instruction stream, i.e. can't emulate in response to a #NPF and
         * instead relies on #NPF(RSVD) being reflected into the guest as #VC
         * (the guest can then do a #VMGEXIT to request MMIO emulation).
         */
        if (!enable_mmio_caching)
                goto out;
Quoted 8 lines
Quoted 2 lines
Would enabling mmio caching in per-VM basis be better?
We need Paolo/Sean to decide.
The thing is TDX guests always require mmio_caching being enabled. For VMX guests, normally we will always enable mmio_caching too. So I think per-VM basis mmio_caching is not that useful.
With per-VM basis enabling, I think we can get rid of the kvm_gfn_shared_mask(kvm) in below code and also in handle_abnormal_pfn()
static inline bool is_mmio_spte(struct kvm *kvm, u64 spte)
{
        return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&
               likely(enable_mmio_caching || kvm_gfn_shared_mask(kvm));
}

Quoted 2 lines

Quoted 2 lines
It needs to go anyway regardless per-VM mmio_caching or not, as explained we need to change to allow enable_mmio_caching to be true even mmio_value is 0.

Yes, the kvm_gfn_shared_mask() in is_mmio_spte() should not exist.

Quoted 7 lines

Or it's better to check enable_mmio_caching is true in kvm_mmu_set_mmio_spte_value() as below.
void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value)
{
        WARN_ON(!enable_mmio_caching);
        kvm->arch.shadow_mmio_value = mmio_value;

Eh, if you're going to bother with that assert, capture the more subtle aspects as well, e.g. that EPT is enabled and that shadow_mmio_mask has been set to the expected value. I would also ditch the helper and just open code this in TDX code, I doubt there's a use case outside of TDX+VMX coexistence.

[email protected]

author

2022-10-30 06:25 UTC #92 lore raw

PATCH v10 36/108 KVM: TDX: Enable mmio spte caching always for TDX

From: Isaku Yamahata <[email protected]>

TDX needs to set shared spte for MMIO GFN to !SUPPRES_VE_BIT | !RWX so that guest TD can get #VE and then issue TDG.VP.VMCALL<MMIO>. Enable mmio caching always for TDX irrelevant the module parameter enable_mmio_caching.

https://lore.kernel.org/lkml/[email protected]/

---
 arch/x86/kvm/mmu/mmu.c     | 3 ++-
 arch/x86/kvm/mmu/spte.h    | 2 +-
 arch/x86/kvm/mmu/tdp_mmu.c | 7 +++++++
 3 files changed, 10 insertions(+), 2 deletions(-)

arch/x86/kvm/mmu/mmu.c +2 -1

index 0d3fa29ccccc..9098f77cdaa4 100644
@@ -3229,7 +3229,8 @@ static int handle_abnormal_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fau
 		 * and only if L1's MAXPHYADDR is inaccurate with respect to
 		 * the hardware's).
 		 */
-		if (unlikely(!enable_mmio_caching) ||
+		if (unlikely(!enable_mmio_caching &&
+			     !kvm_gfn_shared_mask(vcpu->kvm)) ||
 		    unlikely(fault->gfn > kvm_mmu_max_gfn()))
 			return RET_PF_EMULATE;
 	}

arch/x86/kvm/mmu/spte.h +1 -1

index 82f0d5c08b77..fecfdcb5f321 100644
@@ -244,7 +244,7 @@ extern u64 __read_mostly shadow_nonpresent_or_rsvd_lower_gfn_mask;
 static inline bool is_mmio_spte(struct kvm *kvm, u64 spte)
 {
 	return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&
-	       likely(enable_mmio_caching);
+	       likely(enable_mmio_caching || kvm_gfn_shared_mask(kvm));
 }
 
 static inline bool is_shadow_present_pte(u64 pte)

arch/x86/kvm/mmu/tdp_mmu.c +7 -0

index e07f14351d14..3325633b1cb5 100644
@@ -1875,6 +1875,13 @@ int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes,
 
 	*root_level = vcpu->arch.mmu->root_role.level;
 
+	/*
+	 * mmio page fault isn't supported for protected guest because
+	 * instructions in protected guest memory can't be parsed by VMM.
+	 */
+	if (WARN_ON_ONCE(kvm_gfn_shared_mask(vcpu->kvm)))
+		return leaf;
+
 	tdp_mmu_for_each_pte(iter, mmu, gfn, gfn + 1) {
 		leaf = iter.level;
 		sptes[leaf] = iter.old_spte;

Signature · 1 line

2.25.1

Huang, Kai

2022-11-09 12:46 UTC #93 lore raw

Re: [PATCH v10 036/108] KVM: TDX: Enable mmio spte caching always for TDX

Quoted 32 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>
TDX needs to set shared spte for MMIO GFN to !SUPPRES_VE_BIT | !RWX so that guest TD can get #VE and then issue TDG.VP.VMCALL<MMIO>. Enable mmio caching always for TDX irrelevant the module parameter enable_mmio_caching.
Signed-off-by Isaku Yamahata <[email protected]>
---
 arch/x86/kvm/mmu/mmu.c     | 3 ++-
 arch/x86/kvm/mmu/spte.h    | 2 +-
 arch/x86/kvm/mmu/tdp_mmu.c | 7 +++++++
 3 files changed, 10 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -1
index 0d3fa29ccccc..9098f77cdaa4 100644
@@ -3229,7 +3229,8 @@ static int handle_abnormal_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fau
 		 * and only if L1's MAXPHYADDR is inaccurate with respect to
 		 * the hardware's).
 		 */
-		if (unlikely(!enable_mmio_caching) ||
+		if (unlikely(!enable_mmio_caching &&
+			     !kvm_gfn_shared_mask(vcpu->kvm)) ||
 		    unlikely(fault->gfn > kvm_mmu_max_gfn()))
 			return RET_PF_EMULATE;
 	}
arch/x86/kvm/mmu/spte.h +0 -0
index 82f0d5c08b77..fecfdcb5f321 100644

arch/x86/kvm/mmu/spte.h extern u64 __read_mostly shadow_nonpresent_or_rsvd_lower_gfn_mask; · line 248

Quoted 6 lines

arch/x86/kvm/mmu/spte.h +1 -1

@@ -244,7 +244,7 @@ extern u64 __read_mostly shadow_nonpresent_or_rsvd_lower_gfn_mask;
 static inline bool is_mmio_spte(struct kvm *kvm, u64 spte)
 {
 	return (spte & shadow_mmio_mask) == kvm->arch.shadow_mmio_value &&
-	       likely(enable_mmio_caching);
+	       likely(enable_mmio_caching || kvm_gfn_shared_mask(kvm));
 }

Sean suggested we can change to treat "mmio_value == 0" also as MMIO caching being enabled:

Quoted 7 lines

arch/x86/kvm/mmu/spte.h +0 -0
 static inline bool is_shadow_present_pte(u64 pte)
arch/x86/kvm/mmu/tdp_mmu.c +0 -0
index e07f14351d14..3325633b1cb5 100644

arch/x86/kvm/mmu/tdp_mmu.c int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, · line 1884

Quoted 10 lines

arch/x86/kvm/mmu/tdp_mmu.c +7 -0

@@ -1875,6 +1875,13 @@ int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes,
 
 	*root_level = vcpu->arch.mmu->root_role.level;
 
+	/*
+	 * mmio page fault isn't supported for protected guest because
+	 * instructions in protected guest memory can't be parsed by VMM.
+	 */
+	if (WARN_ON_ONCE(kvm_gfn_shared_mask(vcpu->kvm)))
+		return leaf;
+

It's weird to put this here.

I think the logic is, for TDX guest (or similar protected guests), handle_mmio_page_fault() should not be reached at all. So I think we can just add a WARN() against kvm_gfn_shared_mask(), or is_tdx_vm() at the very beginning of handle_mmio_page_fault()?

Quoted 3 lines

arch/x86/kvm/mmu/tdp_mmu.c +0 -0

 	tdp_mmu_for_each_pte(iter, mmu, gfn, gfn + 1) {
 		leaf = iter.level;
 		sptes[leaf] = iter.old_spte;

[email protected]

author

2022-10-30 06:25 UTC #94 lore raw

PATCH v10 37/108 KVM: x86/mmu: Disallow fast page fault on private GPA

From: Isaku Yamahata <[email protected]>

TDX requires TDX SEAMCALL to operate Secure EPT instead of direct memory access and TDX SEAMCALL is heavy operation. Fast page fault on private GPA doesn't make sense. Disallow fast page fault on private GPA.

---
 arch/x86/kvm/mmu/mmu.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

arch/x86/kvm/mmu/mmu.c +10 -2

index 9098f77cdaa4..09defac49bf0 100644
@@ -3238,8 +3238,16 @@ static int handle_abnormal_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fau
 	return RET_PF_CONTINUE;
 }
 
-static bool page_fault_can_be_fast(struct kvm_page_fault *fault)
+static bool page_fault_can_be_fast(struct kvm *kvm, struct kvm_page_fault *fault)
 {
+	/*
+	 * TDX private mapping doesn't support fast page fault because the EPT
+	 * entry is read/written with TDX SEAMCALLs instead of direct memory
+	 * access.
+	 */
+	if (kvm_is_private_gpa(kvm, fault->addr))
+		return false;
+
 	/*
 	 * Page faults with reserved bits set, i.e. faults on MMIO SPTEs, only
 	 * reach the common page fault handler if the SPTE has an invalid MMIO

@@ -3349,7 +3357,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 	u64 *sptep = NULL;
 	uint retry_count = 0;
 
-	if (!page_fault_can_be_fast(fault))
+	if (!page_fault_can_be_fast(vcpu->kvm, fault))
 		return ret;
 
 	walk_shadow_page_lockless_begin(vcpu);

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #95 lore raw

PATCH v10 38/108 KVM: x86/mmu: Allow per-VM override of the TDP max page level

From: Sean Christopherson <[email protected]>

TDX requires special handling to support large private page. For simplicity, only support 4K page for TD guest for now. Add per-VM maximum page level support to support different maximum page sizes for TD guest and conventional VMX guest.

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/kvm_host.h | 1 +
 arch/x86/kvm/mmu/mmu.c          | 1 +
 arch/x86/kvm/mmu/mmu_internal.h | 2 +-
 3 files changed, 3 insertions(+), 1 deletion(-)

arch/x86/include/asm/kvm_host.h +1 -0

index a1c801ca61d3..2e0b23422d63 100644
@@ -1156,6 +1156,7 @@ struct kvm_arch {
 	unsigned long n_requested_mmu_pages;
 	unsigned long n_max_mmu_pages;
 	unsigned int indirect_shadow_pages;
+	int tdp_max_page_level;
 	u8 mmu_valid_gen;
 	struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES];
 	struct list_head active_mmu_pages;

arch/x86/kvm/mmu/mmu.c +1 -0

index 09defac49bf0..0001e921154e 100644
@@ -6092,6 +6092,7 @@ int kvm_mmu_init_vm(struct kvm *kvm)
 	kvm->arch.split_desc_cache.kmem_cache = pte_list_desc_cache;
 	kvm->arch.split_desc_cache.gfp_zero = __GFP_ZERO;
 
+	kvm->arch.tdp_max_page_level = KVM_MAX_HUGEPAGE_LEVEL;
 	return 0;
 }

arch/x86/kvm/mmu/mmu_internal.h +1 -1

index 5cdff5ca546c..57e3ea2b52cc 100644
@@ -275,7 +275,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		.nx_huge_page_workaround_enabled =
 			is_nx_huge_page_enabled(vcpu->kvm),
 
-		.max_level = KVM_MAX_HUGEPAGE_LEVEL,
+		.max_level = vcpu->kvm->arch.tdp_max_page_level,
 		.req_level = PG_LEVEL_4K,
 		.goal_level = PG_LEVEL_4K,
 	};

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #96 lore raw

PATCH v10 39/108 KVM: VMX: Introduce test mode related to EPT violation VE

From: Isaku Yamahata <[email protected]>

To support TDX, KVM is enhanced to operate with #VE. For TDX, KVM programs to inject #VE conditionally and set #VE suppress bit in EPT entry. For VMX case, #VE isn't used. If #VE happens for VMX, it's a bug. To be defensive (test that VMX case isn't broken), introduce option ept_violation_ve_test and when it's set, set error.

---
 arch/x86/include/asm/vmx.h | 12 +++++++
 arch/x86/kvm/vmx/vmcs.h    |  5 +++
 arch/x86/kvm/vmx/vmx.c     | 69 +++++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/vmx/vmx.h     |  6 +++-
 4 files changed, 90 insertions(+), 2 deletions(-)

arch/x86/include/asm/vmx.h +12 -0

index cdbf12c1a83c..752d53652007 100644
@@ -68,6 +68,7 @@
 #define SECONDARY_EXEC_ENCLS_EXITING		VMCS_CONTROL_BIT(ENCLS_EXITING)
 #define SECONDARY_EXEC_RDSEED_EXITING		VMCS_CONTROL_BIT(RDSEED_EXITING)
 #define SECONDARY_EXEC_ENABLE_PML               VMCS_CONTROL_BIT(PAGE_MOD_LOGGING)
+#define SECONDARY_EXEC_EPT_VIOLATION_VE		VMCS_CONTROL_BIT(EPT_VIOLATION_VE)
 #define SECONDARY_EXEC_PT_CONCEAL_VMX		VMCS_CONTROL_BIT(PT_CONCEAL_VMX)
 #define SECONDARY_EXEC_XSAVES			VMCS_CONTROL_BIT(XSAVES)
 #define SECONDARY_EXEC_MODE_BASED_EPT_EXEC	VMCS_CONTROL_BIT(MODE_BASED_EPT_EXEC)

@@ -223,6 +224,8 @@ enum vmcs_field {
 	VMREAD_BITMAP_HIGH              = 0x00002027,
 	VMWRITE_BITMAP                  = 0x00002028,
 	VMWRITE_BITMAP_HIGH             = 0x00002029,
+	VE_INFORMATION_ADDRESS		= 0x0000202A,
+	VE_INFORMATION_ADDRESS_HIGH	= 0x0000202B,
 	XSS_EXIT_BITMAP                 = 0x0000202C,
 	XSS_EXIT_BITMAP_HIGH            = 0x0000202D,
 	ENCLS_EXITING_BITMAP		= 0x0000202E,

@@ -628,4 +631,13 @@ enum vmx_l1d_flush_state {
 
 extern enum vmx_l1d_flush_state l1tf_vmx_mitigation;
 
+struct vmx_ve_information {
+	u32 exit_reason;
+	u32 delivery;
+	u64 exit_qualification;
+	u64 guest_linear_address;
+	u64 guest_physical_address;
+	u16 eptp_index;
+};
+
 #endif

arch/x86/kvm/vmx/vmcs.h +5 -0

index ac290a44a693..9277676057a7 100644
@@ -140,6 +140,11 @@ static inline bool is_nm_fault(u32 intr_info)
 	return is_exception_n(intr_info, NM_VECTOR);
 }
 
+static inline bool is_ve_fault(u32 intr_info)
+{
+	return is_exception_n(intr_info, VE_VECTOR);
+}
+
 /* Undocumented: icebp/int1 */
 static inline bool is_icebp(u32 intr_info)
 {

arch/x86/kvm/vmx/vmx.c +68 -1

124 lines not on this page · raw message

arch/x86/kvm/vmx/vmx.h +5 -1

index d49d0ace9fb8..1813caeb24d8 100644
@@ -359,6 +359,9 @@ struct vcpu_vmx {
 		DECLARE_BITMAP(read, MAX_POSSIBLE_PASSTHROUGH_MSRS);
 		DECLARE_BITMAP(write, MAX_POSSIBLE_PASSTHROUGH_MSRS);
 	} shadow_msr_intercept;
+
+	/* ve_info must be page aligned. */
+	struct vmx_ve_information *ve_info;
 };
 
 struct kvm_vmx {

@@ -570,7 +573,8 @@ static inline u8 vmx_get_rvi(void)
 	 SECONDARY_EXEC_ENABLE_VMFUNC |					\
 	 SECONDARY_EXEC_BUS_LOCK_DETECTION |				\
 	 SECONDARY_EXEC_NOTIFY_VM_EXITING |				\
-	 SECONDARY_EXEC_ENCLS_EXITING)
+	 SECONDARY_EXEC_ENCLS_EXITING |					\
+	 SECONDARY_EXEC_EPT_VIOLATION_VE)
 
 #define KVM_REQUIRED_VMX_TERTIARY_VM_EXEC_CONTROL 0
 #define KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL			\

Signature · 1 line

2.25.1

Binbin Wu

2022-11-03 13:42 UTC #97 lore raw

Re: [PATCH v10 039/108] KVM: VMX: Introduce test mode related to EPT violation VE

Quoted 97 lines · On 2022/10/30 14:22, [email protected] wrote:

From: Isaku Yamahata <[email protected]>

---
  arch/x86/include/asm/vmx.h | 12 +++++++
  arch/x86/kvm/vmx/vmcs.h    |  5 +++
  arch/x86/kvm/vmx/vmx.c     | 69 +++++++++++++++++++++++++++++++++++++-
  arch/x86/kvm/vmx/vmx.h     |  6 +++-
  4 files changed, 90 insertions(+), 2 deletions(-)

arch/x86/include/asm/vmx.h +12 -0

index cdbf12c1a83c..752d53652007 100644
@@ -68,6 +68,7 @@
  #define SECONDARY_EXEC_ENCLS_EXITING		VMCS_CONTROL_BIT(ENCLS_EXITING)
  #define SECONDARY_EXEC_RDSEED_EXITING		VMCS_CONTROL_BIT(RDSEED_EXITING)
  #define SECONDARY_EXEC_ENABLE_PML               VMCS_CONTROL_BIT(PAGE_MOD_LOGGING)
+#define SECONDARY_EXEC_EPT_VIOLATION_VE		VMCS_CONTROL_BIT(EPT_VIOLATION_VE)
  #define SECONDARY_EXEC_PT_CONCEAL_VMX		VMCS_CONTROL_BIT(PT_CONCEAL_VMX)
  #define SECONDARY_EXEC_XSAVES			VMCS_CONTROL_BIT(XSAVES)
  #define SECONDARY_EXEC_MODE_BASED_EPT_EXEC	VMCS_CONTROL_BIT(MODE_BASED_EPT_EXEC)

@@ -223,6 +224,8 @@ enum vmcs_field {
  	VMREAD_BITMAP_HIGH              = 0x00002027,
  	VMWRITE_BITMAP                  = 0x00002028,
  	VMWRITE_BITMAP_HIGH             = 0x00002029,
+	VE_INFORMATION_ADDRESS		= 0x0000202A,
+	VE_INFORMATION_ADDRESS_HIGH	= 0x0000202B,
  	XSS_EXIT_BITMAP                 = 0x0000202C,
  	XSS_EXIT_BITMAP_HIGH            = 0x0000202D,
  	ENCLS_EXITING_BITMAP		= 0x0000202E,

@@ -628,4 +631,13 @@ enum vmx_l1d_flush_state {
 
  extern enum vmx_l1d_flush_state l1tf_vmx_mitigation;
 
+struct vmx_ve_information {
+	u32 exit_reason;
+	u32 delivery;
+	u64 exit_qualification;
+	u64 guest_linear_address;
+	u64 guest_physical_address;
+	u16 eptp_index;
+};
+
  #endif

arch/x86/kvm/vmx/vmcs.h +5 -0

index ac290a44a693..9277676057a7 100644
@@ -140,6 +140,11 @@ static inline bool is_nm_fault(u32 intr_info)
  	return is_exception_n(intr_info, NM_VECTOR);
  }
 
+static inline bool is_ve_fault(u32 intr_info)
+{
+	return is_exception_n(intr_info, VE_VECTOR);
+}
+
  /* Undocumented: icebp/int1 */
  static inline bool is_icebp(u32 intr_info)
  {

arch/x86/kvm/vmx/vmx.c +10 -0

index f890191e8580..dd3fde9d3c32 100644
@@ -126,6 +126,9 @@ module_param(error_on_inconsistent_vmcs_config, bool, 0444);
  static bool __read_mostly dump_invalid_vmcs = 0;
  module_param(dump_invalid_vmcs, bool, 0644);
 
+static bool __read_mostly ept_violation_ve_test;
+module_param(ept_violation_ve_test, bool, 0444);
+
  #define MSR_BITMAP_MODE_X2APIC		1
  #define MSR_BITMAP_MODE_X2APIC_APICV	2

@@ -783,6 +786,13 @@ void vmx_update_exception_bitmap(struct kvm_vcpu *vcpu)
 
  	eb = (1u << PF_VECTOR) | (1u << UD_VECTOR) | (1u << MC_VECTOR) |
  	     (1u << DB_VECTOR) | (1u << AC_VECTOR);
+	/*
+	 * #VE isn't used for VMX, but for TDX.  To test against unexpected
+	 * change related to #VE for VMX, intercept unexpected #VE and warn on
+	 * it.
+	 */
+	if (ept_violation_ve_test)
+		eb |= 1u << VE_VECTOR;
  	/*
  	 * Guest access to VMware backdoor ports could legitimately
  	 * trigger #GP because of TSS I/O permission bitmap.

arch/x86/kvm/vmx/vmx.c static int setup_vmcs_config(struct vmcs_config *vmcs_conf, · line 2658

Quoted 5 lines

arch/x86/kvm/vmx/vmx.c +2 -0

@@ -2644,6 +2654,9 @@ static int setup_vmcs_config(struct vmcs_config *vmcs_conf,
  					&_cpu_based_2nd_exec_control))
  			return -EIO;
  	}
+	if (!ept_violation_ve_test)
+		_cpu_based_exec_control &= ~SECONDARY_EXEC_EPT_VIOLATION_VE;

should be _cpu_based_2nd_exec_control

Quoted 127 lines

arch/x86/kvm/vmx/vmx.c +56 -1

+
  #ifndef CONFIG_X86_64
  	if (!(_cpu_based_2nd_exec_control &
  				SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES))

@@ -2668,6 +2681,7 @@ static int setup_vmcs_config(struct vmcs_config *vmcs_conf,
  			return -EIO;
 
  		vmx_cap->ept = 0;
+		_cpu_based_2nd_exec_control &= ~SECONDARY_EXEC_EPT_VIOLATION_VE;
  	}
  	if (!(_cpu_based_2nd_exec_control & SECONDARY_EXEC_ENABLE_VPID) &&
  	    vmx_cap->vpid) {

@@ -4510,6 +4524,7 @@ static u32 vmx_secondary_exec_control(struct vcpu_vmx *vmx)
  		exec_control &= ~SECONDARY_EXEC_ENABLE_VPID;
  	if (!enable_ept) {
  		exec_control &= ~SECONDARY_EXEC_ENABLE_EPT;
+		exec_control &= ~SECONDARY_EXEC_EPT_VIOLATION_VE;
  		enable_unrestricted_guest = 0;
  	}
  	if (!enable_unrestricted_guest)

@@ -4637,8 +4652,40 @@ static void init_vmcs(struct vcpu_vmx *vmx)
 
  	exec_controls_set(vmx, vmx_exec_control(vmx));
 
-	if (cpu_has_secondary_exec_ctrls())
+	if (cpu_has_secondary_exec_ctrls()) {
  		secondary_exec_controls_set(vmx, vmx_secondary_exec_control(vmx));
+		if (secondary_exec_controls_get(vmx) &
+		    SECONDARY_EXEC_EPT_VIOLATION_VE) {
+			if (!vmx->ve_info) {
+				/* ve_info must be page aligned. */
+				struct page *page;
+
+				BUILD_BUG_ON(sizeof(*vmx->ve_info) > PAGE_SIZE);
+				page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+				if (page)
+					vmx->ve_info = page_to_virt(page);
+			}
+			if (vmx->ve_info) {
+				/*
+				 * Allow #VE delivery. CPU sets this field to
+				 * 0xFFFFFFFF on #VE delivery.  Another #VE can
+				 * occur only if software clears the field.
+				 */
+				vmx->ve_info->delivery = 0;
+				vmcs_write64(VE_INFORMATION_ADDRESS,
+					     __pa(vmx->ve_info));
+			} else {
+				/*
+				 * Because SECONDARY_EXEC_EPT_VIOLATION_VE is
+				 * used only when ept_violation_ve_test is true,
+				 * it's okay to go with the bit disabled.
+				 */
+				pr_err("Failed to allocate ve_info. disabling EPT_VIOLATION_VE.\n");
+				secondary_exec_controls_clearbit(vmx,
+								 SECONDARY_EXEC_EPT_VIOLATION_VE);
+			}
+		}
+	}
 
  	if (cpu_has_tertiary_exec_ctrls())
  		tertiary_exec_controls_set(vmx, vmx_tertiary_exec_control(vmx));

@@ -5118,6 +5165,12 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu)
  	if (is_invalid_opcode(intr_info))
  		return handle_ud(vcpu);
 
+	/*
+	 * #VE isn't supposed to happen.  Although vcpu can send
+	 */
+	if (KVM_BUG_ON(is_ve_fault(intr_info), vcpu->kvm))
+		return -EIO;
+
  	error_code = 0;
  	if (intr_info & INTR_INFO_DELIVER_CODE_MASK)
  		error_code = vmcs_read32(VM_EXIT_INTR_ERROR_CODE);

@@ -6306,6 +6359,18 @@ void dump_vmcs(struct kvm_vcpu *vcpu)
  	if (secondary_exec_control & SECONDARY_EXEC_ENABLE_VPID)
  		pr_err("Virtual processor ID = 0x%04x\n",
  		       vmcs_read16(VIRTUAL_PROCESSOR_ID));
+	if (secondary_exec_control & SECONDARY_EXEC_EPT_VIOLATION_VE) {
+		struct vmx_ve_information *ve_info;
+
+		pr_err("VE info address = 0x%016llx\n",
+		       vmcs_read64(VE_INFORMATION_ADDRESS));
+		ve_info = __va(vmcs_read64(VE_INFORMATION_ADDRESS));
+		pr_err("ve_info: 0x%08x 0x%08x 0x%016llx 0x%016llx 0x%016llx 0x%04x\n",
+		       ve_info->exit_reason, ve_info->delivery,
+		       ve_info->exit_qualification,
+		       ve_info->guest_linear_address,
+		       ve_info->guest_physical_address, ve_info->eptp_index);
+	}
  }
 
  /*

@@ -7302,6 +7367,8 @@ void vmx_vcpu_free(struct kvm_vcpu *vcpu)
  	free_vpid(vmx->vpid);
  	nested_vmx_free_vcpu(vcpu);
  	free_loaded_vmcs(vmx->loaded_vmcs);
+	if (vmx->ve_info)
+		free_page((unsigned long)vmx->ve_info);
  }
 
  int vmx_vcpu_create(struct kvm_vcpu *vcpu)

arch/x86/kvm/vmx/vmx.h +5 -1

index d49d0ace9fb8..1813caeb24d8 100644
@@ -359,6 +359,9 @@ struct vcpu_vmx {
  		DECLARE_BITMAP(read, MAX_POSSIBLE_PASSTHROUGH_MSRS);
  		DECLARE_BITMAP(write, MAX_POSSIBLE_PASSTHROUGH_MSRS);
  	} shadow_msr_intercept;
+
+	/* ve_info must be page aligned. */
+	struct vmx_ve_information *ve_info;
  };
 
  struct kvm_vmx {

@@ -570,7 +573,8 @@ static inline u8 vmx_get_rvi(void)
  	 SECONDARY_EXEC_ENABLE_VMFUNC |					\
  	 SECONDARY_EXEC_BUS_LOCK_DETECTION |				\
  	 SECONDARY_EXEC_NOTIFY_VM_EXITING |				\
-	 SECONDARY_EXEC_ENCLS_EXITING)
+	 SECONDARY_EXEC_ENCLS_EXITING |					\
+	 SECONDARY_EXEC_EPT_VIOLATION_VE)
 
  #define KVM_REQUIRED_VMX_TERTIARY_VM_EXEC_CONTROL 0
  #define KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL			\

Isaku Yamahata

2022-11-03 20:13 UTC #98 lore raw

Re: [PATCH v10 039/108] KVM: VMX: Introduce test mode related to EPT violation VE

Quoted 92 lines · Binbin Wu

Quoted 90 lines · On 2022/10/30 14:22, [email protected] wrote:

From: Isaku Yamahata <[email protected]>

---
  arch/x86/include/asm/vmx.h | 12 +++++++
  arch/x86/kvm/vmx/vmcs.h    |  5 +++
  arch/x86/kvm/vmx/vmx.c     | 69 +++++++++++++++++++++++++++++++++++++-
  arch/x86/kvm/vmx/vmx.h     |  6 +++-
  4 files changed, 90 insertions(+), 2 deletions(-)

arch/x86/include/asm/vmx.h +12 -0

index cdbf12c1a83c..752d53652007 100644
@@ -68,6 +68,7 @@
  #define SECONDARY_EXEC_ENCLS_EXITING		VMCS_CONTROL_BIT(ENCLS_EXITING)
  #define SECONDARY_EXEC_RDSEED_EXITING		VMCS_CONTROL_BIT(RDSEED_EXITING)
  #define SECONDARY_EXEC_ENABLE_PML               VMCS_CONTROL_BIT(PAGE_MOD_LOGGING)
+#define SECONDARY_EXEC_EPT_VIOLATION_VE		VMCS_CONTROL_BIT(EPT_VIOLATION_VE)
  #define SECONDARY_EXEC_PT_CONCEAL_VMX		VMCS_CONTROL_BIT(PT_CONCEAL_VMX)
  #define SECONDARY_EXEC_XSAVES			VMCS_CONTROL_BIT(XSAVES)
  #define SECONDARY_EXEC_MODE_BASED_EPT_EXEC	VMCS_CONTROL_BIT(MODE_BASED_EPT_EXEC)

@@ -223,6 +224,8 @@ enum vmcs_field {
  	VMREAD_BITMAP_HIGH              = 0x00002027,
  	VMWRITE_BITMAP                  = 0x00002028,
  	VMWRITE_BITMAP_HIGH             = 0x00002029,
+	VE_INFORMATION_ADDRESS		= 0x0000202A,
+	VE_INFORMATION_ADDRESS_HIGH	= 0x0000202B,
  	XSS_EXIT_BITMAP                 = 0x0000202C,
  	XSS_EXIT_BITMAP_HIGH            = 0x0000202D,
  	ENCLS_EXITING_BITMAP		= 0x0000202E,

@@ -628,4 +631,13 @@ enum vmx_l1d_flush_state {
  extern enum vmx_l1d_flush_state l1tf_vmx_mitigation;
+struct vmx_ve_information {
+	u32 exit_reason;
+	u32 delivery;
+	u64 exit_qualification;
+	u64 guest_linear_address;
+	u64 guest_physical_address;
+	u16 eptp_index;
+};
+
  #endif

arch/x86/kvm/vmx/vmcs.h +5 -0

index ac290a44a693..9277676057a7 100644
@@ -140,6 +140,11 @@ static inline bool is_nm_fault(u32 intr_info)
  	return is_exception_n(intr_info, NM_VECTOR);
  }
+static inline bool is_ve_fault(u32 intr_info)
+{
+	return is_exception_n(intr_info, VE_VECTOR);
+}
+
  /* Undocumented: icebp/int1 */
  static inline bool is_icebp(u32 intr_info)
  {

arch/x86/kvm/vmx/vmx.c +10 -0

index f890191e8580..dd3fde9d3c32 100644
@@ -126,6 +126,9 @@ module_param(error_on_inconsistent_vmcs_config, bool, 0444);
  static bool __read_mostly dump_invalid_vmcs = 0;
  module_param(dump_invalid_vmcs, bool, 0644);
+static bool __read_mostly ept_violation_ve_test;
+module_param(ept_violation_ve_test, bool, 0444);
+
  #define MSR_BITMAP_MODE_X2APIC		1
  #define MSR_BITMAP_MODE_X2APIC_APICV	2

@@ -783,6 +786,13 @@ void vmx_update_exception_bitmap(struct kvm_vcpu *vcpu)
  	eb = (1u << PF_VECTOR) | (1u << UD_VECTOR) | (1u << MC_VECTOR) |
  	     (1u << DB_VECTOR) | (1u << AC_VECTOR);
+	/*
+	 * #VE isn't used for VMX, but for TDX.  To test against unexpected
+	 * change related to #VE for VMX, intercept unexpected #VE and warn on
+	 * it.
+	 */
+	if (ept_violation_ve_test)
+		eb |= 1u << VE_VECTOR;
  	/*
  	 * Guest access to VMware backdoor ports could legitimately
  	 * trigger #GP because of TSS I/O permission bitmap.

Quoted 8 lines

arch/x86/kvm/vmx/vmx.c static int setup_vmcs_config(struct vmcs_config *vmcs_conf, · line 2658
Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +2 -0
@@ -2644,6 +2654,9 @@ static int setup_vmcs_config(struct vmcs_config *vmcs_conf,
  					&_cpu_based_2nd_exec_control))
  			return -EIO;
  	}
+	if (!ept_violation_ve_test)
+		_cpu_based_exec_control &= ~SECONDARY_EXEC_EPT_VIOLATION_VE;
should be _cpu_based_2nd_exec_control

Oops, thanks.

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:25 UTC #99 lore raw

PATCH v10 40/108 The start of TDX KVM patch series: KVM TDP MMU hooks

From: Isaku Yamahata <[email protected]>

This empty commit is to mark the start of patch series of KVM TDP MMU hooks.

---
 Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Documentation/virt/kvm/intel-tdx-layer-status.rst +2 -2

index df003d2ed89e..d5cace00c433 100644
@@ -25,6 +25,6 @@ Patch Layer status
 * TD vcpu interrupts/exit/hypercall:    Not yet
 
 * KVM MMU GPA shared bits:              Applied
-* KVM TDP refactoring for TDX:          Applying
-* KVM TDP MMU hooks:                    Not yet
+* KVM TDP refactoring for TDX:          Applied
+* KVM TDP MMU hooks:                    Applying
 * KVM TDP MMU MapGPA:                   Not yet

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #100 lore raw

PATCH v10 41/108 KVM: x86/tdp_mmu: refactor kvm_tdp_mmu_map()

From: Isaku Yamahata <[email protected]>

Factor out non-leaf SPTE population logic from kvm_tdp_mmu_map(). MapGPA hypercall needs to populate non-leaf SPTE to record which GPA, private or shared, is allowed in the leaf EPT entry.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

arch/x86/kvm/mmu/tdp_mmu.c +19 -7

index 3325633b1cb5..11b0ec8aebe2 100644
@@ -1157,6 +1157,24 @@ static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter,
 	return 0;
 }
 
+static int tdp_mmu_populate_nonleaf(struct kvm_vcpu *vcpu, struct tdp_iter *iter,
+				    bool account_nx)
+{
+	struct kvm_mmu_page *sp;
+	int ret;
+
+	KVM_BUG_ON(is_shadow_present_pte(iter->old_spte), vcpu->kvm);
+	KVM_BUG_ON(is_removed_spte(iter->old_spte), vcpu->kvm);
+
+	sp = tdp_mmu_alloc_sp(vcpu);
+	tdp_mmu_init_child_sp(sp, iter);
+
+	ret = tdp_mmu_link_sp(vcpu->kvm, iter, sp, account_nx, true);
+	if (ret)
+		tdp_mmu_free_sp(sp);
+	return ret;
+}
+
 /*
  * Handle a TDP page fault (NPT/EPT violation/misconfiguration) by installing
  * page tables and SPTEs to translate the faulting guest physical address.

@@ -1165,7 +1183,6 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 {
 	struct kvm_mmu *mmu = vcpu->arch.mmu;
 	struct tdp_iter iter;
-	struct kvm_mmu_page *sp;
 	int ret;
 
 	kvm_mmu_hugepage_adjust(vcpu, fault);

@@ -1211,13 +1228,8 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 			if (is_removed_spte(iter.old_spte))
 				break;
 
-			sp = tdp_mmu_alloc_sp(vcpu);
-			tdp_mmu_init_child_sp(sp, &iter);
-
-			if (tdp_mmu_link_sp(vcpu->kvm, &iter, sp, account_nx, true)) {
-				tdp_mmu_free_sp(sp);
+			if (tdp_mmu_populate_nonleaf(vcpu, &iter, account_nx))
 				break;
-			}
 		}
 	}

Signature · 1 line

2.25.1

Huang, Kai

2022-11-16 09:42 UTC #101 lore raw

Re: [PATCH v10 041/108] KVM: x86/tdp_mmu: refactor kvm_tdp_mmu_map()

Quoted 5 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>
Factor out non-leaf SPTE population logic from kvm_tdp_mmu_map(). MapGPA hypercall needs to populate non-leaf SPTE to record which GPA, private or shared, is allowed in the leaf EPT entry.

Is this patch still valid/needed since you have changed to use XArray to store whether GFN is private or shared?

Quoted 58 lines

---
 arch/x86/kvm/mmu/tdp_mmu.c | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

arch/x86/kvm/mmu/tdp_mmu.c +19 -7

index 3325633b1cb5..11b0ec8aebe2 100644
@@ -1157,6 +1157,24 @@ static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter,
 	return 0;
 }
 
+static int tdp_mmu_populate_nonleaf(struct kvm_vcpu *vcpu, struct tdp_iter *iter,
+				    bool account_nx)
+{
+	struct kvm_mmu_page *sp;
+	int ret;
+
+	KVM_BUG_ON(is_shadow_present_pte(iter->old_spte), vcpu->kvm);
+	KVM_BUG_ON(is_removed_spte(iter->old_spte), vcpu->kvm);
+
+	sp = tdp_mmu_alloc_sp(vcpu);
+	tdp_mmu_init_child_sp(sp, iter);
+
+	ret = tdp_mmu_link_sp(vcpu->kvm, iter, sp, account_nx, true);
+	if (ret)
+		tdp_mmu_free_sp(sp);
+	return ret;
+}
+
 /*
  * Handle a TDP page fault (NPT/EPT violation/misconfiguration) by installing
  * page tables and SPTEs to translate the faulting guest physical address.

@@ -1165,7 +1183,6 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 {
 	struct kvm_mmu *mmu = vcpu->arch.mmu;
 	struct tdp_iter iter;
-	struct kvm_mmu_page *sp;
 	int ret;
 
 	kvm_mmu_hugepage_adjust(vcpu, fault);

@@ -1211,13 +1228,8 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 			if (is_removed_spte(iter.old_spte))
 				break;
 
-			sp = tdp_mmu_alloc_sp(vcpu);
-			tdp_mmu_init_child_sp(sp, &iter);
-
-			if (tdp_mmu_link_sp(vcpu->kvm, &iter, sp, account_nx, true)) {
-				tdp_mmu_free_sp(sp);
+			if (tdp_mmu_populate_nonleaf(vcpu, &iter, account_nx))
 				break;
-			}
 		}
 	}

Isaku Yamahata

2022-11-17 18:37 UTC #102 lore raw

Re: [PATCH v10 041/108] KVM: x86/tdp_mmu: refactor kvm_tdp_mmu_map()

Quoted 9 lines · Kai

Quoted 5 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
Factor out non-leaf SPTE population logic from kvm_tdp_mmu_map(). MapGPA hypercall needs to populate non-leaf SPTE to record which GPA, private or shared, is allowed in the leaf EPT entry.
Is this patch still valid/needed since you have changed to use XArray to store whether GFN is private or shared?

Because tdp_mmu_populate_nonleaf() won't be touched any more, this patch doesn't have any benefit. Will drop it.

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:25 UTC #103 lore raw

PATCH v10 42/108 KVM: x86/tdp_mmu: Init role member of struct kvm_mmu_page at allocation

From: Isaku Yamahata <[email protected]>

Refactor tdp_mmu_alloc_sp() and tdp_mmu_init_sp and eliminate tdp_mmu_init_child_sp(). Currently tdp_mmu_init_sp() (or tdp_mmu_init_child_sp()) sets kvm_mmu_page.role after tdp_mmu_alloc_sp() allocating struct kvm_mmu_page and its page table page. This patch makes tdp_mmu_alloc_sp() initialize kvm_mmu_page.role instead of tdp_mmu_init_sp().

To handle private page tables, argument of is_private needs to be passed down. Given that already page level is passed down, it would be cumbersome to add one more parameter about sp. Instead replace the level argument with union kvm_mmu_page_role. Thus the number of argument won't be increased and more info about sp can be passed down.

For private sp, secure page table will be also allocated in addition to struct kvm_mmu_page and page table (spt member). The allocation functions (tdp_mmu_alloc_sp() and __tdp_mmu_alloc_sp_for_split()) need to know if the allocation is for the conventional page table or private page table. Pass union kvm_mmu_role to those functions and initialize role member of struct kvm_mmu_page.

---
 arch/x86/kvm/mmu/tdp_iter.h | 12 ++++++++++
 arch/x86/kvm/mmu/tdp_mmu.c  | 44 ++++++++++++++++---------------------
 2 files changed, 31 insertions(+), 25 deletions(-)

arch/x86/kvm/mmu/tdp_iter.h +12 -0

index f0af385c56e0..9e56a5b1024c 100644
@@ -115,4 +115,16 @@ void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root,
 void tdp_iter_next(struct tdp_iter *iter);
 void tdp_iter_restart(struct tdp_iter *iter);
 
+static inline union kvm_mmu_page_role tdp_iter_child_role(struct tdp_iter *iter)
+{
+	union kvm_mmu_page_role child_role;
+	struct kvm_mmu_page *parent_sp;
+
+	parent_sp = sptep_to_sp(rcu_dereference(iter->sptep));
+
+	child_role = parent_sp->role;
+	child_role.level--;
+	return child_role;
+}
+
 #endif /* __KVM_X86_MMU_TDP_ITER_H */

arch/x86/kvm/mmu/tdp_mmu.c +19 ~~-25~~

117 lines not on this page · raw message

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #104 lore raw

PATCH v10 43/108 KVM: x86/mmu: Require TDP MMU for TDX

From: Isaku Yamahata <[email protected]>

Require the TDP MMU for guest TDs, the so called "shadow" MMU does not support mapping guest private memory, i.e. does not support Secure-EPT.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

arch/x86/kvm/mmu/tdp_mmu.c +5 -1

index c6be76f9849c..1bf58288ea79 100644
@@ -18,8 +18,12 @@ int kvm_mmu_init_tdp_mmu(struct kvm *kvm)
 {
 	struct workqueue_struct *wq;
 
+	/*
+	 * Because only the TDP MMU supports TDX, require the TDP MMU for guest
+	 * TDs.
+	 */
 	if (!tdp_enabled || !READ_ONCE(tdp_mmu_enabled))
-		return 0;
+		return kvm->arch.vm_type == KVM_X86_TDX_VM ? -EOPNOTSUPP : 0;
 
 	wq = alloc_workqueue("kvm", WQ_UNBOUND|WQ_MEM_RECLAIM|WQ_CPU_INTENSIVE, 0);
 	if (!wq)

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #105 lore raw

PATCH v10 44/108 KVM: x86/mmu: Add a new is_private member for union kvm_mmu_page_role

From: Isaku Yamahata <[email protected]>

Because TDX support introduces private mapping, add a new member in union kvm_mmu_page_role with access functions to check the member.

---
 arch/x86/include/asm/kvm_host.h | 27 +++++++++++++++++++++++++++
 arch/x86/kvm/mmu/mmu_internal.h | 11 +++++++++++
 2 files changed, 38 insertions(+)

arch/x86/include/asm/kvm_host.h +27 -0

index 2e0b23422d63..ee01add57a6b 100644
@@ -335,7 +335,12 @@ union kvm_mmu_page_role {
 		unsigned ad_disabled:1;
 		unsigned guest_mode:1;
 		unsigned passthrough:1;
+#ifdef CONFIG_KVM_MMU_PRIVATE
+		unsigned is_private:1;
+		unsigned :4;
+#else
 		unsigned :5;
+#endif
 
 		/*
 		 * This is left at the top of the word so that

@@ -347,6 +352,28 @@ union kvm_mmu_page_role {
 	};
 };
 
+#ifdef CONFIG_KVM_MMU_PRIVATE
+static inline bool kvm_mmu_page_role_is_private(union kvm_mmu_page_role role)
+{
+	return !!role.is_private;
+}
+
+static inline void kvm_mmu_page_role_set_private(union kvm_mmu_page_role *role)
+{
+	role->is_private = 1;
+}
+#else
+static inline bool kvm_mmu_page_role_is_private(union kvm_mmu_page_role role)
+{
+	return false;
+}
+
+static inline void kvm_mmu_page_role_set_private(union kvm_mmu_page_role *role)
+{
+	WARN_ON_ONCE(1);
+}
+#endif
+
 /*
  * kvm_mmu_extended_role complements kvm_mmu_page_role, tracking properties
  * relevant to the current MMU configuration.   When loading CR0, CR4, or EFER,

arch/x86/kvm/mmu/mmu_internal.h +11 -0

index 57e3ea2b52cc..b0c4a78404b2 100644
@@ -142,6 +142,17 @@ static inline int kvm_mmu_page_as_id(struct kvm_mmu_page *sp)
 	return kvm_mmu_role_as_id(sp->role);
 }
 
+static inline bool is_private_sp(const struct kvm_mmu_page *sp)
+{
+	return kvm_mmu_page_role_is_private(sp->role);
+}
+
+static inline bool is_private_sptep(u64 *sptep)
+{
+	WARN_ON_ONCE(!sptep);
+	return is_private_sp(sptep_to_sp(sptep));
+}
+
 static inline bool kvm_mmu_page_ad_need_write_protect(struct kvm_mmu_page *sp)
 {
 	/*

Signature · 1 line

2.25.1

[email protected]

author

2022-10-30 06:25 UTC #106 lore raw

PATCH v10 45/108 KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page

From: Isaku Yamahata <[email protected]>

For private GPA, CPU refers a private page table whose contents are encrypted. The dedicated APIs to operate on it (e.g. updating/reading its PTE entry) are used and their cost is expensive.

When KVM resolves KVM page fault, it walks the page tables. To reuse the existing KVM MMU code and mitigate the heavy cost to directly walk protected (encrypted) page table, allocate one more page to copy the protected page table for KVM MMU code to directly walk. Resolve KVM page fault with the existing code, and do additional operations necessary for the protected page table. To distinguish such cases, the existing KVM page table is called a shared page table (i.e. not associated with protected page table), and the page table with protected page table is called a private page table. The relationship is depicted below.

Add a private pointer to struct kvm_mmu_page for protected page table and add helper functions to allocate/initialize/free a protected page table page.

              KVM page fault                     |
                     |                           |
                     V                           |
        -------------+----------                 |
        |                      |                 |
        V                      V                 |
     shared GPA           private GPA            |
        |                      |                 |
        V                      V                 |
    shared PT root      private PT root          |    protected PT root
        |                      |                 |           |
        V                      V                 |           V
     shared PT            private PT ----propagate----> protected PT
        |                      |                 |           |
        |                      \-----------------+------\    |
        |                                        |      |    |
        V                                        |      V    V
  shared guest page                              |    private guest page
                                                 |
                           non-encrypted memory  |    encrypted memory
                                                 |

PT: page table
- Shared PT is visible to KVM and it is used by CPU.
- Protected PT is used by CPU but it is invisible to KVM.
- Private PT is visible to KVM but not used by CPU. It is used to propagate PT change to the actual protected PT which is used by CPU.

---
 arch/x86/include/asm/kvm_host.h |  7 +++
 arch/x86/kvm/mmu/mmu.c          |  8 +++
 arch/x86/kvm/mmu/mmu_internal.h | 90 +++++++++++++++++++++++++++++++--
 arch/x86/kvm/mmu/tdp_mmu.c      |  1 +
 4 files changed, 102 insertions(+), 4 deletions(-)

arch/x86/include/asm/kvm_host.h +7 -0

index ee01add57a6b..381df2c8136d 100644
@@ -754,6 +754,13 @@ struct kvm_vcpu_arch {
 	struct kvm_mmu_memory_cache mmu_shadow_page_cache;
 	struct kvm_mmu_memory_cache mmu_shadowed_info_cache;
 	struct kvm_mmu_memory_cache mmu_page_header_cache;
+	/*
+	 * This cache is to allocate private page table. E.g.  Secure-EPT used
+	 * by the TDX module.  Because the TDX module doesn't trust VMM and
+	 * initializes the pages itself, KVM doesn't initialize them.  Allocate
+	 * pages with garbage and give them to the TDX module.
+	 */
+	struct kvm_mmu_memory_cache mmu_private_spt_cache;
 
 	/*
 	 * QEMU userspace and the guest each have their own FPU state.

arch/x86/kvm/mmu/mmu.c +8 -0

index 0001e921154e..faf69774c7ce 100644
@@ -653,6 +653,13 @@ static int mmu_topup_shadow_page_cache(struct kvm_vcpu *vcpu)
 	struct kvm_mmu_memory_cache *mc = &vcpu->arch.mmu_shadow_page_cache;
 	int start, end, i, r;
 
+	if (kvm_gfn_shared_mask(vcpu->kvm)) {
+		r = kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_private_spt_cache,
+					       PT64_ROOT_MAX_LEVEL);
+		if (r)
+			return r;
+	}
+
 	start = kvm_mmu_memory_cache_nr_free_objects(mc);
 	r = kvm_mmu_topup_memory_cache(mc, PT64_ROOT_MAX_LEVEL);

@@ -702,6 +709,7 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_pte_list_desc_cache);
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_shadow_page_cache);
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_shadowed_info_cache);
+	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_private_spt_cache);
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_header_cache);
 }

arch/x86/kvm/mmu/mmu_internal.h +86 -4

108 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.c +1 -0

index 1bf58288ea79..b2f56110d62d 100644
@@ -71,6 +71,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
 
 static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
 {
+	kvm_mmu_free_private_spt(sp);
 	free_page((unsigned long)sp->spt);
 	kmem_cache_free(mmu_page_header_cache, sp);
 }

Signature · 1 line

2.25.1

Huang, Kai

2022-11-16 10:46 UTC #107 lore raw

Re: [PATCH v10 045/108] KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page

Quoted 60 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>
For private GPA, CPU refers a private page table whose contents are encrypted. The dedicated APIs to operate on it (e.g. updating/reading its PTE entry) are used and their cost is expensive.
When KVM resolves KVM page fault, it walks the page tables. To reuse the existing KVM MMU code and mitigate the heavy cost to directly walk protected (encrypted) page table, allocate one more page to copy the protected page table for KVM MMU code to directly walk. Resolve KVM page fault with the existing code, and do additional operations necessary for the protected page table. To distinguish such cases, the existing KVM page table is called a shared page table (i.e. not associated with protected page table), and the page table with protected page table is called a private page table. The relationship is depicted below.
Add a private pointer to struct kvm_mmu_page for protected page table and add helper functions to allocate/initialize/free a protected page table page.
              KVM page fault                     |
                     |                           |
                     V                           |
        -------------+----------                 |
        |                      |                 |
        V                      V                 |
     shared GPA           private GPA            |
        |                      |                 |
        V                      V                 |
    shared PT root      private PT root          |    protected PT root
        |                      |                 |           |
        V                      V                 |           V
     shared PT            private PT ----propagate----> protected PT
        |                      |                 |           |
        |                      \-----------------+------\    |
        |                                        |      |    |
        V                                        |      V    V
  shared guest page                              |    private guest page
                                                 |
                           non-encrypted memory  |    encrypted memory
                                                 |
PT: page table
- Shared PT is visible to KVM and it is used by CPU.
- Protected PT is used by CPU but it is invisible to KVM.
- Private PT is visible to KVM but not used by CPU. It is used to propagate PT change to the actual protected PT which is used by CPU.
Signed-off-by Isaku Yamahata <[email protected]>
---
 arch/x86/include/asm/kvm_host.h |  7 +++
 arch/x86/kvm/mmu/mmu.c          |  8 +++
 arch/x86/kvm/mmu/mmu_internal.h | 90 +++++++++++++++++++++++++++++++--
 arch/x86/kvm/mmu/tdp_mmu.c      |  1 +
 4 files changed, 102 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -0
index ee01add57a6b..381df2c8136d 100644

arch/x86/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 762

Quoted 9 lines

arch/x86/include/asm/kvm_host.h +6 -0

@@ -754,6 +754,13 @@ struct kvm_vcpu_arch {
 	struct kvm_mmu_memory_cache mmu_shadow_page_cache;
 	struct kvm_mmu_memory_cache mmu_shadowed_info_cache;
 	struct kvm_mmu_memory_cache mmu_page_header_cache;
+	/*
+	 * This cache is to allocate private page table. E.g.  Secure-EPT used
+	 * by the TDX module.  Because the TDX module doesn't trust VMM and
+	 * initializes the pages itself, KVM doesn't initialize them.  Allocate
+	 * pages with garbage and give them to the TDX module.
+	 */

Perhaps put "Because ..." part to kvm_mmu_create() so people can see why there's no initialization for mmu_private_spte_cache?

arch/x86/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 763

Quoted 1 line

arch/x86/include/asm/kvm_host.h +1 -0
+	struct kvm_mmu_memory_cache mmu_private_spt_cache;

Nit:

In changelog you mentioned the page table used by CPU is actually "protected PT". IIUC the sp->private_spt you allocated from this cache is the one that is actually used by CPU. So looks there's some inconsistency here.

[...]

arch/x86/include/asm/kvm_host.h struct kvm_vcpu_arch {

Quoted 4 lines

arch/x86/include/asm/kvm_host.h +4 -0

+
+static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu,
+					     struct kvm_mmu_memory_cache *private_spt_cache,
+					     struct kvm_mmu_page *sp)

This function is very weird in the context of this patch. _Only_ a new vcpu- scope 'mmu_private_spte_cache' is added in this patch, but here you allow caller to pass an additional argument of private_spt_cache. So there must be another cache which is not introduced in this patch?

arch/x86/include/asm/kvm_host.h

Quoted 5 lines

arch/x86/include/asm/kvm_host.h +5 -0

+{
+	/*
+	 * vcpu == NULL means non-root SPT:
+	 * vcpu == NULL is used to split a large SPT into smaller SPT.  Root SPT
+	 * is not a large SPT.

I am guessing this "vcpu == NULL" case is for "Eager Splitting"?

If so, why not adding a global MMU cache for private_spt allocation, and make vcpu->arch.mmu_private_spt_cache point to the global cache? In this case, in the context where you only have 'kvm', you can use the global cache directly. And in the context where you have a 'vcpu', you just use vcpu's cache.

Quoted 24 lines

arch/x86/include/asm/kvm_host.h +24 -0

+	 */
+	bool is_root = vcpu &&
+		vcpu->arch.root_mmu.root_role.level == sp->role.level;
+
+	if (vcpu)
+		private_spt_cache = &vcpu->arch.mmu_private_spt_cache;
+	KVM_BUG_ON(!kvm_mmu_page_role_is_private(sp->role), vcpu->kvm);
+	if (is_root)
+		/*
+		 * Because TDX module assigns root Secure-EPT page and set it to
+		 * Secure-EPTP when TD vcpu is created, secure page table for
+		 * root isn't needed.
+		 */
+		sp->private_spt = NULL;
+	else {
+		sp->private_spt = kvm_mmu_memory_cache_alloc(private_spt_cache);
+		/*
+		 * Because mmu_private_spt_cache is topped up before staring kvm
+		 * page fault resolving, the allocation above shouldn't fail.
+		 */
+		WARN_ON_ONCE(!sp->private_spt);
+	}
+}
+

Huang, Kai

2022-11-16 12:01 UTC #108 lore raw

Re: [PATCH v10 045/108] KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page

Quoted 21 lines · Kai

Review
Quoted 5 lines
arch/x86/kvm/mmu/mmu_internal.h +3 -0
+
+static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu,
+					     struct kvm_mmu_memory_cache
*private_spt_cache,
(quoted diff) +1 -0
+					     struct kvm_mmu_page *sp)
This function is very weird in the context of this patch. _Only_ a new vcpu- scope 'mmu_private_spte_cache' is added in this patch, but here you allow caller
to pass an additional argument of private_spt_cache. So there must be another cache which is not introduced in this patch?
arch/x86/kvm/mmu/mmu_internal.h static inline bool is_private_sptep(u64 *sptep) · line 187
Quoted 6 lines
arch/x86/kvm/mmu/mmu_internal.h +4 -0
+{
+	/*
+	 * vcpu == NULL means non-root SPT:
+	 * vcpu == NULL is used to split a large SPT into smaller SPT. 
Root SPT
arch/x86/kvm/mmu/mmu_internal.h +1 -0
+	 * is not a large SPT.
I am guessing this "vcpu == NULL" case is for "Eager Splitting"?

Quoted 4 lines

If so, why not adding a global MMU cache for private_spt allocation, and make vcpu->arch.mmu_private_spt_cache point to the global cache? In this case, in the context where you only have 'kvm', you can use the global cache directly. And in the context where you have a 'vcpu', you just use vcpu's cache.

So I went through all MMU related patches in this series, but I cannot find a place where this function is called with 'vcpu == NULL' and a valid cache is passed in, if I am reading correctly.

Also checked that "Eager Splitting" uses a kvm-scope cache for legacy MMU, but just uses __get_free_page() for TDP MMU. And in later patch "KVM: x86/tdp_mmu:
Support TDX private mapping for TDP MMU", __get_free_page() is also used to allocate the private_spt (which is consistent with existing eager splitting code).

So IIUC only legacy MMU code will call this function with 'vcpu == NULL' and a valid cache. In this case, please remove the 'private_spt_cache' argument for now, and make the function always allocate from the vcpu-

Quoted 1 line

arch.mmu_private_spt_cache.

You can add the additional argument when TDX gets legacy MMU support.

Also, I think you need to move eager splitting support part (whether that handling is correct is another story) from the later patch to this patch. Otherwise this patch is not complete.

Quoted 30 lines

Quoted 29 lines

arch/x86/kvm/mmu/mmu_internal.h +10 -0

+	 */
+	bool is_root = vcpu &&
+		vcpu->arch.root_mmu.root_role.level == sp->role.level;
+
+	if (vcpu)
+		private_spt_cache = &vcpu->arch.mmu_private_spt_cache;
+	KVM_BUG_ON(!kvm_mmu_page_role_is_private(sp->role), vcpu->kvm);
+	if (is_root)
+		/*
+		 * Because TDX module assigns root Secure-EPT page and set

it to

(quoted diff) +1 -0

+		 * Secure-EPTP when TD vcpu is created, secure page table

for

arch/x86/kvm/mmu/mmu_internal.h +5 -0

+		 * root isn't needed.
+		 */
+		sp->private_spt = NULL;
+	else {
+		sp->private_spt =

kvm_mmu_memory_cache_alloc(private_spt_cache);

(quoted diff) +2 -0

+		/*
+		 * Because mmu_private_spt_cache is topped up before

staring kvm

(quoted diff) +1 -0

+		 * page fault resolving, the allocation above shouldn't

fail.

arch/x86/kvm/mmu/mmu_internal.h +5 -0

+		 */
+		WARN_ON_ONCE(!sp->private_spt);
+	}
+}
+

Isaku Yamahata

2022-11-17 19:25 UTC #109 lore raw

Re: [PATCH v10 045/108] KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page

Quoted 40 lines · Kai

Quoted 21 lines · Kai
Review
Quoted 5 lines
arch/x86/kvm/mmu/mmu_internal.h +3 -0
+
+static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu,
+					     struct kvm_mmu_memory_cache
*private_spt_cache,
(quoted diff) +1 -0
+					     struct kvm_mmu_page *sp)
This function is very weird in the context of this patch. _Only_ a new vcpu- scope 'mmu_private_spte_cache' is added in this patch, but here you allow caller
to pass an additional argument of private_spt_cache. So there must be another cache which is not introduced in this patch?
arch/x86/kvm/mmu/mmu_internal.h static inline bool is_private_sptep(u64 *sptep) · line 187
Quoted 6 lines
arch/x86/kvm/mmu/mmu_internal.h +4 -0
+{
+	/*
+	 * vcpu == NULL means non-root SPT:
+	 * vcpu == NULL is used to split a large SPT into smaller SPT. 
Root SPT
arch/x86/kvm/mmu/mmu_internal.h +1 -0
+	 * is not a large SPT.
I am guessing this "vcpu == NULL" case is for "Eager Splitting"?
Quoted 4 lines
If so, why not adding a global MMU cache for private_spt allocation, and make vcpu->arch.mmu_private_spt_cache point to the global cache? In this case, in the context where you only have 'kvm', you can use the global cache directly. And in the context where you have a 'vcpu', you just use vcpu's cache.
So I went through all MMU related patches in this series, but I cannot find a place where this function is called with 'vcpu == NULL' and a valid cache is passed in, if I am reading correctly.
Also checked that "Eager Splitting" uses a kvm-scope cache for legacy MMU, but just uses __get_free_page() for TDP MMU. And in later patch "KVM: x86/tdp_mmu:
Support TDX private mapping for TDP MMU", __get_free_page() is also used to allocate the private_spt (which is consistent with existing eager splitting code).
So IIUC only legacy MMU code will call this function with 'vcpu == NULL' and a valid cache. In this case, please remove the 'private_spt_cache' argument for now, and make the function always allocate from the vcpu-

Quoted 7 lines

Quoted 1 line
arch.mmu_private_spt_cache.
You can add the additional argument when TDX gets legacy MMU support.
Also, I think you need to move eager splitting support part (whether that handling is correct is another story) from the later patch to this patch. Otherwise this patch is not complete.

Yes, you're right. Somehow legacy mmu part is crept in. I'll remove those from this patch series.

Signature · 1 line

Isaku Yamahata <[email protected]>

[email protected]

author

2022-10-30 06:26 UTC #110 lore raw

PATCH v10 46/108 KVM: Add flags to struct kvm_gfn_range

From: Isaku Yamahata <[email protected]>

kvm_unmap_gfn_range() needs to know the reason of the callback for TDX. mmu notifier, set memattr ioctl or restrictedmem notifier. Based on the reason, TDX changes the behavior. For mmu notifier, it's the operation on shared memory slot to zap shared PTE. For set memattr, private<->shared conversion, zap the original PTE. For restrictedmem, it's a hint that TDX can ignore.

---
 include/linux/kvm_host.h | 8 +++++++-
 virt/kvm/kvm_main.c      | 5 ++++-
 2 files changed, 11 insertions(+), 2 deletions(-)

include/linux/kvm_host.h +7 -1

index 839d98d56632..b658803ea2c7 100644
@@ -247,12 +247,18 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 
 
 #if defined(KVM_ARCH_WANT_MMU_NOTIFIER) || defined(CONFIG_HAVE_KVM_RESTRICTED_MEM)
+#define KVM_GFN_RANGE_FLAGS_RESTRICTED_MEM	BIT(0)
+#define KVM_GFN_RANGE_FLAGS_SET_MEM_ATTR	BIT(1)
 struct kvm_gfn_range {
 	struct kvm_memory_slot *slot;
 	gfn_t start;
 	gfn_t end;
-	pte_t pte;
+	union {
+		pte_t pte;
+		int attr;
+	};
 	bool may_block;
+	unsigned int flags;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif

virt/kvm/kvm_main.c +4 -1

index 3b05a3396f89..dda2f2ec4faa 100644
@@ -676,6 +676,7 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm,
 			gfn_range.start = hva_to_gfn_memslot(hva_start, slot);
 			gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot);
 			gfn_range.slot = slot;
+			gfn_range.flags = 0;
 
 			if (!locked) {
 				locked = true;

@@ -947,8 +948,9 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end,
 	int i;
 	int r = 0;
 
-	gfn_range.pte = __pte(0);
+	gfn_range.attr = attr;
 	gfn_range.may_block = true;
+	gfn_range.flags = KVM_GFN_RANGE_FLAGS_SET_MEM_ATTR;
 
 	for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
 		slots = __kvm_memslots(kvm, i);

@@ -1074,6 +1076,7 @@ static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *no
 	gfn_range.slot = slot;
 	gfn_range.pte = __pte(0);
 	gfn_range.may_block = true;
+	gfn_range.flags = KVM_GFN_RANGE_FLAGS_RESTRICTED_MEM;
 
 	if (kvm_unmap_gfn_range(kvm, &gfn_range))
 		kvm_flush_remote_tlbs(kvm);

Signature · 1 line

2.25.1

Huang, Kai

2022-12-14 10:51 UTC #111 lore raw

Re: [PATCH v10 046/108] KVM: Add flags to struct kvm_gfn_range

Quoted 8 lines · On Sat, 2022-10-29 at 23:22 -0700, [email protected] wrote:

From: Isaku Yamahata <[email protected]>
kvm_unmap_gfn_range() needs to know the reason of the callback for TDX. mmu notifier, set memattr ioctl or restrictedmem notifier. Based on the reason, TDX changes the behavior. For mmu notifier, it's the operation on shared memory slot to zap shared PTE. For set memattr, private<->shared conversion, zap the original PTE. For restrictedmem, it's a hint that TDX can ignore.

Could you elaborate why restricted memfd notifier can be ignored? IIUC if userspace punch a hole, the pages within the hole will be de-allocated. So why can such notifier be ignored?

Btw, I think this patch should be just merged to the patch which consumes those flags (checked it is next patch). It's easier to review when putting them together.

Quoted 16 lines