Skip to contentSkip to conversation

Enable Secure TSC for SEV-SNP

RFC PATCH Superseded Nikunj A Dadhania 3 participants 12 messages Aug 29, 2024 – Nov 22, 2024

Subsystems (3)

v1 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

cover Enable Secure TSC for SEV-SNP

TSC value calculations for the guest are controlled by the hypervisor. A malicious hypervisor can prevent guest from moving forward. The Secure TSC feature for SEV-SNP allows guests to securely use the RDTSC and RDTSCP instructions. This ensures the guest gets a consistent view of time and prevents a malicious hypervisor from making it appear that time rolls backwards, advancing at an unusually fast rate, or employing similar tricks. For more details, please refer to "Secure Nested Paging (SEV-SNP)" section, subsection "Secure TSC" of APM Volume 2

This patchset is also available at:

https://github.com/AMDESE/linux-kvm/tree/sectsc-host-latest

and is based on v6.11-rc5

Testing SecureTSC
-----------------

SecureTSC Guest patches:
https://github.com/AMDESE/linux-kvm/tree/sectsc-guest-latest

QEMU changes:
https://github.com/nikunjad/qemu/tree/snp-securetsc-latest

QEMU commandline SEV-SNP with SecureTSC:

  qemu-system-x86_64 -cpu EPYC-Milan-v2 -smp 4 \
     -object memory-backend-memfd,id=ram1,size=1G,share=true,prealloc=false,reserve=false \
     -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,secure-tsc=on \
     -machine q35,confidential-guest-support=sev0,memory-backend=ram1 \
     ...

Ketan Chaturvedi (2):
KVM: SVM: Set TSC frequency for SecureTSC-enabled guests
KVM: SVM: Add Secure TSC support for SNP Guest

Nikunj A Dadhania (3):
x86/cpufeatures: Add SNP Secure TSC
KVM: SVM: Add GUEST_TSC_FREQ MSR
KVM: SVM: Prevent writes to TSC MSR when Secure TSC is enabled

 arch/x86/include/asm/cpufeatures.h |  1 +
 arch/x86/include/asm/msr-index.h   |  1 +
 arch/x86/include/asm/svm.h         |  1 +
 arch/x86/include/uapi/asm/kvm.h    |  3 ++-
 arch/x86/kvm/svm/sev.c             | 15 +++++++++++++++
 arch/x86/kvm/svm/svm.c             | 12 ++++++++++++
 arch/x86/kvm/svm/svm.h             | 14 +++++++++++++-
 include/linux/psp-sev.h            |  2 ++
 8 files changed, 47 insertions(+), 2 deletions(-)

base-commit: 5be63fc19fcaa4c236b307420483578a56986a37

Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH 1/5 x86/cpufeatures: Add SNP Secure TSC

The Secure TSC feature for SEV-SNP allows guests to securely use the RDTSC and RDTSCP instructions, ensuring that the parameters used cannot be altered by the hypervisor once the guest is launched. More details in the AMD64 APM Vol 2, Section "Secure TSC".

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index dd4682857c12..ed61549e8a11 100644
@@ -444,6 +444,7 @@
 #define X86_FEATURE_VM_PAGE_FLUSH	(19*32+ 2) /* VM Page Flush MSR is supported */
 #define X86_FEATURE_SEV_ES		(19*32+ 3) /* "sev_es" AMD Secure Encrypted Virtualization - Encrypted State */
 #define X86_FEATURE_SEV_SNP		(19*32+ 4) /* "sev_snp" AMD Secure Encrypted Virtualization - Secure Nested Paging */
+#define X86_FEATURE_SNP_SECURE_TSC	(19*32+ 8) /* "" AMD SEV-SNP Secure TSC */
 #define X86_FEATURE_V_TSC_AUX		(19*32+ 9) /* Virtual TSC_AUX */
 #define X86_FEATURE_SME_COHERENT	(19*32+10) /* AMD hardware-enforced cache coherency */
 #define X86_FEATURE_DEBUG_SWAP		(19*32+14) /* "debug_swap" AMD SEV-ES full debug state swap support */
Signature · 1 line
2.34.1
maintainer
#2 lore raw

Re: [RFC PATCH 1/5] x86/cpufeatures: Add SNP Secure TSC

Quoted 15 lines · Nikunj A Dadhania

The Secure TSC feature for SEV-SNP allows guests to securely use the RDTSC and RDTSCP instructions, ensuring that the parameters used cannot be altered by the hypervisor once the guest is launched. More details in the AMD64 APM Vol 2, Section "Secure TSC".

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +0 -0
index dd4682857c12..ed61549e8a11 100644

arch/x86/include/asm/cpufeatures.h line 447

Quoted 4 lines
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -444,6 +444,7 @@
 #define X86_FEATURE_VM_PAGE_FLUSH	(19*32+ 2) /* VM Page Flush MSR is supported */
 #define X86_FEATURE_SEV_ES		(19*32+ 3) /* "sev_es" AMD Secure Encrypted Virtualization - Encrypted State */
 #define X86_FEATURE_SEV_SNP		(19*32+ 4) /* "sev_snp" AMD Secure Encrypted Virtualization - Secure Nested Paging */
+#define X86_FEATURE_SNP_SECURE_TSC	(19*32+ 8) /* "" AMD SEV-SNP Secure TSC */

There's a newline here on purpose - keep it.

Also, you don't need "" anymore.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#3 lore raw

Re: [RFC PATCH 1/5] x86/cpufeatures: Add SNP Secure TSC

Quoted 11 lines · Borislav Petkov
Quoted 10 lines · Nikunj A Dadhania

The Secure TSC feature for SEV-SNP allows guests to securely use the RDTSC and RDTSCP instructions, ensuring that the parameters used cannot be altered by the hypervisor once the guest is launched. More details in the AMD64 APM Vol 2, Section "Secure TSC".

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
Quoted 11 lines

arch/x86/include/asm/cpufeatures.h line 447

Quoted 9 lines
arch/x86/include/asm/cpufeatures.h +1 -0
index dd4682857c12..ed61549e8a11 100644
@@ -444,6 +444,7 @@
 #define X86_FEATURE_VM_PAGE_FLUSH	(19*32+ 2) /* VM Page Flush MSR is supported */
 #define X86_FEATURE_SEV_ES		(19*32+ 3) /* "sev_es" AMD Secure Encrypted Virtualization - Encrypted State */
 #define X86_FEATURE_SEV_SNP		(19*32+ 4) /* "sev_snp" AMD Secure Encrypted Virtualization - Secure Nested Paging */
+#define X86_FEATURE_SNP_SECURE_TSC	(19*32+ 8) /* "" AMD SEV-SNP Secure TSC */

There's a newline here on purpose - keep it.

Sure

Quoted 1 line

Also, you don't need "" anymore.

Ok, do we need to add an entry to tools/arch/x86/kcpuid/cpuid.csv ?

Regards
Nikunj

maintainer
#4 lore raw

Re: [RFC PATCH 1/5] x86/cpufeatures: Add SNP Secure TSC

Quoted 1 line · Nikunj A. Dadhania

Ok, do we need to add an entry to tools/arch/x86/kcpuid/cpuid.csv ?

Already there:

# Leaf 8000001FH
# AMD encrypted memory capabilities enumeration (SME/SEV)

0x8000001f,         0,  eax,       0,    sme                    , Secure Memory Encryption supported
0x8000001f,         0,  eax,       1,    sev                    , Secure Encrypted Virtualization supported
0x8000001f,         0,  eax,       2,    vm_page_flush          , VM Page Flush MSR (0xc001011e) available
0x8000001f,         0,  eax,       3,    sev_es                 , SEV Encrypted State supported
0x8000001f,         0,  eax,       4,    sev_nested_paging      , SEV secure nested paging supported
0x8000001f,         0,  eax,       5,    vm_permission_levels   , VMPL supported
0x8000001f,         0,  eax,       6,    rpmquery               , RPMQUERY instruction supported
0x8000001f,         0,  eax,       7,    vmpl_sss               , VMPL supervisor shadwo stack supported
0x8000001f,         0,  eax,       8,    secure_tsc             , Secure TSC supported

^^^^

but in general if it is not there, most definitely.

This list should contain *all* CPUID definitions.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#5 lore raw

Re: [RFC PATCH 1/5] x86/cpufeatures: Add SNP Secure TSC

Quoted 4 lines · Borislav Petkov
Quoted 1 line · Nikunj A. Dadhania

Ok, do we need to add an entry to tools/arch/x86/kcpuid/cpuid.csv ?

Already there:

Ah ok, in tip/master

Quoted 6 lines

0x8000001f, 0, eax, 8, secure_tsc , Secure TSC supported
^^^^

but in general if it is not there, most definitely.

This list should contain *all* CPUID definitions.

Sure

Regards
Nikunj

maintainer
#6 lore raw

Re: [RFC PATCH 1/5] x86/cpufeatures: Add SNP Secure TSC

Quoted 1 line · Nikunj A. Dadhania

Ah ok, in tip/master

What else?

See, we document these things not to waste space on people's hdds:

Documentation/process/maintainer-tip.rst

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#7 lore raw

RFC PATCH 2/5 KVM: SVM: Set TSC frequency for SecureTSC-enabled guests

In reply to Nikunj A Dadhania

From: Ketan Chaturvedi <[email protected]>

The SNP specification provides a desired TSC frequency parameter that can be set as part of the SNP_LAUNCH_START command. This field has effect only in Secure TSC enabled guests and can be used by the hypervisor to set desired mean TSC frequency in KHz of the guest.

---
 arch/x86/include/asm/svm.h      |  1 +
 arch/x86/include/uapi/asm/kvm.h |  3 ++-
 arch/x86/kvm/svm/sev.c          |  7 +++++++
 arch/x86/kvm/svm/svm.h          | 12 ++++++++++++
 include/linux/psp-sev.h         |  2 ++
 5 files changed, 24 insertions(+), 1 deletion(-)
arch/x86/include/asm/svm.h +1 -0
index f0dea3750ca9..2bb9be9b2d30 100644
@@ -289,6 +289,7 @@ static_assert((X2AVIC_MAX_PHYSICAL_ID & AVIC_PHYSICAL_MAX_INDEX_MASK) == X2AVIC_
 #define SVM_SEV_FEAT_RESTRICTED_INJECTION		BIT(3)
 #define SVM_SEV_FEAT_ALTERNATE_INJECTION		BIT(4)
 #define SVM_SEV_FEAT_DEBUG_SWAP				BIT(5)
+#define SVM_SEV_FEAT_SECURE_TSC				BIT(9)
 
 #define SVM_SEV_FEAT_INT_INJ_MODES		\
 	(SVM_SEV_FEAT_RESTRICTED_INJECTION |	\
arch/x86/include/uapi/asm/kvm.h +2 -1
index bf57a824f722..c15d7c843bfd 100644
@@ -834,7 +834,8 @@ struct kvm_sev_snp_launch_start {
 	__u64 policy;
 	__u8 gosvw[16];
 	__u16 flags;
-	__u8 pad0[6];
+	__u32 desired_tsc_freq;
+	__u8 pad0[2];
 	__u64 pad1[4];
 };
 
arch/x86/kvm/svm/sev.c +7 -0
index 714c517dd4b7..ff82a644b174 100644
@@ -2212,6 +2212,9 @@ static int snp_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	if (sev->snp_context)
 		return -EINVAL;
 
+	if (snp_secure_tsc_enabled(kvm) && !params.desired_tsc_freq)
+		return -EINVAL;
+
 	sev->snp_context = snp_context_create(kvm, argp);
 	if (!sev->snp_context)
 		return -ENOTTY;
@@ -2232,6 +2235,10 @@ static int snp_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
 
 	start.gctx_paddr = __psp_pa(sev->snp_context);
 	start.policy = params.policy;
+
+	if (snp_secure_tsc_enabled(kvm))
+		start.desired_tsc_freq = params.desired_tsc_freq;
+
 	memcpy(start.gosvw, params.gosvw, sizeof(params.gosvw));
 	rc = __sev_issue_cmd(argp->sev_fd, SEV_CMD_SNP_LAUNCH_START, &start, &argp->error);
 	if (rc) {
arch/x86/kvm/svm/svm.h +12 -0
index 76107c7d0595..262b638dfcb8 100644
@@ -378,6 +378,18 @@ static __always_inline bool sev_snp_guest(struct kvm *kvm)
 #endif
 }
 
+static inline bool snp_secure_tsc_enabled(struct kvm *kvm)
+{
+#ifdef CONFIG_KVM_AMD_SEV
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+
+	return (sev->vmsa_features & SVM_SEV_FEAT_SECURE_TSC) &&
+	       !WARN_ON_ONCE(!sev_snp_guest(kvm));
+#else
+	return false;
+#endif
+}
+
 static inline bool ghcb_gpa_is_registered(struct vcpu_svm *svm, u64 val)
 {
 	return svm->sev_es.ghcb_registered_gpa == val;
include/linux/psp-sev.h +2 -0
index 903ddfea8585..fcd710f0baf8 100644
@@ -594,6 +594,7 @@ struct sev_data_snp_addr {
  * @imi_en: launch flow is launching an IMI (Incoming Migration Image) for the
  *          purpose of guest-assisted migration.
  * @rsvd: reserved
+ * @desired_tsc_freq: hypervisor desired mean TSC freq in kHz of the guest
  * @gosvw: guest OS-visible workarounds, as defined by hypervisor
  */
 struct sev_data_snp_launch_start {
@@ -603,6 +604,7 @@ struct sev_data_snp_launch_start {
 	u32 ma_en:1;				/* In */
 	u32 imi_en:1;				/* In */
 	u32 rsvd:30;
+	u32 desired_tsc_freq;			/* In */
 	u8 gosvw[16];				/* In */
 } __packed;
 
Signature · 1 line
2.34.1
#8 lore raw

RFC PATCH 3/5 KVM: SVM: Add GUEST_TSC_FREQ MSR

In reply to Nikunj A Dadhania

TSC calculation in Secure TSC enabled guests uses a new read-only MSR 0xc0010134 (GUEST_TSC_FREQ). Add the GUEST_TSC_FREQ MSR and disable the interception when secure TSC is enabled. Moreover, GUEST_TSC_FREQ MSR is only available to the guest and is not accessible from hypervisor context.

---
 arch/x86/include/asm/msr-index.h | 1 +
 arch/x86/kvm/svm/sev.c           | 2 ++
 arch/x86/kvm/svm/svm.c           | 1 +
 arch/x86/kvm/svm/svm.h           | 2 +-
 4 files changed, 5 insertions(+), 1 deletion(-)
arch/x86/include/asm/msr-index.h +1 -0
index 82c6a4d350e0..b15635de1290 100644
@@ -586,6 +586,7 @@
 #define MSR_AMD_PERF_CTL		0xc0010062
 #define MSR_AMD_PERF_STATUS		0xc0010063
 #define MSR_AMD_PSTATE_DEF_BASE		0xc0010064
+#define MSR_AMD64_GUEST_TSC_FREQ	0xc0010134
 #define MSR_AMD64_OSVW_ID_LENGTH	0xc0010140
 #define MSR_AMD64_OSVW_STATUS		0xc0010141
 #define MSR_AMD_PPIN_CTL		0xc00102f0
arch/x86/kvm/svm/sev.c +2 -0
index ff82a644b174..9adab01d9003 100644
@@ -851,6 +851,8 @@ static int sev_es_sync_vmsa(struct vcpu_svm *svm)
 	save->dr6  = svm->vcpu.arch.dr6;
 
 	save->sev_features = sev->vmsa_features;
+	if (save->sev_features & SVM_SEV_FEAT_SECURE_TSC)
+		set_msr_interception(&svm->vcpu, svm->msrpm, MSR_AMD64_GUEST_TSC_FREQ, 1, 1);
 
 	/*
 	 * Skip FPU and AVX setup with KVM_SEV_ES_INIT to avoid
arch/x86/kvm/svm/svm.c +1 -0
index d6f252555ab3..bf86410b2f43 100644
@@ -142,6 +142,7 @@ static const struct svm_direct_access_msrs {
 	{ .index = X2APIC_MSR(APIC_TMICT),		.always = false },
 	{ .index = X2APIC_MSR(APIC_TMCCT),		.always = false },
 	{ .index = X2APIC_MSR(APIC_TDCR),		.always = false },
+	{ .index = MSR_AMD64_GUEST_TSC_FREQ,		.always = false },
 	{ .index = MSR_INVALID,				.always = false },
 };
 
arch/x86/kvm/svm/svm.h +1 -1
index 262b638dfcb8..9d4280d564e9 100644
@@ -30,7 +30,7 @@
 #define	IOPM_SIZE PAGE_SIZE * 3
 #define	MSRPM_SIZE PAGE_SIZE * 2
 
-#define MAX_DIRECT_ACCESS_MSRS	48
+#define MAX_DIRECT_ACCESS_MSRS	49
 #define MSRPM_OFFSETS	32
 extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly;
 extern bool npt_enabled;
Signature · 1 line
2.34.1
#9 lore raw

RFC PATCH 4/5 KVM: SVM: Prevent writes to TSC MSR when Secure TSC is enabled

In reply to Nikunj A Dadhania

For a Secure TSC enabled SNP guest, writes to MSR_IA32_TSC is not expected. Log the error and return #GP.

---
 arch/x86/kvm/svm/svm.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/x86/kvm/svm/svm.c +11 -0
index bf86410b2f43..f9b2c1956a60 100644
@@ -3151,6 +3151,17 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 
 		svm->tsc_aux = data;
 		break;
+	case MSR_IA32_TSC:
+		/*
+		 * If Secure TSC is enabled, KVM doesn't expect to receive
+		 * a VMEXIT for a TSC write, record the error and return a
+		 * #GP
+		 */
+		if (vcpu->arch.guest_state_protected && snp_secure_tsc_enabled(vcpu->kvm)) {
+			vcpu_unimpl(vcpu, "unimplemented IA32_TSC for secure tsc\n");
+			return 1;
+		}
+		break;
 	case MSR_IA32_DEBUGCTLMSR:
 		if (!lbrv) {
 			kvm_pr_unimpl_wrmsr(vcpu, ecx, data);
Signature · 1 line
2.34.1
#10 lore raw

RFC PATCH 5/5 KVM: SVM: Add Secure TSC support for SNP Guest

In reply to Nikunj A Dadhania

From: Ketan Chaturvedi <[email protected]>

Enable the Secure TSC capability and add it to SEV supported features to allow the userspave to set the Secure TSC feature for SNP guests.

---
 arch/x86/kvm/svm/sev.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/kvm/svm/sev.c +6 -0
index 9adab01d9003..c37fcd164413 100644
@@ -2966,6 +2966,9 @@ void __init sev_set_cpu_caps(void)
 	if (sev_snp_enabled) {
 		kvm_cpu_cap_set(X86_FEATURE_SEV_SNP);
 		kvm_caps.supported_vm_types |= BIT(KVM_X86_SNP_VM);
+
+		if (cpu_feature_enabled(X86_FEATURE_SNP_SECURE_TSC))
+			kvm_cpu_cap_set(X86_FEATURE_SNP_SECURE_TSC);
 	}
 }
 
@@ -3088,6 +3091,9 @@ void __init sev_hardware_setup(void)
 	sev_supported_vmsa_features = 0;
 	if (sev_es_debug_swap_enabled)
 		sev_supported_vmsa_features |= SVM_SEV_FEAT_DEBUG_SWAP;
+
+	if (sev_snp_enabled && cpu_feature_enabled(X86_FEATURE_SNP_SECURE_TSC))
+		sev_supported_vmsa_features |= SVM_SEV_FEAT_SECURE_TSC;
 }
 
 void sev_hardware_unsetup(void)
Signature · 1 line
2.34.1
#11 lore raw
In reply to Nikunj A Dadhania
Quoted 8 lines · Nikunj A Dadhania

TSC value calculations for the guest are controlled by the hypervisor. A malicious hypervisor can prevent guest from moving forward. The Secure TSC feature for SEV-SNP allows guests to securely use the RDTSC and RDTSCP instructions. This ensures the guest gets a consistent view of time and prevents a malicious hypervisor from making it appear that time rolls backwards, advancing at an unusually fast rate, or employing similar tricks. For more details, please refer to "Secure Nested Paging (SEV-SNP)" section, subsection "Secure TSC" of APM Volume 2

Hello. Although I replied at [1], let raise this here too.

Don't we need to prevent the KVM from modifying KVM vcpu tsc offset/multiplier (vcpu->arch.tsc_offset etc.)?

As long as I understand, the spec (APM volume2) says the timer interrupt (TSC deadline timer or local APIC timer) is not virtualized by hardware so that KVM emulates timer interrupt.
If KVM modifies guest offset/multiplier from the original value (the SEV-SNP secure tsc uses or the TDX module uses), the timer interrupt emulation by KVM will be inaccurate. It's injected late or early than the guest expects.

Please notice that kvm_arch_vcpu_create() calls kvm_set_tsc_khz(vcpu, vcpu->kvm->arch.default_tsc_khz) after kvm_x86_call(vcpu_create)().

[1] https://lore.kernel.org/kvm/Zz%2FDGOoo%[email protected]/

Quoted 15 lines

This patchset is also available at:

https://github.com/AMDESE/linux-kvm/tree/sectsc-host-latest

and is based on v6.11-rc5

Testing SecureTSC
-----------------

SecureTSC Guest patches:
https://github.com/AMDESE/linux-kvm/tree/sectsc-guest-latest

QEMU changes:
https://github.com/nikunjad/qemu/tree/snp-securetsc-latest

Quoted 7 lines

QEMU commandline SEV-SNP with SecureTSC:

  qemu-system-x86_64 -cpu EPYC-Milan-v2 -smp 4 \
     -object memory-backend-memfd,id=ram1,size=1G,share=true,prealloc=false,reserve=false \
     -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,secure-tsc=on \
     -machine q35,confidential-guest-support=sev0,memory-backend=ram1 \
     ...

Did you test it with tsc frequency/offset different from the kvm system default value (max_tsc_khz or kvm_caps.default_tsc_scaling_ranio etc.)?

Signature · 1 line
Isaku Yamahata <[email protected]>